答复: Call for discussion: Extend JMap to support parallel and incremental heap scanning.

Hohensee, Paul

2018-11-30 13:48:36 UTC

Hi Lin,

This looks interesting and useful, so I filed JBS RFE https://bugs.openjdk.java.net/browse/JDK-8214535 to track it, using your summary below as the description. This work would fit in with JCâs original Atlantis project proposal, but not one restricted to doing a thread sanitizer. Maybe we need another project for this? Otherwise weâd be passing potentially large patches around.

Thanks,

Paul

From: serviceability-dev <serviceability-dev-***@openjdk.java.net> on behalf of è§ç³ <***@jd.com>
Date: Tuesday, November 27, 2018 at 6:48 PM
To: "serviceability-***@openjdk.java.net" <serviceability-***@openjdk.java.net>
Cc: "***@openjdk.java.net" <***@openjdk.java.net>
Subject: çå€: Call for discussion: Extend JMap to support parallel and incremental heap scanning.

Add serviceability-dev.

Thanks.

________________________________
åä»¶äºº: è§ç³
åéæ¶éŽ: 2018å¹Ž11æ28æ¥ 10:30
æ¶ä»¶äºº: Martijn Verburg
æé: ***@openjdk.java.net
äž»é¢: Re: Call for discussion: Extend JMap to support parallel and incremental heap scanning.

Hi Martijn,

Thanks a lot for you guidance! I will go to serviceability-dev and try to join the Atlantis discussion.

Cheers,

Lin

åä»¶äºº: Martijn Verburg [mailto:***@gmail.com]
åéæ¶éŽ: 2018å¹Ž11æ27æ¥ 23:18
æ¶ä»¶äºº: è§ç³ <***@jd.com>
æé: ***@openjdk.java.net
äž»é¢: Re: Call for discussion: Extend JMap to support parallel and incremental heap scanning.

Hi Lin and welcome to OpenJDK!

There's an ongoing thread discussing project Atlantis (http://mail.openjdk.java.net/pipermail/discuss/2018-November/004904.html) which I think will be looking at potential enhancements like this alongside the serviceability group.

In the meantime, I think the serviceability-dev mailing list is the right place to start your discussion.

Cheers,
Martijn

On Tue, 27 Nov 2018 at 15:08, è§ç³ <***@jd.com<mailto:***@jd.com>> wrote:

Hello,

This is Zang Lin, JVM developer in JD.COM<http://JD.COM>. My department just signed OCA and we will try to contribute to OpenJDK. As I am newbie at openJDK, I'd like to discuss some of my ideas and ask for suggestions.

Recently I tried to use "jmap -histo" to collect histogram info on a large heap at ~200G bytes. it took about 160 seconds, which actually cause our online system suffering "non-responsive" issue. In addition, if we stop jmap with "kill", jmap exits directly (even the attached JVM still working on heap iteration), leave no help info for analysis.

To address the issue above, I proposed following approaches:

1) Extend Jmap to be parallel and incremental, which means Jmap iterates heap in parallel and save the intermediate results incrementally during heap scanning (controlled by threshold to decide when to dump).

2) Make Jmap to accept arguments that enable/disable parallel and partial heap iterating, also make parallel thread number configurable.

3) Make Jmap to dump results to specified file path given by argument, rather than output to stdout directly.

A prototype has been made internally with G1 upon JDK 11, and our preliminary measurement show that parallel iterate ~200GB heap with 4 threads can save ~30-40 seconds, and I am also analyzing the approach to make the time more scaled with parallel threads. Moreover, with the implementation of incremental dump, we can get usefully partial dump info for analyzing, even when jmap hang for a long time, or get killed for some reason.

Several problems were encountered during development. One problem is that it is not easy to dynamically allocate some data structures for thread local operations, such as KlassInfoTable,which is designed to be StackObj that not allowed to be allocated using "new", another problem is that some base classes definition in hotspot has to be modified to support parallel operations, such as ObjectClosure etc. I am not sure whether these modifications can be acceptable.

Any suggestions/guidance would be much appreciated!

BRs,

Lin