Discussion:
答复: Call for discussion: Extend JMap to support parallel and incremental heap scanning.
(too old to reply)
臧琳
2018-11-28 02:47:38 UTC
Permalink
Add serviceability-dev.

Thanks.


________________________________
·¢ŒþÈË: ê°ÁÕ
·¢ËÍʱŒä: 2018Äê11ÔÂ28ÈÕ 10:30
ÊÕŒþÈË: Martijn Verburg
³­ËÍ: ***@openjdk.java.net
Ö÷Ìâ: Re: Call for discussion: Extend JMap to support parallel and incremental heap scanning.


Hi Martijn,

Thanks a lot for you guidance! I will go to serviceability-dev and try to join the Atlantis discussion.



Cheers,

Lin









·¢ŒþÈË: Martijn Verburg [mailto:***@gmail.com]
·¢ËÍʱŒä: 2018Äê11ÔÂ27ÈÕ 23:18
ÊÕŒþÈË: ê°ÁÕ <***@jd.com>
³­ËÍ: ***@openjdk.java.net
Ö÷Ìâ: Re: Call for discussion: Extend JMap to support parallel and incremental heap scanning.



Hi Lin and welcome to OpenJDK!



There's an ongoing thread discussing project Atlantis (http://mail.openjdk.java.net/pipermail/discuss/2018-November/004904.html) which I think will be looking at potential enhancements like this alongside the serviceability group.



In the meantime, I think the serviceability-dev mailing list is the right place to start your discussion.


Cheers,
Martijn





On Tue, 27 Nov 2018 at 15:08, ê°ÁÕ <***@jd.com<mailto:***@jd.com>> wrote:

Hello,

This is Zang Lin, JVM developer in JD.COM<http://JD.COM>. My department just signed OCA and we will try to contribute to OpenJDK. As I am newbie at openJDK, I'd like to discuss some of my ideas and ask for suggestions.


Recently I tried to use "jmap -histo" to collect histogram info on a large heap at ~200G bytes. it took about 160 seconds, which actually cause our online system suffering "non-responsive" issue. In addition, if we stop jmap with "kill", jmap exits directly (even the attached JVM still working on heap iteration), leave no help info for analysis.



To address the issue above, I proposed following approaches:

1) Extend Jmap to be parallel and incremental, which means Jmap iterates heap in parallel and save the intermediate results incrementally during heap scanning (controlled by threshold to decide when to dump).

2) Make Jmap to accept arguments that enable/disable parallel and partial heap iterating, also make parallel thread number configurable.

3) Make Jmap to dump results to specified file path given by argument, rather than output to stdout directly.



A prototype has been made internally with G1 upon JDK 11, and our preliminary measurement show that parallel iterate ~200GB heap with 4 threads can save ~30-40 seconds, and I am also analyzing the approach to make the time more scaled with parallel threads. Moreover, with the implementation of incremental dump, we can get usefully partial dump info for analyzing, even when jmap hang for a long time, or get killed for some reason.



Several problems were encountered during development. One problem is that it is not easy to dynamically allocate some data structures for thread local operations, such as KlassInfoTable,which is designed to be StackObj that not allowed to be allocated using "new", another problem is that some base classes definition in hotspot has to be modified to support parallel operations, such as ObjectClosure etc. I am not sure whether these modifications can be acceptable.



Any suggestions/guidance would be much appreciated!



BRs,

Lin
Hohensee, Paul
2018-11-30 13:48:36 UTC
Permalink
Hi Lin,

This looks interesting and useful, so I filed JBS RFE https://bugs.openjdk.java.net/browse/JDK-8214535 to track it, using your summary below as the description. This work would fit in with JC’s original Atlantis project proposal, but not one restricted to doing a thread sanitizer. Maybe we need another project for this? Otherwise we’d be passing potentially large patches around.

Thanks,

Paul

From: serviceability-dev <serviceability-dev-***@openjdk.java.net> on behalf of 臧琳 <***@jd.com>
Date: Tuesday, November 27, 2018 at 6:48 PM
To: "serviceability-***@openjdk.java.net" <serviceability-***@openjdk.java.net>
Cc: "***@openjdk.java.net" <***@openjdk.java.net>
Subject: 答倍: Call for discussion: Extend JMap to support parallel and incremental heap scanning.


Add serviceability-dev.

Thanks.

________________________________
发件人: 臧琳
发送时闎: 2018幎11月28日 10:30
收件人: Martijn Verburg
抄送: ***@openjdk.java.net
䞻题: Re: Call for discussion: Extend JMap to support parallel and incremental heap scanning.


Hi Martijn,

Thanks a lot for you guidance! I will go to serviceability-dev and try to join the Atlantis discussion.



Cheers,

Lin









发件人: Martijn Verburg [mailto:***@gmail.com]
发送时闎: 2018幎11月27日 23:18
收件人: 臧琳 <***@jd.com>
抄送: ***@openjdk.java.net
䞻题: Re: Call for discussion: Extend JMap to support parallel and incremental heap scanning.



Hi Lin and welcome to OpenJDK!



There's an ongoing thread discussing project Atlantis (http://mail.openjdk.java.net/pipermail/discuss/2018-November/004904.html) which I think will be looking at potential enhancements like this alongside the serviceability group.



In the meantime, I think the serviceability-dev mailing list is the right place to start your discussion.


Cheers,
Martijn





On Tue, 27 Nov 2018 at 15:08, 臧琳 <***@jd.com<mailto:***@jd.com>> wrote:

Hello,

This is Zang Lin, JVM developer in JD.COM<http://JD.COM>. My department just signed OCA and we will try to contribute to OpenJDK. As I am newbie at openJDK, I'd like to discuss some of my ideas and ask for suggestions.


Recently I tried to use "jmap -histo" to collect histogram info on a large heap at ~200G bytes. it took about 160 seconds, which actually cause our online system suffering "non-responsive" issue. In addition, if we stop jmap with "kill", jmap exits directly (even the attached JVM still working on heap iteration), leave no help info for analysis.



To address the issue above, I proposed following approaches:

1) Extend Jmap to be parallel and incremental, which means Jmap iterates heap in parallel and save the intermediate results incrementally during heap scanning (controlled by threshold to decide when to dump).

2) Make Jmap to accept arguments that enable/disable parallel and partial heap iterating, also make parallel thread number configurable.

3) Make Jmap to dump results to specified file path given by argument, rather than output to stdout directly.



A prototype has been made internally with G1 upon JDK 11, and our preliminary measurement show that parallel iterate ~200GB heap with 4 threads can save ~30-40 seconds, and I am also analyzing the approach to make the time more scaled with parallel threads. Moreover, with the implementation of incremental dump, we can get usefully partial dump info for analyzing, even when jmap hang for a long time, or get killed for some reason.



Several problems were encountered during development. One problem is that it is not easy to dynamically allocate some data structures for thread local operations, such as KlassInfoTable,which is designed to be StackObj that not allowed to be allocated using "new", another problem is that some base classes definition in hotspot has to be modified to support parallel operations, such as ObjectClosure etc. I am not sure whether these modifications can be acceptable.



Any suggestions/guidance would be much appreciated!



BRs,

Lin

Loading...