Discussion:
RFR(xs): 8213834: JVMTI ResourceExhausted should not be posted in CompilerThread
(too old to reply)
Daniel D. Daugherty
2018-11-14 15:06:56 UTC
Permalink
Adding serviceability-***@...

Since the proposed solution to the bug is to not post an event, I think
the Serviceability Team (which owns JVM/TI) needs to be involved directly
in the review.

Dan
Dear all,
may I please have reviews on this small patch. Note that this is
borderline serviceability. I try to avoid crossposting, but if you
think this should be looked at by serviceability feel free to forward
it there.
Issue: https://bugs.openjdk.java.net/browse/JDK-8213834
CR: http://cr.openjdk.java.net/~stuefe/webrevs/8213834-jvmti-reourceexhausted-shall-not-be-posted-from-compiler-thread/webrev.00/webrev/
Short description: we may post ResourceExhausted from Compiler
threads. Handlers of this event may call back into the JVM, which may
cause problems since we run in the compiler thread. The proposed
solution is not to post the event in that case.
http://mail.openjdk.java.net/pipermail/serviceability-dev/2018-November/025898.html
Thanks, Thomas
s***@oracle.com
2018-11-14 19:20:34 UTC
Permalink
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">Hi Thomas,<br>
<br>
Thank you for taking care about this issue!<br>
We recently also saw a couple of problems related to the compiler
thread not being hidden.<br>
<br>
I have several comments to the fix.<br>
<br>
It would be really nice if there is any chance to have a unit test
for this.<br>
But I understand that it is not easy to minimize what you have in
your agent.<br> <br> <a class="moz-txt-link-freetext" href="http://cr.openjdk.java.net/~stuefe/webrevs/8213834-jvmti-reourceexhausted-shall-not-be-posted-from-compiler-thread/webrev.00/webrev/src/hotspot/share/prims/jvmtiExport.cpp.udiff.html">http://cr.openjdk.java.net/~stuefe/webrevs/8213834-jvmti-reourceexhausted-shall-not-be-posted-from-compiler-thread/webrev.00/webrev/src/hotspot/share/prims/jvmtiExport.cpp.udiff.html</a><br> <pre> void JvmtiExport::post_resource_exhausted(jint resource_exhausted_flags, const char* description) { <span class="new">+</span> <span class="new">+ // Handlers may call back into the JVM as a reaction to this event, e.g. to print out</span> <span class="new">+ // a class histogram. Most of those actions require the ability to run java though. So</span> <span class="new">+ // lets not post this event in cases where we cannot run java (e.g. when a CompilerThread</span> <span class="new">+ // triggers a Metaspace OOM).</span> <span class="new">+ Thread* thread = Thread::current_or_null();</span> <span class="new">+ if (thread == NULL || !thread-&gt;can_call_java()) {</span>
<span class="new">+ return;</span>
<span class="new">+ }</span>
<span class="new">+

</span>
<span class="new"><span class="new">1. Did you check that your fix does also work with the Graal enabled?</span>
The CompilerThread:</span><span class="new"><span class="new">:can_call_java() should return true for JVMCI is used:
</span>
bool CompilerThread::can_call_java() const {
return _compiler != NULL &amp;&amp; _compiler-&gt;is_jvmci();
}


2. It is better to use the is_hidden_from_external_view() instead of !</span><span class="new"><span class="new">can_call_java()

</span></span>3. The 'thread' is already defined in post_resource_exhausted:

JavaThread *thread = JavaThread::current();
 
But it is in a deeper scope.

void JvmtiExport::post_resource_exhausted(jint resource_exhausted_flags, const char* description) {
EVT_TRIG_TRACE(JVMTI_EVENT_RESOURCE_EXHAUSTED, ("Trg resource exhausted event triggered" ));

JvmtiEnvIterator it;
for (JvmtiEnv* env = it.first(); env != NULL; env = it.next(env)) {
if (env-&gt;is_enabled(JVMTI_EVENT_RESOURCE_EXHAUSTED)) {
EVT_TRACE(JVMTI_EVENT_RESOURCE_EXHAUSTED, ("Evt resource exhausted event sent" ));

JavaThread *thread = JavaThread::current();

I'd suggest to move it up as it is a better place for it anyway.
Also, you do not need the extra check:
<span class="new"> if (thread == NULL || !</span><span class="new"></span></pre>
<br>
Thanks,<br>
Serguei<br>
<br>
<br>
On 11/14/18 07:06, Daniel D. Daugherty wrote:<br>
</div>
<blockquote type="cite"
cite="mid:b0508ce8-e587-0949-f40f-***@oracle.com">Adding
serviceability-***@...
<br>
<br>
Since the proposed solution to the bug is to not post an event, I
think
<br>
the Serviceability Team (which owns JVM/TI) needs to be involved
directly
<br>
in the review.
<br>
<br>
Dan
<br>
<br>
<br>
On 11/14/18 9:28 AM, Thomas Stüfe wrote:
<br>
<blockquote type="cite">Dear all,
<br>
<br>
may I please have reviews on this small patch. Note that this is
<br>
borderline serviceability. I try to avoid crossposting, but if
you
<br>
think this should be looked at by serviceability feel free to
forward
<br>
it there.
<br>
<br>
Issue: <a class="moz-txt-link-freetext" href="https://bugs.openjdk.java.net/browse/JDK-8213834">https://bugs.openjdk.java.net/browse/JDK-8213834</a>
<br>
<br>
CR:
<a class="moz-txt-link-freetext" href="http://cr.openjdk.java.net/~stuefe/webrevs/8213834-jvmti-reourceexhausted-shall-not-be-posted-from-compiler-thread/webrev.00/webrev/">http://cr.openjdk.java.net/~stuefe/webrevs/8213834-jvmti-reourceexhausted-shall-not-be-posted-from-compiler-thread/webrev.00/webrev/</a><br>
<br>
Short description: we may post ResourceExhausted from Compiler
<br>
threads. Handlers of this event may call back into the JVM,
which may
<br>
cause problems since we run in the compiler thread. The proposed
<br>
solution is not to post the event in that case.
<br>
<br>
See previous discussion here:
<br>
<a class="moz-txt-link-freetext" href="http://mail.openjdk.java.net/pipermail/serviceability-dev/2018-November/025898.html">http://mail.openjdk.java.net/pipermail/serviceability-dev/2018-November/025898.html</a>
<br>
<br>
Thanks, Thomas
<br>
</blockquote>
<br>
</blockquote>
<br>
</body>
</html>
s***@oracle.com
2018-11-14 19:31:48 UTC
Permalink
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">On 11/14/18 11:20,
<a class="moz-txt-link-abbreviated" href="mailto:***@oracle.com">***@oracle.com</a> wrote:<br>
</div>
<blockquote type="cite"
cite="mid:bb5f88ff-b9ca-2f34-3adb-***@oracle.com">
<div class="moz-cite-prefix">Hi Thomas,<br>
<br>
Thank you for taking care about this issue!<br>
We recently also saw a couple of problems related to the
compiler thread not being hidden.<br>
<br>
I have several comments to the fix.<br>
<br>
It would be really nice if there is any chance to have a unit
test for this.<br>
But I understand that it is not easy to minimize what you have
in your agent.<br>
<br>
<a class="moz-txt-link-freetext"
href="http://cr.openjdk.java.net/%7Estuefe/webrevs/8213834-jvmti-reourceexhausted-shall-not-be-posted-from-compiler-thread/webrev.00/webrev/src/hotspot/share/prims/jvmtiExport.cpp.udiff.html"
moz-do-not-send="true">http://cr.openjdk.java.net/~stuefe/webrevs/8213834-jvmti-reourceexhausted-shall-not-be-posted-from-compiler-thread/webrev.00/webrev/src/hotspot/share/prims/jvmtiExport.cpp.udiff.html</a><br> <pre> void JvmtiExport::post_resource_exhausted(jint resource_exhausted_flags, const char* description) { <span class="new">+</span> <span class="new">+ // Handlers may call back into the JVM as a reaction to this event, e.g. to print out</span> <span class="new">+ // a class histogram. Most of those actions require the ability to run java though. So</span> <span class="new">+ // lets not post this event in cases where we cannot run java (e.g. when a CompilerThread</span> <span class="new">+ // triggers a Metaspace OOM).</span> <span class="new">+ Thread* thread = Thread::current_or_null();</span> <span class="new">+ if (thread == NULL || !thread-&gt;can_call_java()) {</span>
<span class="new">+ return;</span>
<span class="new">+ }</span>
<span class="new">+

</span>
<span class="new"><span class="new">1. Did you check that your fix does also work with the Graal enabled?</span>
The CompilerThread:</span><span class="new"><span class="new">:can_call_java() should return true for JVMCI is used:
</span>
bool CompilerThread::can_call_java() const {
return _compiler != NULL &amp;&amp; _compiler-&gt;is_jvmci();
}</span></pre>
</div>
</blockquote>
<br>
We may want to use a pattern below:<br>
<br>
bool
JvmtiSampledObjectAllocEventCollector::object_alloc_is_safe_to_sample()
{<br>
  Thread* thread = Thread::current();<br>
  // Really only sample allocations if this is a JavaThread and not
the compiler<br>
  // thread.<br>
  if (!thread-&gt;is_Java_thread() ||
thread-&gt;is_Compiler_thread()) {<br>
    return false;<br>
  }<br>
<br>
<span class="new"></span><span class="new">Thanks,<br>
Serguei<br>
<br>
</span><span class="new"></span>
<blockquote type="cite"
cite="mid:bb5f88ff-b9ca-2f34-3adb-***@oracle.com">
<div class="moz-cite-prefix">
<pre><span class="new">2. It is better to use the is_hidden_from_external_view() instead of !</span><span class="new"><span class="new">can_call_java()

</span></span>3. The 'thread' is already defined in post_resource_exhausted:

JavaThread *thread = JavaThread::current();
 
But it is in a deeper scope.

void JvmtiExport::post_resource_exhausted(jint resource_exhausted_flags, const char* description) {
EVT_TRIG_TRACE(JVMTI_EVENT_RESOURCE_EXHAUSTED, ("Trg resource exhausted event triggered" ));

JvmtiEnvIterator it;
for (JvmtiEnv* env = it.first(); env != NULL; env = it.next(env)) {
if (env-&gt;is_enabled(JVMTI_EVENT_RESOURCE_EXHAUSTED)) {
EVT_TRACE(JVMTI_EVENT_RESOURCE_EXHAUSTED, ("Evt resource exhausted event sent" ));

JavaThread *thread = JavaThread::current();

I'd suggest to move it up as it is a better place for it anyway.
Also, you do not need the extra check:
<span class="new"> if (thread == NULL || !</span><span class="new"></span></pre>
<br>
Thanks,<br>
Serguei<br>
<br>
<br>
On 11/14/18 07:06, Daniel D. Daugherty wrote:<br>
</div>
<blockquote type="cite"
cite="mid:b0508ce8-e587-0949-f40f-***@oracle.com">Adding
serviceability-***@... <br>
<br>
Since the proposed solution to the bug is to not post an event,
I think <br>
the Serviceability Team (which owns JVM/TI) needs to be involved
directly <br>
in the review. <br>
<br>
Dan <br>
<br>
<br>
On 11/14/18 9:28 AM, Thomas Stüfe wrote: <br>
<blockquote type="cite">Dear all, <br>
<br>
may I please have reviews on this small patch. Note that this
is <br>
borderline serviceability. I try to avoid crossposting, but if
you <br>
think this should be looked at by serviceability feel free to
forward <br>
it there. <br>
<br>
Issue: <a class="moz-txt-link-freetext"
href="https://bugs.openjdk.java.net/browse/JDK-8213834"
moz-do-not-send="true">https://bugs.openjdk.java.net/browse/JDK-8213834</a>
<br>
<br>
CR:
<a class="moz-txt-link-freetext"
href="http://cr.openjdk.java.net/%7Estuefe/webrevs/8213834-jvmti-reourceexhausted-shall-not-be-posted-from-compiler-thread/webrev.00/webrev/"
moz-do-not-send="true">http://cr.openjdk.java.net/~stuefe/webrevs/8213834-jvmti-reourceexhausted-shall-not-be-posted-from-compiler-thread/webrev.00/webrev/</a><br>
<br>
Short description: we may post ResourceExhausted from Compiler
<br>
threads. Handlers of this event may call back into the JVM,
which may <br>
cause problems since we run in the compiler thread. The
proposed <br>
solution is not to post the event in that case. <br>
<br>
See previous discussion here: <br>
<a class="moz-txt-link-freetext"
href="http://mail.openjdk.java.net/pipermail/serviceability-dev/2018-November/025898.html"
moz-do-not-send="true">http://mail.openjdk.java.net/pipermail/serviceability-dev/2018-November/025898.html</a>
<br>
<br>
Thanks, Thomas <br>
</blockquote>
<br>
</blockquote>
<br>
</blockquote>
<br>
</body>
</html>
David Holmes
2018-11-14 23:05:59 UTC
Permalink
Hi Serguei,
Post by s***@oracle.com
Hi Thomas,
Thank you for taking care about this issue!
We recently also saw a couple of problems related to the compiler thread
not being hidden.
I have several comments to the fix.
It would be really nice if there is any chance to have a unit test for this.
But I understand that it is not easy to minimize what you have in your
agent.
http://cr.openjdk.java.net/~stuefe/webrevs/8213834-jvmti-reourceexhausted-shall-not-be-posted-from-compiler-thread/webrev.00/webrev/src/hotspot/share/prims/jvmtiExport.cpp.udiff.html
void JvmtiExport::post_resource_exhausted(jint resource_exhausted_flags, const char* description) {
+
+ // Handlers may call back into the JVM as a reaction to this event,
e.g. to print out
+ // a class histogram. Most of those actions require the ability to run
java though. So
+ // lets not post this event in cases where we cannot run java (e.g.
when a CompilerThread
+ // triggers a Metaspace OOM).
+ Thread* thread = Thread::current_or_null();
+ if (thread == NULL || !thread->can_call_java()) {
+ return;
+ }
+
1. Did you check that your fix does also work with the Graal enabled?
The CompilerThread::can_call_java() should return true for JVMCI is
used: bool CompilerThread::can_call_java() const { return _compiler !=
NULL && _compiler->is_jvmci(); }
That's exactly why I suggested checking for can_call_Java() - because it
would work with JVMCI/Graal.
Post by s***@oracle.com
2. It is better to use the is_hidden_from_external_view() instead of !can_call_java()
Why? In terms of being conservative that would be unnecessarily
conservative. If the problem is only executing Java code why go beyond that?

BTW what exactly is the set of hidden threads these days?
Post by s***@oracle.com
JavaThread *thread = JavaThread::current();
But it is in a deeper scope.
void JvmtiExport::post_resource_exhausted(jint resource_exhausted_flags, const char* description) {
EVT_TRIG_TRACE(JVMTI_EVENT_RESOURCE_EXHAUSTED, ("Trg resource exhausted event triggered" ));
JvmtiEnvIterator it;
for (JvmtiEnv* env = it.first(); env != NULL; env = it.next(env)) {
if (env->is_enabled(JVMTI_EVENT_RESOURCE_EXHAUSTED)) {
EVT_TRACE(JVMTI_EVENT_RESOURCE_EXHAUSTED, ("Evt resource exhausted event sent" ));
JavaThread *thread = JavaThread::current();
I'd suggest to move it up as it is a better place for it anyway.
if (thread == NULL || !
Agreed - Thread::current() can not be NULL in this context.

Cheers,
David
Post by s***@oracle.com
Thanks,
Serguei
Post by Daniel D. Daugherty
Since the proposed solution to the bug is to not post an event, I think
the Serviceability Team (which owns JVM/TI) needs to be involved directly
in the review.
Dan
Dear all,
may I please have reviews on this small patch. Note that this is
borderline serviceability. I try to avoid crossposting, but if you
think this should be looked at by serviceability feel free to forward
it there.
Issue: https://bugs.openjdk.java.net/browse/JDK-8213834
http://cr.openjdk.java.net/~stuefe/webrevs/8213834-jvmti-reourceexhausted-shall-not-be-posted-from-compiler-thread/webrev.00/webrev/
Short description: we may post ResourceExhausted from Compiler
threads. Handlers of this event may call back into the JVM, which may
cause problems since we run in the compiler thread. The proposed
solution is not to post the event in that case.
http://mail.openjdk.java.net/pipermail/serviceability-dev/2018-November/025898.html
Thanks, Thomas
s***@oracle.com
2018-11-14 23:58:57 UTC
Permalink
Hi David,
Post by David Holmes
Hi Serguei,
Post by s***@oracle.com
Hi Thomas,
Thank you for taking care about this issue!
We recently also saw a couple of problems related to the compiler
thread not being hidden.
I have several comments to the fix.
It would be really nice if there is any chance to have a unit test for this.
But I understand that it is not easy to minimize what you have in
your agent.
http://cr.openjdk.java.net/~stuefe/webrevs/8213834-jvmti-reourceexhausted-shall-not-be-posted-from-compiler-thread/webrev.00/webrev/src/hotspot/share/prims/jvmtiExport.cpp.udiff.html
  void JvmtiExport::post_resource_exhausted(jint
resource_exhausted_flags, const char* description) {
+
+ // Handlers may call back into the JVM as a reaction to this event,
e.g. to print out
+ // a class histogram. Most of those actions require the ability to
run java though. So
+ // lets not post this event in cases where we cannot run java (e.g.
when a CompilerThread
+ // triggers a Metaspace OOM).
+ Thread* thread = Thread::current_or_null();
+ if (thread == NULL || !thread->can_call_java()) {
+ return;
+ }
+
1. Did you check that your fix does also work with the Graal enabled?
The CompilerThread::can_call_java() should return true for JVMCI is
used: bool CompilerThread::can_call_java() const { return _compiler
!= NULL && _compiler->is_jvmci(); }
That's exactly why I suggested checking for can_call_Java() - because
it would work with JVMCI/Graal.
Does it mean, we want to continue posting such events on the JVMCI/Graal
compiler threads executing Java?
Post by David Holmes
Post by s***@oracle.com
2. It is better to use the is_hidden_from_external_view() instead of !can_call_java()
Why? In terms of being conservative that would be unnecessarily
conservative. If the problem is only executing Java code why go beyond that?
  The is_hidden_from_external_view() means the same as !can_call_java()
CompilerThread's:
  // Hide native compiler threads from external view.
  bool is_hidden_from_external_view() const      { return
!can_call_java(); }

  It seems the ServiceThread and CodeCacheSweeperThread do not follow
the above rule.
  But do we want to post any events on the hidden threads?
Post by David Holmes
BTW what exactly is the set of hidden threads these days?
From my observation the following threads are hidden now:
  ServiceThread
  CodeCacheSweeperThread
  All native CompilerThread's that return false from can_call_java().


Thanks,
Serguei
Post by David Holmes
Post by s***@oracle.com
   JavaThread *thread  = JavaThread::current();
  But it is in a deeper scope.
   void JvmtiExport::post_resource_exhausted(jint
resource_exhausted_flags, const char* description) {
     EVT_TRIG_TRACE(JVMTI_EVENT_RESOURCE_EXHAUSTED, ("Trg resource
exhausted event triggered" ));
     JvmtiEnvIterator it;
     for (JvmtiEnv* env = it.first(); env != NULL; env = it.next(env)) {
       if (env->is_enabled(JVMTI_EVENT_RESOURCE_EXHAUSTED)) {
         EVT_TRACE(JVMTI_EVENT_RESOURCE_EXHAUSTED, ("Evt resource
exhausted event sent" ));
         JavaThread *thread  = JavaThread::current();
I'd suggest to move it up as it is a better place for it anyway.
if (thread == NULL || !
Agreed - Thread::current() can not be NULL in this context.
Cheers,
David
Post by s***@oracle.com
Thanks,
Serguei
Post by Daniel D. Daugherty
Since the proposed solution to the bug is to not post an event, I think
the Serviceability Team (which owns JVM/TI) needs to be involved directly
in the review.
Dan
Dear all,
may I please have reviews on this small patch. Note that this is
borderline serviceability. I try to avoid crossposting, but if you
think this should be looked at by serviceability feel free to forward
it there.
Issue: https://bugs.openjdk.java.net/browse/JDK-8213834
http://cr.openjdk.java.net/~stuefe/webrevs/8213834-jvmti-reourceexhausted-shall-not-be-posted-from-compiler-thread/webrev.00/webrev/
Short description: we may post ResourceExhausted from Compiler
threads. Handlers of this event may call back into the JVM, which may
cause problems since we run in the compiler thread. The proposed
solution is not to post the event in that case.
http://mail.openjdk.java.net/pipermail/serviceability-dev/2018-November/025898.html
Thanks, Thomas
Daniel D. Daugherty
2018-11-14 19:32:03 UTC
Permalink
I have a philosophical problem with a fix like this.

The fix is making the assumption that the handler for this event will want
to run Java code and if the event is posted from a Java thread that cannot
run Java code, then we skip posting the event.

If I happen to have a more conservative agent that does not run Java code
in its handlers, then my agent won't receive this event even though it
should not cause my agent any problems. That might be an unexpected change
in behavior on the part of my agent.

Dan
Post by Daniel D. Daugherty
Since the proposed solution to the bug is to not post an event, I think
the Serviceability Team (which owns JVM/TI) needs to be involved directly
in the review.
Dan
Dear all,
may I please have reviews on this small patch. Note that this is
borderline serviceability. I try to avoid crossposting, but if you
think this should be looked at by serviceability feel free to forward
it there.
Issue: https://bugs.openjdk.java.net/browse/JDK-8213834
http://cr.openjdk.java.net/~stuefe/webrevs/8213834-jvmti-reourceexhausted-shall-not-be-posted-from-compiler-thread/webrev.00/webrev/
Short description: we may post ResourceExhausted from Compiler
threads. Handlers of this event may call back into the JVM, which may
cause problems since we run in the compiler thread. The proposed
solution is not to post the event in that case.
http://mail.openjdk.java.net/pipermail/serviceability-dev/2018-November/025898.html
Thanks, Thomas
Thomas Stüfe
2018-11-14 19:59:12 UTC
Permalink
Hi Dan,

On Wed, Nov 14, 2018 at 8:32 PM Daniel D. Daugherty
Post by Daniel D. Daugherty
I have a philosophical problem with a fix like this.
The fix is making the assumption that the handler for this event will want
to run Java code and if the event is posted from a Java thread that cannot
run Java code, then we skip posting the event.
If I happen to have a more conservative agent that does not run Java code
in its handlers, then my agent won't receive this event even though it
should not cause my agent any problems. That might be an unexpected change
in behavior on the part of my agent.
Dan
I was thinking about that too. In fact, the JVMTI agent I am trying to
add this fix for exists in two variants. The base form,
airlift/jvmkill, just does a plain kill(2). So not problem there. Only
the forked version by cloudfoundry/jvmkill do this fancy stuff. (Note
that both projects are on github, if you care to look them up. They
are not by me. I am just using them).

But the problem is, the JVMTI spec says nothing about what you can or
cannot do in reaction to a ResourceExhausted event. So what
cloudfoundry/jvmkill does is valid and not forbidden.

Therefore I think suppressing ResourceExhausted in this case is the
only choice we have. One might fine-tune the conditions under which we
suppress sending ResourceExhausted: maybe suppress for
CompilerThreads, or only for CompilerThreads getting MetaspaceOOMs...

But I think we should do something here. By neglecting to add
restrictions to the JVMTI spec, we encouraged JVMTI agents to do these
kind of things. The least we can do is minimize the damage.

And then, usually this will not be the last opportunity for
ResourceExhausted to be posted, no? Chances are high that there will
be more OOMs following, in real java threads.

Finally, I find swallowing the ResourceExhausted for compiler threads
is symmetric to the way the compiler thread swallows the OOME itself.
They do not report the OOME but ignore and clear it.

But as you can see, I see your point. Do you have a better proposal?

..Thomas
Post by Daniel D. Daugherty
Post by Daniel D. Daugherty
Since the proposed solution to the bug is to not post an event, I think
the Serviceability Team (which owns JVM/TI) needs to be involved directly
in the review.
Dan
Dear all,
may I please have reviews on this small patch. Note that this is
borderline serviceability. I try to avoid crossposting, but if you
think this should be looked at by serviceability feel free to forward
it there.
Issue: https://bugs.openjdk.java.net/browse/JDK-8213834
http://cr.openjdk.java.net/~stuefe/webrevs/8213834-jvmti-reourceexhausted-shall-not-be-posted-from-compiler-thread/webrev.00/webrev/
Short description: we may post ResourceExhausted from Compiler
threads. Handlers of this event may call back into the JVM, which may
cause problems since we run in the compiler thread. The proposed
solution is not to post the event in that case.
http://mail.openjdk.java.net/pipermail/serviceability-dev/2018-November/025898.html
Thanks, Thomas
Daniel D. Daugherty
2018-11-14 20:13:35 UTC
Permalink
I do not have a better idea at the moment.

The ResourceExhausted event is one of many JVM/TI events. So we put in a
special case for this event because an agent out in the world tries to do
something that is not expressly forbidden by the spec, but it is a bad
thing to do with HotSpot. Okay. What about the next event? What about
the next agent? Once you've added a special case, where do you stop?

Dan
Post by Thomas Stüfe
Hi Dan,
On Wed, Nov 14, 2018 at 8:32 PM Daniel D. Daugherty
Post by Daniel D. Daugherty
I have a philosophical problem with a fix like this.
The fix is making the assumption that the handler for this event will want
to run Java code and if the event is posted from a Java thread that cannot
run Java code, then we skip posting the event.
If I happen to have a more conservative agent that does not run Java code
in its handlers, then my agent won't receive this event even though it
should not cause my agent any problems. That might be an unexpected change
in behavior on the part of my agent.
Dan
I was thinking about that too. In fact, the JVMTI agent I am trying to
add this fix for exists in two variants. The base form,
airlift/jvmkill, just does a plain kill(2). So not problem there. Only
the forked version by cloudfoundry/jvmkill do this fancy stuff. (Note
that both projects are on github, if you care to look them up. They
are not by me. I am just using them).
But the problem is, the JVMTI spec says nothing about what you can or
cannot do in reaction to a ResourceExhausted event. So what
cloudfoundry/jvmkill does is valid and not forbidden.
Therefore I think suppressing ResourceExhausted in this case is the
only choice we have. One might fine-tune the conditions under which we
suppress sending ResourceExhausted: maybe suppress for
CompilerThreads, or only for CompilerThreads getting MetaspaceOOMs...
But I think we should do something here. By neglecting to add
restrictions to the JVMTI spec, we encouraged JVMTI agents to do these
kind of things. The least we can do is minimize the damage.
And then, usually this will not be the last opportunity for
ResourceExhausted to be posted, no? Chances are high that there will
be more OOMs following, in real java threads.
Finally, I find swallowing the ResourceExhausted for compiler threads
is symmetric to the way the compiler thread swallows the OOME itself.
They do not report the OOME but ignore and clear it.
But as you can see, I see your point. Do you have a better proposal?
..Thomas
Post by Daniel D. Daugherty
Post by Daniel D. Daugherty
Since the proposed solution to the bug is to not post an event, I think
the Serviceability Team (which owns JVM/TI) needs to be involved directly
in the review.
Dan
Dear all,
may I please have reviews on this small patch. Note that this is
borderline serviceability. I try to avoid crossposting, but if you
think this should be looked at by serviceability feel free to forward
it there.
Issue: https://bugs.openjdk.java.net/browse/JDK-8213834
http://cr.openjdk.java.net/~stuefe/webrevs/8213834-jvmti-reourceexhausted-shall-not-be-posted-from-compiler-thread/webrev.00/webrev/
Short description: we may post ResourceExhausted from Compiler
threads. Handlers of this event may call back into the JVM, which may
cause problems since we run in the compiler thread. The proposed
solution is not to post the event in that case.
http://mail.openjdk.java.net/pipermail/serviceability-dev/2018-November/025898.html
Thanks, Thomas
David Holmes
2018-11-14 23:01:48 UTC
Permalink
Post by Daniel D. Daugherty
I do not have a better idea at the moment.
The ResourceExhausted event is one of many JVM/TI events. So we put in a
special case for this event because an agent out in the world tries to do
something that is not expressly forbidden by the spec, but it is a bad
thing to do with HotSpot. Okay. What about the next event? What about
the next agent? Once you've added a special case, where do you stop?
I had proposed more broadly not posting any events in threads that can
not run Java but that is potentially more damaging to existing agents
that just happen to work in the compiler thread today.

But it goes beyond just running Java. Can these other threads make JNI
calls, or other JVM TI calls? If they can run Java they can do all those
things. If they can't run Java they may not be able to.

There is no perfect solution here because JVM TI allows too much
freedom. It doesn't put any constraints on the threads that can execute
callbacks, and has very few constraints on what callbacks can be
executed when.

On the other hand an implementation has complete freedom as to when
exactly resource exhaustion events get posted. Does it make sense for an
internal VM thread to post this ResourceExhausted event? Is it just an
accident of implementation that the code path decided to throw OOME and
so also posts the event? You can easily imagine this code not doing that
- and as Thomas points out the compiler thread swallows the exception
anyway!

As there is no way to know what an agent may do ahead of time we are
faced with either:
- breaking some agents and crashing the VM; or
- masking an event

I vote for no breakage.

Cheers,
David
Post by Daniel D. Daugherty
Dan
Post by Thomas Stüfe
Hi Dan,
On Wed, Nov 14, 2018 at 8:32 PM Daniel D. Daugherty
Post by Daniel D. Daugherty
I have a philosophical problem with a fix like this.
The fix is making the assumption that the handler for this event will want
to run Java code and if the event is posted from a Java thread that cannot
run Java code, then we skip posting the event.
If I happen to have a more conservative agent that does not run Java code
in its handlers, then my agent won't receive this event even though it
should not cause my agent any problems. That might be an unexpected change
in behavior on the part of my agent.
Dan
I was thinking about that too. In fact, the JVMTI agent I am trying to
add this fix for exists in two variants. The base form,
airlift/jvmkill, just does a plain kill(2). So not problem there. Only
the forked version by cloudfoundry/jvmkill do this fancy stuff. (Note
that both projects are on github, if you care to look them up. They
are not by me. I am just using them).
But the problem is, the JVMTI spec says nothing about what you can or
cannot do in reaction to a ResourceExhausted event. So what
cloudfoundry/jvmkill does is valid and not forbidden.
Therefore I think suppressing ResourceExhausted in this case is the
only choice we have. One might fine-tune the conditions under which we
suppress sending ResourceExhausted: maybe suppress for
CompilerThreads, or only  for CompilerThreads getting MetaspaceOOMs...
But I think we should do something here. By neglecting to add
restrictions to the JVMTI spec, we encouraged JVMTI agents to do these
kind of things. The least we can do is minimize the damage.
And then, usually this will not be the last opportunity for
ResourceExhausted to be posted, no? Chances are high that there will
be more OOMs following, in real java threads.
Finally, I find swallowing the ResourceExhausted for compiler threads
is symmetric to the way the compiler thread swallows the OOME itself.
They do not report the OOME but ignore and clear it.
But as you can see, I see your point. Do you have a better proposal?
..Thomas
Post by Daniel D. Daugherty
Post by Daniel D. Daugherty
Since the proposed solution to the bug is to not post an event, I think
the Serviceability Team (which owns JVM/TI) needs to be involved directly
in the review.
Dan
Dear all,
may I please have reviews on this small patch. Note that this is
borderline serviceability. I try to avoid crossposting, but if you
think this should be looked at by serviceability feel free to forward
it there.
Issue: https://bugs.openjdk.java.net/browse/JDK-8213834
http://cr.openjdk.java.net/~stuefe/webrevs/8213834-jvmti-reourceexhausted-shall-not-be-posted-from-compiler-thread/webrev.00/webrev/
Short description: we may post ResourceExhausted from Compiler
threads. Handlers of this event may call back into the JVM, which may
cause problems since we run in the compiler thread. The proposed
solution is not to post the event in that case.
http://mail.openjdk.java.net/pipermail/serviceability-dev/2018-November/025898.html
Thanks, Thomas
Daniel D. Daugherty
2018-11-14 23:28:11 UTC
Permalink
Post by David Holmes
Post by Daniel D. Daugherty
I do not have a better idea at the moment.
The ResourceExhausted event is one of many JVM/TI events. So we put in a
special case for this event because an agent out in the world tries to do
something that is not expressly forbidden by the spec, but it is a bad
thing to do with HotSpot. Okay. What about the next event? What about
the next agent? Once you've added a special case, where do you stop?
I had proposed more broadly not posting any events in threads that can
not run Java but that is potentially more damaging to existing agents
that just happen to work in the compiler thread today.
But it goes beyond just running Java. Can these other threads make JNI
calls, or other JVM TI calls? If they can run Java they can do all
those things. If they can't run Java they may not be able to.
There is no perfect solution here because JVM TI allows too much
freedom. It doesn't put any constraints on the threads that can
execute callbacks, and has very few constraints on what callbacks can
be executed when.
On the other hand an implementation has complete freedom as to when
exactly resource exhaustion events get posted. Does it make sense for
an internal VM thread to post this ResourceExhausted event? Is it just
an accident of implementation that the code path decided to throw OOME
and so also posts the event? You can easily imagine this code not
doing that - and as Thomas points out the compiler thread swallows the
exception anyway!
As there is no way to know what an agent may do ahead of time we are
- breaking some agents and crashing the VM; or
- masking an event
I vote for no breakage.
I would have no problem suppressing the event from a "hidden" thread
because those threads intentionally try to hide from JVM/TI. That would
cover the case for the C1 and C2 CompilerThreads, but I don't know
about these newer JVM/CI Compiler threads that actually run Java code...

Dan
Post by David Holmes
Cheers,
David
Post by Daniel D. Daugherty
Dan
Post by Thomas Stüfe
Hi Dan,
On Wed, Nov 14, 2018 at 8:32 PM Daniel D. Daugherty
Post by Daniel D. Daugherty
I have a philosophical problem with a fix like this.
The fix is making the assumption that the handler for this event will want
to run Java code and if the event is posted from a Java thread that cannot
run Java code, then we skip posting the event.
If I happen to have a more conservative agent that does not run Java code
in its handlers, then my agent won't receive this event even though it
should not cause my agent any problems. That might be an unexpected change
in behavior on the part of my agent.
Dan
I was thinking about that too. In fact, the JVMTI agent I am trying to
add this fix for exists in two variants. The base form,
airlift/jvmkill, just does a plain kill(2). So not problem there. Only
the forked version by cloudfoundry/jvmkill do this fancy stuff. (Note
that both projects are on github, if you care to look them up. They
are not by me. I am just using them).
But the problem is, the JVMTI spec says nothing about what you can or
cannot do in reaction to a ResourceExhausted event. So what
cloudfoundry/jvmkill does is valid and not forbidden.
Therefore I think suppressing ResourceExhausted in this case is the
only choice we have. One might fine-tune the conditions under which we
suppress sending ResourceExhausted: maybe suppress for
CompilerThreads, or only  for CompilerThreads getting MetaspaceOOMs...
But I think we should do something here. By neglecting to add
restrictions to the JVMTI spec, we encouraged JVMTI agents to do these
kind of things. The least we can do is minimize the damage.
And then, usually this will not be the last opportunity for
ResourceExhausted to be posted, no? Chances are high that there will
be more OOMs following, in real java threads.
Finally, I find swallowing the ResourceExhausted for compiler threads
is symmetric to the way the compiler thread swallows the OOME itself.
They do not report the OOME but ignore and clear it.
But as you can see, I see your point. Do you have a better proposal?
..Thomas
Post by Daniel D. Daugherty
Post by Daniel D. Daugherty
Since the proposed solution to the bug is to not post an event, I think
the Serviceability Team (which owns JVM/TI) needs to be involved directly
in the review.
Dan
Dear all,
may I please have reviews on this small patch. Note that this is
borderline serviceability. I try to avoid crossposting, but if you
think this should be looked at by serviceability feel free to forward
it there.
Issue: https://bugs.openjdk.java.net/browse/JDK-8213834
http://cr.openjdk.java.net/~stuefe/webrevs/8213834-jvmti-reourceexhausted-shall-not-be-posted-from-compiler-thread/webrev.00/webrev/
Short description: we may post ResourceExhausted from Compiler
threads. Handlers of this event may call back into the JVM, which may
cause problems since we run in the compiler thread. The proposed
solution is not to post the event in that case.
http://mail.openjdk.java.net/pipermail/serviceability-dev/2018-November/025898.html
Thanks, Thomas
David Holmes
2018-11-15 01:47:02 UTC
Permalink
Hi Dan, Serguei,

I'm going to combine my response to you both into one as it's the same
discussion ...
Post by Daniel D. Daugherty
Post by David Holmes
Post by Daniel D. Daugherty
I do not have a better idea at the moment.
The ResourceExhausted event is one of many JVM/TI events. So we put in a
special case for this event because an agent out in the world tries to do
something that is not expressly forbidden by the spec, but it is a bad
thing to do with HotSpot. Okay. What about the next event? What about
the next agent? Once you've added a special case, where do you stop?
I had proposed more broadly not posting any events in threads that can
not run Java but that is potentially more damaging to existing agents
that just happen to work in the compiler thread today.
But it goes beyond just running Java. Can these other threads make JNI
calls, or other JVM TI calls? If they can run Java they can do all
those things. If they can't run Java they may not be able to.
There is no perfect solution here because JVM TI allows too much
freedom. It doesn't put any constraints on the threads that can
execute callbacks, and has very few constraints on what callbacks can
be executed when.
On the other hand an implementation has complete freedom as to when
exactly resource exhaustion events get posted. Does it make sense for
an internal VM thread to post this ResourceExhausted event? Is it just
an accident of implementation that the code path decided to throw OOME
and so also posts the event? You can easily imagine this code not
doing that - and as Thomas points out the compiler thread swallows the
exception anyway!
As there is no way to know what an agent may do ahead of time we are
- breaking some agents and crashing the VM; or
- masking an event
I vote for no breakage.
I would have no problem suppressing the event from a "hidden" thread
because those threads intentionally try to hide from JVM/TI.
Yes but only for certain things:
- not visible in thread stack dumps
- not able to suspend/resume
- not visible in thread lists/groups
- no thread lifecycle events
- not roots for Heap walking

But as far as I can see "hidden" threads still generate all other JVM TI
events, so excluding them here seems to be too broad and marks a change
in behaviour that seems unjustified.
Post by Daniel D. Daugherty
That would
cover the case for the C1 and C2 CompilerThreads, but I don't know
about these newer JVM/CI Compiler threads that actually run Java code...
The JVMCI threads are not hidden. As Serguei points out in his response
the hidden-ness of a compiler thread is based on its ability to run
Java. So although it comes down to the same thing for compiler threads I
still thing the correct test is for can_call_Java().

[Aside: I think JVMCI needs to re-think how it expects internal compiler
threads to interact with things like JVM TI - we're already seeing
issues with tests 'accidentally' suspending compiler threads and then
getting into deadlocks! ]

JC Beyler made a suggestion in Thomas's original query thread: perhaps
we should defer these events for the ServiceThread to execute? That
would still have to be gated on either the thread being hidden or
(preferably to me) not being able to run Java, but at least it should
avoid the problem whilst still causing the event to appear.

That said, ResourceExhausted events have no context information and to
me they appear to be expected to be synchronous events thrown in the
thread that encountered the resource exhaustion. So to have them post in
the ServiceThread might still be somewhat surprising to an agent.

So my preferred point solution is still to skip posting
ResourceExhausted if the thread can not run Java code.

Cheers,
David
-----
Post by Daniel D. Daugherty
Dan
Post by David Holmes
Cheers,
David
Post by Daniel D. Daugherty
Dan
Post by Thomas Stüfe
Hi Dan,
On Wed, Nov 14, 2018 at 8:32 PM Daniel D. Daugherty
Post by Daniel D. Daugherty
I have a philosophical problem with a fix like this.
The fix is making the assumption that the handler for this event will want
to run Java code and if the event is posted from a Java thread that cannot
run Java code, then we skip posting the event.
If I happen to have a more conservative agent that does not run Java code
in its handlers, then my agent won't receive this event even though it
should not cause my agent any problems. That might be an unexpected change
in behavior on the part of my agent.
Dan
I was thinking about that too. In fact, the JVMTI agent I am trying to
add this fix for exists in two variants. The base form,
airlift/jvmkill, just does a plain kill(2). So not problem there. Only
the forked version by cloudfoundry/jvmkill do this fancy stuff. (Note
that both projects are on github, if you care to look them up. They
are not by me. I am just using them).
But the problem is, the JVMTI spec says nothing about what you can or
cannot do in reaction to a ResourceExhausted event. So what
cloudfoundry/jvmkill does is valid and not forbidden.
Therefore I think suppressing ResourceExhausted in this case is the
only choice we have. One might fine-tune the conditions under which we
suppress sending ResourceExhausted: maybe suppress for
CompilerThreads, or only  for CompilerThreads getting MetaspaceOOMs...
But I think we should do something here. By neglecting to add
restrictions to the JVMTI spec, we encouraged JVMTI agents to do these
kind of things. The least we can do is minimize the damage.
And then, usually this will not be the last opportunity for
ResourceExhausted to be posted, no? Chances are high that there will
be more OOMs following, in real java threads.
Finally, I find swallowing the ResourceExhausted for compiler threads
is symmetric to the way the compiler thread swallows the OOME itself.
They do not report the OOME but ignore and clear it.
But as you can see, I see your point. Do you have a better proposal?
..Thomas
Post by Daniel D. Daugherty
Post by Daniel D. Daugherty
Since the proposed solution to the bug is to not post an event, I think
the Serviceability Team (which owns JVM/TI) needs to be involved directly
in the review.
Dan
Dear all,
may I please have reviews on this small patch. Note that this is
borderline serviceability. I try to avoid crossposting, but if you
think this should be looked at by serviceability feel free to forward
it there.
Issue: https://bugs.openjdk.java.net/browse/JDK-8213834
http://cr.openjdk.java.net/~stuefe/webrevs/8213834-jvmti-reourceexhausted-shall-not-be-posted-from-compiler-thread/webrev.00/webrev/
Short description: we may post ResourceExhausted from Compiler
threads. Handlers of this event may call back into the JVM, which may
cause problems since we run in the compiler thread. The proposed
solution is not to post the event in that case.
http://mail.openjdk.java.net/pipermail/serviceability-dev/2018-November/025898.html
Thanks, Thomas
JC Beyler
2018-11-15 04:32:26 UTC
Permalink
For what it's worth I wonder if skipping the event is the best; it is the
easiest to ensure non breaking the agent; it does not really go against
what the spec is saying and another thread that CAN call java will most
likely hit the issue and then all will be good in the world.

However, also for what it's worth I wonder if deferring is not that hard to
accomplish either. There already is the infrastructure for this and we
should be relatively able to do it. Only question I would have is can the
service thread create a JNIEnv for the event but I don't think that's an
issue, is it?

It might however conflict with the description of the JNIEnv in the spec
which says the jni_env is "The JNI environment of the event (current)
thread" though it doesn't say current of what or the event thread of what
really.

However, skipping it also kind of goes against the spec since it says:
"Sent when a VM resource needed by a running application has been
exhausted".Though someone could argue it doesn' t say it is sent when the
resource was first noticed to be exhausted.

So, if I over-read the spec and look at options, it seems that:
- Sending the event via the compiler thread will risk breaking things if
the agent calls Java -> not really an option
- Using the service thread breaks what David calls the synchronous
assumption of the event
- Skipping the event kind of breaks the sentence that the event is sent
when a VM resource has been exhausted

So we come back to probably skipping is the best solution since at least
the event remains "synchronous" when you get it.

(A side note would be perhaps to augment ThreadInfo* with the
"can_call_java" bit and then put in the right spots of the spec that only
threads marked as "can_call_java" can safely call Java).

Thanks,
Jc
Thomas Stüfe
2018-11-19 12:00:25 UTC
Permalink
Hi all,

David and JC already outlined the options we have nicely.

I'd like to add that I do not favor the ServiceThread delayed
deliverance since a common reaction to ResourceExhausted would to
print call stack of the thread running into it or to print a heap
histogram, as jvmkill does. For the former only a synchronous event
delivery makes sense, for the latter at least it helps analyzing.

In cloud foundry, the heap histogram produced by the JVMTI agent can
be helpful, and since in the majority of cases do not entail the
compiler thread running out of metaspace, I'd rather preserve this
ability. So to me, masking this event for this one case is the most
pragmatic approach.

The other option would be not to change the code but to add, in the
JVMTI documentation for ResourceExhausted, the same disclaimer as for
ObjectFree, GCStarted etc: "The event handler must not use JNI
functions and must not use JVM TI functions except those which
specifically allow such use". Then, writers of JVMTI agents like
jvmkill would have to update their agents accordingly.

FWIW, I think JCs idea of exposing the can_call_java attribute somehow
to the outside would also work. But unfortunately not retroactively,
for older releases. Whereas a simple internal patch like "mask
ResourceExhausted" could be backported easily to older releases.

Best Regards, Thomas
For what it's worth I wonder if skipping the event is the best; it is the easiest to ensure non breaking the agent; it does not really go against what the spec is saying and another thread that CAN call java will most likely hit the issue and then all will be good in the world.
However, also for what it's worth I wonder if deferring is not that hard to accomplish either. There already is the infrastructure for this and we should be relatively able to do it. Only question I would have is can the service thread create a JNIEnv for the event but I don't think that's an issue, is it?
It might however conflict with the description of the JNIEnv in the spec which says the jni_env is "The JNI environment of the event (current) thread" though it doesn't say current of what or the event thread of what really.
However, skipping it also kind of goes against the spec since it says: "Sent when a VM resource needed by a running application has been exhausted".Though someone could argue it doesn' t say it is sent when the resource was first noticed to be exhausted.
- Sending the event via the compiler thread will risk breaking things if the agent calls Java -> not really an option
- Using the service thread breaks what David calls the synchronous assumption of the event
- Skipping the event kind of breaks the sentence that the event is sent when a VM resource has been exhausted
So we come back to probably skipping is the best solution since at least the event remains "synchronous" when you get it.
(A side note would be perhaps to augment ThreadInfo* with the "can_call_java" bit and then put in the right spots of the spec that only threads marked as "can_call_java" can safely call Java).
Thanks,
Jc
JC Beyler
2018-11-21 17:03:45 UTC
Permalink
Hi Thomas,
Post by Thomas Stüfe
Hi all,
David and JC already outlined the options we have nicely.
I'd like to add that I do not favor the ServiceThread delayed
deliverance since a common reaction to ResourceExhausted would to
print call stack of the thread running into it or to print a heap
histogram, as jvmkill does. For the former only a synchronous event
delivery makes sense, for the latter at least it helps analyzing.
In cloud foundry, the heap histogram produced by the JVMTI agent can
be helpful, and since in the majority of cases do not entail the
compiler thread running out of metaspace, I'd rather preserve this
ability. So to me, masking this event for this one case is the most
pragmatic approach.
The other option would be not to change the code but to add, in the
JVMTI documentation for ResourceExhausted, the same disclaimer as for
ObjectFree, GCStarted etc: "The event handler must not use JNI
functions and must not use JVM TI functions except those which
specifically allow such use". Then, writers of JVMTI agents like
jvmkill would have to update their agents accordingly.
FWIW, I think JCs idea of exposing the can_call_java attribute somehow
to the outside would also work. But unfortunately not retroactively,
for older releases. Whereas a simple internal patch like "mask
ResourceExhausted" could be backported easily to older releases.
I actually agree with not using the service thread to be honest, resource
exhaustion seems to be something you'd want to know sooner than later.

How about we do both?
- Fix it now so that the compiler thread does not post the event because
we'd rather not have crashes and that can get backported
- Put out a RFE that would add the information to ThreadInfo and work
through the process of a CSR/etc. to get it working for future versions?

Thanks,
Jc
Post by Thomas Stüfe
Best Regards, Thomas
Post by JC Beyler
For what it's worth I wonder if skipping the event is the best; it is
the easiest to ensure non breaking the agent; it does not really go against
what the spec is saying and another thread that CAN call java will most
likely hit the issue and then all will be good in the world.
Post by JC Beyler
However, also for what it's worth I wonder if deferring is not that hard
to accomplish either. There already is the infrastructure for this and we
should be relatively able to do it. Only question I would have is can the
service thread create a JNIEnv for the event but I don't think that's an
issue, is it?
Post by JC Beyler
It might however conflict with the description of the JNIEnv in the spec
which says the jni_env is "The JNI environment of the event (current)
thread" though it doesn't say current of what or the event thread of what
really.
"Sent when a VM resource needed by a running application has been
exhausted".Though someone could argue it doesn' t say it is sent when the
resource was first noticed to be exhausted.
Post by JC Beyler
- Sending the event via the compiler thread will risk breaking things if
the agent calls Java -> not really an option
Post by JC Beyler
- Using the service thread breaks what David calls the synchronous
assumption of the event
Post by JC Beyler
- Skipping the event kind of breaks the sentence that the event is sent
when a VM resource has been exhausted
Post by JC Beyler
So we come back to probably skipping is the best solution since at least
the event remains "synchronous" when you get it.
Post by JC Beyler
(A side note would be perhaps to augment ThreadInfo* with the
"can_call_java" bit and then put in the right spots of the spec that only
threads marked as "can_call_java" can safely call Java).
Post by JC Beyler
Thanks,
Jc
--
Thanks,
Jc
Thomas Stüfe
2018-11-22 07:36:38 UTC
Permalink
Hi JC,
Post by s***@oracle.com
Hi Thomas,
<snip>
Post by s***@oracle.com
I actually agree with not using the service thread to be honest, resource exhaustion seems to be something you'd want to know sooner than later.
How about we do both?
- Fix it now so that the compiler thread does not post the event because we'd rather not have crashes and that can get backported
- Put out a RFE that would add the information to ThreadInfo and work through the process of a CSR/etc. to get it working for future versions?
Thanks,
Jc
Makes sense, sure. But both Dan and Serguei voiced opposition to this
patch. Dan, Serguei, what do you think?

If we do not find an agreement, I would rather close this bug as WNF
than push it in without consensus. I would then just fix it downstream
in our port.

Your proposed RFE would still make sense, but not in jdk12 anymore,
let alone in older releases.

Thanks, Thomas
David Holmes
2018-11-22 07:42:35 UTC
Permalink
Post by Thomas Stüfe
Hi JC,
Post by s***@oracle.com
Hi Thomas,
<snip>
Post by s***@oracle.com
I actually agree with not using the service thread to be honest, resource exhaustion seems to be something you'd want to know sooner than later.
How about we do both?
- Fix it now so that the compiler thread does not post the event because we'd rather not have crashes and that can get backported
- Put out a RFE that would add the information to ThreadInfo and work through the process of a CSR/etc. to get it working for future versions?
Thanks,
Jc
Makes sense, sure. But both Dan and Serguei voiced opposition to this
patch. Dan, Serguei, what do you think?
I note neither Dan nor Serguei responded to my response to them :)

Cheers,
David
-----
Post by Thomas Stüfe
If we do not find an agreement, I would rather close this bug as WNF
than push it in without consensus. I would then just fix it downstream
in our port.
Your proposed RFE would still make sense, but not in jdk12 anymore,
let alone in older releases.
Thanks, Thomas
s***@oracle.com
2018-11-22 08:06:23 UTC
Permalink
Post by David Holmes
Post by Thomas Stüfe
Hi JC,
Post by s***@oracle.com
Hi Thomas,
<snip>
Post by s***@oracle.com
I actually agree with not using the service thread to be honest,
resource exhaustion seems to be something you'd want to know sooner
than later.
How about we do both?
   - Fix it now so that the compiler thread does not post the event
because we'd rather not have crashes and that can get backported
   - Put out a RFE that would add the information to ThreadInfo and
work through the process of a CSR/etc. to get it working for future
versions?
Thanks,
Jc
Makes sense, sure. But both Dan and Serguei voiced opposition to this
patch. Dan, Serguei, what do you think?
Hi David,

I guess, it is again some issue with the mailing system.
Please, find my and Dan's replies to your email in the attachments.

I'm still thinking about what would be a better choice here.

Thanks,
Serguei
Post by David Holmes
Cheers,
David
-----
Post by Thomas Stüfe
If we do not find an agreement, I would rather close this bug as WNF
than push it in without consensus. I would then just fix it downstream
in our port.
Your proposed RFE would still make sense, but not in jdk12 anymore,
let alone in older releases.
Thanks, Thomas
David Holmes
2018-11-22 08:12:51 UTC
Permalink
Hi Serguei,

Must be a mail system issue. This is the reply I wrote in response to
the two replies you had attached:

http://mail.openjdk.java.net/pipermail/serviceability-dev/2018-November/025927.html

"Hi Dan, Serguei,

I'm going to combine my response to you both into one as it's the same
discussion ..."

:)

Cheers,
David
Post by s***@oracle.com
Post by David Holmes
Post by Thomas Stüfe
Hi JC,
Post by s***@oracle.com
Hi Thomas,
<snip>
Post by s***@oracle.com
I actually agree with not using the service thread to be honest,
resource exhaustion seems to be something you'd want to know sooner
than later.
How about we do both?
   - Fix it now so that the compiler thread does not post the event
because we'd rather not have crashes and that can get backported
   - Put out a RFE that would add the information to ThreadInfo and
work through the process of a CSR/etc. to get it working for future
versions?
Thanks,
Jc
Makes sense, sure. But both Dan and Serguei voiced opposition to this
patch. Dan, Serguei, what do you think?
Hi David,
I guess, it is again some issue with the mailing system.
Please, find my and Dan's replies to your email in the attachments.
I'm still thinking about what would be a better choice here.
Thanks,
Serguei
Post by David Holmes
Cheers,
David
-----
Post by Thomas Stüfe
If we do not find an agreement, I would rather close this bug as WNF
than push it in without consensus. I would then just fix it downstream
in our port.
Your proposed RFE would still make sense, but not in jdk12 anymore,
let alone in older releases.
Thanks, Thomas
s***@oracle.com
2018-11-26 03:13:13 UTC
Permalink
Hi David,
Post by David Holmes
Hi Serguei,
Must be a mail system issue. This is the reply I wrote in response to
http://mail.openjdk.java.net/pipermail/serviceability-dev/2018-November/025927.html
In fact, I saw this reply from you.
Sorry for confusion.
I need to catch up on all the emails on this topic and will post my reply.
Sorry for long silence, we have the Holidays in US.


Thanks,
Serguei
Post by David Holmes
"Hi Dan, Serguei,
I'm going to combine my response to you both into one as it's the same
discussion ..."
:)
Cheers,
David
Post by s***@oracle.com
Post by David Holmes
Post by Thomas Stüfe
Hi JC,
Post by s***@oracle.com
Hi Thomas,
<snip>
Post by s***@oracle.com
I actually agree with not using the service thread to be honest,
resource exhaustion seems to be something you'd want to know
sooner than later.
How about we do both?
   - Fix it now so that the compiler thread does not post the
event because we'd rather not have crashes and that can get
backported
   - Put out a RFE that would add the information to ThreadInfo
and work through the process of a CSR/etc. to get it working for
future versions?
Thanks,
Jc
Makes sense, sure. But both Dan and Serguei voiced opposition to this
patch. Dan, Serguei, what do you think?
Hi David,
I guess, it is again some issue with the mailing system.
Please, find my and Dan's replies to your email in the attachments.
I'm still thinking about what would be a better choice here.
Thanks,
Serguei
Post by David Holmes
Cheers,
David
-----
Post by Thomas Stüfe
If we do not find an agreement, I would rather close this bug as WNF
than push it in without consensus. I would then just fix it downstream
in our port.
Your proposed RFE would still make sense, but not in jdk12 anymore,
let alone in older releases.
Thanks, Thomas
Daniel D. Daugherty
2018-11-22 15:21:46 UTC
Permalink
Post by David Holmes
Post by Thomas Stüfe
Hi JC,
Post by s***@oracle.com
Hi Thomas,
<snip>
Post by s***@oracle.com
I actually agree with not using the service thread to be honest,
resource exhaustion seems to be something you'd want to know sooner
than later.
How about we do both?
   - Fix it now so that the compiler thread does not post the event
because we'd rather not have crashes and that can get backported
   - Put out a RFE that would add the information to ThreadInfo and
work through the process of a CSR/etc. to get it working for future
versions?
Thanks,
Jc
Makes sense, sure. But both Dan and Serguei voiced opposition to this
patch. Dan, Serguei, what do you think?
I note neither Dan nor Serguei responded to my response to them :)
I would have no problem suppressing the event from a "hidden" thread
because those threads intentionally try to hide from JVM/TI. That would
cover the case for the C1 and C2 CompilerThreads, but I don't know
about these newer JVM/CI Compiler threads that actually run Java code...
I read your combined response to this as not in conflict with what I said.
Post by David Holmes
So my preferred point solution is still to skip posting ResourceExhausted
if the thread can not run Java code.
matches what I said since the C1 and C2 compiler threads are hidden and
cannot run Java code. We're in agreement.

Dan
Post by David Holmes
Cheers,
David
-----
Post by Thomas Stüfe
If we do not find an agreement, I would rather close this bug as WNF
than push it in without consensus. I would then just fix it downstream
in our port.
Your proposed RFE would still make sense, but not in jdk12 anymore,
let alone in older releases.
Thanks, Thomas
Thomas Stüfe
2018-11-22 16:16:03 UTC
Permalink
Hi all,

would such a patch be acceptable:

http://cr.openjdk.java.net/~stuefe/webrevs/8213834-jvmti-reourceexhausted-shall-not-be-posted-from-compiler-thread/webrev.01/webrev/
?

As has been proposed, I am now using
thread->is_hidden_from_external_view() to suppress the event.

Side note, this function apparently should only ever be called from
JavaThreads? To my knowledge we do not guard metaspace against
allocation from non-java-threads, should we then do that?

I attempted to create a jtreg test for this but this turns out to be difficult:

Attempting to trigger a metaspace OOM from a compiler thread proved
futile - chances for that to happen are just too low compared to other
metaspace users. Only way to do this reliably would be to artificially
allocate a lot of metaspace from the compiler thread - triggered by a
test switch or similar? That would be very ugly though. And then, I
would need to add a jvmti agent to jtreg, or adapt
jtreg/vmTestbase/nsk/jvmti?

Thanks, Thomas
On Thu, Nov 22, 2018 at 4:22 PM Daniel D. Daugherty
Post by Daniel D. Daugherty
Post by David Holmes
Post by Thomas Stüfe
Hi JC,
Post by s***@oracle.com
Hi Thomas,
<snip>
Post by s***@oracle.com
I actually agree with not using the service thread to be honest,
resource exhaustion seems to be something you'd want to know sooner
than later.
How about we do both?
- Fix it now so that the compiler thread does not post the event
because we'd rather not have crashes and that can get backported
- Put out a RFE that would add the information to ThreadInfo and
work through the process of a CSR/etc. to get it working for future
versions?
Thanks,
Jc
Makes sense, sure. But both Dan and Serguei voiced opposition to this
patch. Dan, Serguei, what do you think?
I note neither Dan nor Serguei responded to my response to them :)
I would have no problem suppressing the event from a "hidden" thread
because those threads intentionally try to hide from JVM/TI. That would
cover the case for the C1 and C2 CompilerThreads, but I don't know
about these newer JVM/CI Compiler threads that actually run Java code...
I read your combined response to this as not in conflict with what I said.
Post by David Holmes
So my preferred point solution is still to skip posting ResourceExhausted
if the thread can not run Java code.
matches what I said since the C1 and C2 compiler threads are hidden and
cannot run Java code. We're in agreement.
Dan
Post by David Holmes
Cheers,
David
-----
Post by Thomas Stüfe
If we do not find an agreement, I would rather close this bug as WNF
than push it in without consensus. I would then just fix it downstream
in our port.
Your proposed RFE would still make sense, but not in jdk12 anymore,
let alone in older releases.
Thanks, Thomas
David Holmes
2018-11-22 22:24:19 UTC
Permalink
Hi Thomas,
Post by Thomas Stüfe
Hi all,
http://cr.openjdk.java.net/~stuefe/webrevs/8213834-jvmti-reourceexhausted-shall-not-be-posted-from-compiler-thread/webrev.01/webrev/
?
As has been proposed, I am now using
thread->is_hidden_from_external_view() to suppress the event.
I still disagree with this for reason previously outlined. It not only
excludes the CompilerThreads that can't run Java but all
CompilerThreads, and so excludes JVMCI compiler threads that do
everything in Java and so will quite possibly trigger resource
exhaustion. It also excludes the ServiceThread and
CodeCacheSweeperThread. While the latter seems insignificant I think the
ServiceThread is more significant. Those threads are not excluded from
posting other events AFAICS so I don't see why this event should be
treated specially for them.
Post by Thomas Stüfe
Side note, this function apparently should only ever be called from
JavaThreads? To my knowledge we do not guard metaspace against
allocation from non-java-threads, should we then do that?
Not sure how it is arranged but certainly Metaspace::allocate expects to
only be called from a JavaThread as it immediately checks for

if (HAS_PENDING_EXCEPTION) {

Cheers,
David
-----
Post by Thomas Stüfe
Attempting to trigger a metaspace OOM from a compiler thread proved
futile - chances for that to happen are just too low compared to other
metaspace users. Only way to do this reliably would be to artificially
allocate a lot of metaspace from the compiler thread - triggered by a
test switch or similar? That would be very ugly though. And then, I
would need to add a jvmti agent to jtreg, or adapt
jtreg/vmTestbase/nsk/jvmti?
Thanks, Thomas
On Thu, Nov 22, 2018 at 4:22 PM Daniel D. Daugherty
Post by Daniel D. Daugherty
Post by David Holmes
Post by Thomas Stüfe
Hi JC,
Post by s***@oracle.com
Hi Thomas,
<snip>
Post by s***@oracle.com
I actually agree with not using the service thread to be honest,
resource exhaustion seems to be something you'd want to know sooner
than later.
How about we do both?
- Fix it now so that the compiler thread does not post the event
because we'd rather not have crashes and that can get backported
- Put out a RFE that would add the information to ThreadInfo and
work through the process of a CSR/etc. to get it working for future
versions?
Thanks,
Jc
Makes sense, sure. But both Dan and Serguei voiced opposition to this
patch. Dan, Serguei, what do you think?
I note neither Dan nor Serguei responded to my response to them :)
I would have no problem suppressing the event from a "hidden" thread
because those threads intentionally try to hide from JVM/TI. That would
cover the case for the C1 and C2 CompilerThreads, but I don't know
about these newer JVM/CI Compiler threads that actually run Java code...
I read your combined response to this as not in conflict with what I said.
Post by David Holmes
So my preferred point solution is still to skip posting ResourceExhausted
if the thread can not run Java code.
matches what I said since the C1 and C2 compiler threads are hidden and
cannot run Java code. We're in agreement.
Dan
Post by David Holmes
Cheers,
David
-----
Post by Thomas Stüfe
If we do not find an agreement, I would rather close this bug as WNF
than push it in without consensus. I would then just fix it downstream
in our port.
Your proposed RFE would still make sense, but not in jdk12 anymore,
let alone in older releases.
Thanks, Thomas
Daniel D. Daugherty
2018-11-24 14:41:43 UTC
Permalink
Post by s***@oracle.com
Hi Thomas,
Post by Thomas Stüfe
Hi all,
http://cr.openjdk.java.net/~stuefe/webrevs/8213834-jvmti-reourceexhausted-shall-not-be-posted-from-compiler-thread/webrev.01/webrev/
?
As has been proposed, I am now using
thread->is_hidden_from_external_view() to suppress the event.
I still disagree with this for reason previously outlined. It not only
excludes the CompilerThreads that can't run Java but all
CompilerThreads, and so excludes JVMCI compiler threads that do
everything in Java and so will quite possibly trigger resource exhaustion.
I thought the CompilerThread is_hidden_from_external_view() is built on
top of can_call_java():

  // Hide native compiler threads from external view.
  bool is_hidden_from_external_view() const      { return
!can_call_java(); }

so is_hidden_from_external_view() does not exclude JVMCI...

Of course, the question of whether JVMCI threads should be participating
in JVM/TI events and other stuff is being debugged and explored at the
current time.
Post by s***@oracle.com
It also excludes the ServiceThread and CodeCacheSweeperThread. While
the latter seems insignificant I think the ServiceThread is more
significant. Those threads are not excluded from posting other events
AFAICS so I don't see why this event should be treated specially for
them.
Hmmm... I don't think the ServiceThread should be posting JVM/TI events
either. We definitely don't want the ServiceThread running "arbitrary"
event handler code since taking down the ServiceThread could mess up
various things that rely on it running.

I don't have an opinion about the CodeCacheSweeperThread.

dan
Post by s***@oracle.com
Post by Thomas Stüfe
Side note, this function apparently should only ever be called from
JavaThreads? To my knowledge we do not guard metaspace against
allocation from non-java-threads, should we then do that?
Not sure how it is arranged but certainly Metaspace::allocate expects
to only be called from a JavaThread as it immediately checks for
 if (HAS_PENDING_EXCEPTION) {
Cheers,
David
-----
Post by Thomas Stüfe
Attempting to trigger a metaspace OOM from a compiler thread proved
futile - chances for that to happen are just too low compared to other
metaspace users. Only way to do this reliably would be to artificially
allocate a lot of metaspace from the compiler thread - triggered by a
test switch or similar? That would be very ugly though. And then, I
would need to add a jvmti agent to jtreg, or adapt
jtreg/vmTestbase/nsk/jvmti?
Thanks, Thomas
On Thu, Nov 22, 2018 at 4:22 PM Daniel D. Daugherty
Post by Daniel D. Daugherty
Post by David Holmes
Post by Thomas Stüfe
Hi JC,
Post by s***@oracle.com
Hi Thomas,
<snip>
Post by s***@oracle.com
I actually agree with not using the service thread to be honest,
resource exhaustion seems to be something you'd want to know sooner
than later.
How about we do both?
    - Fix it now so that the compiler thread does not post the event
because we'd rather not have crashes and that can get backported
    - Put out a RFE that would add the information to ThreadInfo and
work through the process of a CSR/etc. to get it working for future
versions?
Thanks,
Jc
Makes sense, sure. But both Dan and Serguei voiced opposition to this
patch. Dan, Serguei, what do you think?
I note neither Dan nor Serguei responded to my response to them :)
I would have no problem suppressing the event from a "hidden" thread
because those threads intentionally try to hide from JVM/TI. That would
cover the case for the C1 and C2 CompilerThreads, but I don't know
about these newer JVM/CI Compiler threads that actually run Java code...
I read your combined response to this as not in conflict with what I said.
  > So my preferred point solution is still to skip posting
ResourceExhausted
  > if the thread can not run Java code.
matches what I said since the C1 and C2 compiler threads are hidden and
cannot run Java code. We're in agreement.
Dan
Post by David Holmes
Cheers,
David
-----
Post by Thomas Stüfe
If we do not find an agreement, I would rather close this bug as WNF
than push it in without consensus. I would then just fix it downstream
in our port.
Your proposed RFE would still make sense, but not in jdk12 anymore,
let alone in older releases.
Thanks, Thomas
David Holmes
2018-11-25 11:57:46 UTC
Permalink
Post by Daniel D. Daugherty
Post by s***@oracle.com
Hi Thomas,
Post by Thomas Stüfe
Hi all,
http://cr.openjdk.java.net/~stuefe/webrevs/8213834-jvmti-reourceexhausted-shall-not-be-posted-from-compiler-thread/webrev.01/webrev/
?
As has been proposed, I am now using
thread->is_hidden_from_external_view() to suppress the event.
I still disagree with this for reason previously outlined. It not only
excludes the CompilerThreads that can't run Java but all
CompilerThreads, and so excludes JVMCI compiler threads that do
everything in Java and so will quite possibly trigger resource exhaustion.
I thought the CompilerThread is_hidden_from_external_view() is built on
  // Hide native compiler threads from external view.
  bool is_hidden_from_external_view() const      { return
!can_call_java(); }
so is_hidden_from_external_view() does not exclude JVMCI...
Yes sorry my mistake.
Post by Daniel D. Daugherty
Of course, the question of whether JVMCI threads should be participating
in JVM/TI events and other stuff is being debugged and explored at the
current time.
Post by s***@oracle.com
It also excludes the ServiceThread and CodeCacheSweeperThread. While
the latter seems insignificant I think the ServiceThread is more
significant. Those threads are not excluded from posting other events
AFAICS so I don't see why this event should be treated specially for
them.
Hmmm... I don't think the ServiceThread should be posting JVM/TI events
But it already does! It's already posting deferred JVM TI events
explicitly as part of its job, and I see nothing that excludes it from
posting JVM TI events as part of any other Java code it runs.

David
-----
Post by Daniel D. Daugherty
either. We definitely don't want the ServiceThread running "arbitrary"
event handler code since taking down the ServiceThread could mess up
various things that rely on it running.
I don't have an opinion about the CodeCacheSweeperThread.
dan
Post by s***@oracle.com
Post by Thomas Stüfe
Side note, this function apparently should only ever be called from
JavaThreads? To my knowledge we do not guard metaspace against
allocation from non-java-threads, should we then do that?
Not sure how it is arranged but certainly Metaspace::allocate expects
to only be called from a JavaThread as it immediately checks for
 if (HAS_PENDING_EXCEPTION) {
Cheers,
David
-----
Post by Thomas Stüfe
Attempting to trigger a metaspace OOM from a compiler thread proved
futile - chances for that to happen are just too low compared to other
metaspace users. Only way to do this reliably would be to artificially
allocate a lot of metaspace from the compiler thread - triggered by a
test switch or similar? That would be very ugly though. And then, I
would need to add a jvmti agent to jtreg, or adapt
jtreg/vmTestbase/nsk/jvmti?
Thanks, Thomas
On Thu, Nov 22, 2018 at 4:22 PM Daniel D. Daugherty
Post by Daniel D. Daugherty
Post by David Holmes
Post by Thomas Stüfe
Hi JC,
Post by s***@oracle.com
Hi Thomas,
<snip>
Post by s***@oracle.com
I actually agree with not using the service thread to be honest,
resource exhaustion seems to be something you'd want to know sooner
than later.
How about we do both?
    - Fix it now so that the compiler thread does not post the event
because we'd rather not have crashes and that can get backported
    - Put out a RFE that would add the information to ThreadInfo and
work through the process of a CSR/etc. to get it working for future
versions?
Thanks,
Jc
Makes sense, sure. But both Dan and Serguei voiced opposition to this
patch. Dan, Serguei, what do you think?
I note neither Dan nor Serguei responded to my response to them :)
I would have no problem suppressing the event from a "hidden" thread
because those threads intentionally try to hide from JVM/TI. That would
cover the case for the C1 and C2 CompilerThreads, but I don't know
about these newer JVM/CI Compiler threads that actually run Java code...
I read your combined response to this as not in conflict with what I said.
  > So my preferred point solution is still to skip posting
ResourceExhausted
  > if the thread can not run Java code.
matches what I said since the C1 and C2 compiler threads are hidden and
cannot run Java code. We're in agreement.
Dan
Post by David Holmes
Cheers,
David
-----
Post by Thomas Stüfe
If we do not find an agreement, I would rather close this bug as WNF
than push it in without consensus. I would then just fix it downstream
in our port.
Your proposed RFE would still make sense, but not in jdk12 anymore,
let alone in older releases.
Thanks, Thomas
Daniel D. Daugherty
2018-11-25 14:17:12 UTC
Permalink
Post by David Holmes
Post by Daniel D. Daugherty
Post by s***@oracle.com
Hi Thomas,
Post by Thomas Stüfe
Hi all,
http://cr.openjdk.java.net/~stuefe/webrevs/8213834-jvmti-reourceexhausted-shall-not-be-posted-from-compiler-thread/webrev.01/webrev/
?
As has been proposed, I am now using
thread->is_hidden_from_external_view() to suppress the event.
I still disagree with this for reason previously outlined. It not
only excludes the CompilerThreads that can't run Java but all
CompilerThreads, and so excludes JVMCI compiler threads that do
everything in Java and so will quite possibly trigger resource exhaustion.
I thought the CompilerThread is_hidden_from_external_view() is built on
   // Hide native compiler threads from external view.
   bool is_hidden_from_external_view() const      { return
!can_call_java(); }
so is_hidden_from_external_view() does not exclude JVMCI...
Yes sorry my mistake.
Post by Daniel D. Daugherty
Of course, the question of whether JVMCI threads should be participating
in JVM/TI events and other stuff is being debugged and explored at the
current time.
Post by s***@oracle.com
It also excludes the ServiceThread and CodeCacheSweeperThread. While
the latter seems insignificant I think the ServiceThread is more
significant. Those threads are not excluded from posting other
events AFAICS so I don't see why this event should be treated
specially for them.
Hmmm... I don't think the ServiceThread should be posting JVM/TI events
But it already does! It's already posting deferred JVM TI events
explicitly as part of its job, and I see nothing that excludes it from
posting JVM TI events as part of any other Java code it runs.
You're right. I completely forgot about the ServiceThread doing proxy
event posting duty. So I think where we are is switching back to using

    can_call_java()

to restrict the posting of ResourceExhausted events. Less limiting than
!is_hidden_from_external_view(), but still limited. I'm reluctantly okay
with this, but we need to hear from Serguei.

Dan
Post by David Holmes
David
-----
Post by Daniel D. Daugherty
either. We definitely don't want the ServiceThread running "arbitrary"
event handler code since taking down the ServiceThread could mess up
various things that rely on it running.
I don't have an opinion about the CodeCacheSweeperThread.
dan
Post by s***@oracle.com
Post by Thomas Stüfe
Side note, this function apparently should only ever be called from
JavaThreads? To my knowledge we do not guard metaspace against
allocation from non-java-threads, should we then do that?
Not sure how it is arranged but certainly Metaspace::allocate
expects to only be called from a JavaThread as it immediately checks
for
 if (HAS_PENDING_EXCEPTION) {
Cheers,
David
-----
Post by Thomas Stüfe
Attempting to trigger a metaspace OOM from a compiler thread proved
futile - chances for that to happen are just too low compared to other
metaspace users. Only way to do this reliably would be to artificially
allocate a lot of metaspace from the compiler thread - triggered by a
test switch or similar? That would be very ugly though. And then, I
would need to add a jvmti agent to jtreg, or adapt
jtreg/vmTestbase/nsk/jvmti?
Thanks, Thomas
On Thu, Nov 22, 2018 at 4:22 PM Daniel D. Daugherty
Post by Daniel D. Daugherty
Post by David Holmes
Post by Thomas Stüfe
Hi JC,
Post by s***@oracle.com
Hi Thomas,
<snip>
Post by s***@oracle.com
I actually agree with not using the service thread to be honest,
resource exhaustion seems to be something you'd want to know sooner
than later.
How about we do both?
    - Fix it now so that the compiler thread does not post the event
because we'd rather not have crashes and that can get backported
    - Put out a RFE that would add the information to ThreadInfo and
work through the process of a CSR/etc. to get it working for future
versions?
Thanks,
Jc
Makes sense, sure. But both Dan and Serguei voiced opposition to this
patch. Dan, Serguei, what do you think?
I note neither Dan nor Serguei responded to my response to them :)
I would have no problem suppressing the event from a "hidden" thread
because those threads intentionally try to hide from JVM/TI. That would
cover the case for the C1 and C2 CompilerThreads, but I don't know
about these newer JVM/CI Compiler threads that actually run Java code...
I read your combined response to this as not in conflict with what I said.
  > So my preferred point solution is still to skip posting
ResourceExhausted
  > if the thread can not run Java code.
matches what I said since the C1 and C2 compiler threads are hidden and
cannot run Java code. We're in agreement.
Dan
Post by David Holmes
Cheers,
David
-----
Post by Thomas Stüfe
If we do not find an agreement, I would rather close this bug as WNF
than push it in without consensus. I would then just fix it downstream
in our port.
Your proposed RFE would still make sense, but not in jdk12 anymore,
let alone in older releases.
Thanks, Thomas
s***@oracle.com
2018-11-26 03:58:04 UTC
Permalink
Hi guys,
Post by Daniel D. Daugherty
Post by David Holmes
Post by Daniel D. Daugherty
Post by s***@oracle.com
Hi Thomas,
Post by Thomas Stüfe
Hi all,
http://cr.openjdk.java.net/~stuefe/webrevs/8213834-jvmti-reourceexhausted-shall-not-be-posted-from-compiler-thread/webrev.01/webrev/
?
As has been proposed, I am now using
thread->is_hidden_from_external_view() to suppress the event.
I still disagree with this for reason previously outlined. It not
only excludes the CompilerThreads that can't run Java but all
CompilerThreads, and so excludes JVMCI compiler threads that do
everything in Java and so will quite possibly trigger resource exhaustion.
I thought the CompilerThread is_hidden_from_external_view() is built on
   // Hide native compiler threads from external view.
   bool is_hidden_from_external_view() const      { return
!can_call_java(); }
so is_hidden_from_external_view() does not exclude JVMCI...
Yes sorry my mistake.
There are many confusions around this.
Unfortunately, I make the same mistakes. :(
Post by Daniel D. Daugherty
Post by David Holmes
Post by Daniel D. Daugherty
Of course, the question of whether JVMCI threads should be
participating
in JVM/TI events and other stuff is being debugged and explored at the
current time.
Post by s***@oracle.com
It also excludes the ServiceThread and CodeCacheSweeperThread.
While the latter seems insignificant I think the ServiceThread is
more significant. Those threads are not excluded from posting other
events AFAICS so I don't see why this event should be treated
specially for them.
Hmmm... I don't think the ServiceThread should be posting JVM/TI events
But it already does! It's already posting deferred JVM TI events
explicitly as part of its job, and I see nothing that excludes it
from posting JVM TI events as part of any other Java code it runs.
You're right. I completely forgot about the ServiceThread doing proxy
event posting duty. So I think where we are is switching back to using
    can_call_java()
to restrict the posting of ResourceExhausted events. Less limiting than
!is_hidden_from_external_view(), but still limited. I'm reluctantly okay
with this, but we need to hear from Serguei.
I've read all the discussion thread.
Thank you to everyone participating in this discussion - it is very helpful.
Now, I'm convinced that using can_call_java (not
is_hidden_from_external_view)
is right (thanks a lot to David for defending this point!)

I still have a concern though.
If we skip a ResourceExhausted event this way (as in the webrev above)
then can we expect similar event posted on another (executing Java) thread?
I have a doubt that we skip it correctly (I need to learn the code more).


Thanks,
Serguei
Post by Daniel D. Daugherty
Dan
Post by David Holmes
David
-----
Post by Daniel D. Daugherty
either. We definitely don't want the ServiceThread running "arbitrary"
event handler code since taking down the ServiceThread could mess up
various things that rely on it running.
I don't have an opinion about the CodeCacheSweeperThread.
dan
Post by s***@oracle.com
Post by Thomas Stüfe
Side note, this function apparently should only ever be called from
JavaThreads? To my knowledge we do not guard metaspace against
allocation from non-java-threads, should we then do that?
Not sure how it is arranged but certainly Metaspace::allocate
expects to only be called from a JavaThread as it immediately
checks for
 if (HAS_PENDING_EXCEPTION) {
Cheers,
David
-----
Post by Thomas Stüfe
Attempting to trigger a metaspace OOM from a compiler thread proved
futile - chances for that to happen are just too low compared to other
metaspace users. Only way to do this reliably would be to
artificially
allocate a lot of metaspace from the compiler thread - triggered by a
test switch or similar? That would be very ugly though. And then, I
would need to add a jvmti agent to jtreg, or adapt
jtreg/vmTestbase/nsk/jvmti?
Thanks, Thomas
On Thu, Nov 22, 2018 at 4:22 PM Daniel D. Daugherty
Post by Daniel D. Daugherty
Post by David Holmes
Post by Thomas Stüfe
Hi JC,
Post by s***@oracle.com
Hi Thomas,
<snip>
Post by s***@oracle.com
I actually agree with not using the service thread to be honest,
resource exhaustion seems to be something you'd want to know sooner
than later.
How about we do both?
    - Fix it now so that the compiler thread does not post the event
because we'd rather not have crashes and that can get backported
    - Put out a RFE that would add the information to ThreadInfo and
work through the process of a CSR/etc. to get it working for future
versions?
Thanks,
Jc
Makes sense, sure. But both Dan and Serguei voiced opposition to this
patch. Dan, Serguei, what do you think?
I note neither Dan nor Serguei responded to my response to them :)
I would have no problem suppressing the event from a "hidden" thread
because those threads intentionally try to hide from JVM/TI. That would
cover the case for the C1 and C2 CompilerThreads, but I don't know
about these newer JVM/CI Compiler threads that actually run Java code...
I read your combined response to this as not in conflict with what I said.
  > So my preferred point solution is still to skip posting
ResourceExhausted
  > if the thread can not run Java code.
matches what I said since the C1 and C2 compiler threads are hidden and
cannot run Java code. We're in agreement.
Dan
Post by David Holmes
Cheers,
David
-----
Post by Thomas Stüfe
If we do not find an agreement, I would rather close this bug as WNF
than push it in without consensus. I would then just fix it downstream
in our port.
Your proposed RFE would still make sense, but not in jdk12 anymore,
let alone in older releases.
Thanks, Thomas
s***@oracle.com
2018-11-26 04:12:08 UTC
Permalink
Post by s***@oracle.com
Hi guys,
Post by Daniel D. Daugherty
Post by David Holmes
Post by Daniel D. Daugherty
Post by s***@oracle.com
Hi Thomas,
Post by Thomas Stüfe
Hi all,
http://cr.openjdk.java.net/~stuefe/webrevs/8213834-jvmti-reourceexhausted-shall-not-be-posted-from-compiler-thread/webrev.01/webrev/
?
As has been proposed, I am now using
thread->is_hidden_from_external_view() to suppress the event.
I still disagree with this for reason previously outlined. It not
only excludes the CompilerThreads that can't run Java but all
CompilerThreads, and so excludes JVMCI compiler threads that do
everything in Java and so will quite possibly trigger resource exhaustion.
I thought the CompilerThread is_hidden_from_external_view() is built on
   // Hide native compiler threads from external view.
   bool is_hidden_from_external_view() const      { return
!can_call_java(); }
so is_hidden_from_external_view() does not exclude JVMCI...
Yes sorry my mistake.
There are many confusions around this.
Unfortunately, I make the same mistakes. :(
Post by Daniel D. Daugherty
Post by David Holmes
Post by Daniel D. Daugherty
Of course, the question of whether JVMCI threads should be
participating
in JVM/TI events and other stuff is being debugged and explored at the
current time.
Post by s***@oracle.com
It also excludes the ServiceThread and CodeCacheSweeperThread.
While the latter seems insignificant I think the ServiceThread is
more significant. Those threads are not excluded from posting
other events AFAICS so I don't see why this event should be
treated specially for them.
Hmmm... I don't think the ServiceThread should be posting JVM/TI events
But it already does! It's already posting deferred JVM TI events
explicitly as part of its job, and I see nothing that excludes it
from posting JVM TI events as part of any other Java code it runs.
You're right. I completely forgot about the ServiceThread doing proxy
event posting duty. So I think where we are is switching back to using
    can_call_java()
to restrict the posting of ResourceExhausted events. Less limiting than
!is_hidden_from_external_view(), but still limited. I'm reluctantly okay
with this, but we need to hear from Serguei.
I've read all the discussion thread.
Thank you to everyone participating in this discussion - it is very helpful.
Now, I'm convinced that using can_call_java (not
is_hidden_from_external_view)
is right (thanks a lot to David for defending this point!)
I still have a concern though.
If we skip a ResourceExhausted event this way (as in the webrev above)
then can we expect similar event posted on another (executing Java) thread?
I have a doubt that we skip it correctly (I need to learn the code more).
Please, skip my concern above.
I do not see any issues in the GC code.

Thanks,
Serguei
Post by s***@oracle.com
Thanks,
Serguei
Post by Daniel D. Daugherty
Dan
Post by David Holmes
David
-----
Post by Daniel D. Daugherty
either. We definitely don't want the ServiceThread running "arbitrary"
event handler code since taking down the ServiceThread could mess up
various things that rely on it running.
I don't have an opinion about the CodeCacheSweeperThread.
dan
Post by s***@oracle.com
Post by Thomas Stüfe
Side note, this function apparently should only ever be called from
JavaThreads? To my knowledge we do not guard metaspace against
allocation from non-java-threads, should we then do that?
Not sure how it is arranged but certainly Metaspace::allocate
expects to only be called from a JavaThread as it immediately
checks for
 if (HAS_PENDING_EXCEPTION) {
Cheers,
David
-----
Post by Thomas Stüfe
Attempting to trigger a metaspace OOM from a compiler thread proved
futile - chances for that to happen are just too low compared to other
metaspace users. Only way to do this reliably would be to
artificially
allocate a lot of metaspace from the compiler thread - triggered by a
test switch or similar? That would be very ugly though. And then, I
would need to add a jvmti agent to jtreg, or adapt
jtreg/vmTestbase/nsk/jvmti?
Thanks, Thomas
On Thu, Nov 22, 2018 at 4:22 PM Daniel D. Daugherty
Post by Daniel D. Daugherty
Post by David Holmes
Post by Thomas Stüfe
Hi JC,
On Wed, Nov 21, 2018 at 6:03 PM JC Beyler
Post by s***@oracle.com
Hi Thomas,
<snip>
Post by s***@oracle.com
I actually agree with not using the service thread to be honest,
resource exhaustion seems to be something you'd want to know sooner
than later.
How about we do both?
    - Fix it now so that the compiler thread does not post the event
because we'd rather not have crashes and that can get backported
    - Put out a RFE that would add the information to ThreadInfo and
work through the process of a CSR/etc. to get it working for future
versions?
Thanks,
Jc
Makes sense, sure. But both Dan and Serguei voiced opposition to this
patch. Dan, Serguei, what do you think?
I note neither Dan nor Serguei responded to my response to them :)
I would have no problem suppressing the event from a "hidden" thread
because those threads intentionally try to hide from JVM/TI. That would
cover the case for the C1 and C2 CompilerThreads, but I don't know
about these newer JVM/CI Compiler threads that actually run Java code...
I read your combined response to this as not in conflict with what I said.
  > So my preferred point solution is still to skip posting
ResourceExhausted
  > if the thread can not run Java code.
matches what I said since the C1 and C2 compiler threads are hidden and
cannot run Java code. We're in agreement.
Dan
Post by David Holmes
Cheers,
David
-----
Post by Thomas Stüfe
If we do not find an agreement, I would rather close this bug as WNF
than push it in without consensus. I would then just fix it downstream
in our port.
Your proposed RFE would still make sense, but not in jdk12 anymore,
let alone in older releases.
Thanks, Thomas
Thomas Stüfe
2018-11-26 08:51:24 UTC
Permalink
Post by s***@oracle.com
Hi guys,
Post by Daniel D. Daugherty
Post by David Holmes
Post by Daniel D. Daugherty
Post by s***@oracle.com
Hi Thomas,
Post by Thomas Stüfe
Hi all,
http://cr.openjdk.java.net/~stuefe/webrevs/8213834-jvmti-reourceexhausted-shall-not-be-posted-from-compiler-thread/webrev.01/webrev/
?
As has been proposed, I am now using
thread->is_hidden_from_external_view() to suppress the event.
I still disagree with this for reason previously outlined. It not
only excludes the CompilerThreads that can't run Java but all
CompilerThreads, and so excludes JVMCI compiler threads that do
everything in Java and so will quite possibly trigger resource exhaustion.
I thought the CompilerThread is_hidden_from_external_view() is built on
// Hide native compiler threads from external view.
bool is_hidden_from_external_view() const { return
!can_call_java(); }
so is_hidden_from_external_view() does not exclude JVMCI...
Yes sorry my mistake.
There are many confusions around this.
Unfortunately, I make the same mistakes. :(
Post by Daniel D. Daugherty
Post by David Holmes
Post by Daniel D. Daugherty
Of course, the question of whether JVMCI threads should be
participating
in JVM/TI events and other stuff is being debugged and explored at the
current time.
Post by s***@oracle.com
It also excludes the ServiceThread and CodeCacheSweeperThread.
While the latter seems insignificant I think the ServiceThread is
more significant. Those threads are not excluded from posting other
events AFAICS so I don't see why this event should be treated
specially for them.
Hmmm... I don't think the ServiceThread should be posting JVM/TI events
But it already does! It's already posting deferred JVM TI events
explicitly as part of its job, and I see nothing that excludes it
from posting JVM TI events as part of any other Java code it runs.
You're right. I completely forgot about the ServiceThread doing proxy
event posting duty. So I think where we are is switching back to using
can_call_java()
to restrict the posting of ResourceExhausted events. Less limiting than
!is_hidden_from_external_view(), but still limited. I'm reluctantly okay
with this, but we need to hear from Serguei.
I've read all the discussion thread.
Thank you to everyone participating in this discussion - it is very helpful.
Now, I'm convinced that using can_call_java (not
is_hidden_from_external_view)
is right (thanks a lot to David for defending this point!)
I still have a concern though.
If we skip a ResourceExhausted event this way (as in the webrev above)
then can we expect similar event posted on another (executing Java) thread?
I think this usually would happen, but there is no guarantee.

Speaking for metaspace: Before metaspace OOM happens, we already tried
to GC and that failed to recover metaspace so there is no immediate
hope and chances are high subsequent metaspace allocations will fail
too.

If the JIT runs into metaspace OOM, it will cope by bailing out and
leave that particular method uncompiled. I argue that this is not
observable to the user and therefore ResourceExhausted can be
suppressed.

If we, as you fear, do not resend ResourceExhausted, that means that
no-one else tried to allocate metaspace, and that this JIT compilation
was by pure chance the last one to ever want metaspace. Unlikely as it
is, in that case I think it okay suppressing ResourceExhausted and
going on with our lives.

But if Metaspace OOM keep occurring, they most probably will occur
from regular java threads too and that will trigger ResourceExhausted.

Finally, I am not sure whether a pathological case is possible where
only the JIT were to allocate metaspace over and over again, failing
each time, silently swallowing all OOMs and just falling back to
interpreted. But that problem exists today too.

Thanks, Thomas
Post by s***@oracle.com
I have a doubt that we skip it correctly (I need to learn the code more).
Thanks,
Serguei
Post by Daniel D. Daugherty
Dan
Post by David Holmes
David
-----
Post by Daniel D. Daugherty
either. We definitely don't want the ServiceThread running "arbitrary"
event handler code since taking down the ServiceThread could mess up
various things that rely on it running.
I don't have an opinion about the CodeCacheSweeperThread.
dan
Post by s***@oracle.com
Post by Thomas Stüfe
Side note, this function apparently should only ever be called from
JavaThreads? To my knowledge we do not guard metaspace against
allocation from non-java-threads, should we then do that?
Not sure how it is arranged but certainly Metaspace::allocate
expects to only be called from a JavaThread as it immediately
checks for
if (HAS_PENDING_EXCEPTION) {
Cheers,
David
-----
Post by Thomas Stüfe
Attempting to trigger a metaspace OOM from a compiler thread proved
futile - chances for that to happen are just too low compared to other
metaspace users. Only way to do this reliably would be to artificially
allocate a lot of metaspace from the compiler thread - triggered by a
test switch or similar? That would be very ugly though. And then, I
would need to add a jvmti agent to jtreg, or adapt
jtreg/vmTestbase/nsk/jvmti?
Thanks, Thomas
On Thu, Nov 22, 2018 at 4:22 PM Daniel D. Daugherty
Post by Daniel D. Daugherty
Post by David Holmes
Post by Thomas Stüfe
Hi JC,
Post by s***@oracle.com
Hi Thomas,
<snip>
Post by s***@oracle.com
I actually agree with not using the service thread to be honest,
resource exhaustion seems to be something you'd want to know sooner
than later.
How about we do both?
- Fix it now so that the compiler thread does not post the event
because we'd rather not have crashes and that can get backported
- Put out a RFE that would add the information to ThreadInfo and
work through the process of a CSR/etc. to get it working for future
versions?
Thanks,
Jc
Makes sense, sure. But both Dan and Serguei voiced opposition to this
patch. Dan, Serguei, what do you think?
I note neither Dan nor Serguei responded to my response to them :)
I would have no problem suppressing the event from a "hidden" thread
because those threads intentionally try to hide from JVM/TI. That would
cover the case for the C1 and C2 CompilerThreads, but I don't know
about these newer JVM/CI Compiler threads that actually run Java code...
I read your combined response to this as not in conflict with what I said.
Post by David Holmes
So my preferred point solution is still to skip posting
ResourceExhausted
Post by David Holmes
if the thread can not run Java code.
matches what I said since the C1 and C2 compiler threads are hidden and
cannot run Java code. We're in agreement.
Dan
Post by David Holmes
Cheers,
David
-----
Post by Thomas Stüfe
If we do not find an agreement, I would rather close this bug as WNF
than push it in without consensus. I would then just fix it downstream
in our port.
Your proposed RFE would still make sense, but not in jdk12 anymore,
let alone in older releases.
Thanks, Thomas
Thomas Stüfe
2018-11-26 08:58:34 UTC
Permalink
Hi all,

just a general question, would a

virtual bool Thread::can_post_JVMTI_events();

(similar to can_call_java()) not be a clearer solution? It clearly
expresses what we want, and we can, in every Thread child class,
explicitly specify the behavior.

..Thomas
Post by Thomas Stüfe
Post by s***@oracle.com
Hi guys,
Post by Daniel D. Daugherty
Post by David Holmes
Post by Daniel D. Daugherty
Post by s***@oracle.com
Hi Thomas,
Post by Thomas Stüfe
Hi all,
http://cr.openjdk.java.net/~stuefe/webrevs/8213834-jvmti-reourceexhausted-shall-not-be-posted-from-compiler-thread/webrev.01/webrev/
?
As has been proposed, I am now using
thread->is_hidden_from_external_view() to suppress the event.
I still disagree with this for reason previously outlined. It not
only excludes the CompilerThreads that can't run Java but all
CompilerThreads, and so excludes JVMCI compiler threads that do
everything in Java and so will quite possibly trigger resource exhaustion.
I thought the CompilerThread is_hidden_from_external_view() is built on
// Hide native compiler threads from external view.
bool is_hidden_from_external_view() const { return
!can_call_java(); }
so is_hidden_from_external_view() does not exclude JVMCI...
Yes sorry my mistake.
There are many confusions around this.
Unfortunately, I make the same mistakes. :(
Post by Daniel D. Daugherty
Post by David Holmes
Post by Daniel D. Daugherty
Of course, the question of whether JVMCI threads should be participating
in JVM/TI events and other stuff is being debugged and explored at the
current time.
Post by s***@oracle.com
It also excludes the ServiceThread and CodeCacheSweeperThread.
While the latter seems insignificant I think the ServiceThread is
more significant. Those threads are not excluded from posting other
events AFAICS so I don't see why this event should be treated
specially for them.
Hmmm... I don't think the ServiceThread should be posting JVM/TI events
But it already does! It's already posting deferred JVM TI events
explicitly as part of its job, and I see nothing that excludes it
from posting JVM TI events as part of any other Java code it runs.
You're right. I completely forgot about the ServiceThread doing proxy
event posting duty. So I think where we are is switching back to using
can_call_java()
to restrict the posting of ResourceExhausted events. Less limiting than
!is_hidden_from_external_view(), but still limited. I'm reluctantly okay
with this, but we need to hear from Serguei.
I've read all the discussion thread.
Thank you to everyone participating in this discussion - it is very helpful.
Now, I'm convinced that using can_call_java (not
is_hidden_from_external_view)
is right (thanks a lot to David for defending this point!)
I still have a concern though.
If we skip a ResourceExhausted event this way (as in the webrev above)
then can we expect similar event posted on another (executing Java) thread?
I think this usually would happen, but there is no guarantee.
Speaking for metaspace: Before metaspace OOM happens, we already tried
to GC and that failed to recover metaspace so there is no immediate
hope and chances are high subsequent metaspace allocations will fail
too.
If the JIT runs into metaspace OOM, it will cope by bailing out and
leave that particular method uncompiled. I argue that this is not
observable to the user and therefore ResourceExhausted can be
suppressed.
If we, as you fear, do not resend ResourceExhausted, that means that
no-one else tried to allocate metaspace, and that this JIT compilation
was by pure chance the last one to ever want metaspace. Unlikely as it
is, in that case I think it okay suppressing ResourceExhausted and
going on with our lives.
But if Metaspace OOM keep occurring, they most probably will occur
from regular java threads too and that will trigger ResourceExhausted.
Finally, I am not sure whether a pathological case is possible where
only the JIT were to allocate metaspace over and over again, failing
each time, silently swallowing all OOMs and just falling back to
interpreted. But that problem exists today too.
Thanks, Thomas
Post by s***@oracle.com
I have a doubt that we skip it correctly (I need to learn the code more).
Thanks,
Serguei
Post by Daniel D. Daugherty
Dan
Post by David Holmes
David
-----
Post by Daniel D. Daugherty
either. We definitely don't want the ServiceThread running "arbitrary"
event handler code since taking down the ServiceThread could mess up
various things that rely on it running.
I don't have an opinion about the CodeCacheSweeperThread.
dan
Post by s***@oracle.com
Post by Thomas Stüfe
Side note, this function apparently should only ever be called from
JavaThreads? To my knowledge we do not guard metaspace against
allocation from non-java-threads, should we then do that?
Not sure how it is arranged but certainly Metaspace::allocate
expects to only be called from a JavaThread as it immediately
checks for
if (HAS_PENDING_EXCEPTION) {
Cheers,
David
-----
Post by Thomas Stüfe
I attempted to create a jtreg test for this but this turns out to
Attempting to trigger a metaspace OOM from a compiler thread proved
futile - chances for that to happen are just too low compared to other
metaspace users. Only way to do this reliably would be to artificially
allocate a lot of metaspace from the compiler thread - triggered by a
test switch or similar? That would be very ugly though. And then, I
would need to add a jvmti agent to jtreg, or adapt
jtreg/vmTestbase/nsk/jvmti?
Thanks, Thomas
On Thu, Nov 22, 2018 at 4:22 PM Daniel D. Daugherty
Post by Daniel D. Daugherty
Post by David Holmes
Post by Thomas Stüfe
Hi JC,
Post by s***@oracle.com
Hi Thomas,
<snip>
Post by s***@oracle.com
I actually agree with not using the service thread to be honest,
resource exhaustion seems to be something you'd want to know sooner
than later.
How about we do both?
- Fix it now so that the compiler thread does not post the
event
because we'd rather not have crashes and that can get backported
- Put out a RFE that would add the information to
ThreadInfo and
work through the process of a CSR/etc. to get it working for future
versions?
Thanks,
Jc
Makes sense, sure. But both Dan and Serguei voiced opposition to this
patch. Dan, Serguei, what do you think?
I note neither Dan nor Serguei responded to my response to them :)
I would have no problem suppressing the event from a "hidden" thread
because those threads intentionally try to hide from JVM/TI. That would
cover the case for the C1 and C2 CompilerThreads, but I don't know
about these newer JVM/CI Compiler threads that actually run Java
code...
I read your combined response to this as not in conflict with
what I said.
Post by David Holmes
So my preferred point solution is still to skip posting
ResourceExhausted
Post by David Holmes
if the thread can not run Java code.
matches what I said since the C1 and C2 compiler threads are hidden and
cannot run Java code. We're in agreement.
Dan
Post by David Holmes
Cheers,
David
-----
Post by Thomas Stüfe
If we do not find an agreement, I would rather close this bug as WNF
than push it in without consensus. I would then just fix it
downstream
in our port.
Your proposed RFE would still make sense, but not in jdk12 anymore,
let alone in older releases.
Thanks, Thomas
s***@oracle.com
2018-11-26 10:01:45 UTC
Permalink
Hi Thomas,

I like this idea.
It will help to control it better and avoid confusions next time.

Thanks,
Serguei
Post by Thomas Stüfe
Hi all,
just a general question, would a
virtual bool Thread::can_post_JVMTI_events();
(similar to can_call_java()) not be a clearer solution? It clearly
expresses what we want, and we can, in every Thread child class,
explicitly specify the behavior.
..Thomas
Post by Thomas Stüfe
Post by s***@oracle.com
Hi guys,
Post by Daniel D. Daugherty
Post by David Holmes
Post by Daniel D. Daugherty
Post by s***@oracle.com
Hi Thomas,
Post by Thomas Stüfe
Hi all,
http://cr.openjdk.java.net/~stuefe/webrevs/8213834-jvmti-reourceexhausted-shall-not-be-posted-from-compiler-thread/webrev.01/webrev/
?
As has been proposed, I am now using
thread->is_hidden_from_external_view() to suppress the event.
I still disagree with this for reason previously outlined. It not
only excludes the CompilerThreads that can't run Java but all
CompilerThreads, and so excludes JVMCI compiler threads that do
everything in Java and so will quite possibly trigger resource exhaustion.
I thought the CompilerThread is_hidden_from_external_view() is built on
// Hide native compiler threads from external view.
bool is_hidden_from_external_view() const { return
!can_call_java(); }
so is_hidden_from_external_view() does not exclude JVMCI...
Yes sorry my mistake.
There are many confusions around this.
Unfortunately, I make the same mistakes. :(
Post by Daniel D. Daugherty
Post by David Holmes
Post by Daniel D. Daugherty
Of course, the question of whether JVMCI threads should be participating
in JVM/TI events and other stuff is being debugged and explored at the
current time.
Post by s***@oracle.com
It also excludes the ServiceThread and CodeCacheSweeperThread.
While the latter seems insignificant I think the ServiceThread is
more significant. Those threads are not excluded from posting other
events AFAICS so I don't see why this event should be treated
specially for them.
Hmmm... I don't think the ServiceThread should be posting JVM/TI events
But it already does! It's already posting deferred JVM TI events
explicitly as part of its job, and I see nothing that excludes it
from posting JVM TI events as part of any other Java code it runs.
You're right. I completely forgot about the ServiceThread doing proxy
event posting duty. So I think where we are is switching back to using
can_call_java()
to restrict the posting of ResourceExhausted events. Less limiting than
!is_hidden_from_external_view(), but still limited. I'm reluctantly okay
with this, but we need to hear from Serguei.
I've read all the discussion thread.
Thank you to everyone participating in this discussion - it is very helpful.
Now, I'm convinced that using can_call_java (not
is_hidden_from_external_view)
is right (thanks a lot to David for defending this point!)
I still have a concern though.
If we skip a ResourceExhausted event this way (as in the webrev above)
then can we expect similar event posted on another (executing Java) thread?
I think this usually would happen, but there is no guarantee.
Speaking for metaspace: Before metaspace OOM happens, we already tried
to GC and that failed to recover metaspace so there is no immediate
hope and chances are high subsequent metaspace allocations will fail
too.
If the JIT runs into metaspace OOM, it will cope by bailing out and
leave that particular method uncompiled. I argue that this is not
observable to the user and therefore ResourceExhausted can be
suppressed.
If we, as you fear, do not resend ResourceExhausted, that means that
no-one else tried to allocate metaspace, and that this JIT compilation
was by pure chance the last one to ever want metaspace. Unlikely as it
is, in that case I think it okay suppressing ResourceExhausted and
going on with our lives.
But if Metaspace OOM keep occurring, they most probably will occur
from regular java threads too and that will trigger ResourceExhausted.
Finally, I am not sure whether a pathological case is possible where
only the JIT were to allocate metaspace over and over again, failing
each time, silently swallowing all OOMs and just falling back to
interpreted. But that problem exists today too.
Thanks, Thomas
Post by s***@oracle.com
I have a doubt that we skip it correctly (I need to learn the code more).
Thanks,
Serguei
Post by Daniel D. Daugherty
Dan
Post by David Holmes
David
-----
Post by Daniel D. Daugherty
either. We definitely don't want the ServiceThread running "arbitrary"
event handler code since taking down the ServiceThread could mess up
various things that rely on it running.
I don't have an opinion about the CodeCacheSweeperThread.
dan
Post by s***@oracle.com
Post by Thomas Stüfe
Side note, this function apparently should only ever be called from
JavaThreads? To my knowledge we do not guard metaspace against
allocation from non-java-threads, should we then do that?
Not sure how it is arranged but certainly Metaspace::allocate
expects to only be called from a JavaThread as it immediately
checks for
if (HAS_PENDING_EXCEPTION) {
Cheers,
David
-----
Post by Thomas Stüfe
I attempted to create a jtreg test for this but this turns out to
Attempting to trigger a metaspace OOM from a compiler thread proved
futile - chances for that to happen are just too low compared to other
metaspace users. Only way to do this reliably would be to artificially
allocate a lot of metaspace from the compiler thread - triggered by a
test switch or similar? That would be very ugly though. And then, I
would need to add a jvmti agent to jtreg, or adapt
jtreg/vmTestbase/nsk/jvmti?
Thanks, Thomas
On Thu, Nov 22, 2018 at 4:22 PM Daniel D. Daugherty
Post by Daniel D. Daugherty
Post by David Holmes
Post by Thomas Stüfe
Hi JC,
Post by s***@oracle.com
Hi Thomas,
<snip>
Post by s***@oracle.com
I actually agree with not using the service thread to be honest,
resource exhaustion seems to be something you'd want to know sooner
than later.
How about we do both?
- Fix it now so that the compiler thread does not post the event
because we'd rather not have crashes and that can get backported
- Put out a RFE that would add the information to ThreadInfo and
work through the process of a CSR/etc. to get it working for future
versions?
Thanks,
Jc
Makes sense, sure. But both Dan and Serguei voiced opposition to this
patch. Dan, Serguei, what do you think?
I note neither Dan nor Serguei responded to my response to them :)
I would have no problem suppressing the event from a "hidden" thread
because those threads intentionally try to hide from JVM/TI. That would
cover the case for the C1 and C2 CompilerThreads, but I don't know
about these newer JVM/CI Compiler threads that actually run Java code...
I read your combined response to this as not in conflict with what I said.
Post by David Holmes
So my preferred point solution is still to skip posting
ResourceExhausted
Post by David Holmes
if the thread can not run Java code.
matches what I said since the C1 and C2 compiler threads are hidden and
cannot run Java code. We're in agreement.
Dan
Post by David Holmes
Cheers,
David
-----
Post by Thomas Stüfe
If we do not find an agreement, I would rather close this bug as WNF
than push it in without consensus. I would then just fix it downstream
in our port.
Your proposed RFE would still make sense, but not in jdk12 anymore,
let alone in older releases.
Thanks, Thomas
David Holmes
2018-11-26 11:28:40 UTC
Permalink
Post by Thomas Stüfe
Hi all,
just a general question, would a
virtual bool Thread::can_post_JVMTI_events();
(similar to can_call_java()) not be a clearer solution? It clearly
expresses what we want, and we can, in every Thread child class,
explicitly specify the behavior.
That would be good for when we update the specification to actually
define what threads can post JVM TI events. But I'd prefer for now to
see just a comment and the can_call_java check.

David
Post by Thomas Stüfe
..Thomas
Post by Thomas Stüfe
Post by s***@oracle.com
Hi guys,
Post by Daniel D. Daugherty
Post by David Holmes
Post by Daniel D. Daugherty
Post by s***@oracle.com
Hi Thomas,
Post by Thomas Stüfe
Hi all,
http://cr.openjdk.java.net/~stuefe/webrevs/8213834-jvmti-reourceexhausted-shall-not-be-posted-from-compiler-thread/webrev.01/webrev/
?
As has been proposed, I am now using
thread->is_hidden_from_external_view() to suppress the event.
I still disagree with this for reason previously outlined. It not
only excludes the CompilerThreads that can't run Java but all
CompilerThreads, and so excludes JVMCI compiler threads that do
everything in Java and so will quite possibly trigger resource exhaustion.
I thought the CompilerThread is_hidden_from_external_view() is built on
// Hide native compiler threads from external view.
bool is_hidden_from_external_view() const { return
!can_call_java(); }
so is_hidden_from_external_view() does not exclude JVMCI...
Yes sorry my mistake.
There are many confusions around this.
Unfortunately, I make the same mistakes. :(
Post by Daniel D. Daugherty
Post by David Holmes
Post by Daniel D. Daugherty
Of course, the question of whether JVMCI threads should be participating
in JVM/TI events and other stuff is being debugged and explored at the
current time.
Post by s***@oracle.com
It also excludes the ServiceThread and CodeCacheSweeperThread.
While the latter seems insignificant I think the ServiceThread is
more significant. Those threads are not excluded from posting other
events AFAICS so I don't see why this event should be treated
specially for them.
Hmmm... I don't think the ServiceThread should be posting JVM/TI events
But it already does! It's already posting deferred JVM TI events
explicitly as part of its job, and I see nothing that excludes it
from posting JVM TI events as part of any other Java code it runs.
You're right. I completely forgot about the ServiceThread doing proxy
event posting duty. So I think where we are is switching back to using
can_call_java()
to restrict the posting of ResourceExhausted events. Less limiting than
!is_hidden_from_external_view(), but still limited. I'm reluctantly okay
with this, but we need to hear from Serguei.
I've read all the discussion thread.
Thank you to everyone participating in this discussion - it is very helpful.
Now, I'm convinced that using can_call_java (not
is_hidden_from_external_view)
is right (thanks a lot to David for defending this point!)
I still have a concern though.
If we skip a ResourceExhausted event this way (as in the webrev above)
then can we expect similar event posted on another (executing Java) thread?
I think this usually would happen, but there is no guarantee.
Speaking for metaspace: Before metaspace OOM happens, we already tried
to GC and that failed to recover metaspace so there is no immediate
hope and chances are high subsequent metaspace allocations will fail
too.
If the JIT runs into metaspace OOM, it will cope by bailing out and
leave that particular method uncompiled. I argue that this is not
observable to the user and therefore ResourceExhausted can be
suppressed.
If we, as you fear, do not resend ResourceExhausted, that means that
no-one else tried to allocate metaspace, and that this JIT compilation
was by pure chance the last one to ever want metaspace. Unlikely as it
is, in that case I think it okay suppressing ResourceExhausted and
going on with our lives.
But if Metaspace OOM keep occurring, they most probably will occur
from regular java threads too and that will trigger ResourceExhausted.
Finally, I am not sure whether a pathological case is possible where
only the JIT were to allocate metaspace over and over again, failing
each time, silently swallowing all OOMs and just falling back to
interpreted. But that problem exists today too.
Thanks, Thomas
Post by s***@oracle.com
I have a doubt that we skip it correctly (I need to learn the code more).
Thanks,
Serguei
Post by Daniel D. Daugherty
Dan
Post by David Holmes
David
-----
Post by Daniel D. Daugherty
either. We definitely don't want the ServiceThread running "arbitrary"
event handler code since taking down the ServiceThread could mess up
various things that rely on it running.
I don't have an opinion about the CodeCacheSweeperThread.
dan
Post by s***@oracle.com
Post by Thomas Stüfe
Side note, this function apparently should only ever be called from
JavaThreads? To my knowledge we do not guard metaspace against
allocation from non-java-threads, should we then do that?
Not sure how it is arranged but certainly Metaspace::allocate
expects to only be called from a JavaThread as it immediately
checks for
if (HAS_PENDING_EXCEPTION) {
Cheers,
David
-----
Post by Thomas Stüfe
I attempted to create a jtreg test for this but this turns out to
Attempting to trigger a metaspace OOM from a compiler thread proved
futile - chances for that to happen are just too low compared to other
metaspace users. Only way to do this reliably would be to artificially
allocate a lot of metaspace from the compiler thread - triggered by a
test switch or similar? That would be very ugly though. And then, I
would need to add a jvmti agent to jtreg, or adapt
jtreg/vmTestbase/nsk/jvmti?
Thanks, Thomas
On Thu, Nov 22, 2018 at 4:22 PM Daniel D. Daugherty
Post by Daniel D. Daugherty
Post by David Holmes
Post by Thomas Stüfe
Hi JC,
Post by s***@oracle.com
Hi Thomas,
<snip>
Post by s***@oracle.com
I actually agree with not using the service thread to be honest,
resource exhaustion seems to be something you'd want to know sooner
than later.
How about we do both?
- Fix it now so that the compiler thread does not post the event
because we'd rather not have crashes and that can get backported
- Put out a RFE that would add the information to ThreadInfo and
work through the process of a CSR/etc. to get it working for future
versions?
Thanks,
Jc
Makes sense, sure. But both Dan and Serguei voiced opposition to this
patch. Dan, Serguei, what do you think?
I note neither Dan nor Serguei responded to my response to them :)
I would have no problem suppressing the event from a "hidden" thread
because those threads intentionally try to hide from JVM/TI. That would
cover the case for the C1 and C2 CompilerThreads, but I don't know
about these newer JVM/CI Compiler threads that actually run Java code...
I read your combined response to this as not in conflict with what I said.
Post by David Holmes
So my preferred point solution is still to skip posting
ResourceExhausted
Post by David Holmes
if the thread can not run Java code.
matches what I said since the C1 and C2 compiler threads are hidden and
cannot run Java code. We're in agreement.
Dan
Post by David Holmes
Cheers,
David
-----
Post by Thomas Stüfe
If we do not find an agreement, I would rather close this bug as WNF
than push it in without consensus. I would then just fix it downstream
in our port.
Your proposed RFE would still make sense, but not in jdk12 anymore,
let alone in older releases.
Thanks, Thomas
Thomas Stüfe
2018-11-26 11:32:41 UTC
Permalink
Okay, I created an RFE for this:
https://bugs.openjdk.java.net/browse/JDK-8214294

And for now I will prepare a new minimal patch based on our
discussions and post it shortly.

..Thomas
Post by David Holmes
Post by Thomas Stüfe
Hi all,
just a general question, would a
virtual bool Thread::can_post_JVMTI_events();
(similar to can_call_java()) not be a clearer solution? It clearly
expresses what we want, and we can, in every Thread child class,
explicitly specify the behavior.
That would be good for when we update the specification to actually
define what threads can post JVM TI events. But I'd prefer for now to
see just a comment and the can_call_java check.
David
Post by Thomas Stüfe
..Thomas
Post by Thomas Stüfe
Post by s***@oracle.com
Hi guys,
Post by Daniel D. Daugherty
Post by David Holmes
Post by Daniel D. Daugherty
Post by s***@oracle.com
Hi Thomas,
Post by Thomas Stüfe
Hi all,
http://cr.openjdk.java.net/~stuefe/webrevs/8213834-jvmti-reourceexhausted-shall-not-be-posted-from-compiler-thread/webrev.01/webrev/
?
As has been proposed, I am now using
thread->is_hidden_from_external_view() to suppress the event.
I still disagree with this for reason previously outlined. It not
only excludes the CompilerThreads that can't run Java but all
CompilerThreads, and so excludes JVMCI compiler threads that do
everything in Java and so will quite possibly trigger resource exhaustion.
I thought the CompilerThread is_hidden_from_external_view() is built on
// Hide native compiler threads from external view.
bool is_hidden_from_external_view() const { return
!can_call_java(); }
so is_hidden_from_external_view() does not exclude JVMCI...
Yes sorry my mistake.
There are many confusions around this.
Unfortunately, I make the same mistakes. :(
Post by Daniel D. Daugherty
Post by David Holmes
Post by Daniel D. Daugherty
Of course, the question of whether JVMCI threads should be participating
in JVM/TI events and other stuff is being debugged and explored at the
current time.
Post by s***@oracle.com
It also excludes the ServiceThread and CodeCacheSweeperThread.
While the latter seems insignificant I think the ServiceThread is
more significant. Those threads are not excluded from posting other
events AFAICS so I don't see why this event should be treated
specially for them.
Hmmm... I don't think the ServiceThread should be posting JVM/TI events
But it already does! It's already posting deferred JVM TI events
explicitly as part of its job, and I see nothing that excludes it
from posting JVM TI events as part of any other Java code it runs.
You're right. I completely forgot about the ServiceThread doing proxy
event posting duty. So I think where we are is switching back to using
can_call_java()
to restrict the posting of ResourceExhausted events. Less limiting than
!is_hidden_from_external_view(), but still limited. I'm reluctantly okay
with this, but we need to hear from Serguei.
I've read all the discussion thread.
Thank you to everyone participating in this discussion - it is very helpful.
Now, I'm convinced that using can_call_java (not
is_hidden_from_external_view)
is right (thanks a lot to David for defending this point!)
I still have a concern though.
If we skip a ResourceExhausted event this way (as in the webrev above)
then can we expect similar event posted on another (executing Java) thread?
I think this usually would happen, but there is no guarantee.
Speaking for metaspace: Before metaspace OOM happens, we already tried
to GC and that failed to recover metaspace so there is no immediate
hope and chances are high subsequent metaspace allocations will fail
too.
If the JIT runs into metaspace OOM, it will cope by bailing out and
leave that particular method uncompiled. I argue that this is not
observable to the user and therefore ResourceExhausted can be
suppressed.
If we, as you fear, do not resend ResourceExhausted, that means that
no-one else tried to allocate metaspace, and that this JIT compilation
was by pure chance the last one to ever want metaspace. Unlikely as it
is, in that case I think it okay suppressing ResourceExhausted and
going on with our lives.
But if Metaspace OOM keep occurring, they most probably will occur
from regular java threads too and that will trigger ResourceExhausted.
Finally, I am not sure whether a pathological case is possible where
only the JIT were to allocate metaspace over and over again, failing
each time, silently swallowing all OOMs and just falling back to
interpreted. But that problem exists today too.
Thanks, Thomas
Post by s***@oracle.com
I have a doubt that we skip it correctly (I need to learn the code more).
Thanks,
Serguei
Post by Daniel D. Daugherty
Dan
Post by David Holmes
David
-----
Post by Daniel D. Daugherty
either. We definitely don't want the ServiceThread running "arbitrary"
event handler code since taking down the ServiceThread could mess up
various things that rely on it running.
I don't have an opinion about the CodeCacheSweeperThread.
dan
Post by s***@oracle.com
Post by Thomas Stüfe
Side note, this function apparently should only ever be called from
JavaThreads? To my knowledge we do not guard metaspace against
allocation from non-java-threads, should we then do that?
Not sure how it is arranged but certainly Metaspace::allocate
expects to only be called from a JavaThread as it immediately
checks for
if (HAS_PENDING_EXCEPTION) {
Cheers,
David
-----
Post by Thomas Stüfe
I attempted to create a jtreg test for this but this turns out to
Attempting to trigger a metaspace OOM from a compiler thread proved
futile - chances for that to happen are just too low compared to other
metaspace users. Only way to do this reliably would be to artificially
allocate a lot of metaspace from the compiler thread - triggered by a
test switch or similar? That would be very ugly though. And then, I
would need to add a jvmti agent to jtreg, or adapt
jtreg/vmTestbase/nsk/jvmti?
Thanks, Thomas
On Thu, Nov 22, 2018 at 4:22 PM Daniel D. Daugherty
Post by David Holmes
Post by Thomas Stüfe
Hi JC,
Post by s***@oracle.com
Hi Thomas,
<snip>
Post by s***@oracle.com
I actually agree with not using the service thread to be honest,
resource exhaustion seems to be something you'd want to know
sooner
than later.
How about we do both?
- Fix it now so that the compiler thread does not post the
event
because we'd rather not have crashes and that can get backported
- Put out a RFE that would add the information to
ThreadInfo and
work through the process of a CSR/etc. to get it working for
future
versions?
Thanks,
Jc
Makes sense, sure. But both Dan and Serguei voiced opposition
to this
patch. Dan, Serguei, what do you think?
I note neither Dan nor Serguei responded to my response to them :)
I didn't think a response from me was needed. The last thing I
Post by David Holmes
I would have no problem suppressing the event from a "hidden" thread
because those threads intentionally try to hide from JVM/TI.
That would
cover the case for the C1 and C2 CompilerThreads, but I don't know
about these newer JVM/CI Compiler threads that actually run Java
code...
I read your combined response to this as not in conflict with
what I said.
Post by David Holmes
So my preferred point solution is still to skip posting
ResourceExhausted
Post by David Holmes
if the thread can not run Java code.
matches what I said since the C1 and C2 compiler threads are hidden and
cannot run Java code. We're in agreement.
Dan
Post by David Holmes
Cheers,
David
-----
Post by Thomas Stüfe
If we do not find an agreement, I would rather close this bug
as WNF
than push it in without consensus. I would then just fix it
downstream
in our port.
Your proposed RFE would still make sense, but not in jdk12 anymore,
let alone in older releases.
Thanks, Thomas
David Holmes
2018-11-22 22:11:54 UTC
Permalink
Hi Dan,

<trimming>
Post by Daniel D. Daugherty
I read your combined response to this as not in conflict with what I said.
Post by David Holmes
So my preferred point solution is still to skip posting
ResourceExhausted
Post by David Holmes
if the thread can not run Java code.
matches what I said since the C1 and C2 compiler threads are hidden and
cannot run Java code. We're in agreement.
We're not quite in agreement. You want this applied to hidden threads, I
only want it applied to threads that can't run Java code.

Cheers,
David
Post by Daniel D. Daugherty
Dan
Post by David Holmes
Cheers,
David
-----
Post by Thomas Stüfe
If we do not find an agreement, I would rather close this bug as WNF
than push it in without consensus. I would then just fix it downstream
in our port.
Your proposed RFE would still make sense, but not in jdk12 anymore,
let alone in older releases.
Thanks, Thomas
Daniel D. Daugherty
2018-11-24 14:43:24 UTC
Permalink
Post by Thomas Stüfe
Hi Dan,
<trimming>
Post by Daniel D. Daugherty
I read your combined response to this as not in conflict with what I said.
 > So my preferred point solution is still to skip posting
ResourceExhausted
 > if the thread can not run Java code.
matches what I said since the C1 and C2 compiler threads are hidden and
cannot run Java code. We're in agreement.
We're not quite in agreement. You want this applied to hidden threads,
I only want it applied to threads that can't run Java code.
Let's close out this fork of the thread. I picked up the
conversation in your next reply.

Dan
Post by Thomas Stüfe
Cheers,
David
Post by Daniel D. Daugherty
Dan
Post by David Holmes
Cheers,
David
-----
Post by Thomas Stüfe
If we do not find an agreement, I would rather close this bug as WNF
than push it in without consensus. I would then just fix it downstream
in our port.
Your proposed RFE would still make sense, but not in jdk12 anymore,
let alone in older releases.
Thanks, Thomas
Thomas Stüfe
2018-11-26 13:07:08 UTC
Permalink
Hi guys,

latest webrev: http://cr.openjdk.java.net/~stuefe/webrevs/8213834-jvmti-reourceexhausted-shall-not-be-posted-from-compiler-thread/webrev.02/webrev/

Back to can_call_java(), since this seems to be the consensus, with a
comment added.

As for the Thread::can_send_jvmti_events() property idea, I created a
RFE to track discussions around this:
https://bugs.openjdk.java.net/browse/JDK-8214294 but decided to
postpone this for later. I would like to close that particular issue,
if possible with a minimal fix which can be easily downported to older
released.

Thanks, Thomas
Post by Thomas Stüfe
Hi all,
David and JC already outlined the options we have nicely.
I'd like to add that I do not favor the ServiceThread delayed
deliverance since a common reaction to ResourceExhausted would to
print call stack of the thread running into it or to print a heap
histogram, as jvmkill does. For the former only a synchronous event
delivery makes sense, for the latter at least it helps analyzing.
In cloud foundry, the heap histogram produced by the JVMTI agent can
be helpful, and since in the majority of cases do not entail the
compiler thread running out of metaspace, I'd rather preserve this
ability. So to me, masking this event for this one case is the most
pragmatic approach.
The other option would be not to change the code but to add, in the
JVMTI documentation for ResourceExhausted, the same disclaimer as for
ObjectFree, GCStarted etc: "The event handler must not use JNI
functions and must not use JVM TI functions except those which
specifically allow such use". Then, writers of JVMTI agents like
jvmkill would have to update their agents accordingly.
FWIW, I think JCs idea of exposing the can_call_java attribute somehow
to the outside would also work. But unfortunately not retroactively,
for older releases. Whereas a simple internal patch like "mask
ResourceExhausted" could be backported easily to older releases.
Best Regards, Thomas
For what it's worth I wonder if skipping the event is the best; it is the easiest to ensure non breaking the agent; it does not really go against what the spec is saying and another thread that CAN call java will most likely hit the issue and then all will be good in the world.
However, also for what it's worth I wonder if deferring is not that hard to accomplish either. There already is the infrastructure for this and we should be relatively able to do it. Only question I would have is can the service thread create a JNIEnv for the event but I don't think that's an issue, is it?
It might however conflict with the description of the JNIEnv in the spec which says the jni_env is "The JNI environment of the event (current) thread" though it doesn't say current of what or the event thread of what really.
However, skipping it also kind of goes against the spec since it says: "Sent when a VM resource needed by a running application has been exhausted".Though someone could argue it doesn' t say it is sent when the resource was first noticed to be exhausted.
- Sending the event via the compiler thread will risk breaking things if the agent calls Java -> not really an option
- Using the service thread breaks what David calls the synchronous assumption of the event
- Skipping the event kind of breaks the sentence that the event is sent when a VM resource has been exhausted
So we come back to probably skipping is the best solution since at least the event remains "synchronous" when you get it.
(A side note would be perhaps to augment ThreadInfo* with the "can_call_java" bit and then put in the right spots of the spec that only threads marked as "can_call_java" can safely call Java).
Thanks,
Jc
Daniel D. Daugherty
2018-11-26 14:53:41 UTC
Permalink
Post by s***@oracle.com
Hi guys,
latest webrev: http://cr.openjdk.java.net/~stuefe/webrevs/8213834-jvmti-reourceexhausted-shall-not-be-posted-from-compiler-thread/webrev.02/webrev/
src/hotspot/share/prims/jvmtiExport.cpp
    No comments.

Thumbs up.

Dan
Post by s***@oracle.com
Back to can_call_java(), since this seems to be the consensus, with a
comment added.
As for the Thread::can_send_jvmti_events() property idea, I created a
https://bugs.openjdk.java.net/browse/JDK-8214294 but decided to
postpone this for later. I would like to close that particular issue,
if possible with a minimal fix which can be easily downported to older
released.
Thanks, Thomas
Post by Thomas Stüfe
Hi all,
David and JC already outlined the options we have nicely.
I'd like to add that I do not favor the ServiceThread delayed
deliverance since a common reaction to ResourceExhausted would to
print call stack of the thread running into it or to print a heap
histogram, as jvmkill does. For the former only a synchronous event
delivery makes sense, for the latter at least it helps analyzing.
In cloud foundry, the heap histogram produced by the JVMTI agent can
be helpful, and since in the majority of cases do not entail the
compiler thread running out of metaspace, I'd rather preserve this
ability. So to me, masking this event for this one case is the most
pragmatic approach.
The other option would be not to change the code but to add, in the
JVMTI documentation for ResourceExhausted, the same disclaimer as for
ObjectFree, GCStarted etc: "The event handler must not use JNI
functions and must not use JVM TI functions except those which
specifically allow such use". Then, writers of JVMTI agents like
jvmkill would have to update their agents accordingly.
FWIW, I think JCs idea of exposing the can_call_java attribute somehow
to the outside would also work. But unfortunately not retroactively,
for older releases. Whereas a simple internal patch like "mask
ResourceExhausted" could be backported easily to older releases.
Best Regards, Thomas
For what it's worth I wonder if skipping the event is the best; it is the easiest to ensure non breaking the agent; it does not really go against what the spec is saying and another thread that CAN call java will most likely hit the issue and then all will be good in the world.
However, also for what it's worth I wonder if deferring is not that hard to accomplish either. There already is the infrastructure for this and we should be relatively able to do it. Only question I would have is can the service thread create a JNIEnv for the event but I don't think that's an issue, is it?
It might however conflict with the description of the JNIEnv in the spec which says the jni_env is "The JNI environment of the event (current) thread" though it doesn't say current of what or the event thread of what really.
However, skipping it also kind of goes against the spec since it says: "Sent when a VM resource needed by a running application has been exhausted".Though someone could argue it doesn' t say it is sent when the resource was first noticed to be exhausted.
- Sending the event via the compiler thread will risk breaking things if the agent calls Java -> not really an option
- Using the service thread breaks what David calls the synchronous assumption of the event
- Skipping the event kind of breaks the sentence that the event is sent when a VM resource has been exhausted
So we come back to probably skipping is the best solution since at least the event remains "synchronous" when you get it.
(A side note would be perhaps to augment ThreadInfo* with the "can_call_java" bit and then put in the right spots of the spec that only threads marked as "can_call_java" can safely call Java).
Thanks,
Jc
Thomas Stüfe
2018-11-26 15:07:08 UTC
Permalink
Thanks Dan!
On Mon, Nov 26, 2018 at 3:54 PM Daniel D. Daugherty
Post by s***@oracle.com
Post by s***@oracle.com
Hi guys,
latest webrev: http://cr.openjdk.java.net/~stuefe/webrevs/8213834-jvmti-reourceexhausted-shall-not-be-posted-from-compiler-thread/webrev.02/webrev/
src/hotspot/share/prims/jvmtiExport.cpp
No comments.
Thumbs up.
Dan
Post by s***@oracle.com
Back to can_call_java(), since this seems to be the consensus, with a
comment added.
As for the Thread::can_send_jvmti_events() property idea, I created a
https://bugs.openjdk.java.net/browse/JDK-8214294 but decided to
postpone this for later. I would like to close that particular issue,
if possible with a minimal fix which can be easily downported to older
released.
Thanks, Thomas
Post by Thomas Stüfe
Hi all,
David and JC already outlined the options we have nicely.
I'd like to add that I do not favor the ServiceThread delayed
deliverance since a common reaction to ResourceExhausted would to
print call stack of the thread running into it or to print a heap
histogram, as jvmkill does. For the former only a synchronous event
delivery makes sense, for the latter at least it helps analyzing.
In cloud foundry, the heap histogram produced by the JVMTI agent can
be helpful, and since in the majority of cases do not entail the
compiler thread running out of metaspace, I'd rather preserve this
ability. So to me, masking this event for this one case is the most
pragmatic approach.
The other option would be not to change the code but to add, in the
JVMTI documentation for ResourceExhausted, the same disclaimer as for
ObjectFree, GCStarted etc: "The event handler must not use JNI
functions and must not use JVM TI functions except those which
specifically allow such use". Then, writers of JVMTI agents like
jvmkill would have to update their agents accordingly.
FWIW, I think JCs idea of exposing the can_call_java attribute somehow
to the outside would also work. But unfortunately not retroactively,
for older releases. Whereas a simple internal patch like "mask
ResourceExhausted" could be backported easily to older releases.
Best Regards, Thomas
For what it's worth I wonder if skipping the event is the best; it is the easiest to ensure non breaking the agent; it does not really go against what the spec is saying and another thread that CAN call java will most likely hit the issue and then all will be good in the world.
However, also for what it's worth I wonder if deferring is not that hard to accomplish either. There already is the infrastructure for this and we should be relatively able to do it. Only question I would have is can the service thread create a JNIEnv for the event but I don't think that's an issue, is it?
It might however conflict with the description of the JNIEnv in the spec which says the jni_env is "The JNI environment of the event (current) thread" though it doesn't say current of what or the event thread of what really.
However, skipping it also kind of goes against the spec since it says: "Sent when a VM resource needed by a running application has been exhausted".Though someone could argue it doesn' t say it is sent when the resource was first noticed to be exhausted.
- Sending the event via the compiler thread will risk breaking things if the agent calls Java -> not really an option
- Using the service thread breaks what David calls the synchronous assumption of the event
- Skipping the event kind of breaks the sentence that the event is sent when a VM resource has been exhausted
So we come back to probably skipping is the best solution since at least the event remains "synchronous" when you get it.
(A side note would be perhaps to augment ThreadInfo* with the "can_call_java" bit and then put in the right spots of the spec that only threads marked as "can_call_java" can safely call Java).
Thanks,
Jc
s***@oracle.com
2018-11-26 17:57:16 UTC
Permalink
Hi Thomas,

+1

Thanks,
Serguei
Post by s***@oracle.com
Post by s***@oracle.com
Hi guys,
http://cr.openjdk.java.net/~stuefe/webrevs/8213834-jvmti-reourceexhausted-shall-not-be-posted-from-compiler-thread/webrev.02/webrev/
src/hotspot/share/prims/jvmtiExport.cpp
    No comments.
Thumbs up.
Dan
Post by s***@oracle.com
Back to can_call_java(), since this seems to be the consensus, with a
comment added.
As for the Thread::can_send_jvmti_events() property idea, I created a
https://bugs.openjdk.java.net/browse/JDK-8214294 but decided to
postpone this for later. I would like to close that particular issue,
if possible with a minimal fix which can be easily downported to older
released.
Thanks, Thomas
On Mon, Nov 19, 2018 at 1:00 PM Thomas Stüfe
Post by Thomas Stüfe
Hi all,
David and JC already outlined the options we have nicely.
I'd like to add that I do not favor the ServiceThread delayed
deliverance since a common reaction to ResourceExhausted would to
print call stack of the thread running into it or to print a heap
histogram, as jvmkill does. For the former only a synchronous event
delivery makes sense, for the latter at least it helps analyzing.
In cloud foundry, the heap histogram produced by the JVMTI agent can
be helpful, and since in the majority of cases do not entail the
compiler thread running out of metaspace, I'd rather preserve this
ability. So to me, masking this event for this one case is the most
pragmatic approach.
The other option would be not to change the code but to add, in the
JVMTI documentation for ResourceExhausted, the same disclaimer as for
ObjectFree, GCStarted etc: "The event handler must not use JNI
functions and must not use JVM TI functions except those which
specifically allow such use". Then, writers of JVMTI agents like
jvmkill would have to update their agents accordingly.
FWIW, I think JCs idea of exposing the can_call_java attribute somehow
to the outside would also work. But unfortunately not retroactively,
for older releases. Whereas a simple internal patch like "mask
ResourceExhausted" could be backported easily to older releases.
Best Regards, Thomas
Post by JC Beyler
For what it's worth I wonder if skipping the event is the best; it
is the easiest to ensure non breaking the agent; it does not really
go against what the spec is saying and another thread that CAN call
java will most likely hit the issue and then all will be good in
the world.
However, also for what it's worth I wonder if deferring is not that
hard to accomplish either. There already is the infrastructure for
this and we should be relatively able to do it. Only question I
would have is can the service thread create a JNIEnv for the event
but I don't think that's an issue, is it?
It might however conflict with the description of the JNIEnv in the
spec which says the jni_env is "The JNI environment of the event
(current) thread" though it doesn't say current of what or the
event thread of what really.
However, skipping it also kind of goes against the spec since it
says: "Sent when a VM resource needed by a running application has
been exhausted".Though someone could argue it doesn' t say it is
sent when the resource was first noticed to be exhausted.
- Sending the event via the compiler thread will risk breaking
things if the agent calls Java    -> not really an option
- Using the service thread breaks what David calls the synchronous
assumption of the event
- Skipping the event kind of breaks the sentence that the event is
sent when a VM resource has been exhausted
So we come back to probably skipping is the best solution since at
least the event remains "synchronous" when you get it.
(A side note would be perhaps to augment ThreadInfo* with the
"can_call_java" bit and then put in the right spots of the spec
that only threads marked as "can_call_java" can safely call Java).
Thanks,
Jc
David Holmes
2018-11-26 23:10:35 UTC
Permalink
+1 more

Thanks,
David
Post by s***@oracle.com
Hi Thomas,
+1
Thanks,
Serguei
Post by s***@oracle.com
Post by s***@oracle.com
Hi guys,
http://cr.openjdk.java.net/~stuefe/webrevs/8213834-jvmti-reourceexhausted-shall-not-be-posted-from-compiler-thread/webrev.02/webrev/
src/hotspot/share/prims/jvmtiExport.cpp
    No comments.
Thumbs up.
Dan
Post by s***@oracle.com
Back to can_call_java(), since this seems to be the consensus, with a
comment added.
As for the Thread::can_send_jvmti_events() property idea, I created a
https://bugs.openjdk.java.net/browse/JDK-8214294 but decided to
postpone this for later. I would like to close that particular issue,
if possible with a minimal fix which can be easily downported to older
released.
Thanks, Thomas
On Mon, Nov 19, 2018 at 1:00 PM Thomas Stüfe
Post by Thomas Stüfe
Hi all,
David and JC already outlined the options we have nicely.
I'd like to add that I do not favor the ServiceThread delayed
deliverance since a common reaction to ResourceExhausted would to
print call stack of the thread running into it or to print a heap
histogram, as jvmkill does. For the former only a synchronous event
delivery makes sense, for the latter at least it helps analyzing.
In cloud foundry, the heap histogram produced by the JVMTI agent can
be helpful, and since in the majority of cases do not entail the
compiler thread running out of metaspace, I'd rather preserve this
ability. So to me, masking this event for this one case is the most
pragmatic approach.
The other option would be not to change the code but to add, in the
JVMTI documentation for ResourceExhausted, the same disclaimer as for
ObjectFree, GCStarted etc: "The event handler must not use JNI
functions and must not use JVM TI functions except those which
specifically allow such use". Then, writers of JVMTI agents like
jvmkill would have to update their agents accordingly.
FWIW, I think JCs idea of exposing the can_call_java attribute somehow
to the outside would also work. But unfortunately not retroactively,
for older releases. Whereas a simple internal patch like "mask
ResourceExhausted" could be backported easily to older releases.
Best Regards, Thomas
Post by JC Beyler
For what it's worth I wonder if skipping the event is the best; it
is the easiest to ensure non breaking the agent; it does not really
go against what the spec is saying and another thread that CAN call
java will most likely hit the issue and then all will be good in
the world.
However, also for what it's worth I wonder if deferring is not that
hard to accomplish either. There already is the infrastructure for
this and we should be relatively able to do it. Only question I
would have is can the service thread create a JNIEnv for the event
but I don't think that's an issue, is it?
It might however conflict with the description of the JNIEnv in the
spec which says the jni_env is "The JNI environment of the event
(current) thread" though it doesn't say current of what or the
event thread of what really.
However, skipping it also kind of goes against the spec since it
says: "Sent when a VM resource needed by a running application has
been exhausted".Though someone could argue it doesn' t say it is
sent when the resource was first noticed to be exhausted.
- Sending the event via the compiler thread will risk breaking
things if the agent calls Java    -> not really an option
- Using the service thread breaks what David calls the synchronous
assumption of the event
- Skipping the event kind of breaks the sentence that the event is
sent when a VM resource has been exhausted
So we come back to probably skipping is the best solution since at
least the event remains "synchronous" when you get it.
(A side note would be perhaps to augment ThreadInfo* with the
"can_call_java" bit and then put in the right spots of the spec
that only threads marked as "can_call_java" can safely call Java).
Thanks,
Jc
JC Beyler
2018-11-27 05:50:19 UTC
Permalink
Hi Thomas,

Looks good to me too!
Jc
Post by David Holmes
+1 more
Thanks,
David
Post by s***@oracle.com
Hi Thomas,
+1
Thanks,
Serguei
Post by s***@oracle.com
Post by s***@oracle.com
Hi guys,
http://cr.openjdk.java.net/~stuefe/webrevs/8213834-jvmti-reourceexhausted-shall-not-be-posted-from-compiler-thread/webrev.02/webrev/
Post by s***@oracle.com
Post by s***@oracle.com
src/hotspot/share/prims/jvmtiExport.cpp
No comments.
Thumbs up.
Dan
Post by s***@oracle.com
Back to can_call_java(), since this seems to be the consensus, with a
comment added.
As for the Thread::can_send_jvmti_events() property idea, I created a
https://bugs.openjdk.java.net/browse/JDK-8214294 but decided to
postpone this for later. I would like to close that particular issue,
if possible with a minimal fix which can be easily downported to older
released.
Thanks, Thomas
On Mon, Nov 19, 2018 at 1:00 PM Thomas StÃŒfe
Post by Thomas Stüfe
Hi all,
David and JC already outlined the options we have nicely.
I'd like to add that I do not favor the ServiceThread delayed
deliverance since a common reaction to ResourceExhausted would to
print call stack of the thread running into it or to print a heap
histogram, as jvmkill does. For the former only a synchronous event
delivery makes sense, for the latter at least it helps analyzing.
In cloud foundry, the heap histogram produced by the JVMTI agent can
be helpful, and since in the majority of cases do not entail the
compiler thread running out of metaspace, I'd rather preserve this
ability. So to me, masking this event for this one case is the most
pragmatic approach.
The other option would be not to change the code but to add, in the
JVMTI documentation for ResourceExhausted, the same disclaimer as for
ObjectFree, GCStarted etc: "The event handler must not use JNI
functions and must not use JVM TI functions except those which
specifically allow such use". Then, writers of JVMTI agents like
jvmkill would have to update their agents accordingly.
FWIW, I think JCs idea of exposing the can_call_java attribute somehow
to the outside would also work. But unfortunately not retroactively,
for older releases. Whereas a simple internal patch like "mask
ResourceExhausted" could be backported easily to older releases.
Best Regards, Thomas
Post by JC Beyler
For what it's worth I wonder if skipping the event is the best; it
is the easiest to ensure non breaking the agent; it does not really
go against what the spec is saying and another thread that CAN call
java will most likely hit the issue and then all will be good in
the world.
However, also for what it's worth I wonder if deferring is not that
hard to accomplish either. There already is the infrastructure for
this and we should be relatively able to do it. Only question I
would have is can the service thread create a JNIEnv for the event
but I don't think that's an issue, is it?
It might however conflict with the description of the JNIEnv in the
spec which says the jni_env is "The JNI environment of the event
(current) thread" though it doesn't say current of what or the
event thread of what really.
However, skipping it also kind of goes against the spec since it
says: "Sent when a VM resource needed by a running application has
been exhausted".Though someone could argue it doesn' t say it is
sent when the resource was first noticed to be exhausted.
- Sending the event via the compiler thread will risk breaking
things if the agent calls Java -> not really an option
- Using the service thread breaks what David calls the synchronous
assumption of the event
- Skipping the event kind of breaks the sentence that the event is
sent when a VM resource has been exhausted
So we come back to probably skipping is the best solution since at
least the event remains "synchronous" when you get it.
(A side note would be perhaps to augment ThreadInfo* with the
"can_call_java" bit and then put in the right spots of the spec
that only threads marked as "can_call_java" can safely call Java).
Thanks,
Jc
--
Thanks,
Jc
s***@oracle.com
2018-11-14 23:02:00 UTC
Permalink
Hi Dan,

I thought the same event is going to be posted again on a different thread.
It was wrong assumption, so you are right, it is not a good approach to
fix this issue.

Thanks,
Serguei
Post by Daniel D. Daugherty
I have a philosophical problem with a fix like this.
The fix is making the assumption that the handler for this event will want
to run Java code and if the event is posted from a Java thread that cannot
run Java code, then we skip posting the event.
If I happen to have a more conservative agent that does not run Java code
in its handlers, then my agent won't receive this event even though it
should not cause my agent any problems. That might be an unexpected change
in behavior on the part of my agent.
Dan
Post by Daniel D. Daugherty
Since the proposed solution to the bug is to not post an event, I think
the Serviceability Team (which owns JVM/TI) needs to be involved directly
in the review.
Dan
Dear all,
may I please have reviews on this small patch. Note that this is
borderline serviceability. I try to avoid crossposting, but if you
think this should be looked at by serviceability feel free to forward
it there.
Issue: https://bugs.openjdk.java.net/browse/JDK-8213834
http://cr.openjdk.java.net/~stuefe/webrevs/8213834-jvmti-reourceexhausted-shall-not-be-posted-from-compiler-thread/webrev.00/webrev/
Short description: we may post ResourceExhausted from Compiler
threads. Handlers of this event may call back into the JVM, which may
cause problems since we run in the compiler thread. The proposed
solution is not to post the event in that case.
http://mail.openjdk.java.net/pipermail/serviceability-dev/2018-November/025898.html
Thanks, Thomas
Loading...