s***@oracle.com
2018-10-23 07:21:14 UTC
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">Hi,<br>
<br>
I've added the seviceability-dev mailing list.<br>
It can be interesting for the SVC folks. :)<br>
<br>
<br>
On 10/22/18 22:14, Leonid Mesnik wrote:<br>
</div>
<blockquote type="cite"
cite="mid:6FF8CDAA-B83B-4BFC-8352-***@oracle.com">
<div dir="auto" class="">
<div dir="auto" class="">Hi</div>
<div dir="auto" class=""><br class="">
</div>
<div dir="auto" class="">Seems last version also crashes with 2
other different symptoms.
<div class=""><a
href="http://java.se.oracle.com:10065/mdash/jobs/lmesnik-ks8-20181021-0638-7157/results?search=status:failed+AND+-state:invalid"
class="" moz-do-not-send="true">http://java.se.oracle.com:10065/mdash/jobs/lmesnik-ks8-20181021-0638-7157/results?search=status%3Afailed+AND+-state%3Ainvalid</a></div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">Also it might hangs with stack attached. Seems
that test might be blocked because it invoke 2 jvmti
methods. Can jvmti agent invoke jvmti methods from different
threads?</div>
</div>
</div>
</blockquote>
<br>
Yes, in general.<br>
However, you have to be careful when using debugging features.<br>
Below, one thread is enabling single stepping while another thread
is being suspended.<br>
Both are blocked at a safepoint which is Okay in general but not
Okay if they hold any lock.<br>
For instance, the thread #152 is holding the monitor
JvmtiThreadState.<br>
<br>
Also, I see a couple of more threads that are interesting as well:<br>
<br>
Thread 159 (Thread 0x2ae40b78f700 (LWP 27962)):<br>
#0 0x00002ae3927b5945 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> () from
/lib64/libpthread.so.0<br>
#1 0x00002ae393ba8d63 in os::PlatformEvent::park
(this=***@entry=0x2ae3984c9100) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/os/posix/os_posix.cpp:1897<br>
#2 0x00002ae393b50920 in ParkCommon (timo=0, ev=0x2ae3984c9100) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:399<br>
#3 Monitor::ILock (this=0x2ae398024f10, Self=0x2ae3984c7800) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:461<br>
#4 0x00002ae393b512c1 in lock (Self=0x2ae3984c7800,
this=0x2ae398024f10) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:910<br>
#5 Monitor::lock (this=***@entry=0x2ae398024f10) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:919<br>
#6 0x00002ae39350510c in MutexLocker (mutex=0x2ae398024f10,
this=<synthetic pointer>) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutexLocker.hpp:182<br>
#7 ciEnv::cache_jvmti_state (this=***@entry=0x2ae40b78eb30) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/ci/ciEnv.cpp:229<br>
#8 0x00002ae3935d3294 in CompileBroker::invoke_compiler_on_method
(task=***@entry=0x2ae48800ff40) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/compiler/compileBroker.cpp:2084<br>
#9 0x00002ae3935d4f48 in CompileBroker::compiler_thread_loop () at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/compiler/compileBroker.cpp:1798<br>
#10 0x00002ae393d7338a in JavaThread::thread_main_inner
(this=***@entry=0x2ae3984c7800) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/thread.cpp:1795<br>
#11 0x00002ae393d736c6 in JavaThread::run (this=0x2ae3984c7800) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/thread.cpp:1775<br>
#12 0x00002ae393ba0070 in thread_native_entry
(thread=0x2ae3984c7800) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/os/linux/os_linux.cpp:698<br>
#13 0x00002ae3927b1e25 in start_thread () from
/lib64/libpthread.so.0<br>
#14 0x00002ae392cc234d in clone () from /lib64/libc.so.6<br>
<br>
Thread 158 (Thread 0x2ae40b890700 (LWP 27963)):<br>
#0 0x00002ae3927b5945 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> () from
/lib64/libpthread.so.0<br>
#1 0x00002ae393ba8d63 in os::PlatformEvent::park
(this=***@entry=0x2ae3984cbb00) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/os/posix/os_posix.cpp:1897<br>
#2 0x00002ae393b50920 in ParkCommon (timo=0, ev=0x2ae3984cbb00) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:399<br>
#3 Monitor::ILock (this=0x2ae398024f10, Self=0x2ae3984ca800) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:461<br>
#4 0x00002ae393b512c1 in lock (Self=0x2ae3984ca800,
this=0x2ae398024f10) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:910<br>
#5 Monitor::lock (this=***@entry=0x2ae398024f10) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:919<br>
#6 0x00002ae39350510c in MutexLocker (mutex=0x2ae398024f10,
this=<synthetic pointer>) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutexLocker.hpp:182<br>
#7 ciEnv::cache_jvmti_state (this=***@entry=0x2ae40b88fb30) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/ci/ciEnv.cpp:229<br>
#8 0x00002ae3935d3294 in CompileBroker::invoke_compiler_on_method
(task=***@entry=0x2ae49c00a670) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/compiler/compileBroker.cpp:2084<br>
#9 0x00002ae3935d4f48 in CompileBroker::compiler_thread_loop () at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/compiler/compileBroker.cpp:1798<br>
#10 0x00002ae393d7338a in JavaThread::thread_main_inner
(this=***@entry=0x2ae3984ca800) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/thread.cpp:1795<br>
#11 0x00002ae393d736c6 in JavaThread::run (this=0x2ae3984ca800) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/thread.cpp:1775<br>
#12 0x00002ae393ba0070 in thread_native_entry
(thread=0x2ae3984ca800) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/os/linux/os_linux.cpp:698<br>
#13 0x00002ae3927b1e25 in start_thread () from
/lib64/libpthread.so.0<br>
#14 0x00002ae392cc234d in clone () from /lib64/libc.so.6<br>
<br>
Thread 51 (Thread 0x2ae49549b700 (LWP 29678)):<br>
#0 0x00002ae3927b5945 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> () from
/lib64/libpthread.so.0<br>
#1 0x00002ae393ba8d63 in os::PlatformEvent::park
(this=***@entry=0x2ae460061c00) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/os/posix/os_posix.cpp:1897<br>
#2 0x00002ae393b50920 in ParkCommon (timo=0, ev=0x2ae460061c00) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:399<br>
#3 Monitor::ILock (this=0x2ae398024f10, Self=0x2ae4600c2800) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:461<br>
#4 0x00002ae393b512c1 in lock (Self=0x2ae4600c2800,
this=0x2ae398024f10) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:910<br>
#5 Monitor::lock (this=***@entry=0x2ae398024f10) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:919<br>
#6 0x00002ae393999682 in MutexLocker (mutex=0x2ae398024f10,
this=<synthetic pointer>) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutexLocker.hpp:182<br>
#7 thread_started (thread=0x2ae4600c2800) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiEventController.cpp:668<br>
#8 JvmtiEventController::thread_started (thread=0x2ae4600c2800) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiEventController.cpp:1027<br>
#9 0x00002ae39399f3a0 in JvmtiExport::post_thread_start
(thread=***@entry=0x2ae4600c2800) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiExport.cpp:1395<br>
#10 0x00002ae393d737d8 in JavaThread::run (this=0x2ae4600c2800) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/thread.cpp:1764<br>
#11 0x00002ae393ba0070 in thread_native_entry
(thread=0x2ae4600c2800) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/os/linux/os_linux.cpp:698<br>
#12 0x00002ae3927b1e25 in start_thread () from
/lib64/libpthread.so.0<br>
#13 0x00002ae392cc234d in clone () from /lib64/libc.so.6<br>
<br>
<br>
These two thread are blocked on the monitor JvmtiThreadState_lock in
the function ciEnv::cache_jvmti_state().<br>
Also, there are many threads (like #51) that are executing
JvmtiExport::post_thread_start and blocked on the same monitor.<br>
<br>
<br>
Now, the question is why this safepoint can not start?<br>
What thread is blocking it? Or in reverse, what thread this
safepoint is waiting for?<br>
<br>
I think, this safepoint operation is waiting for all threads that
are blocked on the JvmtiThreadState_lock.<br>
<br>
Conclusion:<br>
<br>
The deadlock is:<br>
<br>
Thread #152:<br>
- grabbed the monitor JvmtiThreadState_lock<br>
- blocked in the VM_GetCurrentLocation in the function
JvmtiEnvThreadState::reset_current_location()<br>
<br>
Many other threads:<br>
- blocked on the monitor JvmtiThreadState_lock<br>
- can not reach the blocked at a safepoint state (all threads have
to reach this state for this safepoint to happen)<br>
<br>
It seems to me, this is a bug which has to be filed.<br>
<br>
My guess is that this will stop to reproduce after if you turn off
the single stepping for thread #152.<br>
Please, let me know about the results.<br>
<br>
<br>
<blockquote type="cite"
cite="mid:6FF8CDAA-B83B-4BFC-8352-***@oracle.com">
<div dir="auto" class="">
<div dir="auto" class="">
<div class="">Assuming that crashes look like VM bugs I think
it make sense to integrate jvmti changes but *don't* enabled
jvmti module by default.</div>
</div>
</div>
</blockquote>
<br>
This one is a deadlock.<br>
However, the root cause is a race condition that can potentially
result in both deadlocks and crashes.<br>
So, I'm curious if you observed crashes as well.<br>
<br>
<br>
<blockquote type="cite"
cite="mid:6FF8CDAA-B83B-4BFC-8352-***@oracle.com">
<div dir="auto" class="">
<div dir="auto" class="">
<div class=""> And add to more tests with jvmti enabled.</div>
<div class="">So anyone could easily run them to reproduce
crashes. This test would be out of CI to don't introduce
any bugs. Does it make sense?</div>
<div class=""><br class="">
</div>
<div class="">Consider hang - I think that it might be product
bug since I don't see any locking on my monitors. But I am
not sure. Is it possible that any my code jvmti agent
prevent VM to get into safepoint?</div>
<div class="">Could we discuss it tomorrow or his week when
you have a time?</div>
</div>
</div>
</blockquote>
<br>
Yes, of course.<br>
Let's find some time tomorrow.<br>
<br>
<br>
<blockquote type="cite"
cite="mid:6FF8CDAA-B83B-4BFC-8352-***@oracle.com">
<div dir="auto" class="">
<div dir="auto" class="">
<div class=""> Any suggestion how to diagnose deadlock would
be great.</div>
</div>
</div>
</blockquote>
<br>
Analysis of stack traces is needed.<br>
It is non-trivial in this particular case as there are so many
threads executed at the same time.<br>
<br>
<br>
<blockquote type="cite"
cite="mid:6FF8CDAA-B83B-4BFC-8352-***@oracle.com">
<div dir="auto" class="">
<div dir="auto" class="">
<div class="">Part of stack trace with 2 my threads only:</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">Thread 136 (Thread 0x2ae494100700 (LWP
28023)):</div>
<div class="">#0 0x00002ae3927b5945 in
<a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> () from
/lib64/libpthread.so.0</div>
<div class="">#1 0x00002ae393ba8d63 in
os::PlatformEvent::park (this=***@entry=0x2ae454005800)
at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/os/posix/os_posix.cpp:1897</div>
<div class="">#2 0x00002ae393b50cf8 in ParkCommon (timo=0,
ev=0x2ae454005800) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:399</div>
<div class="">#3 Monitor::IWait
(this=***@entry=0x2ae398023c10,
Self=***@entry=0x2ae454004800, timo=***@entry=0) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:76\</div>
<div class="">8</div>
<div class="">#4 0x00002ae393b51f2e in Monitor::wait
(this=***@entry=0x2ae398023c10,
no_safepoint_check=<optimized out>,
timeout=***@entry=0,
as_suspend_equivalent=***@en\</div>
<div class="">try=false) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:1106</div>
<div class="">#5 0x00002ae393de7867 in VMThread::execute
(op=***@entry=0x2ae4940ffb10) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/vmThread.cpp:657</div>
<div class="">#6 0x00002ae393d6a3bd in
JavaThread::java_suspend (this=***@entry=0x2ae3985f2000)
at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/thread.cpp:2321</div>
<div class="">#7 0x00002ae3939ad7e1 in
JvmtiSuspendControl::suspend
(java_thread=***@entry=0x2ae3985f2000) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiImpl.cpp:8\</div>
<div class="">47</div>
<div class="">#8 0x00002ae3939887ae in
JvmtiEnv::SuspendThread (this=***@entry=0x2ae39801b270,
java_thread=0x2ae3985f2000) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiE\</div>
<div class="">nv.cpp:955</div>
<div class="">#9 0x00002ae39393a8c6 in jvmti_SuspendThread
(env=0x2ae39801b270, thread=0x2ae49929fdf8) at
/scratch/lmesnik/ws/hs-bigapps/build/linux-x64/hotspot/variant-server/gensrc/jvmtifiles\</div>
<div class="">/jvmtiEnter.cpp:527</div>
<div class="">#10 0x00002ae394d973ee in agent_sampler
(jvmti=0x2ae39801b270, env=<optimized out>,
p=<optimized out>) at
/scratch/lmesnik/ws/hs-bigapps/closed/test/hotspot/jtreg/applications/kitc\</div>
<div class="">hensink/process/stress/modules/libJvmtiStressModule.c:274</div>
<div class="">#11 0x00002ae3939ab24d in call_start_function
(this=0x2ae454004800) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiImpl.cpp:85</div>
<div class="">#12 JvmtiAgentThread::start_function_wrapper
(thread=0x2ae454004800, __the_thread__=<optimized
out>) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiImpl.cpp:79</div>
<div class="">#13 0x00002ae393d7338a in
JavaThread::thread_main_inner
(this=***@entry=0x2ae454004800) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/thread.cpp:1795</div>
<div class="">#14 0x00002ae393d736c6 in JavaThread::run
(this=0x2ae454004800) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/thread.cpp:1775</div>
<div class="">#15 0x00002ae393ba0070 in thread_native_entry
(thread=0x2ae454004800) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/os/linux/os_linux.cpp:698</div>
<div class="">#16 0x00002ae3927b1e25 in start_thread () from
/lib64/libpthread.so.0</div>
<div class="">#17 0x00002ae392cc234d in clone () from
/lib64/libc.so.6</div>
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">Thread 152 (Thread 0x2ae427060700 (LWP
27995)):</div>
<div class="">#0 0x00002ae3927b5945 in
<a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> () from
/lib64/libpthread.so.0</div>
<div class="">#1 0x00002ae393ba8d63 in
os::PlatformEvent::park (this=***@entry=0x2ae3985e7400)
at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/os/posix/os_posix.cpp:1897</div>
<div class="">#2 0x00002ae393b50cf8 in ParkCommon (timo=0,
ev=0x2ae3985e7400) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:399</div>
<div class="">#3 Monitor::IWait
(this=***@entry=0x2ae398023c10,
Self=***@entry=0x2ae3985e6000, timo=***@entry=0) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:76\</div>
<div class="">8</div>
<div class="">#4 0x00002ae393b51f2e in Monitor::wait
(this=***@entry=0x2ae398023c10,
no_safepoint_check=<optimized out>,
timeout=***@entry=0,
as_suspend_equivalent=***@en\</div>
<div class="">try=false) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:1106</div>
<div class="">#5 0x00002ae393de7867 in VMThread::execute
(op=0x2ae42705f500) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/vmThread.cpp:657</div>
<div class="">#6 0x00002ae3939965f3 in
JvmtiEnvThreadState::reset_current_location
(this=***@entry=0x2ae6bc000d80,
event_type=***@entry=JVMTI_EVENT_SINGLE_STEP,
enabled=***@entry=tr\</div>
<div class="">ue) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiEnvThreadState.cpp:312</div>
<div class="">#7 0x00002ae393997acf in
recompute_env_thread_enabled (state=0x2ae6bc000cd0,
ets=0x2ae6bc000d80) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiEventControlle\</div>
<div class="">r.cpp:490</div>
<div class="">#8
JvmtiEventControllerPrivate::recompute_thread_enabled
(state=***@entry=0x2ae6bc000cd0) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiEventController.cpp\</div>
<div class="">:523</div>
<div class="">#9 0x00002ae393998168 in
JvmtiEventControllerPrivate::recompute_enabled () at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiEventController.cpp:598</div>
<div class="">#10 0x00002ae39399a244 in set_user_enabled
(enabled=true, event_type=JVMTI_EVENT_SINGLE_STEP,
thread=0x0, env=0x2ae39801b270) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/sha\</div>
<div class="">re/prims/jvmtiEventController.cpp:818</div>
<div class="">#11 JvmtiEventController::set_user_enabled
(env=0x2ae39801b270, thread=0x0,
event_type=JVMTI_EVENT_SINGLE_STEP, enabled=<optimized
out>) at /scratch/lmesnik/ws/hs-bigapps/open/src/\</div>
<div class="">hotspot/share/prims/jvmtiEventController.cpp:963</div>
</div>
<div class="">
<div class="">#12 0x00002ae393987d2d in
JvmtiEnv::SetEventNotificationMode
(this=***@entry=0x2ae39801b270,
mode=***@entry=JVMTI_ENABLE,
event_type=***@entry=JVMTI_EVENT_SINGLE_STEP, eve\</div>
<div class="">nt_thread=***@entry=0x0) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiEnv.cpp:543</div>
<div class="">#13 0x00002ae3939414eb in
jvmti_SetEventNotificationMode (env=0x2ae39801b270,
mode=***@entry=JVMTI_ENABLE,
event_type=***@entry=JVMTI_EVENT_SINGLE_STEP,
event_thread=event_\</div>
<div class="">***@entry=0x0) at
/scratch/lmesnik/ws/hs-bigapps/build/linux-x64/hotspot/variant-server/gensrc/jvmtifiles/jvmtiEnter.cpp:5389</div>
<div class="">#14 0x00002ae394d97989 in enable_events () at
/scratch/lmesnik/ws/hs-bigapps/closed/test/hotspot/jtreg/applications/kitchensink/process/stress/modules/libJvmtiStressModule.c:519</div>
<div class="">#15 0x00002ae394d98070 in
Java_applications_kitchensink_process_stress_modules_JvmtiStressModule_startIteration
(env=<optimized out>, this=<optimized out>) at
/scratch/lmesnik/ws/h\</div>
<div class="">s-bigapps/closed/test/hotspot/jtreg/applications/kitchensink/process/stress/modules/libJvmtiStressModule.c:697</div>
<div class="">#16 0x00002ae3a43ef257 in ?? ()</div>
<div class="">#17 0x00002ae3a43eede1 in ?? ()</div>
<div class="">#18 0x00002ae42705f878 in ?? ()</div>
<div class="">#19 0x00002ae40ad334e0 in ?? ()</div>
<div class="">#20 0x00002ae42705f8e0 in ?? ()</div>
<div class="">#21 0x00002ae40ad33c68 in ?? ()</div>
<div class="">#22 0x0000000000000000 in ?? ()</div>
</div>
</div>
</div>
</blockquote>
<br>
Thanks,<br>
Serguei<br>
<br>
<blockquote type="cite"
cite="mid:6FF8CDAA-B83B-4BFC-8352-***@oracle.com">
<div dir="auto" class="">
<div dir="auto" class="">
<div class=""> </div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<div dir="auto" class="">
<div dir="auto" class="">
<div class=""><br class="">
</div>
<div class="">Leonid</div>
<div class=""><br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Oct 9, 2018, at 4:52 PM, <a
href="mailto:***@oracle.com" class=""
moz-do-not-send="true">***@oracle.com</a>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">Hi Leonid,<br class="">
<br class="">
There is an existing bug:<br class="">
<a
href="https://bugs.openjdk.java.net/browse/JDK-8043571"
class="" moz-do-not-send="true">https://bugs.openjdk.java.net/browse/JDK-8043571</a><br
class="">
<br class="">
Thanks,<br class="">
Serguei<br class="">
<br class="">
<br class="">
On 10/9/18 16:11, Leonid Mesnik wrote:<br class="">
<blockquote type="cite" class="">Hi<br class="">
<br class="">
During fixing kitchensink I get<br class="">
assert(_cur_stack_depth == count_frames()) failed:
cur_stack_depth out of sync<br class="">
<br class="">
Do you know if i might be bug in my jvmti agent?<br
class="">
<br class="">
Leonid<br class="">
<br class="">
<br class="">
#<br class="">
# A fatal error has been detected by the Java
Runtime Environment:<br class="">
#<br class="">
# Internal Error
(/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiThreadState.cpp:277),
pid=13926, tid=13962<br class="">
# assert(_cur_stack_depth == count_frames())
failed: cur_stack_depth out of sync<br class="">
#<br class="">
# JRE version: Java(TM) SE Runtime Environment
(12.0) (fastdebug build
12-internal+0-2018-10-08-2342517.lmesnik.hs-bigapps)<br
class="">
# Java VM: Java HotSpot(TM) 64-Bit Server VM
(fastdebug
12-internal+0-2018-10-08-2342517.lmesnik.hs-bigapps,
mixed mode, tiered, compressed oops, g1 gc,
linux-amd64)<br class="">
# Core dump will be written. Default location:
Core dumps may be processed with
"/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t e
%P %I %h" (or dumping to
/scratch/lmesnik/ws/hs-bigapps/build/linux-x64/test-support/jtreg_closed_test_hotspot_jtreg_applications_kitchensink_KitchensinkSanity_java/scratch/0/core.13926)<br
class="">
#<br class="">
# If you would like to submit a bug report, please
visit:<br class="">
# <a
href="http://bugreport.java.com/bugreport/crash.jsp"
class="" moz-do-not-send="true">http://bugreport.java.com/bugreport/crash.jsp</a><br
class="">
#<br class="">
<br class="">
--------------- S U M M A R Y ------------<br
class="">
<br class="">
Command Line: -XX:MaxRAMPercentage=2
-XX:MaxRAMPercentage=50
-XX:+CrashOnOutOfMemoryError
-Djava.net.preferIPv6Addresses=false
-XX:-PrintVMOptions -XX:+DisplayVMOutputToStderr
-XX:+UsePerfData
-Xlog:gc*,gc+heap=debug:gc.log:uptime,timemillis,level,tags
-XX:+DisableExplicitGC -XX:+PrintFlagsFinal
-XX:+StartAttachListener
-XX:NativeMemoryTracking=detail
-XX:+FlightRecorder
--add-exports=java.base/java.lang=ALL-UNNAMED
--add-opens=java.base/java.lang=ALL-UNNAMED
--add-exports=java.xml/com.sun.org.apache.xerces.internal.parsers=ALL-UNNAMED
--add-exports=java.xml/com.sun.org.apache.xerces.internal.util=ALL-UNNAMED
-Djava.io.tmpdir=/scratch/lmesnik/ws/hs-bigapps/build/linux-x64/test-support/jtreg_closed_test_hotspot_jtreg_applications_kitchensink_KitchensinkSanity_java/scratch/0/java.io.tmpdir
-Duser.home=/scratch/lmesnik/ws/hs-bigapps/build/linux-x64/test-support/jtreg_closed_test_hotspot_jtreg_applications_kitchensink_KitchensinkSanity_java/scratch/0/user.home
-agentpath:/scratch/lmesnik/ws/hs-bigapps/build/linux-x64/images/test/hotspot/jtreg/native/libJvmtiStressModule.so
applications.kitchensink.process.stress.Main
/scratch/lmesnik/ws/hs-bigapps/build/linux-x64/test-support/jtreg_closed_test_hotspot_jtreg_applications_kitchensink_KitchensinkSanity_java/scratch/0/kitchensink.final.properties<br
class="">
<br class="">
Host: <a href="http://scaaa118.us.oracle.com"
class="" moz-do-not-send="true">scaaa118.us.oracle.com</a>,
Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz, 32
cores, 235G, Oracle Linux Server release 7.3<br
class="">
Time: Tue Oct 9 16:06:07 2018 PDT elapsed time:
31 seconds (0d 0h 0m 31s)<br class="">
<br class="">
--------------- T H R E A D ---------------<br
class="">
<br class="">
Current thread (0x00002af3dc6ac800): VMThread "VM
Thread" [stack:
0x00002af44f10a000,0x00002af44f20a000] [id=13962]
_threads_hazard_ptr=0x00002af4ac090eb0,
_nested_threads_hazard_ptr_cnt=0<br class="">
<br class="">
Stack: [0x00002af44f10a000,0x00002af44f20a000],
sp=0x00002af44f208720, free space=1017k<br
class="">
Native frames: (J=compiled Java code, A=aot
compiled Java code, j=interpreted, Vv=VM code,
C=native code)<br class="">
V [libjvm.so+0x18c4923]
VMError::report_and_die(int, char const*, char
const*, __va_list_tag*, Thread*, unsigned char*,
void*, void*, char const*, int, unsigned
long)+0x2c3<br class="">
V [libjvm.so+0x18c56ef]
VMError::report_and_die(Thread*, void*, char
const*, int, char const*, char const*,
__va_list_tag*)+0x2f<br class="">
V [libjvm.so+0xb55aa0] report_vm_error(char
const*, int, char const*, char const*, ...)+0x100<br
class="">
V [libjvm.so+0x11f2cfe]
JvmtiThreadState::cur_stack_depth()+0x14e<br
class="">
V [libjvm.so+0x11f3257]
JvmtiThreadState::update_for_pop_top_frame()+0x27<br
class="">
V [libjvm.so+0x119af99]
VM_UpdateForPopTopFrame::doit()+0xb9<br class="">
V [libjvm.so+0x1908982]
VM_Operation::evaluate()+0x132<br class="">
V [libjvm.so+0x19040be]
VMThread::evaluate_operation(VM_Operation*)
[clone .constprop.51]+0x18e<br class="">
V [libjvm.so+0x1904960] VMThread::loop()+0x4c0<br
class="">
V [libjvm.so+0x1904f53] VMThread::run()+0xd3<br
class="">
V [libjvm.so+0x14e8300]
thread_native_entry(Thread*)+0x100<br class="">
<br class="">
VM_Operation (0x00002af4d8502910):
UpdateForPopTopFrame, mode: safepoint, requested
by thread 0x00002af4dc008800<br class="">
</blockquote>
<br class="">
<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</blockquote>
<br>
</body>
</html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">Hi,<br>
<br>
I've added the seviceability-dev mailing list.<br>
It can be interesting for the SVC folks. :)<br>
<br>
<br>
On 10/22/18 22:14, Leonid Mesnik wrote:<br>
</div>
<blockquote type="cite"
cite="mid:6FF8CDAA-B83B-4BFC-8352-***@oracle.com">
<div dir="auto" class="">
<div dir="auto" class="">Hi</div>
<div dir="auto" class=""><br class="">
</div>
<div dir="auto" class="">Seems last version also crashes with 2
other different symptoms.
<div class=""><a
href="http://java.se.oracle.com:10065/mdash/jobs/lmesnik-ks8-20181021-0638-7157/results?search=status:failed+AND+-state:invalid"
class="" moz-do-not-send="true">http://java.se.oracle.com:10065/mdash/jobs/lmesnik-ks8-20181021-0638-7157/results?search=status%3Afailed+AND+-state%3Ainvalid</a></div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">Also it might hangs with stack attached. Seems
that test might be blocked because it invoke 2 jvmti
methods. Can jvmti agent invoke jvmti methods from different
threads?</div>
</div>
</div>
</blockquote>
<br>
Yes, in general.<br>
However, you have to be careful when using debugging features.<br>
Below, one thread is enabling single stepping while another thread
is being suspended.<br>
Both are blocked at a safepoint which is Okay in general but not
Okay if they hold any lock.<br>
For instance, the thread #152 is holding the monitor
JvmtiThreadState.<br>
<br>
Also, I see a couple of more threads that are interesting as well:<br>
<br>
Thread 159 (Thread 0x2ae40b78f700 (LWP 27962)):<br>
#0 0x00002ae3927b5945 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> () from
/lib64/libpthread.so.0<br>
#1 0x00002ae393ba8d63 in os::PlatformEvent::park
(this=***@entry=0x2ae3984c9100) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/os/posix/os_posix.cpp:1897<br>
#2 0x00002ae393b50920 in ParkCommon (timo=0, ev=0x2ae3984c9100) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:399<br>
#3 Monitor::ILock (this=0x2ae398024f10, Self=0x2ae3984c7800) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:461<br>
#4 0x00002ae393b512c1 in lock (Self=0x2ae3984c7800,
this=0x2ae398024f10) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:910<br>
#5 Monitor::lock (this=***@entry=0x2ae398024f10) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:919<br>
#6 0x00002ae39350510c in MutexLocker (mutex=0x2ae398024f10,
this=<synthetic pointer>) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutexLocker.hpp:182<br>
#7 ciEnv::cache_jvmti_state (this=***@entry=0x2ae40b78eb30) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/ci/ciEnv.cpp:229<br>
#8 0x00002ae3935d3294 in CompileBroker::invoke_compiler_on_method
(task=***@entry=0x2ae48800ff40) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/compiler/compileBroker.cpp:2084<br>
#9 0x00002ae3935d4f48 in CompileBroker::compiler_thread_loop () at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/compiler/compileBroker.cpp:1798<br>
#10 0x00002ae393d7338a in JavaThread::thread_main_inner
(this=***@entry=0x2ae3984c7800) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/thread.cpp:1795<br>
#11 0x00002ae393d736c6 in JavaThread::run (this=0x2ae3984c7800) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/thread.cpp:1775<br>
#12 0x00002ae393ba0070 in thread_native_entry
(thread=0x2ae3984c7800) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/os/linux/os_linux.cpp:698<br>
#13 0x00002ae3927b1e25 in start_thread () from
/lib64/libpthread.so.0<br>
#14 0x00002ae392cc234d in clone () from /lib64/libc.so.6<br>
<br>
Thread 158 (Thread 0x2ae40b890700 (LWP 27963)):<br>
#0 0x00002ae3927b5945 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> () from
/lib64/libpthread.so.0<br>
#1 0x00002ae393ba8d63 in os::PlatformEvent::park
(this=***@entry=0x2ae3984cbb00) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/os/posix/os_posix.cpp:1897<br>
#2 0x00002ae393b50920 in ParkCommon (timo=0, ev=0x2ae3984cbb00) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:399<br>
#3 Monitor::ILock (this=0x2ae398024f10, Self=0x2ae3984ca800) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:461<br>
#4 0x00002ae393b512c1 in lock (Self=0x2ae3984ca800,
this=0x2ae398024f10) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:910<br>
#5 Monitor::lock (this=***@entry=0x2ae398024f10) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:919<br>
#6 0x00002ae39350510c in MutexLocker (mutex=0x2ae398024f10,
this=<synthetic pointer>) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutexLocker.hpp:182<br>
#7 ciEnv::cache_jvmti_state (this=***@entry=0x2ae40b88fb30) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/ci/ciEnv.cpp:229<br>
#8 0x00002ae3935d3294 in CompileBroker::invoke_compiler_on_method
(task=***@entry=0x2ae49c00a670) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/compiler/compileBroker.cpp:2084<br>
#9 0x00002ae3935d4f48 in CompileBroker::compiler_thread_loop () at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/compiler/compileBroker.cpp:1798<br>
#10 0x00002ae393d7338a in JavaThread::thread_main_inner
(this=***@entry=0x2ae3984ca800) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/thread.cpp:1795<br>
#11 0x00002ae393d736c6 in JavaThread::run (this=0x2ae3984ca800) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/thread.cpp:1775<br>
#12 0x00002ae393ba0070 in thread_native_entry
(thread=0x2ae3984ca800) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/os/linux/os_linux.cpp:698<br>
#13 0x00002ae3927b1e25 in start_thread () from
/lib64/libpthread.so.0<br>
#14 0x00002ae392cc234d in clone () from /lib64/libc.so.6<br>
<br>
Thread 51 (Thread 0x2ae49549b700 (LWP 29678)):<br>
#0 0x00002ae3927b5945 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> () from
/lib64/libpthread.so.0<br>
#1 0x00002ae393ba8d63 in os::PlatformEvent::park
(this=***@entry=0x2ae460061c00) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/os/posix/os_posix.cpp:1897<br>
#2 0x00002ae393b50920 in ParkCommon (timo=0, ev=0x2ae460061c00) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:399<br>
#3 Monitor::ILock (this=0x2ae398024f10, Self=0x2ae4600c2800) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:461<br>
#4 0x00002ae393b512c1 in lock (Self=0x2ae4600c2800,
this=0x2ae398024f10) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:910<br>
#5 Monitor::lock (this=***@entry=0x2ae398024f10) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:919<br>
#6 0x00002ae393999682 in MutexLocker (mutex=0x2ae398024f10,
this=<synthetic pointer>) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutexLocker.hpp:182<br>
#7 thread_started (thread=0x2ae4600c2800) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiEventController.cpp:668<br>
#8 JvmtiEventController::thread_started (thread=0x2ae4600c2800) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiEventController.cpp:1027<br>
#9 0x00002ae39399f3a0 in JvmtiExport::post_thread_start
(thread=***@entry=0x2ae4600c2800) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiExport.cpp:1395<br>
#10 0x00002ae393d737d8 in JavaThread::run (this=0x2ae4600c2800) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/thread.cpp:1764<br>
#11 0x00002ae393ba0070 in thread_native_entry
(thread=0x2ae4600c2800) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/os/linux/os_linux.cpp:698<br>
#12 0x00002ae3927b1e25 in start_thread () from
/lib64/libpthread.so.0<br>
#13 0x00002ae392cc234d in clone () from /lib64/libc.so.6<br>
<br>
<br>
These two thread are blocked on the monitor JvmtiThreadState_lock in
the function ciEnv::cache_jvmti_state().<br>
Also, there are many threads (like #51) that are executing
JvmtiExport::post_thread_start and blocked on the same monitor.<br>
<br>
<br>
Now, the question is why this safepoint can not start?<br>
What thread is blocking it? Or in reverse, what thread this
safepoint is waiting for?<br>
<br>
I think, this safepoint operation is waiting for all threads that
are blocked on the JvmtiThreadState_lock.<br>
<br>
Conclusion:<br>
<br>
The deadlock is:<br>
<br>
Thread #152:<br>
- grabbed the monitor JvmtiThreadState_lock<br>
- blocked in the VM_GetCurrentLocation in the function
JvmtiEnvThreadState::reset_current_location()<br>
<br>
Many other threads:<br>
- blocked on the monitor JvmtiThreadState_lock<br>
- can not reach the blocked at a safepoint state (all threads have
to reach this state for this safepoint to happen)<br>
<br>
It seems to me, this is a bug which has to be filed.<br>
<br>
My guess is that this will stop to reproduce after if you turn off
the single stepping for thread #152.<br>
Please, let me know about the results.<br>
<br>
<br>
<blockquote type="cite"
cite="mid:6FF8CDAA-B83B-4BFC-8352-***@oracle.com">
<div dir="auto" class="">
<div dir="auto" class="">
<div class="">Assuming that crashes look like VM bugs I think
it make sense to integrate jvmti changes but *don't* enabled
jvmti module by default.</div>
</div>
</div>
</blockquote>
<br>
This one is a deadlock.<br>
However, the root cause is a race condition that can potentially
result in both deadlocks and crashes.<br>
So, I'm curious if you observed crashes as well.<br>
<br>
<br>
<blockquote type="cite"
cite="mid:6FF8CDAA-B83B-4BFC-8352-***@oracle.com">
<div dir="auto" class="">
<div dir="auto" class="">
<div class=""> And add to more tests with jvmti enabled.</div>
<div class="">So anyone could easily run them to reproduce
crashes. This test would be out of CI to don't introduce
any bugs. Does it make sense?</div>
<div class=""><br class="">
</div>
<div class="">Consider hang - I think that it might be product
bug since I don't see any locking on my monitors. But I am
not sure. Is it possible that any my code jvmti agent
prevent VM to get into safepoint?</div>
<div class="">Could we discuss it tomorrow or his week when
you have a time?</div>
</div>
</div>
</blockquote>
<br>
Yes, of course.<br>
Let's find some time tomorrow.<br>
<br>
<br>
<blockquote type="cite"
cite="mid:6FF8CDAA-B83B-4BFC-8352-***@oracle.com">
<div dir="auto" class="">
<div dir="auto" class="">
<div class=""> Any suggestion how to diagnose deadlock would
be great.</div>
</div>
</div>
</blockquote>
<br>
Analysis of stack traces is needed.<br>
It is non-trivial in this particular case as there are so many
threads executed at the same time.<br>
<br>
<br>
<blockquote type="cite"
cite="mid:6FF8CDAA-B83B-4BFC-8352-***@oracle.com">
<div dir="auto" class="">
<div dir="auto" class="">
<div class="">Part of stack trace with 2 my threads only:</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">Thread 136 (Thread 0x2ae494100700 (LWP
28023)):</div>
<div class="">#0 0x00002ae3927b5945 in
<a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> () from
/lib64/libpthread.so.0</div>
<div class="">#1 0x00002ae393ba8d63 in
os::PlatformEvent::park (this=***@entry=0x2ae454005800)
at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/os/posix/os_posix.cpp:1897</div>
<div class="">#2 0x00002ae393b50cf8 in ParkCommon (timo=0,
ev=0x2ae454005800) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:399</div>
<div class="">#3 Monitor::IWait
(this=***@entry=0x2ae398023c10,
Self=***@entry=0x2ae454004800, timo=***@entry=0) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:76\</div>
<div class="">8</div>
<div class="">#4 0x00002ae393b51f2e in Monitor::wait
(this=***@entry=0x2ae398023c10,
no_safepoint_check=<optimized out>,
timeout=***@entry=0,
as_suspend_equivalent=***@en\</div>
<div class="">try=false) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:1106</div>
<div class="">#5 0x00002ae393de7867 in VMThread::execute
(op=***@entry=0x2ae4940ffb10) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/vmThread.cpp:657</div>
<div class="">#6 0x00002ae393d6a3bd in
JavaThread::java_suspend (this=***@entry=0x2ae3985f2000)
at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/thread.cpp:2321</div>
<div class="">#7 0x00002ae3939ad7e1 in
JvmtiSuspendControl::suspend
(java_thread=***@entry=0x2ae3985f2000) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiImpl.cpp:8\</div>
<div class="">47</div>
<div class="">#8 0x00002ae3939887ae in
JvmtiEnv::SuspendThread (this=***@entry=0x2ae39801b270,
java_thread=0x2ae3985f2000) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiE\</div>
<div class="">nv.cpp:955</div>
<div class="">#9 0x00002ae39393a8c6 in jvmti_SuspendThread
(env=0x2ae39801b270, thread=0x2ae49929fdf8) at
/scratch/lmesnik/ws/hs-bigapps/build/linux-x64/hotspot/variant-server/gensrc/jvmtifiles\</div>
<div class="">/jvmtiEnter.cpp:527</div>
<div class="">#10 0x00002ae394d973ee in agent_sampler
(jvmti=0x2ae39801b270, env=<optimized out>,
p=<optimized out>) at
/scratch/lmesnik/ws/hs-bigapps/closed/test/hotspot/jtreg/applications/kitc\</div>
<div class="">hensink/process/stress/modules/libJvmtiStressModule.c:274</div>
<div class="">#11 0x00002ae3939ab24d in call_start_function
(this=0x2ae454004800) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiImpl.cpp:85</div>
<div class="">#12 JvmtiAgentThread::start_function_wrapper
(thread=0x2ae454004800, __the_thread__=<optimized
out>) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiImpl.cpp:79</div>
<div class="">#13 0x00002ae393d7338a in
JavaThread::thread_main_inner
(this=***@entry=0x2ae454004800) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/thread.cpp:1795</div>
<div class="">#14 0x00002ae393d736c6 in JavaThread::run
(this=0x2ae454004800) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/thread.cpp:1775</div>
<div class="">#15 0x00002ae393ba0070 in thread_native_entry
(thread=0x2ae454004800) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/os/linux/os_linux.cpp:698</div>
<div class="">#16 0x00002ae3927b1e25 in start_thread () from
/lib64/libpthread.so.0</div>
<div class="">#17 0x00002ae392cc234d in clone () from
/lib64/libc.so.6</div>
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">Thread 152 (Thread 0x2ae427060700 (LWP
27995)):</div>
<div class="">#0 0x00002ae3927b5945 in
<a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> () from
/lib64/libpthread.so.0</div>
<div class="">#1 0x00002ae393ba8d63 in
os::PlatformEvent::park (this=***@entry=0x2ae3985e7400)
at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/os/posix/os_posix.cpp:1897</div>
<div class="">#2 0x00002ae393b50cf8 in ParkCommon (timo=0,
ev=0x2ae3985e7400) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:399</div>
<div class="">#3 Monitor::IWait
(this=***@entry=0x2ae398023c10,
Self=***@entry=0x2ae3985e6000, timo=***@entry=0) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:76\</div>
<div class="">8</div>
<div class="">#4 0x00002ae393b51f2e in Monitor::wait
(this=***@entry=0x2ae398023c10,
no_safepoint_check=<optimized out>,
timeout=***@entry=0,
as_suspend_equivalent=***@en\</div>
<div class="">try=false) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:1106</div>
<div class="">#5 0x00002ae393de7867 in VMThread::execute
(op=0x2ae42705f500) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/vmThread.cpp:657</div>
<div class="">#6 0x00002ae3939965f3 in
JvmtiEnvThreadState::reset_current_location
(this=***@entry=0x2ae6bc000d80,
event_type=***@entry=JVMTI_EVENT_SINGLE_STEP,
enabled=***@entry=tr\</div>
<div class="">ue) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiEnvThreadState.cpp:312</div>
<div class="">#7 0x00002ae393997acf in
recompute_env_thread_enabled (state=0x2ae6bc000cd0,
ets=0x2ae6bc000d80) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiEventControlle\</div>
<div class="">r.cpp:490</div>
<div class="">#8
JvmtiEventControllerPrivate::recompute_thread_enabled
(state=***@entry=0x2ae6bc000cd0) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiEventController.cpp\</div>
<div class="">:523</div>
<div class="">#9 0x00002ae393998168 in
JvmtiEventControllerPrivate::recompute_enabled () at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiEventController.cpp:598</div>
<div class="">#10 0x00002ae39399a244 in set_user_enabled
(enabled=true, event_type=JVMTI_EVENT_SINGLE_STEP,
thread=0x0, env=0x2ae39801b270) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/sha\</div>
<div class="">re/prims/jvmtiEventController.cpp:818</div>
<div class="">#11 JvmtiEventController::set_user_enabled
(env=0x2ae39801b270, thread=0x0,
event_type=JVMTI_EVENT_SINGLE_STEP, enabled=<optimized
out>) at /scratch/lmesnik/ws/hs-bigapps/open/src/\</div>
<div class="">hotspot/share/prims/jvmtiEventController.cpp:963</div>
</div>
<div class="">
<div class="">#12 0x00002ae393987d2d in
JvmtiEnv::SetEventNotificationMode
(this=***@entry=0x2ae39801b270,
mode=***@entry=JVMTI_ENABLE,
event_type=***@entry=JVMTI_EVENT_SINGLE_STEP, eve\</div>
<div class="">nt_thread=***@entry=0x0) at
/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiEnv.cpp:543</div>
<div class="">#13 0x00002ae3939414eb in
jvmti_SetEventNotificationMode (env=0x2ae39801b270,
mode=***@entry=JVMTI_ENABLE,
event_type=***@entry=JVMTI_EVENT_SINGLE_STEP,
event_thread=event_\</div>
<div class="">***@entry=0x0) at
/scratch/lmesnik/ws/hs-bigapps/build/linux-x64/hotspot/variant-server/gensrc/jvmtifiles/jvmtiEnter.cpp:5389</div>
<div class="">#14 0x00002ae394d97989 in enable_events () at
/scratch/lmesnik/ws/hs-bigapps/closed/test/hotspot/jtreg/applications/kitchensink/process/stress/modules/libJvmtiStressModule.c:519</div>
<div class="">#15 0x00002ae394d98070 in
Java_applications_kitchensink_process_stress_modules_JvmtiStressModule_startIteration
(env=<optimized out>, this=<optimized out>) at
/scratch/lmesnik/ws/h\</div>
<div class="">s-bigapps/closed/test/hotspot/jtreg/applications/kitchensink/process/stress/modules/libJvmtiStressModule.c:697</div>
<div class="">#16 0x00002ae3a43ef257 in ?? ()</div>
<div class="">#17 0x00002ae3a43eede1 in ?? ()</div>
<div class="">#18 0x00002ae42705f878 in ?? ()</div>
<div class="">#19 0x00002ae40ad334e0 in ?? ()</div>
<div class="">#20 0x00002ae42705f8e0 in ?? ()</div>
<div class="">#21 0x00002ae40ad33c68 in ?? ()</div>
<div class="">#22 0x0000000000000000 in ?? ()</div>
</div>
</div>
</div>
</blockquote>
<br>
Thanks,<br>
Serguei<br>
<br>
<blockquote type="cite"
cite="mid:6FF8CDAA-B83B-4BFC-8352-***@oracle.com">
<div dir="auto" class="">
<div dir="auto" class="">
<div class=""> </div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<div dir="auto" class="">
<div dir="auto" class="">
<div class=""><br class="">
</div>
<div class="">Leonid</div>
<div class=""><br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Oct 9, 2018, at 4:52 PM, <a
href="mailto:***@oracle.com" class=""
moz-do-not-send="true">***@oracle.com</a>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">Hi Leonid,<br class="">
<br class="">
There is an existing bug:<br class="">
<a
href="https://bugs.openjdk.java.net/browse/JDK-8043571"
class="" moz-do-not-send="true">https://bugs.openjdk.java.net/browse/JDK-8043571</a><br
class="">
<br class="">
Thanks,<br class="">
Serguei<br class="">
<br class="">
<br class="">
On 10/9/18 16:11, Leonid Mesnik wrote:<br class="">
<blockquote type="cite" class="">Hi<br class="">
<br class="">
During fixing kitchensink I get<br class="">
assert(_cur_stack_depth == count_frames()) failed:
cur_stack_depth out of sync<br class="">
<br class="">
Do you know if i might be bug in my jvmti agent?<br
class="">
<br class="">
Leonid<br class="">
<br class="">
<br class="">
#<br class="">
# A fatal error has been detected by the Java
Runtime Environment:<br class="">
#<br class="">
# Internal Error
(/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiThreadState.cpp:277),
pid=13926, tid=13962<br class="">
# assert(_cur_stack_depth == count_frames())
failed: cur_stack_depth out of sync<br class="">
#<br class="">
# JRE version: Java(TM) SE Runtime Environment
(12.0) (fastdebug build
12-internal+0-2018-10-08-2342517.lmesnik.hs-bigapps)<br
class="">
# Java VM: Java HotSpot(TM) 64-Bit Server VM
(fastdebug
12-internal+0-2018-10-08-2342517.lmesnik.hs-bigapps,
mixed mode, tiered, compressed oops, g1 gc,
linux-amd64)<br class="">
# Core dump will be written. Default location:
Core dumps may be processed with
"/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t e
%P %I %h" (or dumping to
/scratch/lmesnik/ws/hs-bigapps/build/linux-x64/test-support/jtreg_closed_test_hotspot_jtreg_applications_kitchensink_KitchensinkSanity_java/scratch/0/core.13926)<br
class="">
#<br class="">
# If you would like to submit a bug report, please
visit:<br class="">
# <a
href="http://bugreport.java.com/bugreport/crash.jsp"
class="" moz-do-not-send="true">http://bugreport.java.com/bugreport/crash.jsp</a><br
class="">
#<br class="">
<br class="">
--------------- S U M M A R Y ------------<br
class="">
<br class="">
Command Line: -XX:MaxRAMPercentage=2
-XX:MaxRAMPercentage=50
-XX:+CrashOnOutOfMemoryError
-Djava.net.preferIPv6Addresses=false
-XX:-PrintVMOptions -XX:+DisplayVMOutputToStderr
-XX:+UsePerfData
-Xlog:gc*,gc+heap=debug:gc.log:uptime,timemillis,level,tags
-XX:+DisableExplicitGC -XX:+PrintFlagsFinal
-XX:+StartAttachListener
-XX:NativeMemoryTracking=detail
-XX:+FlightRecorder
--add-exports=java.base/java.lang=ALL-UNNAMED
--add-opens=java.base/java.lang=ALL-UNNAMED
--add-exports=java.xml/com.sun.org.apache.xerces.internal.parsers=ALL-UNNAMED
--add-exports=java.xml/com.sun.org.apache.xerces.internal.util=ALL-UNNAMED
-Djava.io.tmpdir=/scratch/lmesnik/ws/hs-bigapps/build/linux-x64/test-support/jtreg_closed_test_hotspot_jtreg_applications_kitchensink_KitchensinkSanity_java/scratch/0/java.io.tmpdir
-Duser.home=/scratch/lmesnik/ws/hs-bigapps/build/linux-x64/test-support/jtreg_closed_test_hotspot_jtreg_applications_kitchensink_KitchensinkSanity_java/scratch/0/user.home
-agentpath:/scratch/lmesnik/ws/hs-bigapps/build/linux-x64/images/test/hotspot/jtreg/native/libJvmtiStressModule.so
applications.kitchensink.process.stress.Main
/scratch/lmesnik/ws/hs-bigapps/build/linux-x64/test-support/jtreg_closed_test_hotspot_jtreg_applications_kitchensink_KitchensinkSanity_java/scratch/0/kitchensink.final.properties<br
class="">
<br class="">
Host: <a href="http://scaaa118.us.oracle.com"
class="" moz-do-not-send="true">scaaa118.us.oracle.com</a>,
Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz, 32
cores, 235G, Oracle Linux Server release 7.3<br
class="">
Time: Tue Oct 9 16:06:07 2018 PDT elapsed time:
31 seconds (0d 0h 0m 31s)<br class="">
<br class="">
--------------- T H R E A D ---------------<br
class="">
<br class="">
Current thread (0x00002af3dc6ac800): VMThread "VM
Thread" [stack:
0x00002af44f10a000,0x00002af44f20a000] [id=13962]
_threads_hazard_ptr=0x00002af4ac090eb0,
_nested_threads_hazard_ptr_cnt=0<br class="">
<br class="">
Stack: [0x00002af44f10a000,0x00002af44f20a000],
sp=0x00002af44f208720, free space=1017k<br
class="">
Native frames: (J=compiled Java code, A=aot
compiled Java code, j=interpreted, Vv=VM code,
C=native code)<br class="">
V [libjvm.so+0x18c4923]
VMError::report_and_die(int, char const*, char
const*, __va_list_tag*, Thread*, unsigned char*,
void*, void*, char const*, int, unsigned
long)+0x2c3<br class="">
V [libjvm.so+0x18c56ef]
VMError::report_and_die(Thread*, void*, char
const*, int, char const*, char const*,
__va_list_tag*)+0x2f<br class="">
V [libjvm.so+0xb55aa0] report_vm_error(char
const*, int, char const*, char const*, ...)+0x100<br
class="">
V [libjvm.so+0x11f2cfe]
JvmtiThreadState::cur_stack_depth()+0x14e<br
class="">
V [libjvm.so+0x11f3257]
JvmtiThreadState::update_for_pop_top_frame()+0x27<br
class="">
V [libjvm.so+0x119af99]
VM_UpdateForPopTopFrame::doit()+0xb9<br class="">
V [libjvm.so+0x1908982]
VM_Operation::evaluate()+0x132<br class="">
V [libjvm.so+0x19040be]
VMThread::evaluate_operation(VM_Operation*)
[clone .constprop.51]+0x18e<br class="">
V [libjvm.so+0x1904960] VMThread::loop()+0x4c0<br
class="">
V [libjvm.so+0x1904f53] VMThread::run()+0xd3<br
class="">
V [libjvm.so+0x14e8300]
thread_native_entry(Thread*)+0x100<br class="">
<br class="">
VM_Operation (0x00002af4d8502910):
UpdateForPopTopFrame, mode: safepoint, requested
by thread 0x00002af4dc008800<br class="">
</blockquote>
<br class="">
<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</blockquote>
<br>
</body>
</html>