Discussion:
crash in JvmtiExport::post_compiled_method_load
nezih yigitbasi
2018-05-10 00:21:02 UTC
Permalink
Thanks everyone for their input. Here is the full list of JVM args that
were used:
https://gist.github.com/nezihyigitbasi/2cc666dff55c663dbfc157e789a3a6f7

Nezih
Hi Nezih,
https://bugs.openjdk.java.net/browse/JDK-8202883
crash in the JvmtiExport::post_compiled_method_load
Please, feel free to add any details when you have them.
They are very important to reproduce the crash.
For instance, what GC was used in scenario with this crash?
Thank you for reporting the issue!
Serguei
Hi Nezih,
Crashing in the JvmtiExport::post_compiled_method_load function
does not prove itself that it is a VM issue.
And there are mo specific details in the log that help to understand this.
Filing a bug with this log is almost useless as it would not help to get
to the root cause.
Also, there is still a pretty big chance the real problem is in the agent.
My advice is to get as more details about this crash as possible.
The best you can do is to provide a stand alone test case.
Even if the issue is in the agent it will help to figure it out.
Thanks,
Serguei
Hi Serguei,
We don't have a repro unfortunately. We have seen this only once so far in
our production environment.
Do you think this can be an agent issue? I thought like if it was an agent
issue the stack trace would show it. The stack trace, however, ends in the
VM code -- the crash happens in the "JvmtiExport::post_compiled_method_load"
method.
Thanks,
Nezih
Hi Nezih,
You error file with log does not help much.
Could you, please, reproduce this issue with the debug or fastdebug build?
Also, a standalone test case is better to provide to
increase chances the issue to be investigated and fixed.
In general, a JVMTI agent has to be written very carefully as it can
easily cause the VM to crash.
So, it makes sense to look at the agent implementation first to make sure
it does not do anything wrong.
It is what we normally do first and why a test case that demonstrates the
problem is needed.
Thanks,
Serguei
Hi,
First of all, sorry for bringing up a crash issue in this dev mailing
list, but the crash report submission page (https://bugreport.java.com/bu
greport/crash.jsp) doesn't list Java 9 in the release drop down, so I
couldn't report it there.
We recently got a crash with Java 9.0.1+11 with an interesting stack
ending at "JvmtiExport::post_compiled_method_load()" (entire error file
is here
<https://gist.github.com/nezihyigitbasi/52a58698cc9acfcab21c69d00bd0cef2>).
A google search didn't end up with much info, so I just wanted to check
with this mailing list to see whether anyone has any ideas to investigate
this further.
Thanks,
Nezih
JC Beyler
2018-05-09 17:36:51 UTC
Permalink
Not that this helps but I was trying to see if I could get a repro for this
and understand a bit more how things work. I failed to do so but here was
what I did in case it helps get closer to an actual repro. I did a code
that had a lot of methods and reduced the various knobs to get to a case
where the JVM would keep loading/unloading methods:

final class TestCompileEvent {
public long cnt;
public static void main(String[] args) {
TestCompileEvent elem = new TestAOT();
for (int i = 0; i < 20000; i++) {
elem.foo0();
elem.foo1();
....
elem.foo3999();
}
System.out.println(elem.cnt);
}

where each foo method something like:

private void foo2() {
for (int i = 0; i < 2; i++) {
cnt += 2 + i / (2 + i + 1) + (2 * (i + 3.14));
}
}

The knobs I used were:
-XX:InitialCodeCacheSize=4k -XX:ReservedCodeCacheSize=2000k
-XX:CodeCacheMinimumUseSpace=1k -XX:CodeCacheExpansionSize=32k
-XX:-SegmentedCodeCache -XX:CompileThreshold=1 -XX:-Inline

This with a JVMTI agent logging compiled method load/unload seemed to keep
the code in a state of continuous loading/unloading of the various
methods. But I did not see the bug show up. I wonder if I need to add some
allocations in the methods and reduce the Xms/Xmx to get the GC to start
freaking out a bit.

Anyway, that was my analysis of this. Perhaps it helps?
Jc
Okay.
More details on this.
/ Locks an nmethod so its code will not get removed and it will not
// be made into a zombie, even if it is a not_entrant method. After the
// nmethod becomes a zombie, if CompiledMethodUnload event processing
// needs to be done, then lock_nmethod() is used directly to keep the
// generated code from being reused too early.
class nmethodLocker : public StackObj {
CompiledMethod* _nm;
// note: nm can be NULL
// Only JvmtiDeferredEvent::compiled_method_unload_event()
// should pass zombie_ok == true.
static void lock_nmethod(CompiledMethod* nm, bool zombie_ok = false);
static void unlock_nmethod(CompiledMethod* nm); // (ditto)
nmethodLocker(address pc); // derive nm from pc
nmethodLocker(nmethod *nm) { _nm = nm; lock_nmethod(_nm); }
nmethodLocker(CompiledMethod *nm) {
_nm = nm;
lock(_nm);
}
static void lock(CompiledMethod* method) {
if (method == NULL) return;
lock_nmethod(method);
}
static void unlock(CompiledMethod* method) {
if (method == NULL) return;
unlock_nmethod(method);
}
nmethodLocker() { _nm = NULL; }
~nmethodLocker() {
unlock(_nm);
}
CompiledMethod* code() { return _nm; }
void set_code(CompiledMethod* new_nm) {
unlock(_nm); // note: This works even if _nm==new_nm.
_nm = new_nm;
lock(_nm);
}
};
// QQQ might we make this work from a frame??
nmethodLocker::nmethodLocker(address pc) {
CodeBlob* cb = CodeCache::find_blob(pc);
guarantee(cb != NULL && cb->is_compiled(), "bad pc for a nmethod
found");
_nm = cb->as_compiled_method();
lock_nmethod(_nm);
}
// Only JvmtiDeferredEvent::compiled_method_unload_event()
// should pass zombie_ok == true.
void nmethodLocker::lock_nmethod(CompiledMethod* cm, bool zombie_ok) {
if (cm == NULL) return;
if (cm->is_aot()) return; // FIXME: Revisit once _lock_count is
added to aot_method
nmethod* nm = cm->as_nmethod();
Atomic::inc(&nm->_lock_count);
assert(zombie_ok || !nm->is_zombie(), "cannot lock a zombie method");
}
void nmethodLocker::unlock_nmethod(CompiledMethod* cm) {
if (cm == NULL) return;
if (cm->is_aot()) return; // FIXME: Revisit once _lock_count is
added to aot_method
nmethod* nm = cm->as_nmethod();
Atomic::dec(&nm->_lock_count);
assert(nm->_lock_count >= 0, "unmatched nmethod lock/unlock");
}
bool is_locked_by_vm() const { return _lock_count
0; }
The function is_locked_by_vm() must be used to prevent unloading an
nmethod in use.
NMethodSweeper::process_compiled_method()
called from NMethodSweeper::sweep_code_cache()
called from NMethodSweeper::possibly_sweep()
called from NMethodSweeper::sweeper_loop()
called from sweeper_thread_entry()
called from
CodeCacheSweeperThread::CodeCacheSweeperThread()
called from CompileBroker::make_thread()
called from
CompileBroker::init_compiler_sweeper_threads()
NMethodSweeper::possibly_flush()
called from NMethodSweeper::process_compiled_method()
From the other hand, it seems, the nmethod::make_unloaded() cleans up
// If _method is already NULL the Method* is about to be unloaded,
// so we don't have to break the cycle. Note that it is possible to
// have the Method* live here, in case we unload the nmethod because
// it is pointing to some oop (other than the Method*) being unloaded.
if (_method != NULL) {
// OSR methods point to the Method*, but the Method* does not
// point back!
if (_method->code() == this) {
_method->clear_code(); // Break a cycle
}
_method = NULL; // Clear the method of this dead
nmethod <== !!
}
but it does not care about the nmethod locking api above.
// If this oop is not live, the nmethod can be unloaded.
bool nmethod::can_unload(BoolObjectClosure* is_alive, oop* root, bool
unloading_occurred) {
assert(root != NULL, "just checking");
oop obj = *root;
if (obj == NULL || is_alive->do_object_b(obj)) {
return false;
}
// If ScavengeRootsInCode is true, an nmethod might be unloaded
// simply because one of its constant oops has gone dead.
// No actual classes need to be unloaded in order for this to occur.
assert(unloading_occurred || ScavengeRootsInCode, "Inconsistency in
unloading");
make_unloaded(is_alive,
obj); <== !!
return true;
}
bool nmethod::unload_if_dead_at(RelocIterator* iter_at_oop,
BoolObjectClosure *is_alive, bool unloading_occurred) {
assert(iter_at_oop->type() == relocInfo::oop_type, "Wrong relocation
type");
oop_Relocation* r = iter_at_oop->oop_reloc();
// Traverse those oops directly embedded in the code.
// Other oops (oop_index>0) are seen as part of scopes_oops.
assert(1 == (r->oop_is_immediate()) +
(r->oop_addr() >= oops_begin() && r->oop_addr() < oops_end()),
"oop must be found in exactly one place");
if (r->oop_is_immediate() && r->oop_value() != NULL) {
// Unload this nmethod if the oop is dead.
if (can_unload(is_alive, r->oop_addr(), unloading_occurred)) {
return true;;
}
}
return false;
}
bool nmethod::do_unloading_scopes(BoolObjectClosure* is_alive, bool
unloading_occurred) {
// Scopes
for (oop* p = oops_begin(); p < oops_end(); p++) {
if (*p == Universe::non_oop_word()) continue; // skip non-oops
if (can_unload(is_alive, p, unloading_occurred)) {
return true;
}
}
return false;
}
#if INCLUDE_JVMCI
bool nmethod::do_unloading_jvmci(BoolObjectClosure* is_alive, bool
unloading_occurred) {
bool is_unloaded = false;
// Follow JVMCI method
BarrierSet* bs = Universe::heap()->barrier_set();
if (_jvmci_installed_code != NULL) {
if (_jvmci_installed_code->is_a(HotSpotNmethod::klass()) &&
HotSpotNmethod::isDefault(_jvmci_installed_code)) {
if (!is_alive->do_object_b(_jvmci_installed_code)) {
clear_jvmci_installed_code();
}
} else {
if (can_unload(is_alive, (oop*)&_jvmci_installed_code,
unloading_occurred)) {
return true;
}
}
}
if (_speculation_log != NULL) {
if (!is_alive->do_object_b(_speculation_log)) {
bs->write_ref_nmethod_pre(&_speculation_log, this);
_speculation_log = NULL;
bs->write_ref_nmethod_post(&_speculation_log, this);
}
}
return is_unloaded;
}
#endif
The only caller of the the unload_if_dead_at() and do_unloading_scopes()
bool nmethod::do_unloading_oops(address low_boundary, BoolObjectClosure*
is_alive, bool unloading_occurred) {
// Compiled code
{
RelocIterator iter(this, low_boundary);
while (iter.next()) {
if (iter.type() == relocInfo::oop_type) {
if (unload_if_dead_at(&iter, is_alive, unloading_occurred)) {
return true;
}
}
}
}
return do_unloading_scopes(is_alive, unloading_occurred);
}
Let's skip AOT part.
Both
nmethod::do_unloading_oops() and
nmethod::do_unloading_jvmci()
called from CompiledMethod::do_unloading()
and CompiledMethod::do_unloading_parallel().
CompiledMethod::do_unloading_parallel()
called from G1CodeCacheUnloadingTask::clean_nmethod()
called from G1CodeCacheUnloadingTask::work_first_pass()
and G1CodeCacheUnloadingTask::work_second_pass()
called from G1ParallelCleaningTask::G1ParallelCleaningTask()
called from G1CollectedHeap::parallel_cleaning()
called from
G1ConcurrentMark::weakRefsWorkParallelPart()
called from G1ConcurrentMark::weakRefsWork()
called from
G1ConcurrentMark::checkpointRootsFinal()
. . .
Now, I'm lost...
It is hard to find out how all these sub-call-tree should transitively
call the NMethodSweeper::process_compiled_method() function which uses
the function is_locked_by_vm() to synchronize with
post_compiled_method_load().
Just some initial (very stupid) analysis... :-)
Thanks,
Serguei
Hi Ioi and David,
Ioi, thank you for the disassembly.
It helps for sure!
Below are 3 call sites for the
void ciEnv::register_method(ciMethod* target, . . .) {
. . .
if (nm != NULL) {
// JVMTI -- compiled method notification (must be done outside lock)
nm->post_compiled_method_load_event(); <== !!
} else {
. . .
}
}
JVMCIEnv::CodeInstallResult JVMCIEnv::register_method(. . .) {
. . .
// JVMTI -- compiled method notification (must be done outside lock)
if (nm != NULL) {
nm->post_compiled_method_load_event(); <== !!
. . .
}
return result;
}
void AdapterHandlerLibrary::create_native_wrapper(const methodHandle&
method) {
. . .
if (nm != NULL) {
. . .
nm->post_compiled_method_load_event(); <== !!
}
}
The nmethod::post_compiled_method_load_event() creates a
JvmtiDeferredEvent
compiled_load_event entry and enqueues it for the ServiceThread
processing.
void nmethod::post_compiled_method_load_event() {
Method* moop = method();
HOTSPOT_COMPILED_METHOD_LOAD(
(char *) moop->klass_name()->bytes(),
moop->klass_name()->utf8_length(),
(char *) moop->name()->bytes(),
moop->name()->utf8_length(),
(char *) moop->signature()->bytes(),
moop->signature()->utf8_length(),
insts_begin(), insts_size());
if (JvmtiExport::should_post_compiled_method_load() ||
JvmtiExport::should_post_compiled_method_unload()) {
get_and_cache_jmethod_id();
}
if (JvmtiExport::should_post_compiled_method_load()) {
// Let the Service thread (which is a real Java thread) post the
event
MutexLockerEx ml(Service_lock, Mutex::_no_safepoint_check_flag);
JvmtiDeferredEventQueue::enqueue(
JvmtiDeferredEvent::compiled_method_load_event(this)); <== !!
}
}
The vmtiDeferredEvent::compiled_method_load_event() below uses
nmethodLocker::lock_nmethod(nm) to protect the nmethod* nm
JvmtiDeferredEvent JvmtiDeferredEvent::compiled_method_load_event(
nmethod* nm) {
JvmtiDeferredEvent event =
JvmtiDeferredEvent(TYPE_COMPILED_METHOD_LOAD);
event._event_data.compiled_method_load =
nm; <== !!
// Keep the nmethod alive until the ServiceThread can process
// this deferred event.
nmethodLocker::lock_nmethod(nm); <== !!
return event;
}
void JvmtiDeferredEventQueue::enqueue(const JvmtiDeferredEvent& event) {
assert(Service_lock->owned_by_self(), "Must own Service_lock");
process_pending_events();
// Events get added to the end of the queue (and are pulled off the
front).
QueueNode* node = new QueueNode(event);
if (_queue_tail == NULL) {
_queue_tail = _queue_head = node;
} else {
assert(_queue_tail->next() == NULL, "Must be the last element in
the list");
_queue_tail->set_next(node);
_queue_tail = node;
}
Service_lock->notify_all();
assert((_queue_head == NULL) == (_queue_tail == NULL),
"Inconsistent queue markers");
}
JvmtiDeferredEvent JvmtiDeferredEventQueue::dequeue() {
assert(Service_lock->owned_by_self(), "Must own Service_lock");
process_pending_events();
assert(_queue_head != NULL, "Nothing to dequeue");
if (_queue_head == NULL) {
// Just in case this happens in product; it shouldn't but let's
not crash
return JvmtiDeferredEvent();
}
QueueNode* node = _queue_head;
_queue_head = _queue_head->next();
if (_queue_head == NULL) {
_queue_tail = NULL;
}
assert((_queue_head == NULL) == (_queue_tail == NULL),
"Inconsistent queue markers");
JvmtiDeferredEvent event = node->event();
delete node;
return event;
}
The ServiceThread (different thread which is a JavaThread)
takes the event from the JvmtiDeferredEventQueue queue
and posts it.
void ServiceThread::service_thread_entry(JavaThread* jt, TRAPS) {
. . .
if (has_jvmti_events) {
jvmti_event = JvmtiDeferredEventQueue::dequeue();
<== !!
}
}
if (has_jvmti_events) {
jvmti_event.post(); <== !!
}
. . .
}
JvmtiExport::post_compiled_method_load(nm);
and only then unlocks the nmethod nm with
nmethodLocker::unlock_nmethod(nm).
void JvmtiDeferredEvent::post() {
assert(ServiceThread::is_service_thread(Thread::current()),
"Service thread must post enqueued events");
switch(_type) {
case TYPE_COMPILED_METHOD_LOAD: {
nmethod* nm = _event_data.compiled_method_load;
JvmtiExport::post_compiled_method_load(nm); <== !!
// done with the deferred event so unlock the nmethod
nmethodLocker::unlock_nmethod(nm); <== !!
break;
}
. . .
ShouldNotReachHere();
}
}
The printed disassembly shows where we observe the crash in the
void JvmtiExport::post_compiled_method_load(nmethod *nm) {
if (JvmtiEnv::get_phase() < JVMTI_PHASE_PRIMORDIAL) {
return;
}
JavaThread* thread = JavaThread::current();
EVT_TRIG_TRACE(JVMTI_EVENT_COMPILED_METHOD_LOAD,
("[%s] method compile load event triggered",
JvmtiTrace::safe_get_thread_name(thread)));
JvmtiEnvIterator it;
for (JvmtiEnv* env = it.first(); env != NULL; env = it.next(env)) {
if (env->is_enabled(JVMTI_EVENT_COMPILED_METHOD_LOAD)) {
if (env->phase() == JVMTI_PHASE_PRIMORDIAL) {
continue;
}
EVT_TRACE(JVMTI_EVENT_COMPILED_METHOD_LOAD,
("[%s] class compile method load event sent %s.%s ",
JvmtiTrace::safe_get_thread_name(thread),
nm->method()->klass_name()->as_C_string(),
nm->method()->name()->as_C_string()));
ResourceMark rm(thread);
HandleMark hm(thread);
// Add inlining information
jvmtiCompiledMethodLoadInlineRecord* inlinerecord =
create_inline_record(nm);
// Pass inlining information through the void pointer
JvmtiCompiledMethodLoadEventMark jem(thread, nm,
inlinerecord); <== !!
JvmtiJavaThreadEventTransition jet(thread);
jvmtiEventCompiledMethodLoad callback =
env->callbacks()->CompiledMethodLoad;
if (callback != NULL) {
(*callback)(env->jvmti_external(), jem.jni_methodID(),
jem.code_size(), jem.code_data(), jem.map_length(),
jem.map(), jem.compile_info());
}
}
}
}
class JvmtiCompiledMethodLoadEventMark : public JvmtiMethodEventMark {
. . .
JvmtiCompiledMethodLoadEventMark(JavaThread *thread, nmethod *nm,
void* compile_info_ptr = NULL)
: JvmtiMethodEventMark(thread,methodHandle(thread,
nm->method())) { <== !!
. . .
}
. . .
}
class JvmtiMethodEventMark : public JvmtiThreadEventMark {
. . .
JvmtiThreadEventMark(thread),
_mid(to_jmethodID(method))
{}; <== line 234 !!
jmethodID jni_methodID() { return _mid; }
};
All the fragments above show the code path correctly operates with
nmethod
by protecting it with the nmethodLocker::lock_nmethod(nm) until the event
has been posted.
- the above is correct but some other code abuses the convention and
updates the nmethod without grabbing the nmethod lock
- the above operations with the nmethod is incorrect (miss some steps)
- some other reason (I doubt it is the case)
One conclusion is that it is unlikely an issue in the JVMTI agent as I
suspected initially.
It would still be valuable to have more details about a scenario that
reproduced this issue.
It is probably enough information to file a bug.
Any opinions on the above?
Any ideas about what can be broken?
Thanks,
Serguei
Serguei & Nezih,
I've looked at the crash log and it seems like nm->method() is NULL,
which caused the crash.
I am not familiar with the JVMTI internals or nmethods, so I am just
guessing here -- is it legal for nm->method() to be NULL? Or, is it
possible that someone has cleared nm->method() before it's processed
by post_compiled_method_load?
http://hg.openjdk.java.net/jdk/jdk/file/ae0ebd3cf949/src/hotspot/share/runtime/sweeper.hpp#l42
for the nmethod lifecycle. Given this is actually a deferred event, I
too wonder if the nmethod may have been made not-entrant or even
zombie, since the event was posted.
David
-----
=================================
Native frames: (J=compiled Java code, A=aot compiled Java code,
j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x99e365]
JvmtiExport::post_compiled_method_load(nmethod*)+0x275
V [libjvm.so+0x9a6614] JvmtiDeferredEvent::post()+0x44
V [libjvm.so+0xc2c507]
ServiceThread::service_thread_entry(JavaThread*, Thread*)+0x337
V [libjvm.so+0xceee58] JavaThread::thread_main_inner()+0xd8
V [libjvm.so+0xb6de12] thread_native_entry(Thread*)+0xf2
C [libpthread.so.0+0x7e25] start_thread+0xc5
...
R14=0x0000000000000000 is an unknown value
=================================
Here's a disassembly of JvmtiExport::post_compiled_method_load from
JDK 9.0.1
http://hg.openjdk.java.net/jdk9/jdk9/hotspot/file/b756e7a2ec33/src/share/vm/prims/jvmtiExport.cpp
2051 HandleMark hm(thread);
0x00007ffff67dd239 <+329>: mov %rax,-0x140(%rbp)
0x00007ffff67dd240 <+336>: callq 0x7ffff66149d0
<HandleMark::initialize(Thread*)>
2054 jvmtiCompiledMethodLoadInlineRecord* inlinerecord =
create_inline_record(nm);
0x00007ffff67dd245 <+341>: mov %r12,%rdi
0x00007ffff67dd248 <+344>: callq 0x7ffff67d4090
<create_inline_record(nmethod*)>
2055 // Pass inlining information through the void pointer
2056 JvmtiCompiledMethodLoadEventMark jem(thread, nm,
inlinerecord);
0x00007ffff67dd24d <+349>: mov %rax,-0x1a8(%rbp)
0x00007ffff67dd254 <+356>: mov 0x70(%r12),%rax
0x00007ffff67dd259 <+361>: mov %rbx,-0x178(%rbp)
0x00007ffff67dd260 <+368>: test %rax,%rax
0x00007ffff67dd263 <+371>: mov %rax,-0x180(%rbp)
0x00007ffff67dd26a <+378>: je 0x7ffff67dd282
<JvmtiExport::post_compiled_method_load(nmethod*)+402>
0x00007ffff67dd26c <+380>: mov %rax,-0xa0(%rbp)
0x00007ffff67dd273 <+387>: mov 0x160(%rbx),%rdi
0x00007ffff67dd27a <+394>: mov %r13,%rsi
0x00007ffff67dd27d <+397>: callq 0x7ffff671d5f0
<GrowableArray<Metadata*>::append(Metadata* const&)>
0x00007ffff67dd282 <+402>: lea 0x218(%rbx),%rax
0x00007ffff67dd289 <+409>: mov %rbx,-0xf0(%rbp)
0x00007ffff67dd290 <+416>: movl $0x0,-0xe0(%rbp)
0x00007ffff67dd29a <+426>: mov %rax,-0xe8(%rbp)
0x00007ffff67dd2a1 <+433>: mov 0x3c8(%rbx),%rax
0x00007ffff67dd2a8 <+440>: test %rax,%rax
0x00007ffff67dd2ab <+443>: je 0x7ffff67dd2b6
<JvmtiExport::post_compiled_method_load(nmethod*)+454>
0x00007ffff67dd2ad <+445>: mov 0x10(%rax),%eax
0x00007ffff67dd2b0 <+448>: mov %eax,-0xe0(%rbp)
0x00007ffff67dd2b6 <+454>: mov 0x38(%rbx),%r14
0x00007ffff67dd2ba <+458>: mov %rbx,%rdi
0x00007ffff67dd2bd <+461>: callq 0x7ffff6707d50
<JNIHandleBlock::allocate_block(Thread*)>
0x00007ffff67dd2c2 <+466>: lea 0x1f8(%rbx),%rdi
0x00007ffff67dd2c9 <+473>: mov %rbx,%rsi
0x00007ffff67dd2cc <+476>: mov %r14,0x118(%rax)
0x00007ffff67dd2d3 <+483>: mov %rax,0x38(%rbx)
0x00007ffff67dd2d7 <+487>: callq 0x7ffff6559410
<JavaFrameAnchor::make_walkable(JavaThread*)>
0x00007ffff67dd2dc <+492>: mov 0x1f0(%rbx),%rsi
0x00007ffff67dd2e3 <+499>: mov -0xf0(%rbp),%rdi
0x00007ffff67dd2ea <+506>: callq 0x7ffff6708520
<JNIHandles::make_local(Thread*, oopDesc*)>
0x00007ffff67dd2ef <+511>: mov -0x1a0(%rbp),%rsi
0x00007ffff67dd2f6 <+518>: mov %rax,-0xd8(%rbp)
0x00007ffff67dd2fd <+525>: lea -0x170(%rbp),%rax
0x00007ffff67dd304 <+532>: mov %rax,%rdi
0x00007ffff67dd307 <+535>: mov %rax,-0x1b0(%rbp)
// JvmtiThreadEventMark(thread),
234 _mid(to_jmethodID(method)) {}; <-- this line
0x00007ffff67dd30e <+542>: callq 0x7ffff66140a0
<methodHandle::methodHandle(methodHandle const&)>
0x00007ffff67dd313 <+547>: mov -0x170(%rbp),%r14
0x00007ffff67dd31a <+554>: movq $0x0,-0x98(%rbp)
0x00007ffff67dd325 <+565>: test %r14,%r14
0x00007ffff67dd328 <+568>: mov %r14,-0xa0(%rbp)
0x00007ffff67dd32f <+575>: je 0x7ffff67dd365
<JvmtiExport::post_compiled_method_load(nmethod*)+629>
0x00007ffff67dd331 <+577>: data16 lea
0x946ca7(%rip),%rdi # 0x7ffff7123fe0
0x00007ffff67dd339 <+585>: data16 data16 callq 0x7ffff60b11d0
0x00007ffff67dd341 <+593>: lea -0x188(%rbp),%rsi
0x00007ffff67dd348 <+600>: mov %r14,-0x188(%rbp)
0x00007ffff67dd34f <+607>: mov (%rax),%rax
0x00007ffff67dd352 <+610>: mov %rax,-0x98(%rbp)
0x00007ffff67dd359 <+617>: mov 0x160(%rax),%rdi
0x00007ffff67dd360 <+624>: callq 0x7ffff671d5f0
<GrowableArray<Metadata*>::append(Metadata* const&)>
CRASH %r14 == 0x0
198 jmethodID to_jmethodID(methodHandle method) { return
method->jmethod_id(); }
0x00007ffff67dd365 <+629>: mov 0x8(%r14),%rax
Hope this helps
- Ioi
Hi Nezih,
Crashing in the JvmtiExport::post_compiled_method_load function
does not prove itself that it is a VM issue.
And there are mo specific details in the log that help to
understand this.
Filing a bug with this log is almost useless as it would not help
to get to the root cause.
Also, there is still a pretty big chance the real problem is in the
agent.
My advice is to get as more details about this crash as possible.
The best you can do is to provide a stand alone test case.
Even if the issue is in the agent it will help to figure it out.
Thanks,
Serguei
Hi Serguei,
We don't have a repro unfortunately. We have seen this only once
so far in our production environment.
Do you think this can be an agent issue? I thought like if it was
an agent issue the stack trace would show it. The stack trace,
however, ends in the VM code -- the crash happens in the
"JvmtiExport::post_compiled_method_load" method.
Thanks,
Nezih
Hi Nezih,
You error file with log does not help much.
Could you, please, reproduce this issue with the debug or
fastdebug build?
Also, a standalone test case is better to provide to
increase chances the issue to be investigated and fixed.
In general, a JVMTI agent has to be written very carefully as it
can easily cause the VM to crash.
So, it makes sense to look at the agent implementation first to
make sure it does not do anything wrong.
It is what we normally do first and why a test case that
demonstrates the problem is needed.
Thanks,
Serguei
Hi,
First of all, sorry for bringing up a crash issue in this dev
mailing list, but the crash report submission page
(https://bugreport.java.com/bugreport/crash.jsp
<https://bugreport.java.com/bugreport/crash.jsp>) doesn't list
Java 9 in the release drop down, so I couldn't report it there.
We recently got a crash with Java 9.0.1+11 with an interesting
stack ending at "JvmtiExport::post_compiled_method_load()"
(entire error file is here
<
https://gist.github.com/nezihyigitbasi/52a58698cc9acfcab21c69d00bd0cef2>).
A google search didn't end up with much info, so I just wanted
to check with this mailing list to see whether anyone has any
ideas to investigate this further.
Thanks,
Nezih
s***@oracle.com
2018-05-10 00:45:19 UTC
Permalink
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">I've added the JVM args list to the bug
report description.<br>
<br>
Thanks,<br>
Serguei<br>
<br>
<br>
On 5/9/18 17:21, nezih yigitbasi wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CALur2hUxFq86GeXixCG3-BBDqPZpCOMcwbFv9ZHK5-8Q_QCt=***@mail.gmail.com">
<meta http-equiv="Context-Type" content="text/html; charset=UTF-8">
<div dir="ltr">Thanks everyone for their input. Here is the full
list of JVM args that were used: <a
href="https://gist.github.com/nezihyigitbasi/2cc666dff55c663dbfc157e789a3a6f7"
moz-do-not-send="true">https://gist.github.com/nezihyigitbasi/2cc666dff55c663dbfc157e789a3a6f7</a>
<div><br>
</div>
<div>Nezih</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">2018-05-09 16:52 GMT-07:00 <a
href="mailto:***@oracle.com"
moz-do-not-send="true">***@oracle.com</a> <span
dir="ltr">&lt;<a href="mailto:***@oracle.com"
target="_blank" moz-do-not-send="true">***@oracle.com</a>&gt;</span>:<br>
<blockquote class="gmail_quote">
<div>
<div class="m_7401702676096723713moz-cite-prefix">Hi
Nezih,<br>
<br>
I've filed this bug:<br>
  <a class="m_7401702676096723713moz-txt-link-freetext"
href="https://bugs.openjdk.java.net/browse/JDK-8202883" target="_blank"
moz-do-not-send="true">https://bugs.openjdk.java.net/<wbr>browse/JDK-8202883</a><br>
    crash in the JvmtiExport::post_compiled_<wbr>method_load<span
class="m_7401702676096723713overlay-icon
m_7401702676096723713aui-icon
m_7401702676096723713aui-icon-small
m_7401702676096723713aui-iconfont-edit"></span><br>
<br>
Please, feel free to add any details when you have them.<br>
They are very important to reproduce the crash.<br>
For instance, what GC was used in scenario with this
crash?<br>
<br>
Thank you for reporting the issue!<span class="HOEnZb"><br>
Serguei</span>
<div>
<div class="h5"><br>
<br>
<br>
On 5/8/18 16:51, <a
class="m_7401702676096723713moz-txt-link-abbreviated"
href="mailto:***@oracle.com"
target="_blank" moz-do-not-send="true">***@oracle.com</a>
wrote:<br>
</div>
</div>
</div>
<div>
<div class="h5">
<blockquote type="cite">
<div class="m_7401702676096723713moz-cite-prefix">Hi
Nezih,<br>
<br>
Crashing in the JvmtiExport::post_compiled_<wbr>method_load
function<br>
does not prove itself that it is a VM issue.<br>
And there are mo specific details in the log that
help to understand this.<br>
Filing a bug with this log is almost useless as it
would not help to get to the root cause.<br>
Also, there is still a pretty big chance the real
problem is in the agent.<br>
<br>
My advice is to get as more details about this
crash as possible.<br>
The best you can do is to provide a stand alone
test case.<br>
Even if the issue is in the agent it will help to
figure it out.<br>
<br>
Thanks,<br>
Serguei<br>
<br>
<br>
On 5/8/18 16:38, nezih yigitbasi wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hi Serguei,
<div>We don't have a repro unfortunately. We
have seen this only once so far in our
production environment. </div>
<div><br>
<div>Do you think this can be an agent issue?
I thought like if it was an agent issue the
stack trace would show it. The stack trace,
however, ends in the VM code -- the crash
happens in the "JvmtiExport::post_compiled_<wbr>method_load"
method.</div>
</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Nezih</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">2018-05-08 16:34
GMT-07:00 <a
href="mailto:***@oracle.com"
target="_blank" moz-do-not-send="true">***@oracle.com</a>
<span dir="ltr">&lt;<a
href="mailto:***@oracle.com"
target="_blank" moz-do-not-send="true">***@oracle.com</a>&gt;</span>:<br>
<blockquote class="gmail_quote">
<div>
<div
class="m_7401702676096723713m_5351634827283183755moz-cite-prefix">Hi
Nezih,<br>
<br>
You error file with log does not help
much.<br>
Could you, please, reproduce this issue
with the debug or fastdebug build?<br>
Also, a standalone test case is better
to provide to<br>
increase chances the issue to be
investigated and fixed.<br>
<br>
In general, a JVMTI agent has to be
written very carefully as it can easily
cause the VM to crash.<br>
So, it makes sense to look at the agent
implementation first to make sure it
does not do anything wrong.<br>
It is what we normally do first and why
a test case that demonstrates the
problem is needed.<br>
<br>
Thanks,<br>
Serguei
<div>
<div class="m_7401702676096723713h5"><br>
<br>
<br>
On 5/8/18 16:00, nezih yigitbasi
wrote:<br>
</div>
</div>
</div>
<div>
<div class="m_7401702676096723713h5">
<blockquote type="cite">
<div dir="ltr">Hi,
<div><br>
</div>
<div>First of all, sorry for
bringing up a crash issue in
this dev mailing list, but the
crash report submission page (<a
href="https://bugreport.java.com/bugreport/crash.jsp" target="_blank"
moz-do-not-send="true">https://bugreport.java.com/bu<wbr>greport/crash.jsp</a>)
doesn't list Java 9 in the
release drop down, so I couldn't
report it there.</div>
<div><br>
</div>
<div>We recently got a crash with
Java 9.0.1+11 with an
interesting stack ending at
"JvmtiExport::post_compiled_me<wbr>thod_load()"
(entire error file is <a
href="https://gist.github.com/nezihyigitbasi/52a58698cc9acfcab21c69d00bd0cef2"
target="_blank"
moz-do-not-send="true">here</a>).
A google search didn't end up
with much info, so I just wanted
to check with this mailing list
to see whether anyone has any
ideas to investigate this
further.</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Nezih</div>
<div><br>
</div>
</div>
</blockquote>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</blockquote>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</body>
</html>
nezih yigitbasi
2018-10-12 22:28:27 UTC
Permalink
Hi,
We have observed the same issue again on Java 10.0+46. I just wanted to
share the new error file and the jvm args in case these provide additional
information.
hs_err file:
https://gist.github.com/nezihyigitbasi/912fc3e7f2aff933f5828f5e913dae64
jvm args:
https://gist.github.com/nezihyigitbasi/301716e9bbcb272d0c2cbadc45938a68

Thanks!
Nezih
I've added the JVM args list to the bug report description.
Thanks,
Serguei
Thanks everyone for their input. Here is the full list of JVM args that
https://gist.github.com/nezihyigitbasi/2cc666dff55c663dbfc157e789a3a6f7
Nezih
Hi Nezih,
https://bugs.openjdk.java.net/browse/JDK-8202883
crash in the JvmtiExport::post_compiled_method_load
Please, feel free to add any details when you have them.
They are very important to reproduce the crash.
For instance, what GC was used in scenario with this crash?
Thank you for reporting the issue!
Serguei
Hi Nezih,
Crashing in the JvmtiExport::post_compiled_method_load function
does not prove itself that it is a VM issue.
And there are mo specific details in the log that help to understand this.
Filing a bug with this log is almost useless as it would not help to get
to the root cause.
Also, there is still a pretty big chance the real problem is in the agent.
My advice is to get as more details about this crash as possible.
The best you can do is to provide a stand alone test case.
Even if the issue is in the agent it will help to figure it out.
Thanks,
Serguei
Hi Serguei,
We don't have a repro unfortunately. We have seen this only once so far
in our production environment.
Do you think this can be an agent issue? I thought like if it was an
agent issue the stack trace would show it. The stack trace, however, ends
in the VM code -- the crash happens in the
"JvmtiExport::post_compiled_method_load" method.
Thanks,
Nezih
Hi Nezih,
You error file with log does not help much.
Could you, please, reproduce this issue with the debug or fastdebug build?
Also, a standalone test case is better to provide to
increase chances the issue to be investigated and fixed.
In general, a JVMTI agent has to be written very carefully as it can
easily cause the VM to crash.
So, it makes sense to look at the agent implementation first to make
sure it does not do anything wrong.
It is what we normally do first and why a test case that demonstrates
the problem is needed.
Thanks,
Serguei
Hi,
First of all, sorry for bringing up a crash issue in this dev mailing
list, but the crash report submission page (
https://bugreport.java.com/bugreport/crash.jsp) doesn't list Java 9 in
the release drop down, so I couldn't report it there.
We recently got a crash with Java 9.0.1+11 with an interesting stack
ending at "JvmtiExport::post_compiled_method_load()" (entire error file is
here
<https://gist.github.com/nezihyigitbasi/52a58698cc9acfcab21c69d00bd0cef2>).
A google search didn't end up with much info, so I just wanted to check
with this mailing list to see whether anyone has any ideas to investigate
this further.
Thanks,
Nezih
s***@oracle.com
2018-10-12 23:02:18 UTC
Permalink
Hi Nezih,

I've added a comment to the bug report with your details.

Thanks!
Serguei
Post by nezih yigitbasi
Hi,
We have observed the same issue again on Java 10.0+46. I just wanted
to share the new error file and the jvm args in case these provide
additional information.
https://gist.github.com/nezihyigitbasi/912fc3e7f2aff933f5828f5e913dae64
https://gist.github.com/nezihyigitbasi/301716e9bbcb272d0c2cbadc45938a68
Thanks!
Nezih
I've added the JVM args list to the bug report description.
Thanks,
Serguei
Post by nezih yigitbasi
Thanks everyone for their input. Here is the full list of JVM
https://gist.github.com/nezihyigitbasi/2cc666dff55c663dbfc157e789a3a6f7
Nezih
Hi Nezih,
https://bugs.openjdk.java.net/browse/JDK-8202883
    crash in the JvmtiExport::post_compiled_method_load
Please, feel free to add any details when you have them.
They are very important to reproduce the crash.
For instance, what GC was used in scenario with this crash?
Thank you for reporting the issue!
Serguei
Hi Nezih,
Crashing in the JvmtiExport::post_compiled_method_load function
does not prove itself that it is a VM issue.
And there are mo specific details in the log that help to
understand this.
Filing a bug with this log is almost useless as it would not
help to get to the root cause.
Also, there is still a pretty big chance the real problem is
in the agent.
My advice is to get as more details about this crash as possible.
The best you can do is to provide a stand alone test case.
Even if the issue is in the agent it will help to figure it out.
Thanks,
Serguei
Hi Serguei,
We don't have a repro unfortunately. We have seen this only
once so far in our production environment.
Do you think this can be an agent issue? I thought like if
it was an agent issue the stack trace would show it. The
stack trace, however, ends in the VM code -- the crash
happens in the "JvmtiExport::post_compiled_method_load" method.
Thanks,
Nezih
Hi Nezih,
You error file with log does not help much.
Could you, please, reproduce this issue with the debug
or fastdebug build?
Also, a standalone test case is better to provide to
increase chances the issue to be investigated and fixed.
In general, a JVMTI agent has to be written very
carefully as it can easily cause the VM to crash.
So, it makes sense to look at the agent implementation
first to make sure it does not do anything wrong.
It is what we normally do first and why a test case
that demonstrates the problem is needed.
Thanks,
Serguei
Hi,
First of all, sorry for bringing up a crash issue in
this dev mailing list, but the crash report submission
page (https://bugreport.java.com/bugreport/crash.jsp)
doesn't list Java 9 in the release drop down, so I
couldn't report it there.
We recently got a crash with Java 9.0.1+11 with an
interesting stack ending at
"JvmtiExport::post_compiled_method_load()" (entire
error file is here
<https://gist.github.com/nezihyigitbasi/52a58698cc9acfcab21c69d00bd0cef2>).
A google search didn't end up with much info, so I
just wanted to check with this mailing list to see
whether anyone has any ideas to investigate this further.
Thanks,
Nezih
Loading...