Discussion:
What actions are allowed in an JVMTI ResourceExhausted event handler?
(too old to reply)
Thomas Stüfe
2018-11-13 20:50:21 UTC
Permalink
Hi all,

We have a client using CloudFoundry and its "jvmkill" agent. That is a
tiny JVMTI agent (see https://github.com/cloudfoundry/jvmkill) which
subscribes to the JVMTI ResourceExhausted Event. In the handler it
then does call JVMTI FollowReferences() to produce a heap histogram.

The thing is, at our client we seem to run out of Metaspace in a
compiler thread. That thread normally would swallow the Metaspace OOM
and just bailout from the compilation. But as part of the metaspace
OOME handling the ResourceExhausted event gets posted, the handler
then uses JVMTI FollowReferences() and attempts to print out the heap
histogram, then runs into a guarantee since the compiler thread cannot
call java methods.

My question is: are there any limitations about what one can do inside
a ResourceExhausted event handler?

I checked the https://docs.oracle.com/javase/8/docs/platform/jvmti/jvmti.html
documentation, but I cannot find any mentioning of limitations in that
case.

Thanks and Best Regards, Thomas
David Holmes
2018-11-14 05:32:08 UTC
Permalink
Hi Thomas,
Post by Thomas Stüfe
Hi all,
We have a client using CloudFoundry and its "jvmkill" agent. That is a
tiny JVMTI agent (see https://github.com/cloudfoundry/jvmkill) which
subscribes to the JVMTI ResourceExhausted Event. In the handler it
then does call JVMTI FollowReferences() to produce a heap histogram.
The thing is, at our client we seem to run out of Metaspace in a
compiler thread. That thread normally would swallow the Metaspace OOM
and just bailout from the compilation. But as part of the metaspace
OOME handling the ResourceExhausted event gets posted, the handler
then uses JVMTI FollowReferences() and attempts to print out the heap
histogram, then runs into a guarantee since the compiler thread cannot
call java methods.
My question is: are there any limitations about what one can do inside
a ResourceExhausted event handler?
Not specified no. But the reality of JVM TI is that you can't anticipate
every execution context and there are times when there are implicit
constraints imposed by the implementation.

In this case I think we have a mismatch between the fact we post the
event from the compiler thread, but that a compiler thread is not a true
"Java thread" and so can not execute arbitrary JNI or JVM TI code, or in
particular can not lead to executing Java code. I think we should not be
posting the event from the compiler thread in this case.

Cheers,
David
Post by Thomas Stüfe
I checked the https://docs.oracle.com/javase/8/docs/platform/jvmti/jvmti.html
documentation, but I cannot find any mentioning of limitations in that
case.
Thanks and Best Regards, Thomas
Thomas Stüfe
2018-11-14 05:37:52 UTC
Permalink
Post by David Holmes
Hi Thomas,
Post by Thomas Stüfe
Hi all,
We have a client using CloudFoundry and its "jvmkill" agent. That is a
tiny JVMTI agent (see https://github.com/cloudfoundry/jvmkill) which
subscribes to the JVMTI ResourceExhausted Event. In the handler it
then does call JVMTI FollowReferences() to produce a heap histogram.
The thing is, at our client we seem to run out of Metaspace in a
compiler thread. That thread normally would swallow the Metaspace OOM
and just bailout from the compilation. But as part of the metaspace
OOME handling the ResourceExhausted event gets posted, the handler
then uses JVMTI FollowReferences() and attempts to print out the heap
histogram, then runs into a guarantee since the compiler thread cannot
call java methods.
My question is: are there any limitations about what one can do inside
a ResourceExhausted event handler?
Not specified no. But the reality of JVM TI is that you can't anticipate
every execution context and there are times when there are implicit
constraints imposed by the implementation.
In this case I think we have a mismatch between the fact we post the
event from the compiler thread, but that a compiler thread is not a true
"Java thread" and so can not execute arbitrary JNI or JVM TI code, or in
particular can not lead to executing Java code. I think we should not be
posting the event from the compiler thread in this case.
Cheers,
David
Hi David,

Yes I thought so too. I'll prepare a fix.

Thanks, Thomas
Post by David Holmes
Post by Thomas Stüfe
I checked the
https://docs.oracle.com/javase/8/docs/platform/jvmti/jvmti.html
Post by Thomas Stüfe
documentation, but I cannot find any mentioning of limitations in that
case.
Thanks and Best Regards, Thomas
David Holmes
2018-11-14 05:58:31 UTC
Permalink
Post by David Holmes
Hi Thomas,
Post by Thomas Stüfe
Hi all,
We have a client using CloudFoundry and its "jvmkill" agent. That
is a
Post by Thomas Stüfe
tiny JVMTI agent (see https://github.com/cloudfoundry/jvmkill) which
subscribes to the JVMTI ResourceExhausted Event. In the handler it
then does call JVMTI FollowReferences() to produce a heap histogram.
The thing is, at our client we seem to run out of Metaspace in a
compiler thread. That thread normally would swallow the Metaspace OOM
and just bailout from the compilation. But as part of the metaspace
OOME handling the ResourceExhausted event gets posted, the handler
then uses JVMTI FollowReferences() and attempts to print out the heap
histogram, then runs into a guarantee since the compiler thread
cannot
Post by Thomas Stüfe
call java methods.
My question is: are there any limitations about what one can do
inside
Post by Thomas Stüfe
a ResourceExhausted event handler?
Not specified no. But the reality of JVM TI is that you can't anticipate
every execution context and there are times when there are implicit
constraints imposed by the implementation.
In this case I think we have a mismatch between the fact we post the
event from the compiler thread, but that a compiler thread is not a true
"Java thread" and so can not execute arbitrary JNI or JVM TI code, or in
particular can not lead to executing Java code. I think we should not be
posting the event from the compiler thread in this case.
Cheers,
David
Hi David,
Yes I thought so too. I'll prepare a fix.
My thought on the fix is that we need to check if
Thread::current()->can_call_java(). And that should probably be inside
the JvmtiExport::should_post_xxx implementation.

Cheers,
David
Post by David Holmes
Thanks, Thomas
Post by Thomas Stüfe
I checked the
https://docs.oracle.com/javase/8/docs/platform/jvmti/jvmti.html
Post by Thomas Stüfe
documentation, but I cannot find any mentioning of limitations in
that
Post by Thomas Stüfe
case.
Thanks and Best Regards, Thomas
Thomas Stüfe
2018-11-14 06:38:00 UTC
Permalink
Post by David Holmes
Post by David Holmes
Hi Thomas,
Post by Thomas Stüfe
Hi all,
We have a client using CloudFoundry and its "jvmkill" agent. That
is a
Post by Thomas Stüfe
tiny JVMTI agent (see https://github.com/cloudfoundry/jvmkill) which
subscribes to the JVMTI ResourceExhausted Event. In the handler it
then does call JVMTI FollowReferences() to produce a heap histogram.
The thing is, at our client we seem to run out of Metaspace in a
compiler thread. That thread normally would swallow the Metaspace OOM
and just bailout from the compilation. But as part of the metaspace
OOME handling the ResourceExhausted event gets posted, the handler
then uses JVMTI FollowReferences() and attempts to print out the heap
histogram, then runs into a guarantee since the compiler thread
cannot
Post by Thomas Stüfe
call java methods.
My question is: are there any limitations about what one can do
inside
Post by Thomas Stüfe
a ResourceExhausted event handler?
Not specified no. But the reality of JVM TI is that you can't anticipate
every execution context and there are times when there are implicit
constraints imposed by the implementation.
In this case I think we have a mismatch between the fact we post the
event from the compiler thread, but that a compiler thread is not a true
"Java thread" and so can not execute arbitrary JNI or JVM TI code, or in
particular can not lead to executing Java code. I think we should not be
posting the event from the compiler thread in this case.
Cheers,
David
Hi David,
Yes I thought so too. I'll prepare a fix.
My thought on the fix is that we need to check if
Thread::current()->can_call_java(). And that should probably be inside
the JvmtiExport::should_post_xxx implementation.
I wonder whether that may be too harsh. JVMTI agents may not
necessarily call into java as reaction to ResourceExhausted. I would
have limited this to !CompilerThread, and only in Metaspace.

Also, looking at CompilerThread::can_call_java(), I see that we return
true for jvmci compilers. Still do we want to post it there?

But I am not sure. What do you think?
Post by David Holmes
Cheers,
David
Post by David Holmes
Thanks, Thomas
Post by Thomas Stüfe
I checked the
https://docs.oracle.com/javase/8/docs/platform/jvmti/jvmti.html
Post by Thomas Stüfe
documentation, but I cannot find any mentioning of limitations in
that
Post by Thomas Stüfe
case.
Thanks and Best Regards, Thomas
David Holmes
2018-11-14 06:44:45 UTC
Permalink
Post by Thomas Stüfe
Post by David Holmes
Post by David Holmes
Hi Thomas,
Post by Thomas Stüfe
Hi all,
We have a client using CloudFoundry and its "jvmkill" agent. That
is a
Post by Thomas Stüfe
tiny JVMTI agent (see https://github.com/cloudfoundry/jvmkill) which
subscribes to the JVMTI ResourceExhausted Event. In the handler it
then does call JVMTI FollowReferences() to produce a heap histogram.
The thing is, at our client we seem to run out of Metaspace in a
compiler thread. That thread normally would swallow the Metaspace OOM
and just bailout from the compilation. But as part of the metaspace
OOME handling the ResourceExhausted event gets posted, the handler
then uses JVMTI FollowReferences() and attempts to print out the heap
histogram, then runs into a guarantee since the compiler thread
cannot
Post by Thomas Stüfe
call java methods.
My question is: are there any limitations about what one can do
inside
Post by Thomas Stüfe
a ResourceExhausted event handler?
Not specified no. But the reality of JVM TI is that you can't anticipate
every execution context and there are times when there are implicit
constraints imposed by the implementation.
In this case I think we have a mismatch between the fact we post the
event from the compiler thread, but that a compiler thread is not a true
"Java thread" and so can not execute arbitrary JNI or JVM TI code, or in
particular can not lead to executing Java code. I think we should not be
posting the event from the compiler thread in this case.
Cheers,
David
Hi David,
Yes I thought so too. I'll prepare a fix.
My thought on the fix is that we need to check if
Thread::current()->can_call_java(). And that should probably be inside
the JvmtiExport::should_post_xxx implementation.
I wonder whether that may be too harsh. JVMTI agents may not
necessarily call into java as reaction to ResourceExhausted. I would
have limited this to !CompilerThread, and only in Metaspace.
Also, looking at CompilerThread::can_call_java(), I see that we return
true for jvmci compilers. Still do we want to post it there?
But I am not sure. What do you think?
The JVM TI spec is really only aware of "Java threads", though it does
allow for an implementation specific thread to be used in places. So
posting events from a thread that may not be able to execute
JNI/JVM-TI/Java code successfully seems a very risky proposition. So I'd
say "don't do that".

But if we want to deal only with the callsite in question there are
various ugly hacks we could put in there.

But the basic constraint is the ability to run Java code. JVMCI compiler
threads can run Java code so we don't need to exclude them unnecessarily
- hence the check for can_call_Java().

Cheers,
David
Post by Thomas Stüfe
Post by David Holmes
Cheers,
David
Post by David Holmes
Thanks, Thomas
Post by Thomas Stüfe
I checked the
https://docs.oracle.com/javase/8/docs/platform/jvmti/jvmti.html
Post by Thomas Stüfe
documentation, but I cannot find any mentioning of limitations in
that
Post by Thomas Stüfe
case.
Thanks and Best Regards, Thomas
Thomas Stüfe
2018-11-14 07:03:21 UTC
Permalink
I did open a bug to track this: https://bugs.openjdk.java.net/browse/JDK-8213834
Post by Thomas Stüfe
Post by David Holmes
Post by David Holmes
Hi Thomas,
Post by Thomas Stüfe
Hi all,
We have a client using CloudFoundry and its "jvmkill" agent. That
is a
Post by Thomas Stüfe
tiny JVMTI agent (see https://github.com/cloudfoundry/jvmkill) which
subscribes to the JVMTI ResourceExhausted Event. In the handler it
then does call JVMTI FollowReferences() to produce a heap histogram.
The thing is, at our client we seem to run out of Metaspace in a
compiler thread. That thread normally would swallow the Metaspace OOM
and just bailout from the compilation. But as part of the metaspace
OOME handling the ResourceExhausted event gets posted, the handler
then uses JVMTI FollowReferences() and attempts to print out the heap
histogram, then runs into a guarantee since the compiler thread
cannot
Post by Thomas Stüfe
call java methods.
My question is: are there any limitations about what one can do
inside
Post by Thomas Stüfe
a ResourceExhausted event handler?
Not specified no. But the reality of JVM TI is that you can't anticipate
every execution context and there are times when there are implicit
constraints imposed by the implementation.
In this case I think we have a mismatch between the fact we post the
event from the compiler thread, but that a compiler thread is not a true
"Java thread" and so can not execute arbitrary JNI or JVM TI code, or in
particular can not lead to executing Java code. I think we should not be
posting the event from the compiler thread in this case.
Cheers,
David
Hi David,
Yes I thought so too. I'll prepare a fix.
My thought on the fix is that we need to check if
Thread::current()->can_call_java(). And that should probably be inside
the JvmtiExport::should_post_xxx implementation.
I wonder whether that may be too harsh. JVMTI agents may not
necessarily call into java as reaction to ResourceExhausted. I would
have limited this to !CompilerThread, and only in Metaspace.
Also, looking at CompilerThread::can_call_java(), I see that we return
true for jvmci compilers. Still do we want to post it there?
But I am not sure. What do you think?
Post by David Holmes
Cheers,
David
Post by David Holmes
Thanks, Thomas
Post by Thomas Stüfe
I checked the
https://docs.oracle.com/javase/8/docs/platform/jvmti/jvmti.html
Post by Thomas Stüfe
documentation, but I cannot find any mentioning of limitations in
that
Post by Thomas Stüfe
case.
Thanks and Best Regards, Thomas
JC Beyler
2018-11-14 14:56:06 UTC
Permalink
It seems what we do with other events that might have this type of "risk"
is to defer the event to the ServiceThread, which is a Java thread, no? But
perhaps for a resource exhausted just ignoring it for the compiler thread
and letting another "Java thread" be aware of it and posting is a better
choice?

Thanks,
Jc
Post by Thomas Stüfe
https://bugs.openjdk.java.net/browse/JDK-8213834
Post by Thomas Stüfe
Post by David Holmes
Post by David Holmes
Hi Thomas,
Post by Thomas Stüfe
Hi all,
We have a client using CloudFoundry and its "jvmkill" agent.
That
Post by Thomas Stüfe
Post by David Holmes
Post by David Holmes
is a
Post by Thomas Stüfe
tiny JVMTI agent (see https://github.com/cloudfoundry/jvmkill)
which
Post by Thomas Stüfe
Post by David Holmes
Post by David Holmes
Post by Thomas Stüfe
subscribes to the JVMTI ResourceExhausted Event. In the
handler it
Post by Thomas Stüfe
Post by David Holmes
Post by David Holmes
Post by Thomas Stüfe
then does call JVMTI FollowReferences() to produce a heap
histogram.
Post by Thomas Stüfe
Post by David Holmes
Post by David Holmes
Post by Thomas Stüfe
The thing is, at our client we seem to run out of Metaspace
in a
Post by Thomas Stüfe
Post by David Holmes
Post by David Holmes
Post by Thomas Stüfe
compiler thread. That thread normally would swallow the
Metaspace OOM
Post by Thomas Stüfe
Post by David Holmes
Post by David Holmes
Post by Thomas Stüfe
and just bailout from the compilation. But as part of the
metaspace
Post by Thomas Stüfe
Post by David Holmes
Post by David Holmes
Post by Thomas Stüfe
OOME handling the ResourceExhausted event gets posted, the
handler
Post by Thomas Stüfe
Post by David Holmes
Post by David Holmes
Post by Thomas Stüfe
then uses JVMTI FollowReferences() and attempts to print out
the heap
Post by Thomas Stüfe
Post by David Holmes
Post by David Holmes
Post by Thomas Stüfe
histogram, then runs into a guarantee since the compiler
thread
Post by Thomas Stüfe
Post by David Holmes
Post by David Holmes
cannot
Post by Thomas Stüfe
call java methods.
My question is: are there any limitations about what one can
do
Post by Thomas Stüfe
Post by David Holmes
Post by David Holmes
inside
Post by Thomas Stüfe
a ResourceExhausted event handler?
Not specified no. But the reality of JVM TI is that you can't
anticipate
every execution context and there are times when there are
implicit
Post by Thomas Stüfe
Post by David Holmes
Post by David Holmes
constraints imposed by the implementation.
In this case I think we have a mismatch between the fact we post
the
Post by Thomas Stüfe
Post by David Holmes
Post by David Holmes
event from the compiler thread, but that a compiler thread is
not a
Post by Thomas Stüfe
Post by David Holmes
Post by David Holmes
true
"Java thread" and so can not execute arbitrary JNI or JVM TI
code,
Post by Thomas Stüfe
Post by David Holmes
Post by David Holmes
or in
particular can not lead to executing Java code. I think we should
not be
posting the event from the compiler thread in this case.
Cheers,
David
Hi David,
Yes I thought so too. I'll prepare a fix.
My thought on the fix is that we need to check if
Thread::current()->can_call_java(). And that should probably be inside
the JvmtiExport::should_post_xxx implementation.
I wonder whether that may be too harsh. JVMTI agents may not
necessarily call into java as reaction to ResourceExhausted. I would
have limited this to !CompilerThread, and only in Metaspace.
Also, looking at CompilerThread::can_call_java(), I see that we return
true for jvmci compilers. Still do we want to post it there?
But I am not sure. What do you think?
Post by David Holmes
Cheers,
David
Post by David Holmes
Thanks, Thomas
Post by Thomas Stüfe
I checked the
https://docs.oracle.com/javase/8/docs/platform/jvmti/jvmti.html
Post by Thomas Stüfe
documentation, but I cannot find any mentioning of
limitations in
Post by Thomas Stüfe
Post by David Holmes
Post by David Holmes
that
Post by Thomas Stüfe
case.
Thanks and Best Regards, Thomas
--
Thanks,
Jc
Thomas Stüfe
2018-11-14 15:00:51 UTC
Permalink
Hi JC,
It seems what we do with other events that might have this type of "risk" is to defer the event to the ServiceThread, which is a Java thread, no? But perhaps for a resource exhausted just ignoring it for the compiler thread and letting another "Java thread" be aware of it and posting is a better choice?
Thanks,
Jc
in case of ResourceExhausted I think this is safe to be ignored for
the compiler thread. If the problem persists, surely a java thread
will hit it next.

...Thomas
Post by Thomas Stüfe
I did open a bug to track this: https://bugs.openjdk.java.net/browse/JDK-8213834
Post by Thomas Stüfe
Post by David Holmes
Post by David Holmes
Hi Thomas,
Post by Thomas Stüfe
Hi all,
We have a client using CloudFoundry and its "jvmkill" agent. That
is a
Post by Thomas Stüfe
tiny JVMTI agent (see https://github.com/cloudfoundry/jvmkill) which
subscribes to the JVMTI ResourceExhausted Event. In the handler it
then does call JVMTI FollowReferences() to produce a heap histogram.
The thing is, at our client we seem to run out of Metaspace in a
compiler thread. That thread normally would swallow the Metaspace OOM
and just bailout from the compilation. But as part of the metaspace
OOME handling the ResourceExhausted event gets posted, the handler
then uses JVMTI FollowReferences() and attempts to print out the heap
histogram, then runs into a guarantee since the compiler thread
cannot
Post by Thomas Stüfe
call java methods.
My question is: are there any limitations about what one can do
inside
Post by Thomas Stüfe
a ResourceExhausted event handler?
Not specified no. But the reality of JVM TI is that you can't
anticipate
every execution context and there are times when there are implicit
constraints imposed by the implementation.
In this case I think we have a mismatch between the fact we post the
event from the compiler thread, but that a compiler thread is not a
true
"Java thread" and so can not execute arbitrary JNI or JVM TI code,
or in
particular can not lead to executing Java code. I think we should
not be
posting the event from the compiler thread in this case.
Cheers,
David
Hi David,
Yes I thought so too. I'll prepare a fix.
My thought on the fix is that we need to check if
Thread::current()->can_call_java(). And that should probably be inside
the JvmtiExport::should_post_xxx implementation.
I wonder whether that may be too harsh. JVMTI agents may not
necessarily call into java as reaction to ResourceExhausted. I would
have limited this to !CompilerThread, and only in Metaspace.
Also, looking at CompilerThread::can_call_java(), I see that we return
true for jvmci compilers. Still do we want to post it there?
But I am not sure. What do you think?
Post by David Holmes
Cheers,
David
Post by David Holmes
Thanks, Thomas
Post by Thomas Stüfe
I checked the
https://docs.oracle.com/javase/8/docs/platform/jvmti/jvmti.html
Post by Thomas Stüfe
documentation, but I cannot find any mentioning of limitations in
that
Post by Thomas Stüfe
case.
Thanks and Best Regards, Thomas
--
Thanks,
Jc
Chris Plummer
2018-11-14 05:37:05 UTC
Permalink
Post by David Holmes
Hi Thomas,
Post by Thomas Stüfe
Hi all,
We have a client using CloudFoundry and its "jvmkill" agent. That is a
tiny JVMTI agent (see https://github.com/cloudfoundry/jvmkill) which
subscribes to the JVMTI ResourceExhausted Event. In the handler it
then does call JVMTI FollowReferences() to produce a heap histogram.
The thing is, at our client we seem to run out of Metaspace in a
compiler thread. That thread normally would swallow the Metaspace OOM
and just bailout from the compilation. But as part of the metaspace
OOME handling the ResourceExhausted event gets posted, the handler
then uses JVMTI FollowReferences() and attempts to print out the heap
histogram, then runs into a guarantee since the compiler thread cannot
call java methods.
My question is: are there any limitations about what one can do inside
a ResourceExhausted event handler?
Not specified no. But the reality of JVM TI is that you can't
anticipate every execution context and there are times when there are
implicit constraints imposed by the implementation.
In this case I think we have a mismatch between the fact we post the
event from the compiler thread, but that a compiler thread is not a
true "Java thread" and so can not execute arbitrary JNI or JVM TI
code, or in particular can not lead to executing Java code. I think we
should not be posting the event from the compiler thread in this case.
Does the recent fix for https://bugs.openjdk.java.net/browse/JDK-8193126
address the metaspace failure you are seeing?

Chris
Post by David Holmes
Cheers,
David
Post by Thomas Stüfe
I checked the
https://docs.oracle.com/javase/8/docs/platform/jvmti/jvmti.html
documentation, but I cannot find any mentioning of limitations in that
case.
Thanks and Best Regards, Thomas
Thomas Stüfe
2018-11-14 06:19:35 UTC
Permalink
Hi Chris,

I do not think so. The MetaspaceSize we see when we fail has the
expected value, it is not downgraded somehow.

Also, would that bug apply in a normal VM? Is UseJVMCICompiler not off
by default?

I think with this bug the chance for this error to happen may increase
but the bug itself (the fact that we post ResourceExhausted from
Compiler Thread) is older and also predates JVMCI. We see this also
with older VMs.

Thanks Thomas
Post by Chris Plummer
Post by David Holmes
Hi Thomas,
Post by Thomas Stüfe
Hi all,
We have a client using CloudFoundry and its "jvmkill" agent. That is a
tiny JVMTI agent (see https://github.com/cloudfoundry/jvmkill) which
subscribes to the JVMTI ResourceExhausted Event. In the handler it
then does call JVMTI FollowReferences() to produce a heap histogram.
The thing is, at our client we seem to run out of Metaspace in a
compiler thread. That thread normally would swallow the Metaspace OOM
and just bailout from the compilation. But as part of the metaspace
OOME handling the ResourceExhausted event gets posted, the handler
then uses JVMTI FollowReferences() and attempts to print out the heap
histogram, then runs into a guarantee since the compiler thread cannot
call java methods.
My question is: are there any limitations about what one can do inside
a ResourceExhausted event handler?
Not specified no. But the reality of JVM TI is that you can't
anticipate every execution context and there are times when there are
implicit constraints imposed by the implementation.
In this case I think we have a mismatch between the fact we post the
event from the compiler thread, but that a compiler thread is not a
true "Java thread" and so can not execute arbitrary JNI or JVM TI
code, or in particular can not lead to executing Java code. I think we
should not be posting the event from the compiler thread in this case.
Does the recent fix for https://bugs.openjdk.java.net/browse/JDK-8193126
address the metaspace failure you are seeing?
Chris
Post by David Holmes
Cheers,
David
Post by Thomas Stüfe
I checked the
https://docs.oracle.com/javase/8/docs/platform/jvmti/jvmti.html
documentation, but I cannot find any mentioning of limitations in that
case.
Thanks and Best Regards, Thomas
Loading...