Following this post: -
one of my colleagues asked if/how HC could be set to work in so-called "headless" mode, where the HC agent pumps its data out to a file for offline reviewing.
Using this: -
I updated my BPM AppCluster JVM ( via Servers > Application servers > AppClusterMember1 > Process definition > Java Virtual Machine > Generic JVM arguments ), from this: -
${IBMSCMX} ${IBMGCPOLICY_GENCON} ${IBMJITPMC} -Xmns256m -Xmnx768m -Xgc:preferredHeapBase=0x100000000 -Xdisableexplicitgc -Xss2048k -Dsun.net.http.allowRestrictedHeaders=true -Declipse.bundle.setTCCL=false -Xhealthcenter:transport=jrmp
to this: -
${IBMSCMX} ${IBMGCPOLICY_GENCON} ${IBMJITPMC} -Xmns256m -Xmnx768m -Xgc:preferredHeapBase=0x100000000 -Xdisableexplicitgc -Xss2048k -Dsun.net.http.allowRestrictedHeaders=true -Declipse.bundle.setTCCL=false -Xhealthcenter:level=headless -Dcom.ibm.java.diagnostics.healthcenter.headless.output.directory=${SERVER_LOG_ROOT}/healthcenter -Dcom.ibm.java.diagnostics.healthcenter.headless.run.duration=15 -Dcom.ibm.diagnostics.healthcenter.data.profiling=off -Dcom.ibm.java.diagnostics.healthcenter.allocation.threshold.low=10000000 -Dcom.ibm.java.diagnostics.healthcenter.stack.trace.depth=20 -Dcom.ibm.java.diagnostics.healthcenter.headless.files.to.keep=0
Having restarted the AppCluster, and tailed the logs: -
tail -f /opt/ibm/WebSphereProfiles/AppSrv01/logs/AppClusterMember1/SystemOut.log
...
[02/11/17 12:06:05:885 GMT] 00000001 WsServerImpl A WSVR0001I: Server AppClusterMember1 open for e-business
[02/11/17 12:06:05:903 GMT] 00000144 InstanceManag W The dependency javax.servlet.ServletContext is not supported!
[02/11/17 12:06:05:903 GMT] 00000145 InstanceManag W The dependency javax.servlet.ServletContext is not supported!
[02/11/17 12:06:06:011 GMT] 00000145 ControllerSer I WebDav support is ENABLED
[02/11/17 12:06:06:023 GMT] 00000144 ControllerSer I WebDav support is ENABLED
[02/11/17 12:06:06:058 GMT] 00000145 WASPlatform I WASPlatform.getServerIdentifier: (uncompressed) PCCell1Node1AppClusterMember1
[02/11/17 12:06:06:058 GMT] 00000145 WASPlatform I WASPlatform.getServerIdentifier: (compressed) ce6f324b3c0e2195173d4d2acec5566c
[02/11/17 12:06:06:059 GMT] 00000145 WASPlatform I WASPlatform.getServerIdentifier: ce6f324b3c0e2195173d4d2acec5566c
[02/11/17 12:06:06:065 GMT] 00000144 WASPlatform I WASPlatform.getServerIdentifier: (uncompressed) PCCell1Node1AppClusterMember1
[02/11/17 12:06:06:065 GMT] 00000144 WASPlatform I WASPlatform.getServerIdentifier: (compressed) ce6f324b3c0e2195173d4d2acec5566c
[02/11/17 12:06:06:066 GMT] 00000144 WASPlatform I WASPlatform.getServerIdentifier: ce6f324b3c0e2195173d4d2acec5566c
[02/11/17 12:06:06:152 GMT] 00000138 authz I CWWIM2000I Initialization of the authorization component completed successfully.
[02/11/17 12:08:34:470 GMT] 00000128 Heartbeat I CWLLG0615I: Heartbeat resumed.
[02/11/17 12:08:41:017 GMT] 00000129 TaskLoader I CWLLG0597I: Trying to acquire synchronous queue SYNC_QUEUE_1.
[02/11/17 12:08:41:022 GMT] 00000129 TaskLoader I CWLLG0581I: Acquired synchronous queue SYNC_QUEUE_1.
[02/11/17 12:08:56:620 GMT] 00000129 TaskLoader I CWLLG0597I: Trying to acquire synchronous queue SYNC_QUEUE_2.
[02/11/17 12:08:56:621 GMT] 00000129 TaskLoader I CWLLG0581I: Acquired synchronous queue SYNC_QUEUE_2.
[02/11/17 12:09:12:662 GMT] 00000129 TaskLoader I CWLLG0597I: Trying to acquire synchronous queue SYNC_QUEUE_3.
[02/11/17 12:09:12:663 GMT] 00000129 TaskLoader I CWLLG0581I: Acquired synchronous queue SYNC_QUEUE_3.
...
[02/11/17 12:06:05:903 GMT] 00000144 InstanceManag W The dependency javax.servlet.ServletContext is not supported!
[02/11/17 12:06:05:903 GMT] 00000145 InstanceManag W The dependency javax.servlet.ServletContext is not supported!
[02/11/17 12:06:06:011 GMT] 00000145 ControllerSer I WebDav support is ENABLED
[02/11/17 12:06:06:023 GMT] 00000144 ControllerSer I WebDav support is ENABLED
[02/11/17 12:06:06:058 GMT] 00000145 WASPlatform I WASPlatform.getServerIdentifier: (uncompressed) PCCell1Node1AppClusterMember1
[02/11/17 12:06:06:058 GMT] 00000145 WASPlatform I WASPlatform.getServerIdentifier: (compressed) ce6f324b3c0e2195173d4d2acec5566c
[02/11/17 12:06:06:059 GMT] 00000145 WASPlatform I WASPlatform.getServerIdentifier: ce6f324b3c0e2195173d4d2acec5566c
[02/11/17 12:06:06:065 GMT] 00000144 WASPlatform I WASPlatform.getServerIdentifier: (uncompressed) PCCell1Node1AppClusterMember1
[02/11/17 12:06:06:065 GMT] 00000144 WASPlatform I WASPlatform.getServerIdentifier: (compressed) ce6f324b3c0e2195173d4d2acec5566c
[02/11/17 12:06:06:066 GMT] 00000144 WASPlatform I WASPlatform.getServerIdentifier: ce6f324b3c0e2195173d4d2acec5566c
[02/11/17 12:06:06:152 GMT] 00000138 authz I CWWIM2000I Initialization of the authorization component completed successfully.
[02/11/17 12:08:34:470 GMT] 00000128 Heartbeat I CWLLG0615I: Heartbeat resumed.
[02/11/17 12:08:41:017 GMT] 00000129 TaskLoader I CWLLG0597I: Trying to acquire synchronous queue SYNC_QUEUE_1.
[02/11/17 12:08:41:022 GMT] 00000129 TaskLoader I CWLLG0581I: Acquired synchronous queue SYNC_QUEUE_1.
[02/11/17 12:08:56:620 GMT] 00000129 TaskLoader I CWLLG0597I: Trying to acquire synchronous queue SYNC_QUEUE_2.
[02/11/17 12:08:56:621 GMT] 00000129 TaskLoader I CWLLG0581I: Acquired synchronous queue SYNC_QUEUE_2.
[02/11/17 12:09:12:662 GMT] 00000129 TaskLoader I CWLLG0597I: Trying to acquire synchronous queue SYNC_QUEUE_3.
[02/11/17 12:09:12:663 GMT] 00000129 TaskLoader I CWLLG0581I: Acquired synchronous queue SYNC_QUEUE_3.
...
I can then see the HC data: -
ls -al /opt/ibm/WebSphereProfiles/AppSrv01/logs/AppClusterMember1
total 672
drwxr-xr-x 3 wasadmin wasadmins 4096 Nov 2 12:05 .
drwxr-xr-x 9 wasadmin wasadmins 124 Nov 1 14:00 ..
-rw-r--r-- 1 wasadmin wasadmins 5 Nov 2 12:06 AppClusterMember1.pid
-rw-r--r-- 1 wasadmin wasadmins 9304 Nov 2 12:06 btrace.1
drwxr-xr-x 3 wasadmin wasadmins 31 Nov 2 12:05 healthcenter
-rw-r--r-- 1 wasadmin wasadmins 127820 Nov 2 12:07 native_stderr.log
-rw-r--r-- 1 wasadmin wasadmins 2605 Nov 2 12:05 native_stdout.log
-rw-r--r-- 1 wasadmin wasadmins 123210 Nov 2 12:06 objects
-rw-r--r-- 1 wasadmin wasadmins 871 Nov 2 12:05 SystemErr.log
-rw-r--r-- 1 wasadmin wasadmins 31 Nov 2 12:05 SystemErr.log.owner
-rw-r--r-- 1 wasadmin wasadmins 325877 Nov 2 12:09 SystemOut.log
-rw-r--r-- 1 wasadmin wasadmins 31 Nov 2 12:05 SystemOut.log.owner
drwxr-xr-x 3 wasadmin wasadmins 4096 Nov 2 12:05 .
drwxr-xr-x 9 wasadmin wasadmins 124 Nov 1 14:00 ..
-rw-r--r-- 1 wasadmin wasadmins 5 Nov 2 12:06 AppClusterMember1.pid
-rw-r--r-- 1 wasadmin wasadmins 9304 Nov 2 12:06 btrace.1
drwxr-xr-x 3 wasadmin wasadmins 31 Nov 2 12:05 healthcenter
-rw-r--r-- 1 wasadmin wasadmins 127820 Nov 2 12:07 native_stderr.log
-rw-r--r-- 1 wasadmin wasadmins 2605 Nov 2 12:05 native_stdout.log
-rw-r--r-- 1 wasadmin wasadmins 123210 Nov 2 12:06 objects
-rw-r--r-- 1 wasadmin wasadmins 871 Nov 2 12:05 SystemErr.log
-rw-r--r-- 1 wasadmin wasadmins 31 Nov 2 12:05 SystemErr.log.owner
-rw-r--r-- 1 wasadmin wasadmins 325877 Nov 2 12:09 SystemOut.log
-rw-r--r-- 1 wasadmin wasadmins 31 Nov 2 12:05 SystemOut.log.owner
ls -al /opt/ibm/WebSphereProfiles/AppSrv01/logs/AppClusterMember1/healthcenter
total 4
drwxr-xr-x 3 wasadmin wasadmins 31 Nov 2 12:05 .
drwxr-xr-x 3 wasadmin wasadmins 4096 Nov 2 12:05 ..
drwxr-xr-x 2 wasadmin wasadmins 108 Nov 2 12:05 tmp_021117_120504_
drwxr-xr-x 3 wasadmin wasadmins 31 Nov 2 12:05 .
drwxr-xr-x 3 wasadmin wasadmins 4096 Nov 2 12:05 ..
drwxr-xr-x 2 wasadmin wasadmins 108 Nov 2 12:05 tmp_021117_120504_
ls -al /opt/ibm/WebSphereProfiles/AppSrv01/logs/AppClusterMember1/healthcenter/tmp_021117_120504_
total 39652
drwxr-xr-x 2 wasadmin wasadmins 108 Nov 2 12:05 .
drwxr-xr-x 3 wasadmin wasadmins 31 Nov 2 12:05 ..
-rw-r--r-- 1 wasadmin wasadmins 0 Nov 2 12:05 cpu
-rw-r--r-- 1 wasadmin wasadmins 154377 Nov 2 12:05 environment
-rw-r--r-- 1 wasadmin wasadmins 2044765 Nov 2 12:12 locking
-rw-r--r-- 1 wasadmin wasadmins 8310 Nov 2 12:10 memory
-rw-r--r-- 1 wasadmin wasadmins 195028 Nov 2 12:12 memorycounters
-rw-r--r-- 1 wasadmin wasadmins 2149750 Nov 2 12:12 threads
-rw-r--r-- 1 wasadmin wasadmins 24217364 Nov 2 12:12 trace
drwxr-xr-x 2 wasadmin wasadmins 108 Nov 2 12:05 .
drwxr-xr-x 3 wasadmin wasadmins 31 Nov 2 12:05 ..
-rw-r--r-- 1 wasadmin wasadmins 0 Nov 2 12:05 cpu
-rw-r--r-- 1 wasadmin wasadmins 154377 Nov 2 12:05 environment
-rw-r--r-- 1 wasadmin wasadmins 2044765 Nov 2 12:12 locking
-rw-r--r-- 1 wasadmin wasadmins 8310 Nov 2 12:10 memory
-rw-r--r-- 1 wasadmin wasadmins 195028 Nov 2 12:12 memorycounters
-rw-r--r-- 1 wasadmin wasadmins 2149750 Nov 2 12:12 threads
-rw-r--r-- 1 wasadmin wasadmins 24217364 Nov 2 12:12 trace
and dig into specific logs: -
tail -f /opt/ibm/WebSphereProfiles/AppSrv01/logs/AppClusterMember1/healthcenter/tmp_021117_120504_/memory
MemorySource,1509624551432,totalphysicalmemory=8185933824,physicalmemory=1264852992,privatememory=4677050368,virtualmemory=4781912064,freephysicalmemory=309432320
MemorySource,1509624557436,totalphysicalmemory=8185933824,physicalmemory=1265037312,privatememory=4677046272,virtualmemory=4781912064,freephysicalmemory=308670464
MemorySource,1509624563448,totalphysicalmemory=8185933824,physicalmemory=1274904576,privatememory=4687523840,virtualmemory=4792397824,freephysicalmemory=299327488
MemorySource,1509624569450,totalphysicalmemory=8185933824,physicalmemory=1265676288,privatememory=4677033984,virtualmemory=4781912064,freephysicalmemory=308559872
MemorySource,1509624575451,totalphysicalmemory=8185933824,physicalmemory=1265774592,privatememory=4677033984,virtualmemory=4781912064,freephysicalmemory=307417088
MemorySource,1509624581455,totalphysicalmemory=8185933824,physicalmemory=1265831936,privatememory=4677033984,virtualmemory=4781912064,freephysicalmemory=306622464
MemorySource,1509624587457,totalphysicalmemory=8185933824,physicalmemory=1265840128,privatememory=4677033984,virtualmemory=4781912064,freephysicalmemory=306241536
MemorySource,1509624593460,totalphysicalmemory=8185933824,physicalmemory=1266036736,privatememory=4677029888,virtualmemory=4781912064,freephysicalmemory=307363840
MemorySource,1509624599461,totalphysicalmemory=8185933824,physicalmemory=1266126848,privatememory=4677025792,virtualmemory=4781912064,freephysicalmemory=307052544
MemorySource,1509624605464,totalphysicalmemory=8185933824,physicalmemory=1277919232,privatememory=4688560128,virtualmemory=4793446400,freephysicalmemory=295735296
MemorySource,1509624557436,totalphysicalmemory=8185933824,physicalmemory=1265037312,privatememory=4677046272,virtualmemory=4781912064,freephysicalmemory=308670464
MemorySource,1509624563448,totalphysicalmemory=8185933824,physicalmemory=1274904576,privatememory=4687523840,virtualmemory=4792397824,freephysicalmemory=299327488
MemorySource,1509624569450,totalphysicalmemory=8185933824,physicalmemory=1265676288,privatememory=4677033984,virtualmemory=4781912064,freephysicalmemory=308559872
MemorySource,1509624575451,totalphysicalmemory=8185933824,physicalmemory=1265774592,privatememory=4677033984,virtualmemory=4781912064,freephysicalmemory=307417088
MemorySource,1509624581455,totalphysicalmemory=8185933824,physicalmemory=1265831936,privatememory=4677033984,virtualmemory=4781912064,freephysicalmemory=306622464
MemorySource,1509624587457,totalphysicalmemory=8185933824,physicalmemory=1265840128,privatememory=4677033984,virtualmemory=4781912064,freephysicalmemory=306241536
MemorySource,1509624593460,totalphysicalmemory=8185933824,physicalmemory=1266036736,privatememory=4677029888,virtualmemory=4781912064,freephysicalmemory=307363840
MemorySource,1509624599461,totalphysicalmemory=8185933824,physicalmemory=1266126848,privatememory=4677025792,virtualmemory=4781912064,freephysicalmemory=307052544
MemorySource,1509624605464,totalphysicalmemory=8185933824,physicalmemory=1277919232,privatememory=4688560128,virtualmemory=4793446400,freephysicalmemory=295735296
tail -f /opt/ibm/WebSphereProfiles/AppSrv01/logs/AppClusterMember1/healthcenter/tmp_021117_120504_/environment
system.sun.arch.data.model=64
system.com.ibm.packed.version=2
security.networkaddress.cache.ttl=
system.osgi.framework.extensions=com.ibm.cds,com.ibm.ws.eclipse.adaptors
environment.USER_INSTALL_PROP=-Duser.install.root=/opt/ibm/WebSphereProfiles/AppSrv01
system.java.awt.headless=true
native.library.date=Oct 17 2016 14:42:06
pid=42955
dump.options=system:events=gpf+abort+traceassert+corruptcache,file=/opt/ibm/WebSphereProfiles/AppSrv01/core.%Y%m%d.%H%M%S.%pid.%seq.dmp,range=1..0,priority=999,request=serial@@@system:events=systhrow,filter=java/lang/OutOfMemoryError,file=/opt/ibm/WebSphereProfiles/AppSrv01/core.%Y%m%d.%H%M%S.%pid.%seq.dmp,range=1..1,priority=999,request=exclusive+compact+prepwalk@@@heap:events=systhrow,filter=java/lang/OutOfMemoryError,file=/opt/ibm/WebSphereProfiles/AppSrv01/heapdump.%Y%m%d.%H%M%S.%pid.%seq.phd,range=1..4,priority=500,request=exclusive+compact+prepwalk,opts=PHD@@@java:events=gpf+user+abort+traceassert+corruptcache,file=/opt/ibm/WebSphereProfiles/AppSrv01/javacore.%Y%m%d.%H%M%S.%pid.%seq.txt,range=1..0,priority=400,request=exclusive+preempt@@@java:events=systhrow,filter=java/lang/OutOfMemoryError,file=/opt/ibm/WebSphereProfiles/AppSrv01/javacore.%Y%m%d.%H%M%S.%pid.%seq.txt,range=1..4,priority=400,request=exclusive+preempt@@@snap:events=gpf+abort+traceassert+corruptcache,file=/opt/ibm/WebSphereProfiles/AppSrv01/Snap.%Y%m%d.%H%M%S.%pid.%seq.trc,range=1..0,priority=300,request=serial@@@snap:events=systhrow,filter=java/lang/OutOfMemoryError,file=/opt/ibm/WebSphereProfiles/AppSrv01/Snap.%Y%m%d.%H%M%S.%pid.%seq.trc,range=1..4,priority=300,request=serial@@@jit:events=gpf+abort,file=/opt/ibm/WebSphereProfiles/AppSrv01/jitdump.%Y%m%d.%H%M%S.%pid.%seq.dmp,range=1..0,priority=200,request=serial@@@silent:events=allocation,filter=#10000000,file=-,range=1..0,priority=5,request=
etc.
or combine two or three logs together: -
tail -f /opt/ibm/WebSphereProfiles/AppSrv01/logs/AppClusterMember1/healthcenter/tmp_021117_120504_/environment /opt/ibm/WebSphereProfiles/AppSrv01/logs/AppClusterMember1/healthcenter/tmp_021117_120504_/memory /opt/ibm/WebSphereProfiles/AppSrv01/logs/AppClusterMember1/healthcenter/tmp_021117_120504_/threads
and then watch the box under load: -
==> /opt/ibm/WebSphereProfiles/AppSrv01/logs/AppClusterMember1/healthcenter/tmp_021117_120504_/environment <==
system.sun.arch.data.model=64
system.com.ibm.packed.version=2
security.networkaddress.cache.ttl=
system.osgi.framework.extensions=com.ibm.cds,com.ibm.ws.eclipse.adaptors
environment.USER_INSTALL_PROP=-Duser.install.root=/opt/ibm/WebSphereProfiles/AppSrv01
system.java.awt.headless=true
native.library.date=Oct 17 2016 14:42:06
pid=42955
dump.options=system:events=gpf+abort+traceassert+corruptcache,file=/opt/ibm/WebSphereProfiles/AppSrv01/core.%Y%m%d.%H%M%S.%pid.%seq.dmp,range=1..0,priority=999,request=serial@@@system:events=systhrow,filter=java/lang/OutOfMemoryError,file=/opt/ibm/WebSphereProfiles/AppSrv01/core.%Y%m%d.%H%M%S.%pid.%seq.dmp,range=1..1,priority=999,request=exclusive+compact+prepwalk@@@heap:events=systhrow,filter=java/lang/OutOfMemoryError,file=/opt/ibm/WebSphereProfiles/AppSrv01/heapdump.%Y%m%d.%H%M%S.%pid.%seq.phd,range=1..4,priority=500,request=exclusive+compact+prepwalk,opts=PHD@@@java:events=gpf+user+abort+traceassert+corruptcache,file=/opt/ibm/WebSphereProfiles/AppSrv01/javacore.%Y%m%d.%H%M%S.%pid.%seq.txt,range=1..0,priority=400,request=exclusive+preempt@@@java:events=systhrow,filter=java/lang/OutOfMemoryError,file=/opt/ibm/WebSphereProfiles/AppSrv01/javacore.%Y%m%d.%H%M%S.%pid.%seq.txt,range=1..4,priority=400,request=exclusive+preempt@@@snap:events=gpf+abort+traceassert+corruptcache,file=/opt/ibm/WebSphereProfiles/AppSrv01/Snap.%Y%m%d.%H%M%S.%pid.%seq.trc,range=1..0,priority=300,request=serial@@@snap:events=systhrow,filter=java/lang/OutOfMemoryError,file=/opt/ibm/WebSphereProfiles/AppSrv01/Snap.%Y%m%d.%H%M%S.%pid.%seq.trc,range=1..4,priority=300,request=serial@@@jit:events=gpf+abort,file=/opt/ibm/WebSphereProfiles/AppSrv01/jitdump.%Y%m%d.%H%M%S.%pid.%seq.dmp,range=1..0,priority=200,request=serial@@@silent:events=allocation,filter=#10000000,file=-,range=1..0,priority=5,request=
==> /opt/ibm/WebSphereProfiles/AppSrv01/logs/AppClusterMember1/healthcenter/tmp_021117_120504_/memory <==
MemorySource,1509624857597,totalphysicalmemory=8185933824,physicalmemory=1271615488,privatememory=4676968448,virtualmemory=4781912064,freephysicalmemory=292753408
MemorySource,1509624863601,totalphysicalmemory=8185933824,physicalmemory=1271721984,privatememory=4676968448,virtualmemory=4781912064,freephysicalmemory=292544512
MemorySource,1509624869603,totalphysicalmemory=8185933824,physicalmemory=1271734272,privatememory=4676968448,virtualmemory=4781912064,freephysicalmemory=292659200
MemorySource,1509624875604,totalphysicalmemory=8185933824,physicalmemory=1271754752,privatememory=4676968448,virtualmemory=4781912064,freephysicalmemory=292532224
MemorySource,1509624881605,totalphysicalmemory=8185933824,physicalmemory=1271775232,privatememory=4676968448,virtualmemory=4781912064,freephysicalmemory=292253696
MemorySource,1509624887607,totalphysicalmemory=8185933824,physicalmemory=1271816192,privatememory=4676968448,virtualmemory=4781912064,freephysicalmemory=292134912
MemorySource,1509624893609,totalphysicalmemory=8185933824,physicalmemory=1271816192,privatememory=4676968448,virtualmemory=4781912064,freephysicalmemory=292167680
MemorySource,1509624899612,totalphysicalmemory=8185933824,physicalmemory=1271836672,privatememory=4676968448,virtualmemory=4781912064,freephysicalmemory=292802560
MemorySource,1509624905615,totalphysicalmemory=8185933824,physicalmemory=1271836672,privatememory=4676968448,virtualmemory=4781912064,freephysicalmemory=292818944
MemorySource,1509624911617,totalphysicalmemory=8185933824,physicalmemory=1271885824,privatememory=4676968448,virtualmemory=4781912064,freephysicalmemory=292548608
==> /opt/ibm/WebSphereProfiles/AppSrv01/logs/AppClusterMember1/healthcenter/tmp_021117_120504_/threads <==
startThreads3.0.7@#1509624925622@#238@#18@#321@#8@#Timer-187@#WAITING4@#16@#java.lang.Object4@#wait11@#Object.java-2@#16@#java.lang.Object4@#wait11@#Object.java172@#21@#java.util.TimerThread8@#mainLoop10@#Timer.java537@#21@#java.util.TimerThread3@#run10@#Timer.java516@#1@#19@#java.util.TaskQueue-138882087@#2@#0@#28@#java.util.TaskQueue@f7b8d3d9
startThreads3.0.7@#1509624925622@#238@#20@#322@#10@#Thread-1417@#WAITING5@#16@#java.lang.Object4@#wait11@#Object.java-2@#16@#java.lang.Object4@#wait11@#Object.java172@#44@#com.ibm.ws.wsgroup.p2p.SonP2PShimImpl$Worker7@#oneStep19@#SonP2PShimImpl.java1274@#44@#com.ibm.ws.wsgroup.p2p.SonP2PShimImpl$Worker3@#run19@#SonP2PShimImpl.java1213@#16@#java.lang.Thread3@#run11@#Thread.java785@#1@#43@#com.ibm.ws.wsgroup.p2p.SonP2PShimImpl$Lock1-619790904@#0@#0@#52@#com.ibm.ws.wsgroup.p2p.SonP2PShimImpl$Lock1@db0ebdc8
startThreads3.0.7@#1509624925622@#238@#24@#324@#16@#WebContainer : 013@#TIMED_WAITING8@#15@#sun.misc.Unsafe4@#park11@#Unsafe.java-2@#38@#java.util.concurrent.locks.LockSupport9@#parkNanos16@#LockSupport.java226@#69@#java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject5@#await31@#AbstractQueuedSynchronizer.java2174@#42@#com.ibm.ws.util.BoundedBuffer$GetQueueLock5@#await18@#BoundedBuffer.java298@#29@#com.ibm.ws.util.BoundedBuffer8@#waitGet_18@#BoundedBuffer.java455@#29@#com.ibm.ws.util.BoundedBuffer4@#take18@#BoundedBuffer.java853@#26@#com.ibm.ws.util.ThreadPool7@#getTask15@#ThreadPool.java1059@#33@#com.ibm.ws.util.ThreadPool$Worker3@#run15@#ThreadPool.java1916@#0@#0@#78@#java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@8fed6033
startThreads3.0.7@#1509624925622@#238@#18@#325@#16@#WebContainer : 18@#RUNNABLE5@#29@#com.ibm.io.async.AsyncLibrary12@#aio_getioev317@#AsyncLibrary.java-2@#29@#com.ibm.io.async.AsyncLibrary18@#getCompletionData317@#AsyncLibrary.java658@#30@#com.ibm.io.async.ResultHandler22@#runEventProcessingLoop18@#ResultHandler.java530@#32@#com.ibm.io.async.ResultHandler$23@#run18@#ResultHandler.java905@#33@#com.ibm.ws.util.ThreadPool$Worker3@#run15@#ThreadPool.java1892@#0@#0@#0@#
startThreads3.0.7@#1509624925622@#238@#24@#328@#16@#WebContainer : 213@#TIMED_WAITING8@#15@#sun.misc.Unsafe4@#park11@#Unsafe.java-2@#38@#java.util.concurrent.locks.LockSupport9@#parkNanos16@#LockSupport.java226@#69@#java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject5@#await31@#AbstractQueuedSynchronizer.java2174@#42@#com.ibm.ws.util.BoundedBuffer$GetQueueLock5@#await18@#BoundedBuffer.java298@#29@#com.ibm.ws.util.BoundedBuffer8@#waitGet_18@#BoundedBuffer.java455@#29@#com.ibm.ws.util.BoundedBuffer4@#take18@#BoundedBuffer.java853@#26@#com.ibm.ws.util.ThreadPool7@#getTask15@#ThreadPool.java1059@#33@#com.ibm.ws.util.ThreadPool$Worker3@#run15@#ThreadPool.java1916@#0@#0@#78@#java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5b3f22bd
startThreads3.0.7@#1509624925622@#238@#26@#330@#24@#ProductInsights-thread-77@#WAITING9@#15@#sun.misc.Unsafe4@#park11@#Unsafe.java-2@#38@#java.util.concurrent.locks.LockSupport4@#park16@#LockSupport.java186@#69@#java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject5@#await31@#AbstractQueuedSynchronizer.java2050@#65@#java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue4@#take32@#ScheduledThreadPoolExecutor.java1099@#65@#java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue4@#take32@#ScheduledThreadPoolExecutor.java820@#39@#java.util.concurrent.ThreadPoolExecutor7@#getTask23@#ThreadPoolExecutor.java1078@#39@#java.util.concurrent.ThreadPoolExecutor9@#runWorker23@#ThreadPoolExecutor.java1138@#46@#java.util.concurrent.ThreadPoolExecutor$Worker3@#run23@#ThreadPoolExecutor.java628@#16@#java.lang.Thread3@#run11@#Thread.java785@#0@#0@#78@#java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@ded3f825
startThreads3.0.7@#1509624925622@#238@#26@#331@#24@#ProductInsights-thread-87@#WAITING9@#15@#sun.misc.Unsafe4@#park11@#Unsafe.java-2@#38@#java.util.concurrent.locks.LockSupport4@#park16@#LockSupport.java186@#69@#java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject5@#await31@#AbstractQueuedSynchronizer.java2050@#65@#java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue4@#take32@#ScheduledThreadPoolExecutor.java1099@#65@#java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue4@#take32@#ScheduledThreadPoolExecutor.java820@#39@#java.util.concurrent.ThreadPoolExecutor7@#getTask23@#ThreadPoolExecutor.java1078@#39@#java.util.concurrent.ThreadPoolExecutor9@#runWorker23@#ThreadPoolExecutor.java1138@#46@#java.util.concurrent.ThreadPoolExecutor$Worker3@#run23@#ThreadPoolExecutor.java628@#16@#java.lang.Thread3@#run11@#Thread.java785@#0@#0@#78@#java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@ded3f825
startThreads3.0.7@#1509624925622@#238@#24@#332@#11@#Default : 413@#TIMED_WAITING8@#15@#sun.misc.Unsafe4@#park11@#Unsafe.java-2@#38@#java.util.concurrent.locks.LockSupport9@#parkNanos16@#LockSupport.java226@#69@#java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject5@#await31@#AbstractQueuedSynchronizer.java2174@#42@#com.ibm.ws.util.BoundedBuffer$GetQueueLock5@#await18@#BoundedBuffer.java298@#29@#com.ibm.ws.util.BoundedBuffer8@#waitGet_18@#BoundedBuffer.java455@#29@#com.ibm.ws.util.BoundedBuffer4@#take18@#BoundedBuffer.java853@#26@#com.ibm.ws.util.ThreadPool7@#getTask15@#ThreadPool.java1059@#33@#com.ibm.ws.util.ThreadPool$Worker3@#run15@#ThreadPool.java1916@#0@#0@#78@#java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7939f80b
startThreads3.0.7@#1509624925622@#238@#18@#341@#20@#SIBFAPThreadPool : 38@#RUNNABLE5@#29@#com.ibm.io.async.AsyncLibrary12@#aio_getioev317@#AsyncLibrary.java-2@#29@#com.ibm.io.async.AsyncLibrary18@#getCompletionData317@#AsyncLibrary.java658@#30@#com.ibm.io.async.ResultHandler22@#runEventProcessingLoop18@#ResultHandler.java530@#32@#com.ibm.io.async.ResultHandler$23@#run18@#ResultHandler.java905@#33@#com.ibm.ws.util.ThreadPool$Worker3@#run15@#ThreadPool.java1892@#0@#0@#0@#
startThreads3.0.7@#1509624925622@#238@#22@#349@#20@#Deferrable Alarm : 08@#RUNNABLE6@#16@#java.lang.Object4@#wait11@#Object.java-2@#16@#java.lang.Object4@#wait11@#Object.java201@#46@#com.ibm.ws.wssecurity.util.timer.BoundedBuffer8@#waitGet_18@#BoundedBuffer.java168@#46@#com.ibm.ws.wssecurity.util.timer.BoundedBuffer4@#take18@#BoundedBuffer.java519@#43@#com.ibm.ws.wssecurity.util.timer.ThreadPool7@#getTask15@#ThreadPool.java893@#50@#com.ibm.ws.wssecurity.util.timer.ThreadPool$Worker3@#run15@#ThreadPool.java1617@#1@#16@#java.lang.Object1566472653@#0@#0@#0@#
system.sun.arch.data.model=64
system.com.ibm.packed.version=2
security.networkaddress.cache.ttl=
system.osgi.framework.extensions=com.ibm.cds,com.ibm.ws.eclipse.adaptors
environment.USER_INSTALL_PROP=-Duser.install.root=/opt/ibm/WebSphereProfiles/AppSrv01
system.java.awt.headless=true
native.library.date=Oct 17 2016 14:42:06
pid=42955
dump.options=system:events=gpf+abort+traceassert+corruptcache,file=/opt/ibm/WebSphereProfiles/AppSrv01/core.%Y%m%d.%H%M%S.%pid.%seq.dmp,range=1..0,priority=999,request=serial@@@system:events=systhrow,filter=java/lang/OutOfMemoryError,file=/opt/ibm/WebSphereProfiles/AppSrv01/core.%Y%m%d.%H%M%S.%pid.%seq.dmp,range=1..1,priority=999,request=exclusive+compact+prepwalk@@@heap:events=systhrow,filter=java/lang/OutOfMemoryError,file=/opt/ibm/WebSphereProfiles/AppSrv01/heapdump.%Y%m%d.%H%M%S.%pid.%seq.phd,range=1..4,priority=500,request=exclusive+compact+prepwalk,opts=PHD@@@java:events=gpf+user+abort+traceassert+corruptcache,file=/opt/ibm/WebSphereProfiles/AppSrv01/javacore.%Y%m%d.%H%M%S.%pid.%seq.txt,range=1..0,priority=400,request=exclusive+preempt@@@java:events=systhrow,filter=java/lang/OutOfMemoryError,file=/opt/ibm/WebSphereProfiles/AppSrv01/javacore.%Y%m%d.%H%M%S.%pid.%seq.txt,range=1..4,priority=400,request=exclusive+preempt@@@snap:events=gpf+abort+traceassert+corruptcache,file=/opt/ibm/WebSphereProfiles/AppSrv01/Snap.%Y%m%d.%H%M%S.%pid.%seq.trc,range=1..0,priority=300,request=serial@@@snap:events=systhrow,filter=java/lang/OutOfMemoryError,file=/opt/ibm/WebSphereProfiles/AppSrv01/Snap.%Y%m%d.%H%M%S.%pid.%seq.trc,range=1..4,priority=300,request=serial@@@jit:events=gpf+abort,file=/opt/ibm/WebSphereProfiles/AppSrv01/jitdump.%Y%m%d.%H%M%S.%pid.%seq.dmp,range=1..0,priority=200,request=serial@@@silent:events=allocation,filter=#10000000,file=-,range=1..0,priority=5,request=
==> /opt/ibm/WebSphereProfiles/AppSrv01/logs/AppClusterMember1/healthcenter/tmp_021117_120504_/memory <==
MemorySource,1509624857597,totalphysicalmemory=8185933824,physicalmemory=1271615488,privatememory=4676968448,virtualmemory=4781912064,freephysicalmemory=292753408
MemorySource,1509624863601,totalphysicalmemory=8185933824,physicalmemory=1271721984,privatememory=4676968448,virtualmemory=4781912064,freephysicalmemory=292544512
MemorySource,1509624869603,totalphysicalmemory=8185933824,physicalmemory=1271734272,privatememory=4676968448,virtualmemory=4781912064,freephysicalmemory=292659200
MemorySource,1509624875604,totalphysicalmemory=8185933824,physicalmemory=1271754752,privatememory=4676968448,virtualmemory=4781912064,freephysicalmemory=292532224
MemorySource,1509624881605,totalphysicalmemory=8185933824,physicalmemory=1271775232,privatememory=4676968448,virtualmemory=4781912064,freephysicalmemory=292253696
MemorySource,1509624887607,totalphysicalmemory=8185933824,physicalmemory=1271816192,privatememory=4676968448,virtualmemory=4781912064,freephysicalmemory=292134912
MemorySource,1509624893609,totalphysicalmemory=8185933824,physicalmemory=1271816192,privatememory=4676968448,virtualmemory=4781912064,freephysicalmemory=292167680
MemorySource,1509624899612,totalphysicalmemory=8185933824,physicalmemory=1271836672,privatememory=4676968448,virtualmemory=4781912064,freephysicalmemory=292802560
MemorySource,1509624905615,totalphysicalmemory=8185933824,physicalmemory=1271836672,privatememory=4676968448,virtualmemory=4781912064,freephysicalmemory=292818944
MemorySource,1509624911617,totalphysicalmemory=8185933824,physicalmemory=1271885824,privatememory=4676968448,virtualmemory=4781912064,freephysicalmemory=292548608
==> /opt/ibm/WebSphereProfiles/AppSrv01/logs/AppClusterMember1/healthcenter/tmp_021117_120504_/threads <==
startThreads3.0.7@#1509624925622@#238@#18@#321@#8@#Timer-187@#WAITING4@#16@#java.lang.Object4@#wait11@#Object.java-2@#16@#java.lang.Object4@#wait11@#Object.java172@#21@#java.util.TimerThread8@#mainLoop10@#Timer.java537@#21@#java.util.TimerThread3@#run10@#Timer.java516@#1@#19@#java.util.TaskQueue-138882087@#2@#0@#28@#java.util.TaskQueue@f7b8d3d9
startThreads3.0.7@#1509624925622@#238@#20@#322@#10@#Thread-1417@#WAITING5@#16@#java.lang.Object4@#wait11@#Object.java-2@#16@#java.lang.Object4@#wait11@#Object.java172@#44@#com.ibm.ws.wsgroup.p2p.SonP2PShimImpl$Worker7@#oneStep19@#SonP2PShimImpl.java1274@#44@#com.ibm.ws.wsgroup.p2p.SonP2PShimImpl$Worker3@#run19@#SonP2PShimImpl.java1213@#16@#java.lang.Thread3@#run11@#Thread.java785@#1@#43@#com.ibm.ws.wsgroup.p2p.SonP2PShimImpl$Lock1-619790904@#0@#0@#52@#com.ibm.ws.wsgroup.p2p.SonP2PShimImpl$Lock1@db0ebdc8
startThreads3.0.7@#1509624925622@#238@#24@#324@#16@#WebContainer : 013@#TIMED_WAITING8@#15@#sun.misc.Unsafe4@#park11@#Unsafe.java-2@#38@#java.util.concurrent.locks.LockSupport9@#parkNanos16@#LockSupport.java226@#69@#java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject5@#await31@#AbstractQueuedSynchronizer.java2174@#42@#com.ibm.ws.util.BoundedBuffer$GetQueueLock5@#await18@#BoundedBuffer.java298@#29@#com.ibm.ws.util.BoundedBuffer8@#waitGet_18@#BoundedBuffer.java455@#29@#com.ibm.ws.util.BoundedBuffer4@#take18@#BoundedBuffer.java853@#26@#com.ibm.ws.util.ThreadPool7@#getTask15@#ThreadPool.java1059@#33@#com.ibm.ws.util.ThreadPool$Worker3@#run15@#ThreadPool.java1916@#0@#0@#78@#java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@8fed6033
startThreads3.0.7@#1509624925622@#238@#18@#325@#16@#WebContainer : 18@#RUNNABLE5@#29@#com.ibm.io.async.AsyncLibrary12@#aio_getioev317@#AsyncLibrary.java-2@#29@#com.ibm.io.async.AsyncLibrary18@#getCompletionData317@#AsyncLibrary.java658@#30@#com.ibm.io.async.ResultHandler22@#runEventProcessingLoop18@#ResultHandler.java530@#32@#com.ibm.io.async.ResultHandler$23@#run18@#ResultHandler.java905@#33@#com.ibm.ws.util.ThreadPool$Worker3@#run15@#ThreadPool.java1892@#0@#0@#0@#
startThreads3.0.7@#1509624925622@#238@#24@#328@#16@#WebContainer : 213@#TIMED_WAITING8@#15@#sun.misc.Unsafe4@#park11@#Unsafe.java-2@#38@#java.util.concurrent.locks.LockSupport9@#parkNanos16@#LockSupport.java226@#69@#java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject5@#await31@#AbstractQueuedSynchronizer.java2174@#42@#com.ibm.ws.util.BoundedBuffer$GetQueueLock5@#await18@#BoundedBuffer.java298@#29@#com.ibm.ws.util.BoundedBuffer8@#waitGet_18@#BoundedBuffer.java455@#29@#com.ibm.ws.util.BoundedBuffer4@#take18@#BoundedBuffer.java853@#26@#com.ibm.ws.util.ThreadPool7@#getTask15@#ThreadPool.java1059@#33@#com.ibm.ws.util.ThreadPool$Worker3@#run15@#ThreadPool.java1916@#0@#0@#78@#java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5b3f22bd
startThreads3.0.7@#1509624925622@#238@#26@#330@#24@#ProductInsights-thread-77@#WAITING9@#15@#sun.misc.Unsafe4@#park11@#Unsafe.java-2@#38@#java.util.concurrent.locks.LockSupport4@#park16@#LockSupport.java186@#69@#java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject5@#await31@#AbstractQueuedSynchronizer.java2050@#65@#java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue4@#take32@#ScheduledThreadPoolExecutor.java1099@#65@#java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue4@#take32@#ScheduledThreadPoolExecutor.java820@#39@#java.util.concurrent.ThreadPoolExecutor7@#getTask23@#ThreadPoolExecutor.java1078@#39@#java.util.concurrent.ThreadPoolExecutor9@#runWorker23@#ThreadPoolExecutor.java1138@#46@#java.util.concurrent.ThreadPoolExecutor$Worker3@#run23@#ThreadPoolExecutor.java628@#16@#java.lang.Thread3@#run11@#Thread.java785@#0@#0@#78@#java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@ded3f825
startThreads3.0.7@#1509624925622@#238@#26@#331@#24@#ProductInsights-thread-87@#WAITING9@#15@#sun.misc.Unsafe4@#park11@#Unsafe.java-2@#38@#java.util.concurrent.locks.LockSupport4@#park16@#LockSupport.java186@#69@#java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject5@#await31@#AbstractQueuedSynchronizer.java2050@#65@#java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue4@#take32@#ScheduledThreadPoolExecutor.java1099@#65@#java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue4@#take32@#ScheduledThreadPoolExecutor.java820@#39@#java.util.concurrent.ThreadPoolExecutor7@#getTask23@#ThreadPoolExecutor.java1078@#39@#java.util.concurrent.ThreadPoolExecutor9@#runWorker23@#ThreadPoolExecutor.java1138@#46@#java.util.concurrent.ThreadPoolExecutor$Worker3@#run23@#ThreadPoolExecutor.java628@#16@#java.lang.Thread3@#run11@#Thread.java785@#0@#0@#78@#java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@ded3f825
startThreads3.0.7@#1509624925622@#238@#24@#332@#11@#Default : 413@#TIMED_WAITING8@#15@#sun.misc.Unsafe4@#park11@#Unsafe.java-2@#38@#java.util.concurrent.locks.LockSupport9@#parkNanos16@#LockSupport.java226@#69@#java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject5@#await31@#AbstractQueuedSynchronizer.java2174@#42@#com.ibm.ws.util.BoundedBuffer$GetQueueLock5@#await18@#BoundedBuffer.java298@#29@#com.ibm.ws.util.BoundedBuffer8@#waitGet_18@#BoundedBuffer.java455@#29@#com.ibm.ws.util.BoundedBuffer4@#take18@#BoundedBuffer.java853@#26@#com.ibm.ws.util.ThreadPool7@#getTask15@#ThreadPool.java1059@#33@#com.ibm.ws.util.ThreadPool$Worker3@#run15@#ThreadPool.java1916@#0@#0@#78@#java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7939f80b
startThreads3.0.7@#1509624925622@#238@#18@#341@#20@#SIBFAPThreadPool : 38@#RUNNABLE5@#29@#com.ibm.io.async.AsyncLibrary12@#aio_getioev317@#AsyncLibrary.java-2@#29@#com.ibm.io.async.AsyncLibrary18@#getCompletionData317@#AsyncLibrary.java658@#30@#com.ibm.io.async.ResultHandler22@#runEventProcessingLoop18@#ResultHandler.java530@#32@#com.ibm.io.async.ResultHandler$23@#run18@#ResultHandler.java905@#33@#com.ibm.ws.util.ThreadPool$Worker3@#run15@#ThreadPool.java1892@#0@#0@#0@#
startThreads3.0.7@#1509624925622@#238@#22@#349@#20@#Deferrable Alarm : 08@#RUNNABLE6@#16@#java.lang.Object4@#wait11@#Object.java-2@#16@#java.lang.Object4@#wait11@#Object.java201@#46@#com.ibm.ws.wssecurity.util.timer.BoundedBuffer8@#waitGet_18@#BoundedBuffer.java168@#46@#com.ibm.ws.wssecurity.util.timer.BoundedBuffer4@#take18@#BoundedBuffer.java519@#43@#com.ibm.ws.wssecurity.util.timer.ThreadPool7@#getTask15@#ThreadPool.java893@#50@#com.ibm.ws.wssecurity.util.timer.ThreadPool$Worker3@#run15@#ThreadPool.java1617@#1@#16@#java.lang.Object1566472653@#0@#0@#0@#
If needed, I can then pull individual files down to my Mac for offline analysis: -
scp wasadmin@bpm86:/opt/ibm/WebSphereProfiles/AppSrv01/logs/AppClusterMember1/healthcenter/tmp_021117_120504_/trace .
wasadmin@bpm86's password:
trace 100% 240KB 34.2MB/s 00:00
trace 100% 240KB 34.2MB/s 00:00
and then open that file directly into Eclipse: -
There's plenty of useful material about HC here: -
Enjoy !
No comments:
Post a Comment