IBM WebSphere Application Server - Java Core Debugging using IBM Thread and Monitor Dump Analyzer for Java

And yet more on the theme of WAS performance, following this: -

IBM WebSphere Application Server - Tuning and Monitoring and Tuning - IBM Health Center

IBM WebSphere Application Server - Tuning and Monitoring and Tuning - IBM Garbage Collection and Memory Visualizer (GCMV)

WebSphere Application Server Performance Cookbook

we have this: -

IBM Thread and Monitor Dump Analyze for Java

Having downloaded the tool ( which comes down as a Java Archive  ), and run it from my Mac: -

I then triggered a Java core on one of my IBM BPM JVMs.

Firstly I checked the Process ID (PID) of the relevant JVM: -

ps -elf | grep AppClusterMember1

0 S wasadmin   7975   2113  2  80   0 - 1185215 futex_ 13:08 ?      00:08:22 /opt/ibm/WebSphere/AppServer/java_1.8_64/bin/java -Declipse.security -Dwas.status.socket=36374 -Dosgi.install.area=/opt/ibm/WebSphere/AppServer -Dosgi.configuration.area=/opt/ibm/WebSphereProfiles/AppSrv01/servers/AppClusterMember1/configuration -Dosgi.framework.extensions=com.ibm.cds,com.ibm.ws.eclipse.adaptors -Xshareclasses:name=webspherev85_1.8_64_%g,nonFatal -Dcom.ibm.xtq.processor.overrideSecureProcessing=true -Xcheck:dump -Djava.security.properties=/opt/ibm/WebSphere/AppServer/properties/java.security -Xbootclasspath/p:/opt/ibm/WebSphere/AppServer/java_1.8_64/jre/lib/ibmorb.jar -classpath /opt/ibm/WebSphereProfiles/AppSrv01/properties:/opt/ibm/WebSphere/AppServer/properties:/opt/ibm/WebSphere/AppServer/lib/startup.jar:/opt/ibm/WebSphere/AppServer/lib/bootstrap.jar:/opt/ibm/WebSphere/AppServer/lib/jsf-nls.jar:/opt/ibm/WebSphere/AppServer/lib/lmproxy.jar:/opt/ibm/WebSphere/AppServer/lib/urlprotocols.jar:/opt/ibm/WebSphere/AppServer/deploytool/itp/batchboot.jar:/opt/ibm/WebSphere/AppServer/deploytool/itp/batch2.jar:/opt/ibm/WebSphere/AppServer/java_1.8_64/lib/tools.jar -Dibm.websphere.internalClassAccessMode=allow -verbose:gc -Xms1536m -Xmx2560m -Xcompressedrefs -Xscmaxaot4M -Dws.ext.dirs=/opt/ibm/WebSphere/AppServer/java_1.8_64/lib:/opt/ibm/WebSphereProfiles/AppSrv01/classes:/opt/ibm/WebSphere/AppServer/classes:/opt/ibm/WebSphere/AppServer/lib:/opt/ibm/WebSphere/AppServer/installedChannels:/opt/ibm/WebSphere/AppServer/lib/ext:/opt/ibm/WebSphere/AppServer/web/help:/opt/ibm/WebSphere/AppServer/deploytool/itp/plugins/com.ibm.etools.ejbdeploy/runtime -Dderby.system.home=/opt/ibm/WebSphere/AppServer/derby -Dcom.ibm.itp.location=/opt/ibm/WebSphere/AppServer/bin -Djava.util.logging.configureByServer=true -Duser.install.root=/opt/ibm/WebSphereProfiles/AppSrv01 -Djava.ext.dirs=/opt/ibm/WebSphere/AppServer/tivoli/tam:/opt/ibm/WebSphere/AppServer/java_1.8_64/jre/lib/ext -Djavax.management.builder.initial=com.ibm.ws.management.PlatformMBeanServerBuilder -Dpython.cachedir=/opt/ibm/WebSphereProfiles/AppSrv01/temp/cachedir -Dwas.install.root=/opt/ibm/WebSphere/AppServer -Djava.util.logging.manager=com.ibm.ws.bootstrap.WsLogManager -Dserver.root=/opt/ibm/WebSphereProfiles/AppSrv01 -Dcom.ibm.security.jgss.debug=off -Dcom.ibm.security.krb5.Krb5Debug=off -Dcom.ibm.ws.webservices.startup.wsdl.suppressMessages=INFO -Dcom.ibm.ejs.j2c.J2CServerListener.deferEndpointActivation=true -Dcom.ibm.ws.runtime.enableRestartableCULibraries=true -DHttpSessionIdReuse=true -Dteamworks.tester.appServerPort=9081 -Dteamworks.server.home=/opt/ibm/WebSphereProfiles/AppSrv01/config/cells/PCCell1/nodes/Node1/servers/AppClusterMember1/process-center -Dorg.apache.axis.EngineConfigurationFactory=com.lombardisoftware.core.axis.configuration.EngineConfigurationFactoryProxy -Daxis.Compiler=org.apache.axis.components.compiler.Javac -Djava.awt.headless=true -Dclient.encoding.override=UTF-8 -Dcom.ibm.websphere.naming.jndi.security.jndionly=true -Dorg.eclipse.emf.common.util.URI.archiveSchemes=archive jar zip wsjar -Dcom.ibm.websphere.webservices.attachment.tempfile.expiration=600 -Xscmx120m -Xgcpolicy:gencon -Xjit:iprofilerMemoryConsumptionLimit=67108864 -Xmns256m -Xmnx768m -Xgc:preferredHeapBase=0x100000000 -Xdisableexplicitgc -Xss2048k -Dsun.net.http.allowRestrictedHeaders=true -Declipse.bundle.setTCCL=false -Xhealthcenter:transport=jrmp -Djava.library.path=/opt/ibm/WebSphere/AppServer/lib/native/linux/x86_64/:/opt/ibm/WebSphere/AppServer/java_1.8_64/jre/lib/amd64/compressedrefs:/opt/ibm/WebSphere/AppServer/java_1.8_64/jre/lib/amd64:/opt/ibm/WebSphere/AppServer/bin:/opt/ibm/WebSphere/AppServer/nulldllsdir:/usr/lib64:/usr/lib: -Djava.endorsed.dirs=/opt/ibm/WebSphere/AppServer/endorsed_apis:/opt/ibm/WebSphere/AppServer/java_1.8_64/jre/lib/endorsed -Djava.security.auth.login.config=/opt/ibm/WebSphereProfiles/AppSrv01/properties/wsjaas.conf -Djava.security.policy=/opt/ibm/WebSphereProfiles/AppSrv01/properties/server.policy com.ibm.wsspi.bootstrap.WSPreLauncher -nosplash -application com.ibm.ws.bootstrap.WSLauncher com.ibm.ws.runtime.WsServer /opt/ibm/WebSphereProfiles/AppSrv01/config PCCell1 Node1 AppClusterMember1
0 S wasadmin  28283  28235  0  80   0 - 28165 pipe_w 19:20 pts/0    00:00:00 grep --color=auto AppClusterMember1

and then killed the JVM: -

kill -3 7975

and then checked the core dump etc.

ls -altrc /opt/ibm/WebSphereProfiles/AppSrv01

total 9396
drwxr-xr-x  4 wasadmin wasadmins      34 Oct  9 13:23 ..
drwxr-xr-x  2 wasadmin wasadmins       6 Oct  9 13:24 installedConnectors
drwxr-xr-x  2 wasadmin wasadmins       6 Oct  9 13:24 installableApps
drwxr-xr-x  2 wasadmin wasadmins    4096 Oct  9 13:24 firststeps
drwxr-xr-x  3 wasadmin wasadmins    8192 Oct  9 13:25 bin
drwxr-xr-x  8 wasadmin wasadmins    4096 Oct  9 13:25 properties
drwxr-xr-x  2 wasadmin wasadmins       6 Oct  9 13:25 staticContent
drwxr-xr-x 10 wasadmin wasadmins     132 Oct  9 13:30 config
drwxr-xr-x  3 wasadmin wasadmins      20 Oct  9 13:30 installedApps
drwxr-xr-x  2 wasadmin wasadmins       6 Oct  9 13:43 expandedBundles
drwxr-xr-x  3 wasadmin wasadmins      16 Oct  9 13:44 installedFilters
drwxr-xr-x  5 wasadmin wasadmins      71 Oct  9 13:44 xd_discovery
drwxr-xr-x  3 wasadmin wasadmins    4096 Oct  9 13:44 etc
drwxr-xr-x  3 wasadmin wasadmins      20 Oct  9 13:44 tranlog
drwxr-xr-x  3 wasadmin wasadmins      20 Oct  9 13:45 recoveryLogs
drwxr-xr-x  3 wasadmin wasadmins      23 Oct  9 13:46 BusinessSpace
drwxr-xr-x  3 wasadmin wasadmins      51 Oct  9 13:47 FileNet
drwxr-xr-x  4 wasadmin wasadmins      32 Oct  9 13:49 searchIndex
drwxr-xr-x  4 wasadmin wasadmins      31 Oct  9 15:49 BusinessSpace.system
drwxr-xr-x 12 wasadmin wasadmins    4096 Oct 19 16:10 temp
drwxr-xr-x  7 wasadmin wasadmins    4096 Nov  1 11:53 configuration
drwxr-xr-x 18 wasadmin wasadmins    4096 Nov  1 12:02 workspace
drwxr-xr-x  7 wasadmin wasadmins     114 Nov  1 12:41 servers
-rw-r--r--  1 wasadmin wasadmins   11017 Nov  1 12:46 orbtrc.01112017.1246.38.txt
drwxr-xr-x  9 wasadmin wasadmins     124 Nov  1 14:00 logs
drwxr-xr-x  8 wasadmin wasadmins     134 Nov  1 14:21 wstemp
drwxr-xr-x 26 wasadmin wasadmins    4096 Nov  1 19:21 .
-rw-r--r--  1 wasadmin wasadmins 9568001 Nov  1 19:21 javacore.20171101.192106.7975.0001.txt

and then pulled the javacore onto my Mac: -

scp wasadmin@bpm86:/opt/ibm/WebSphereProfiles/AppSrv01/javacore* .

wasadmin@bpm86's password: 
javacore.20171101.192106.7975.0001.txt                                                                                                                                    100% 9344KB  36.1MB/s   00:00    

and then started the JCA ( Java Core Analyzer ) tool: -

java -jar jca457.jar 

I then used the tool to open up the javacore: -

and then started analysing: -

Which is nice :-)