ADML0004E: An exception occurred when attempting to expand variable ${IBMSCMX} com.ibm.wsspi.runtime.variable.UndefinedVariableException: Undefined variable IBMSCMX

Having installed BPM 8.6 CF2017-12, as part of my ongoing testing of IBM BPM 8.5.5 to 8.6 side-by-side upgrades, I started seeing: -

Continuing with unexpanded genericJvmArguments '-Xscmx120m -Xgcpolicy:gencon -Xjit:iprofilerMemoryConsumptionLimit=67108864 ${IBMSCMX} ${IBMGCPOLICY_GENCON} ${IBMJITPMC}'
ADML0004E: An exception occurred when attempting to expand variable ${IBMSCMX} com.ibm.wsspi.runtime.variable.UndefinedVariableException: Undefined variable IBMSCMX

ADML0004E: An exception occurred when attempting to expand variable ${IBMGCPOLICY_GENCON} com.ibm.wsspi.runtime.variable.UndefinedVariableException: Undefined variable IBMGCPOLICY_GENCON

ADML0004E: An exception occurred when attempting to expand variable ${IBMJITPMC} com.ibm.wsspi.runtime.variable.UndefinedVariableException: Undefined variable IBMJITPMC

This manifested itself whenever I ran a Java process such as startManager.sh or startNode.sh or DBUpgrade.sh

For the record, this is what I had installed: -

com.ibm.cic.agent_1.8.8000.20171130_1105
com.ibm.bpm.ADV.v85_8.6.201712.20171211_1726
com.ibm.websphere.ND.v85_8.5.5012.20170627_1018
8.5.5.12-WS-WASND-IFPI82630_8.5.5012.20170718_1447

As part of my testing, I'm using the Deployment Environment exported from a BPM 8.5.5 environment as input into the 8.6 DE creation: -

/opt/ibm/WebSphere/AppServer/bin/BPMConfig.sh -create -de ~/BPM855DE/De1.properties

Interestingly, this was NOT a problem when I had the base 8.6 binary ( com.ibm.bpm.ADV.v85_8.6.0.20170918_1207 ) installed.

In essence, the problem was that WAS was looking for three variables: -

IBMJITPMC

IBMGCPOLICY_GENCON

IBMSCMX

none of which existed within the WAS cell configuration.

I "cheated" by adding the following three lines to the resources.xml : -

vi /opt/ibm/WebSphereProfiles/Dmgr01/config/cells/PCCell1/nodes/Dmgr/variables.xml
vi /opt/ibm/WebSphereProfiles/Dmgr01/config/cells/PCCell1/nodes/Node1/variables.xml

appending: -

<entries xmi:id="VariableSubstitutionEntry_1515008456894" symbolicName="IBMJITPMC" value="-Xjit:iprofilerMemoryConsumptionLimit=67108864"/>
<entries xmi:id="VariableSubstitutionEntry_1515008456717" symbolicName="IBMGCPOLICY_GENCON" value="-Xgcpolicy:gencon"/>
<entries xmi:id="VariableSubstitutionEntry_1515008456476" symbolicName="IBMSCMX" value="-Xscmx120m"/>

in both cases.

However, that felt too much like a "hack" so I dug further.

Looking back at the De1.properties file that I'd generated from the 8.5.5 environment: -

/opt/ibm/WebSphere/AppServer/bin/BPMConfig.sh -export -profile Dmgr01 -de De1 -outputDir ~/BPM855DE

I noticed this: -

##################
# JVM properties #
##################
# Do not modify this value. If running on z/OS, the name of the process definition these settings are for. On non-z/OS platforms, this value should be empty.
bpm.dmgr.jvmSettings.1.name=
bpm.dmgr.jvmSettings.1.jvmArgs=${IBMSCMX} ${IBMGCPOLICY_GENCON} ${IBMJITPMC}
bpm.dmgr.jvmSettings.1.initialHeapSize=768
bpm.dmgr.jvmSettings.1.maximumHeapSize=2048
bpm.dmgr.jvmSettings.1.verboseModeGarbageCollection=false

##################
# JVM properties #
##################
# Do not modify this value. If running on z/OS, the name of the process definition these settings are for. On non-z/OS platforms, this value should be empty.
bpm.de.node.1.clusterMember.1.jvmSettings.1.name=
bpm.de.node.1.clusterMember.1.jvmSettings.1.jvmArgs=${IBMSCMX} ${IBMGCPOLICY_GENCON} ${IBMJITPMC}
bpm.de.node.1.clusterMember.1.jvmSettings.1.initialHeapSize=
bpm.de.node.1.clusterMember.1.jvmSettings.1.maximumHeapSize=
bpm.de.node.1.clusterMember.1.jvmSettings.1.verboseModeGarbageCollection=false

##################
# JVM properties #
##################
# Do not modify this value. If running on z/OS, the name of the process definition these settings are for. On non-z/OS platforms, this value should be empty.
bpm.de.node.1.clusterMember.2.jvmSettings.1.name=
bpm.de.node.1.clusterMember.2.jvmSettings.1.jvmArgs=${IBMSCMX} ${IBMGCPOLICY_GENCON} ${IBMJITPMC} -Dsun.net.http.allowRestrictedHeaders=true
bpm.de.node.1.clusterMember.2.jvmSettings.1.initialHeapSize=768
bpm.de.node.1.clusterMember.2.jvmSettings.1.maximumHeapSize=2048
bpm.de.node.1.clusterMember.2.jvmSettings.1.verboseModeGarbageCollection=false

##################
# JVM properties #
##################
# Do not modify this value. If running on z/OS, the name of the process definition these settings are for. On non-z/OS platforms, this value should be empty.
bpm.de.node.1.clusterMember.3.jvmSettings.1.name=
bpm.de.node.1.clusterMember.3.jvmSettings.1.jvmArgs=${IBMSCMX} ${IBMGCPOLICY_GENCON} ${IBMJITPMC}
bpm.de.node.1.clusterMember.3.jvmSettings.1.initialHeapSize=768
bpm.de.node.1.clusterMember.3.jvmSettings.1.maximumHeapSize=2048
bpm.de.node.1.clusterMember.3.jvmSettings.1.verboseModeGarbageCollection=false

In other words, the variables had been generated by the -export process.

For some reason, these didn't cause a problem for the September 2017 release of 8.6, BUT did cause a problem with the December 2017 release.

I'm not 100% sure why.

However, the mitigation was to remove the references in the De1.properties, and re-do the DE creation process.

*UPDATE* Having checked with a colleague in IBM, this is a known issue, which should be mitigated in the CF2018-03 Cumulative Fix. In the meantime, the circumvention is to manually amend the exported DE configuration, and remove references to the IBMSCMX, IBMGBPOLICY_GENCON and IBMJITPMC variables is the way to go.

If in doubt, please raise a PMR with IBM Support *UPDATE*