RestCEIAuthLocator: security.xml not found

Hmmm, so I saw this during the startup of a previously more-than-happy WAS cluster member, specifically a member of my IBM BPM  8.5.5 Support cluster: -

[17/08/15 08:33:46:944 BST] 00000001 ModelMgr      I   WSVR0801I: Initializing all server configuration models
[17/08/15 08:33:47:042 BST] 00000001 Ffdc          I com.ibm.ffdc.util.provider.FfdcOnDirProvider logIncident FFDC1003I: FFDC Incident emitted on /opt/ibm/WebSphereProfiles/PSCell1AppSrv01/logs/ffdc/ffdc.2593104506022466010.txt com.ibm.wbimonitor.authutils.RestCEIAuthLocator.initialize 50
[17/08/15 08:33:47:046 BST] 00000001 SystemOut     O RestCEIAuthLocator: security.xml not found: java.lang.IllegalStateException: java.lang.IllegalStateException: java.lang.IllegalStateException: java.lang.reflect.InvocationTargetException
[17/08/15 08:33:47:046 BST] 00000001 ModelMgr      I   WSVR0801I: Initializing all server configuration models
[17/08/15 08:33:47:196 BST] 00000001 ModelMgr      I   WSVR0801I: Initializing all server configuration models
[17/08/15 08:33:47:306 BST] 00000001 wtp           E com.ibm.etools.commonarchive.impl.CommonarchiveFactoryImpl initializeExtensions ERROR: Bindings & Extensions failed to initialize.
[17/08/15 08:33:47:374 BST] 00000001 ModelMgr      I   WSVR0800I: Initializing core configuration models
[17/08/15 08:33:47:379 BST] 00000001 WsServerImpl  E   WSVR0009E: Error occurred during startup

This was a surprise, given that the cluster member had been starting OK previously, BUT I had been making some configuration changes last week, when I enabled transaction/partner logging in DB2.

I did the usual things, including turning it off and on again (!), and also comparing / contrasting the configuration between it and it's happily working counterpart. I also fully resynchronised the node configuration from the Deployment Manager.

But to no avail .....

Even Google did NOT have the answer this time - I did find this: -

Cannot start Application Server

but, alas, the answer wasn't the one for which I was looking: -

<snip>

Hi, It looks like your App Server may be corrupted.  

[5/19/15 12:01:28:260 GMT+02:00] 00000001 SystemOut     O RestCEIAuthLocator: security.xml not found: java.lang.IllegalStateException: java.lang.IllegalStateException: java.lang.IllegalStateException: java.lang.reflect.InvocationTargetException

the security.xml file is missing for some reason.  Try doing a full sync from the node to the dmgr.  If that does not work, you can probably copy the file over from the dmgr to the node.

</snip>

as I'd already checked that (a) security.xml was definitely available to the failing JVM and (b) that it was identical to the cell-level copy on the Deployment Manager AND the other (working) node.

I then checked a few internal IBM sites, and found something that caught my eye, pertaining to a BPM JVM failing to start AFTER IBM Tivoli Composite Application Manager (ITCAM) had been installed.

Given that we DO use ITCAM's successor, IBM Application Performance Management, I read on further, and found reference to this ITCAM APAR: -

PI27187: WEBSPHERE BPM SERVER FAILS TO RESTART AFTER TERMINATING ABNORMALLY (E.G. CRASH, KILL),

Whilst this references a specific ITCAM fix, it also mentioned a circumvention: -

Local Fix/Workaround:

   Clearing the OSGi cache using the

   <profile_home>/bin/osgiCfgInit.sh

   prior to restarting the server instance resolves the issue.

I took the chance, and ran: -

/opt/IBM/WebSphereProfiles/AppSrv01/bin/osgiCfgInit.sh

which cleared the OSGI cache for ALL the JVMs in that particular profile, including the failing SupClusterMember1, and then tried to start the JVM again.

This time .... yep, it came up clean and green.

So I don't know precisely what happened, and I will be checking the ITCAM fix with my APM colleagues, in case there's a corresponding APM patch.

But I have a working cluster, and that's all good then.