WebSphere Application Server and the Case of the Bad File Store

We saw an interesting situation today, with a client's IBM WebSphere Application Server (WAS) 8.5.5.13 environment.

Whereas most clients, in my recent experience, tend to use a relational database such as DB2, Oracle or SQL Server, to host the Service Integration Bus (SIB) Messaging Engine (ME) datastore, this particular client was using the local Linux box's file-system.

When the cluster hosting the SIB came up, a message similar to this: -

CWSIS1561E:The messaging engine's unique id does not match that found in the filestore. ME_UUID=A85B64156006973D, ME_UUID(FS)=AAF19AC8C805638A

was posted in SystemOut.log.

Having checked the SIB/ME configuration, I could see that we had three file-system-based artefacts: -

/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/msglog/Log

/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/msgstore/PermanentStore

/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/tempmsgstore/TemporaryStore

which translates as: -

Log File

Permanent Store File

Temporary Store File

as per this: -

File stores

Thankfully, this developerWorks Answers post provided the insight: -

Why does my messaging engine fail to start after restarting WAS?

…

CWSIS1561E errors are usually caused by deleting the messaging engine, without deleting its filestore, then recreating the messaging engine with the same name. To fix the issue, you will have to delete the existing filestore files for the messaging engine and restart the JVM. A new filestore will be created during the messaging engine startup.

Note1: You must treat the log file, the temporary store file, and the permanent store file as one unit; that is, delete operation must be performed on all three files.

Note2:By performing the suggested action, all the messages in the filestore will be lost.

…

To be on the safe side, we backed up the existing files ( all of which were there, with the correct ownership and permissions ), and then deleted the subdirectories: -

/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/msglog/
/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/msgstore/
/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/tempmsgstore/

and bounced WAS.

Job done :-)

This also provided some useful context: -

Messaging Engine starting problem with the error message: CWSOM1042E