Problem with BusinessSpace after IBM Business Monitor 8.0.1.0 to 8.0.1.2 upgrade

So this problem has parallels with an issue about which I blogged a year or so ago: -

Problems with IBM Business Process Manager ( Business Space ) following 8.0.1.1 upgrade 

albeit for IBM Business Monitor rather than IBM Business Process Manager.

This is what I see in Business Space: -

This is what I see in my logs: -

/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/logs/DepEnvBAM801.WebApp.AppSrv01Node.0/SystemOut.log

...

[18/02/14 15:28:23:529 GMT] 00000031 ControllerSer I com.ibm.mm.framework.rest.next.servlet.ControllerServlet logException A checked exception of type [com.ibm.mm.framework.rest.next.exception.LocalizedIOException] occurred during the resolution process, do not log the exception trace to avoid log file spamming. Enable trace logging for [com.ibm.mm.framework.rest.next.exception.ExceptionHelper=all] to log the exceptions. The exception message was: [404: Space not found.]

[18/02/14 15:28:24:000 GMT] 00000031 ControllerSer I com.ibm.mm.framework.rest.next.servlet.ControllerServlet logException A checked exception of type [com.ibm.wps.resolver.servlet.exceptions.SAXIOException] occurred during the resolution process, do not log the exception trace to avoid log file spamming. Enable trace logging for [com.ibm.mm.framework.rest.next.exception.ExceptionHelper=all] to log the exceptions. The exception message was: [400: Illegal character in authority at index 7: http://{com.ibm.bspace}mmServerRootId/widget-catalog/templateLayout.xml?pragma=cache&max-age=1209600&cache-scope=public&vary=none&user-context=false]

…

So, assuming that the root cause of the problem was the same as before, I thought I'd force BusinessSpace to reset it's templates/spaces/themes, as follows: -

Stop the cluster

/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/bin/stopServer.sh DepEnvBAM801.WebApp.AppSrv01Node.0

Change the oobLoadedStatus.properties file

/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/BusinessSpace/DepEnvBAM801.WebApp/mm.runtime.prof/public/oobLoadedStatus.properties 

from: -

#Tue Feb 18 15:23:39 GMT 2014

importTemplates.txt=false

importSpaces.txt=false

importThemes.txt=false

to: -

#Tue Feb 18 15:23:39 GMT 2014

importTemplates.txt=true

importSpaces.txt=true

importThemes.txt=true

Start the cluster

/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/bin/startServer.sh DepEnvBAM801.WebApp.AppSrv01Node.0

Sadly the same problem with BS persists, BUT we're down to one exception: -

...

[18/02/14 16:12:50:011 GMT] 00000022 ControllerSer I com.ibm.mm.framework.rest.next.servlet.ControllerServlet logException A checked exception of type [com.ibm.wps.resolver.servlet.exceptions.SAXIOException] occurred during the resolution process, do not log the exception trace to avoid log file spamming. Enable trace logging for [com.ibm.mm.framework.rest.next.exception.ExceptionHelper=all] to log the exceptions. The exception message was: [400: Illegal character in authority at index 7: http://{com.ibm.bspace}mmServerRootId/widget-catalog/templateLayout.xml?pragma=cache&max-age=1209600&cache-scope=public&vary=none&user-context=false]

…

When I looked at the exception in more detail: -

The exception message was: [400: Illegal character in authority at index 7: http://{com.ibm.bspace}mmServerRootId/widget-catalog/templateLayout.xml?pragma=cache&max-age=1209600&cache-scope=public&vary=none&user-context=false]

I wondered if the problem MIGHT be with the REST endpoints, so I checked the REST configuration: -

That could definitely be the issue as port 8443 is an HTTPS port, whereas I have the protocol specified as HTTP.

Having changed the protocol to HTTPS I restarted the cluster again: -

/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/bin/stopServer.sh DepEnvBAM801.WebApp.AppSrv01Node.0

/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/bin/startServer.sh DepEnvBAM801.WebApp.AppSrv01Node.0

but to no avail :-(

I still the same spinny thing, and the same exception.

I again did the oobLoadedStatus.properties shuffle but again to no avail :-(

In order to double-check whether the problem was related to the front-end IBM HTTP Server (IHS) box, I also confirmed that I see the same issue when accessing BusinessSpace directly: -

https://rhel6.uk.ibm.com:9446/BusinessSpace

Finally, I re-ran the process to upgrade the WebApp cluster: -

cd /opt/IBM/WebSphere/AppServer
./bin/ws_ant.sh -f util/BPMProfileUpgrade.ant -profileName AppSrv01 -Dupgrade=true -Dcluster=DepEnvBAM801.WebApp

but no change :-(

Sadly, I've NOT been able to work out WHY this is occurring.

Rather than spending yet more time on, I'm going to roll back to my last VMware snapshot, and then re-run the update process.

As a complete aside: -

I then checked the logs for the Support cluster, and found: -

com.ibm.ws.exception.RuntimeWarning: com.ibm.ws.webcontainer.exception.WebAppNotLoadedException: Failed to load webapp: Failed to load webapp: Context root /rest/bpm/events/* is already bound. Cannot start application EventEmitterREST

I have two applications: -

one of which is failing to start, both of which listen on the same context root - /rest/bpm/events.

That was the root (!) cause; the first application - IBM_BPM_EMITTER_SERVICE - should've had a context root of /unsupported. Once I changed that, the application started normally :-)

I suspect that was (l)user error - possibly something I mucked up during the upgrade :-(

**UPDATE 5 MARCH 2014**

So, thanks to some excellent support from IBM Support via a PMR, I managed to work out what I was doing wrong.

It was all down to the fact that I was attempting to run the cluster upgrade process against the AppSrv01 profile rather than the Dmgr01 profile.

That also explains why I thought that it was necessary to have the DM started in order for the cluster upgrade process to complete.

The commands to upgrade the clusters needed to be as follows: -

/opt/IBM/WebSphere/AppServer/bin/ws_ant.sh -f /opt/IBM/WebSphere/AppServer/util/BPMProfileUpgrade.ant -profileName Dmgr01 -Dupgrade=true -Dcluster=DepEnvBAM801.AppTarget

/opt/IBM/WebSphere/AppServer/bin/ws_ant.sh -f /opt/IBM/WebSphere/AppServer/util/BPMProfileUpgrade.ant -profileName Dmgr01 -Dupgrade=true -Dcluster=DepEnvBAM801.Support

/opt/IBM/WebSphere/AppServer/bin/ws_ant.sh -f /opt/IBM/WebSphere/AppServer/util/BPMProfileUpgrade.ant -profileName Dmgr01 -Dupgrade=true -Dcluster=DepEnvBAM801.Messaging

/opt/IBM/WebSphere/AppServer/bin/ws_ant.sh -f /opt/IBM/WebSphere/AppServer/util/BPMProfileUpgrade.ant -profileName Dmgr01 -Dupgrade=true -Dcluster=DepEnvBAM801.WebApp

Having run back through the 8.0.1.0 to 8.0.1.2 process, I no longer see the issue with Business Space, which is nice.

Lesson learned, thanks to IBM Support :-)