Hmmm, Process Portal Not Playing Nicely - IBM BPM Advanced 8.5.5.0

Problem

Hitting Process Portal, either via IHS ( on port 8443 ) or directly ( on port 9443 / 9446 ) results in a garbled page, with most of the artefacts down the left-hand side of the page.
Problem only seen when one explicitly logs into Process Portal

Browser ( IE or Firefox ) makes no difference.

Direct - via AppClusterMember1

https://bpm855.uk.ibm.com:9443/ProcessPortal/jsp/index.jsp

Direct - via AppClusterMember2

https://bpm855.uk.ibm.com:9446/ProcessPortal/jsp/index.jsp

IHS

https://bpm855.uk.ibm.com:8443/ProcessPortal/jsp/index.jsp

Exceptions seen in AppClusterMemberX SystemOut.log

...
[19/10/15 10:13:37:834 BST] 000001c9 JMSClassifica W   XJMS0021E: Unable to map destination and bus to workclass. Aborting the xd classification filter for the current request. Request context information is com.ibm.ws.xd.jms.proxy.impl.AsynchJMSServiceContextImpl@b7558001<bus=BPM.PSCell1De1.Bus,destination=PortalWebMessagingTopicSpace.AppCluster>.
...
[19/10/15 10:18:27:625 BST]     FFDC Exception:java.io.IOException SourceId:com.ibm.ws.webcontainer.channel.WCCByteBufferInputStream
ProbeId:102 Reporter:com.ibm.ws.webcontainer.channel.WCCByteBufferInputStream@af8277d8
java.io.IOException: Async IO operation failed (1), reason: RC: 76  Socket is not connected
        at com.ibm.io.async.AsyncLibrary$IOExceptionCache.<init>(AsyncLibrary.java:924)
        at com.ibm.io.async.AsyncLibrary$IOExceptionCache.get(AsyncLibrary.java:937)
...
[19/10/15 11:43:34:983 BST] 0000014a JSONTransport W org.cometd.server.transport.LongPollingTransport handleJSONParseException Error
parsing JSON:
                                 java.lang.RuntimeException: java.io.IOException: Async IO operation failed (3), reason: RC: 76  Socket is not connected
        at org.eclipse.jetty.util.ajax.JSON$ReaderSource.getNext(JSON.java:1519)
        at org.eclipse.jetty.util.ajax.JSON$ReaderSource.hasNext(JSON.java:1486)
        at org.eclipse.jetty.util.ajax.JSON.parse(JSON.java:789)
        at org.cometd.common.JettyJSONContext.parse(JettyJSONContext.java:57)
        at org.cometd.server.AbstractServerTransport.parseMessages(AbstractServerTransport.java:154)
        at org.cometd.server.transport.JSONTransport.parseMessages(JSONTransport.java:72)
        at org.cometd.server.transport.LongPollingTransport.handle(LongPollingTransport.java:192)
        at org.cometd.server.CometdServlet.service(CometdServlet.java:166)
        at com.ibm.bpm.socialbus.servlet.SocialBusServiceServlet.service(SocialBusServiceServlet.java:183)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
        at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1230)
        at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:779)
        at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:478)
        at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)
        at com.ibm.ws.webcontainer.filter.WebAppFilterChain.invokeTarget(WebAppFilterChain.java:136)
        at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:97)
        at com.ibm.bpm.servlet.filters.GenericSecurityServletFilter.doFilter(GenericSecurityServletFilter.java:84)
        at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195)
        at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:91)
        at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:964)
        at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1104)
        at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:3901)
        at com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(WebGroup.java:304)
        at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:981)
        at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1662)
        at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:200)
        at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:463)
        at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:530)
...

Potential Indication of Root Cause - java.io.IOException: Async IO operation failed (1), reason: RC: 76  Socket is not connected

Feels like client-server issue, perhaps problem at IP stack

Definitely red herring - known "problem" covered in multiple Technotes

[19/10/15 11:17:04:847 BST] 00000141 JMSClassifica W   XJMS0021E: Unable to map destination and bus to workclass. Aborting the xd classification filter for the current
request. Request context information is

com.ibm.ws.xd.jms.proxy.impl.AsynchJMSServiceContextImpl@fe11fe03<bus=BPM.PSCell1De1.Bus,destination=PortalWebMessagingTopicSpace.AppCluster>.

A XJMS0021E error occurs in the SystemOut.log file for IBM Business Process Manager (BPM) Version 8.5

XJMS0021E destination and bus workclass problem is logged to WAS systemOUT

Further Analysis

Used Firefox Debugger ( [Fn] [F12] )

Solution

Force BusinessSpace to reload BusinessSpace - templates, spaces and themes

Edit: -

/opt/ibm/WebSphereProfiles/PSCell1AppSrv01/BusinessSpace/AppCluster/mm.runtime.prof/public/oobLoadedStatus.properties

and change from: -

#Mon Oct 19 14:08:27 BST 2015
importTemplates.txt=false
importSpaces.txt=false
importThemes.txt=false

to: -

#Mon Oct 19 14:08:27 BST 2015
importTemplates.txt=true
importSpaces.txt=true
importThemes.txt=true

and restart AppCluster

( File is then reset back to false setting once cluster has been restarted - once templates, spaces etc. have been reset to their OOB defaults )

Problem does *NOT* appear to recur, including java.io.IOException exception

Re-tested Process Portal ( both direct and via IHS )

Direct - via AppClusterMember1

https://bpm855.uk.ibm.com:9443/ProcessPortal/jsp/index.jsp

Direct - via AppClusterMember2

https://bpm855.uk.ibm.com:9446/ProcessPortal/jsp/index.jsp

IHS

https://bpm855.uk.ibm.com:8443/ProcessPortal/jsp/index.jsp

Also tested Business Space

Direct - via AppClusterMember1

https://bpm855.uk.ibm.com:9443/mum/enabler

Direct - via AppClusterMember2

https://bpm855.uk.ibm.com:9446/mum/enabler

IHS

https://bpm855.uk.ibm.com:8443/mum/enabler