java.lang.UnsatisfiedLinkError: JCAM_Crypto_JNI (libCCLCore.so: cannot open shared object file: No such file or directory)

This caught me out during my scripted installation of IBM Business Monitor 8.0.1 ( with a back-end DB2 LUW 9.7.0.5 database ) on Red Hat Enterprise Linux.

During the startup of my clustered environment, I saw this in the SystemOut.log: -

….

[31/01/13 15:51:27:379 GMT] 00000015 FfdcProvider  W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/logs/ffdc/PCSR011.Support_858d66b1_13.01.31_15.51.27.3254262725509703725002.txt com.ibm.ws.webcontainer.servlet.ServletInstance.init 181
[31/01/13 15:51:27:381 GMT] 00000015 servlet       E com.ibm.ws.webcontainer.servlet.ServletWrapper init Uncaught.init.exception.thrown.by.servlet
[31/01/13 15:51:27:387 GMT] 00000015 webapp        E com.ibm.ws.webcontainer.webapp.WebApp commonInitializationFinally SRVE0266E: Error occured while initializing servlets: {0}
                                 javax.servlet.ServletException: SRVE0207E: Uncaught initialization exception created by servlet
        at com.ibm.ws.webcontainer.servlet.ServletWrapper.init(ServletWrapper.java:398)
        at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.init(ServletWrapperImpl.java:168)
        at com.ibm.ws.webcontainer.servlet.ServletWrapper.loadOnStartupCheck(ServletWrapper.java:1351)
        at com.ibm.ws.webcontainer.webapp.WebApp.doLoadOnStartupActions(WebApp.java:609)
        at com.ibm.ws.webcontainer.webapp.WebApp.commonInitializationFinally(WebApp.java:578)
        at com.ibm.ws.webcontainer.webapp.WebAppImpl.initialize(WebAppImpl.java:421)
        at com.ibm.ws.webcontainer.webapp.WebGroupImpl.addWebApplication(WebGroupImpl.java:88)
        at com.ibm.ws.webcontainer.VirtualHostImpl.addWebApplication(VirtualHostImpl.java:169)
        at com.ibm.ws.webcontainer.WSWebContainer.addWebApp(WSWebContainer.java:749)
        at com.ibm.ws.webcontainer.WSWebContainer.addWebApplication(WSWebContainer.java:634)
        at com.ibm.ws.webcontainer.component.WebContainerImpl.install(WebContainerImpl.java:422)
        at com.ibm.ws.webcontainer.component.WebContainerImpl.start(WebContainerImpl.java:714)
        at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:1160)
        at com.ibm.ws.runtime.component.DeployedApplicationImpl.fireDeployedObjectStart(DeployedApplicationImpl.java:1369)
        at com.ibm.ws.runtime.component.DeployedModuleImpl.start(DeployedModuleImpl.java:639)
        at com.ibm.ws.runtime.component.DeployedApplicationImpl.start(DeployedApplicationImpl.java:967)
        at com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplication(ApplicationMgrImpl.java:766)
        at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:2153)
        at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:445)
        at com.ibm.ws.runtime.component.CompositionUnitImpl.start(CompositionUnitImpl.java:123)
        at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:388)
        at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.access$500(CompositionUnitMgrImpl.java:116)
        at com.ibm.ws.runtime.component.CompositionUnitMgrImpl$CUInitializer.run(CompositionUnitMgrImpl.java:994)
        at com.ibm.wsspi.runtime.component.WsComponentImpl$_AsynchInitializer.run(WsComponentImpl.java:349)
        at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1690)
Caused by: java.lang.UnsatisfiedLinkError: JCAM_Crypto_JNI (libCCLCore.so: cannot open shared object file: No such file or directory)
        at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1053)
        at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:1017)
        at java.lang.System.loadLibrary(System.java:509)
        at com.cognos.accman.jcam.crypto.jni.JNISystemProperties.<clinit>(JNISystemProperties.java:27)
        at java.lang.J9VMInternals.initializeImpl(Native Method)
        at java.lang.J9VMInternals.initialize(J9VMInternals.java:228)
        at com.cognos.accman.jcam.crypto.SystemProtectionSession._getRawKey(SystemProtectionSession.java:244)
        at com.cognos.accman.jcam.crypto.SystemProtectionSession.<init>(SystemProtectionSession.java:57)
        at com.cognos.accman.jcam.crypto.misc.Configuration.<init>(Configuration.java:70)
        at com.cognos.accman.jcam.crypto.misc.Configuration.getInstanceWithDefaultConfig(Configuration.java:113)
        at com.cognos.accman.jcam.crypto.misc.JVMEnvironment.isFIPSSupportedOS(JVMEnvironment.java:212)
        at com.cognos.accman.jcam.crypto.CAMFactory.checkFIPSSupport(CAMFactory.java:2814)
        at com.cognos.accman.jcam.crypto.CAMFactory.initialize(CAMFactory.java:125)
        at com.cognos.indications.LogIPFControl.initCAMCrypto(LogIPFControl.java:566)
        at com.cognos.indications.LogIPFControl.initialize(LogIPFControl.java:255)
        at com.cognos.indications.LogIPFControl.start(LogIPFControl.java:235)
        at com.cognos.p2plb.config.ConfigStoreServlet.init(ConfigStoreServlet.java:122)
        at com.cognos.pogo.isolation.ServletWrapper.initServlet(ServletWrapper.java:121)
        at com.cognos.pogo.isolation.ServletWrapper.init(ServletWrapper.java:101)
        at com.ibm.ws.webcontainer.servlet.ServletWrapper.init(ServletWrapper.java:336)
        ... 24 more

[31/01/13 15:51:27:397 GMT] 00000015 webcontainer  I com.ibm.ws.webcontainer.VirtualHostImpl addWebApplication SRVE0250I: Web Module IBM Cognos has been bound to default_host[*:8444,*:8443,*:443].
[31/01/13 15:51:27:466 GMT] 00000015 ApplicationMg A   WSVR0221I: Application started: IBM Cognos

….

Following this Technote: -

Unable to run cogconfig.sh: Can't find library JCAM_Crypto_JNI

I used the ldd command against the Linux library that was failing to load - libCCLCore.so : -

$ locate libCCLCore.so

/opt/IBM/WebSphere/AppServer/cognos/bin/libCCLCore.so
/opt/IBM/WebSphere/AppServer/cognos/bin64/libCCLCore.so
/opt/IBM/WebSphere/AppServer/cognos/cgi-bin/libCCLCore.so

$ ldd /opt/IBM/WebSphere/AppServer/cognos/bin/libCCLCore.so

linux-gate.so.1 =>  (0x00644000)
libz.so.1 => /lib/libz.so.1 (0x001c0000)
libicucogi18n.so.40 => not found
libicucoguc.so.40 => not found
libX11.so.6 => /usr/lib/libX11.so.6 (0x00e3b000)
libdl.so.2 => /lib/libdl.so.2 (0x00ba7000)
libnsl.so.1 => /lib/libnsl.so.1 (0x00a76000)
libpthread.so.0 => /lib/libpthread.so.0 (0x006b3000)
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00395000)
libm.so.6 => /lib/libm.so.6 (0x0014d000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00177000)
libc.so.6 => /lib/libc.so.6 (0x001d4000)
libxcb.so.1 => /usr/lib/libxcb.so.1 (0x00195000)
/lib/ld-linux.so.2 (0x008ca000)
libXau.so.6 => /usr/lib/libXau.so.6 (0x001b3000)

$ ldd /opt/IBM/WebSphere/AppServer/cognos/bin64/libCCLCore.so

linux-vdso.so.1 =>  (0x00007fff04aac000)
libz.so.1 => /lib64/libz.so.1 (0x00007ffbf50c3000)
libicucogi18n.so.40 => not found
libicucoguc.so.40 => not found
libX11.so.6 => /usr/lib64/libX11.so.6 (0x00007ffbf4d83000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007ffbf4b7e000)
libnsl.so.1 => /lib64/libnsl.so.1 (0x00007ffbf4965000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ffbf4748000)
libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007ffbf4441000)
libm.so.6 => /lib64/libm.so.6 (0x00007ffbf41bd000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007ffbf3fa7000)
libc.so.6 => /lib64/libc.so.6 (0x00007ffbf3c13000)
libxcb.so.1 => /usr/lib64/libxcb.so.1 (0x00007ffbf39f8000)
/lib64/ld-linux-x86-64.so.2 (0x00000032d4400000)
libXau.so.6 => /usr/lib64/libXau.so.6 (0x00007ffbf37f4000)

$ ldd /opt/IBM/WebSphere/AppServer/cognos/cgi-bin/libCCLCore.so

linux-gate.so.1 =>  (0x006cb000)
libz.so.1 => /lib/libz.so.1 (0x00e73000)
libicucogi18n.so.40 => not found
libicucoguc.so.40 => not found
libX11.so.6 => /usr/lib/libX11.so.6 (0x00952000)
libdl.so.2 => /lib/libdl.so.2 (0x00cf3000)
libnsl.so.1 => /lib/libnsl.so.1 (0x00da1000)
libpthread.so.0 => /lib/libpthread.so.0 (0x002d2000)
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x002ed000)
libm.so.6 => /lib/libm.so.6 (0x0088d000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x003d8000)
libc.so.6 => /lib/libc.so.6 (0x003f6000)
libxcb.so.1 => /usr/lib/libxcb.so.1 (0x0058c000)
/lib/ld-linux.so.2 (0x008ca000)
libXau.so.6 => /usr/lib/libXau.so.6 (0x005aa000)

I went around this loop for a while, using the locate and ldd commands to track down the libicucogi18n.so.40 and libicucoguc.so.40 libraries, but to no avail.

I then did some Googling, as one does, and found this developerWorks forum post: -

Oracle Data Source Connect Fails libcogudaor

which led me to this Technote: -

Unable to test an Oracle 10g data source connection: RQP-DEF-0068 UDA-SQL-0432

Now I'm not using Oracle but the latter article said something: -

The LD_LIBRARY_PATH does not reference the 32-bit library for the Oracle Client. The 64-bit client is being referenced instead

that made me think.

I checked the IBM Business Monitor 8.0.1 Installation Guide, which said: -

Following that, I updated the LD_LIBRARY_PATH property ( Servers > Clusters > WebSphere application server clusters > PCSR01.Support > Cluster members > PCSR011.Support > Process definition > Environment Entries > LD_LIBRARY_PATH ) for the PCSR01.Support cluster member ( that which runs the Cognos BI code ) from: -

/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/cognos/PCSR011.Support/bin64

to: -

/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/cognos/PCSR011.Support/bin64:/opt/ibm/db2/V9.7/lib32/

and restarted my cluster.

As they say in the movies, Job Done !

Next step is to script the same using wsadmin and Jython.

Security Bulletin: An IBM Business Process Manager SSL connection can be established without host name verification: CVE-2012-5785

I'm reposting this from this IBM Flash: -

Security Bulletin: An IBM Business Process Manager SSL connection can be established without host name verification: CVE-2012-5785

Abstract

A Secure Sockets Layer (SSL) connection can be established without host name verfication, which an make the connection vulnerable to a man-in-the-middle attack.

Content

While obtaining an SSL connection, the IBM Business Process Management (BPM) system does not validate the host name of the target connection against the SubjectDN of the certificate. This situation can make the connection vulnerable to a man-in-the-middle attack.

CVE ID: 2012-5785
CVSS Base Score: 4.3
CVSS Temporal Score: See http://xforce.iss.net/xforce/xfdb/79830 for the current score.
CVSS Environmental Score*: Undefined
CVSS Vector: (AV:N/AC:M/Au:N/C:N/I:P/A:N)

REMEDIATION

To eliminate a man-in-the-middle attack, apply Interim Fixes JR45329, JR45216, and JR45071, or apply a Fix Pack that contains these APARS. These changes verify the host name against the certificate SubjectDN value. Using the following links, download the interim fixes from IBM Fix Central for IBM Integration Designer, Business Space (IBM Business Monitor) and your applicable IBM Business Process Manager product:

• IBM Integration Designer: APAR JR45329

• Business Space: APAR JR45216

• IBM Business Process Manager Standard: APAR JR45071

• IBM Business Process Manager Express: APAR JR45071

• IBM Business Process Manager Advanced: APAR JR45071

Please please refer the IBM Flash for further details, including the recommended fixes.

Patching IBM Business Process Manager Advanced 8.0.1 on Linux

This is an aide memoire as I know I will need it before too long: -

Download Required Fixes

At the time of writing, there's only two iFixes for IBM BPM Advanced 8.0.1 on IBM Fix Central.

*UPDATE* I need to include a reference to this iFix Flash: -

Security Bulletin: An IBM Business Process Manager SSL connection can be established without host name verification: CVE-2012-5785

Install Fixes

$ mkdir /tmp/bpm801fixes
$ mkdir /tmp/bpm801fixes/IFJR44937
$ mkdir /tmp/bpm801fixes/IFJR45071

$ unzip ~/8.0.1.0-WS-BPM-IFJR44937.zip -d /tmp/bpm801fixes/IFJR44937/

$ unzip ~/8.0.1.0-WS-BPM-IFJR45071.zip -d /tmp/bpm801fixes/IFJR45071/

$ /opt/IBM/InstallationManager/eclipse/tools/imcl install 8.0.1.0-WS-BPM-IFJR44937 -repositories /tmp/bpm801fixes/IFJR44937/ -installationDirectory /opt/IBM/WebSphere/AppServer -log ~/IFJR44937.txt

Installed 8.0.1.0-WS-BPM-IFJR44937_8.0.1000.20121203_1324 to the /opt/IBM/WebSphere/AppServer directory.

$ /opt/IBM/InstallationManager/eclipse/tools/imcl install 8.0.1.0-WS-BPM-IFJR45071 -repositories /tmp/bpm801fixes/IFJR45071/ -installationDirectory /opt/IBM/WebSphere/AppServer -log ~/IFJR45071.txt

Installed 8.0.1.0-WS-BPM-IFJR45071_8.0.1000.20121218_1447 to the /opt/IBM/WebSphere/AppServer directory.

Job Done!

IBM Process Designer - Not playing nicely with IBM Process Center ( BPM Advanced 8.0.1 )

I've seen this a few times now, and wanted to get my initial thoughts down on "paper".

As part of an IBM Business Process Manager Advanced 8.0.1 installation, there's normally a requirement to use the IBM Process Designer (IPD) developer client to create and deploy processes up to the Process Center server.

BPM makes this even easier by including a link to the Process Designer tool from the Process Center page: -

https://bpmpc.uk.ibm.com:8443/ProcessCenter/repository/com.lombardisoftware.repository.Repository/Repository.jsp

One thing that has been bugging me is that the downloaded Process Designer bundle: -

-rw-r--r--@  1 hayd  staff   423M 30 Jan 09:41 IBM Process Designer.zip

( which includes Eclipse )

doesn't seem to automagically connect to the right Process Center URL.

Once one expands the downloaded ZIP file, there's a configuration file - eclipse.ini e.g. /Users/hayd/Downloads/IBM Process Designer/eclipse.ini which contains the following line: -

-install
file:.
-configuration
configuration
-name
IBM BPM
-consoleLog
-dir
ltr
-clean
-vm
./AppClient/java/jre/bin/javaw.exe
-vmargs
-Xms128m
-Xmx512m
-XX:PermSize=128m
-XX:MaxPermSize=128m
-Djavax.net.ssl.keyStoreType=PKCS12
-Djavax.net.ssl.keyStore=./etc/key.p12
-Djavax.net.ssl.keyStorePassword=WebAS
-Djavax.net.ssl.trustStoreType=PKCS12
-Djavax.net.ssl.trustStore=./etc/trust.p12
-Djavax.net.ssl.trustStorePassword=WebAS
-Dcom.ibm.CORBA.ConfigURL="file:resources/sas.client.props"
-Dcom.ibm.CORBA.FragmentSize="0"
-Dcom.ibm.SSL.ConfigURL="file:resources/ssl.client.props"
-Djava.security.auth.login.config="file:resources/wsjaas_client.conf"
-Djava.naming.factory.initial=com.ibm.websphere.naming.WsnInitialContextFactory
-Dcom.ibm.bpm.processcenter.url=https://bpmpc.uk.ibm.com:8443
-Djava.ext.dirs="./AppClient/java/jre/lib/ext;./AppClient/lib;./AppClient/plugins"
-Dcom.lombardisoftware.core.TWEnvironment.environmentName=AUTHORING_ENVIRONMENT
-DentityExpansionLimit=2147483647

which looks good.

However, upon startup, IPD was still trying to connect to the wrong port with the wrong protocol during the authentication process.

Using the above example, IPD would try and authenticate against: -

rhel6.uk.ibm.com:9080

rather than: -

bpmpc.uk.ibm.com:8443

Talking with a learned colleague in Germany, it looks like IPD is making a request to the Process Center on the right URL - bpmpc.uk.ibm.com:8443 - *BUT* is then retrieving the wrong bootstrap host/port combination: -

the initial call is to /repo, which will respond with the server's bootstrap address, which then will be used for EJB and JMS communication.
It that bootstrap address contained an unresolvable host name, you'd see the described behaviour.

He also directed me to a new ( to me ) configuration file: -

/opt/IBM/WebSphere/AppServer/profiles/PCDMProfile/config/cells/PCCELL/nodes/PCNODE1/servers/PCSR011.AppTarget/process-center/TeamWorksConfiguration.running.xml

which contains a whole slew of WRONG host/port combinations: -

    <images-prefix>http://rhel6.uk.ibm.com:9080/teamworks</images-prefix>
    <portal-prefix>http://rhel6.uk.ibm.com:9080/portal</portal-prefix>
    <repository-prefix>http://rhel6.uk.ibm.com:9080/ProcessCenter</repository-prefix>
    <servlet-prefix>http://rhel6.uk.ibm.com:9080/teamworks</servlet-prefix>
    <webapi-prefix>http://rhel6.uk.ibm.com:9080/webapi</webapi-prefix>
      <base-url>http://rhel6.uk.ibm.com:9080/teamworks/webservices</base-url>
TeamWorks Link:  http://rhel6.uk.ibm.com:9080/portal/jsp/getProcessDetails.do?bpdInstanceId=<#= tw.system.process.instanceId #>
    <portal-prefix>http://rhel6.uk.ibm.com:9080/portal</portal-prefix>
    <process-admin-prefix>http://rhel6.uk.ibm.com:9080/ProcessAdmin</process-admin-prefix>
    <teamworks-webapp-prefix>http://rhel6.uk.ibm.com:9080/teamworks</teamworks-webapp-prefix>
      <default-namespace-uri>http://rhel6.uk.ibm.com:9080/schema/</default-namespace-uri>
    <coach-designer-xsl-url>http://rhel6.uk.ibm.com:9080/teamworks/coachdesigner/transform/CoachDesigner.xsl</coach-designer-xsl-url>
      <provider-url>corbaname:iiop:rhel6.uk.ibm.com:2810</provider-url>
    <jndi-url>corbaname:iiop:rhel6.uk.ibm.com:2810</jndi-url>
        <url>jdbc:db2://rhel6.uk.ibm.com:50000/BPMDB</url>
        <client-link>http://rhel6.uk.ibm.com:9080/teamworks</client-link>
    <repository-server-url>http://rhel6.uk.ibm.com:9080/ProcessCenter</repository-server-url>
    <server-host>rhel6.uk.ibm.com</server-host>
      <env key="java.naming.provider.url" value="corbaname:iiop:rhel6.uk.ibm.com:2810" />

I've yet to find out if I can automagically update this file during the installation and configuration phase BUT at least I know where to look.

Meantime, here's some required reading: -

Configuration file overview and explanation for Lombardi Teamworks, WebSphere Lombardi Edition (WLE), and the IBM Business Process Manager (BPM) products

Changing the database information for Process Server and the Performance Data Warehouse for the IBM Business Process Manager (BPM) products

*UPDATE* Reading the last link, it looks like TeamWorksConfiguration.running.xml gets its configuration from 99Local.xml and 100Custom.xml, so that's the first place I'm going to look :-)

*UPDATE AGAIN*

Thanks to some help from my mate, Bob, it's now working OK.

It's all down to 100Custom.xml AND the need to perform a FULL resynchronization of the cell configuration.

This is the DM Profile cell/node/server copy: -

-rw-r--r-- 1 wasadmin wasadmins 3880 Jan 29 16:09 /opt/IBM/WebSphere/AppServer/profiles/PCDMProfile/config/cells/PCCELL/nodes/PCNODE1/servers/PCSR011.AppTarget/process-center/config/100Custom.xml

and this is the Node Profile cell/node/server copy: -

-rw-r--r-- 1 wasadmin wasadmins 2757 Jan 29 13:52 /opt/IBM/WebSphere/AppServer/profiles/PCN1Profile/config/cells/PCCELL/nodes/PCNODE1/servers/PCSR011.AppTarget/process-center/config/100Custom.xml
I did a full resynch. and this is what I now have: -

-rw-r--r-- 1 wasadmin wasadmins 3880 Jan 29 16:09 /opt/IBM/WebSphere/AppServer/profiles/PCN1Profile/config/cells/PCCELL/nodes/PCNODE1/servers/PCSR011.AppTarget/process-center/config/100Custom.xml

So the full resynch. was the missing link :-)

After restarting the AppTarget cluster, I'm now seeing that the Node Profile copy of TeamWorksConfiguration.running.xml is getting updated: -

-rw-r--r-- 1 wasadmin wasadmins 373599 Jan 30 17:13 /opt/IBM/WebSphere/AppServer/profiles/PCN1Profile/config/cells/PCCELL/nodes/PCNODE1/servers/PCSR011.AppTarget/process-center/TeamWorksConfiguration.running.xml

and it's got the correct entries from 100Custom.xml: -

...
    https://bpm801.uk.ibm.com:8443/teamworks
    https://bpm801.uk.ibm.com:8443/portal
    https://bpm801.uk.ibm.com:8443/ProcessCenter
    https://bpm801.uk.ibm.com:8443/teamworks
    true
    https://bpm801.uk.ibm.com:8443/webapi
...

Now to download a new copy of Process Designer, and see whether it connects OK.

IBM Business Process Manager Advanced 8.0.1 - Bootstrapping not …. strapping ?

I saw this earlier: -

INFO:Error creating bean with name 'ejbCore.RepositoryServicesCore' defined in class path resource [registry.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.lombardisoftware.server.ejb.repositoryservices.RepositoryServicesCore]: Constructor threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ejbCore.PersistenceServicesCore' defined in class path resource [registry.xml]: Cannot resolve reference to bean 'handlersMap' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'handlersMap': Cannot create inner bean 'com.lombardisoftware.server.ejb.persistence.PSDefaultHandler#675fd7ca' of type [com.lombardisoftware.server.ejb.persistence.PSDefaultHandler] while setting bean property 'sourceMap' with key [TypedStringValue: value [Task], target type [null]]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.lombardisoftware.server.ejb.persistence.PSDefaultHandler#675fd7ca#1' defined in class path resource [registry.persistence.xml]: Cannot resolve reference to bean 'dao.task' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dao.task' defined in class path resource [registry.persistence.xml]: Instantiation of bean failed; nested exception is java.lang.NoClassDefFoundError: com.lombardisoftware.server.ejb.persistence.dao.TaskDAO (initialization failure)
SEVERE:

in: -

/opt/IBM/WebSphere/AppServer/profiles/PCDMProfile/logs/bootstrapProcesServerData.PCSR01.AppTarget.Tue-Jan-29-14.26.29-2013.log 

using the bootstrapProcessServerData.sh command.

Now I know that this worked, as I'd done the same thing earlier.

I had to dig back through the console log for the session in which I was running the bootstrapProcessServerData command, and found: -

…

Caused by: org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.lombardisoftware.bpd.runtime.engine.message.DefaultMessageRoutingCache]: Constructor threw exception; nested exception is org.springframework.jdbc.BadSqlGrammarException: PreparedStatementCallback; bad SQL grammar [select propvalue from lsw_system where propkey=?]; nested exception is com.ibm.db2.jcc.am.SqlSyntaxErrorException: DB2 SQL Error: SQLCODE=-204, SQLSTATE=42704, SQLERRMC=DB2INST1.LSW_SYSTEM, DRIVER=4.11.69
…

At this point, I realised that ( DOH! ), I hadn't actually created the relevant DB2 database tables - DB2INST1.LSW_SYSTEM etc.

I quickly resolved this by executing: -

$ db2 -tvf /opt/IBM/WebSphere/AppServer/profiles/PCDMProfile/dbscripts/ProcessServer/DB2/BPMDB/createTable_ProcessServer.sql 

and: -

…

The bootstrapping of data completed.....
'BootstrapProcessServerData admin command completed.....'

…

What was annoying is that there was no obvious mention of the issue in any of the WAS logs e.g. /opt/IBM/WebSphere/AppServer/profiles/PCDMProfile/logs, /opt/IBM/WebSphere/AppServer/profiles/PCDMProfile/ffdc etc.

IBM Business Process Manager Advanced 8.0.1 - Database Schemas for the Service Integration bus Messaging Engines

Just writing this down for future reference.

Whilst in BPM Standard, we only have two schemas ( schemae ? ) for the SIbus Messaging Engines - - with Advanced, we have a few more: -

Schema CMNBM00

JNDI Name of JDBC Data Source jdbc/com.ibm.ws.sib/PCSR01.Messaging-BPC.PCCELL.Bus
JDBC Data Source Business Process Choreographer ME data source 
Database CMNDB

Schema CMNCM00

JNDI Name of JDBC Data Source jdbc/com.ibm.ws.sib/PCSR01.Messaging-CEI.PCCELL.BUS
JDBC Data Source CEI ME data source 
Database CMNDB

Schema CMNPE00

JNDI Name of JDBC Data Source jdbc/com.ibm.ws.sib/twperfsvr_bus
JDBC Data Source Performance Data Warehouse ME data source
Database CMNDB

Schema CMNPS00

JNDI Name of JDBC Data Source jdbc/com.ibm.ws.sib/twprocsvr_bus
JDBC Data Source Process Server ME data source 
Database CMNDB

Schema CMNSA00

JNDI Name of JDBC Data Source jdbc/com.ibm.ws.sib/PCSR01.Messaging-SCA.APPLICATION.PCCELL.Bus
JDBC Data Source SCA Application Bus ME data source 
Database CMNDB

Schema CMNSS00

JNDI Name of JDBC Data Source jdbc/com.ibm.ws.sib/PCSR01.Messaging-SCA.SYSTEM.PCCELL.Bus
JDBC Data Source SCA System Bus ME data source
Database CMNDB

The SQL scripts required to create these six schemas can be generated using the sibDDLGenerator.sh script, as follows: -

$ cd /opt/IBM/WebSphere/AppServer/profiles/PCDMProfile/bin/sibDDLGenerator.sh

$ ./sibDDLGenerator.sh -system db2 -version 9.7 -platform unix -schema CMNBM00 >> ~/createBPM801Schemas.sql -statementend ";"

$ ./sibDDLGenerator.sh -system db2 -version 9.7 -platform unix -schema CMNCM00 >> ~/createBPM801Schemas.sql -statementend ";"
$ ./sibDDLGenerator.sh -system db2 -version 9.7 -platform unix -schema CMNPE00 >> ~/createBPM801Schemas.sql -statementend ";"
$ ./sibDDLGenerator.sh -system db2 -version 9.7 -platform unix -schema CMNPS00 >> ~/createBPM801Schemas.sql -statementend ";"
$ ./sibDDLGenerator.sh -system db2 -version 9.7 -platform unix -schema CMNSA00 >> ~/createBPM801Schemas.sql -statementend ";"
$ ./sibDDLGenerator.sh -system db2 -version 9.7 -platform unix -schema CMNSS00 >> ~/createBPM801Schemas.sql -statementend ";"

WSVR0016W for pdq.jar and pdqmgmr.jar during creation of an IBM BPM Advanced 8.0.1 Deployment Manager profile

I am using IBM Business Process Manager Advanced 8.0.1 on Red Hat Enterprise Linux 6.3 (x86-64). During the installation, I have created a Deployment Manager ( Process Center ) profile using the following template: -

/opt/IBM/WebSphere/AppServer/profileTemplates/BPM/dmgr.procctr.adv

During the start-up of the DMGR profile, I see the following two exceptions: -

[29/01/13 12:57:10:993 GMT] 00000000 ResourceMgrIm W   WSVR0016W: Classpath entry, ${PUREQUERY_PATH}/pdq.jar, in Resource, DB2 Using IBM JCC Driver (XA), located at /opt/IBM/WebSphere/AppServer/profiles/PCDMProfile/config/cells/PCCELL/resources.xml has an invalid variable
[29/01/13 12:57:10:998 GMT] 00000000 ResourceMgrIm W   WSVR0016W: Classpath entry, ${PUREQUERY_PATH}/pdqmgmt.jar, in Resource, DB2 Using IBM JCC Driver (XA), located at /opt/IBM/WebSphere/AppServer/profiles/PCDMProfile/config/cells/PCCELL/resources.xml has an invalid variable

These two JAR files - pdq.jar and pdqmgmr.jar - appear to relate to the IBM Optim pureQuery run-time.

As far as I am aware, this isn't something that IBM BPM Advanced 8.0.1 requires.

The two JARs are not installed with the underlying WebSphere Application Server Network Deployment 8.0.0.5 product, or with the IBM BPM Advanced 8.0.1 code.

This is what I have installed: -

$ /opt/IBM/InstallationManager/eclipse/tools/imcl listInstalledPackages

com.ibm.cic.agent_1.6.0.20120831_1216

com.ibm.websphere.IHS.v80_8.0.5.20121022_1902

com.ibm.bpm.ADV.V80_8.0.1000.20121102_2136

com.ibm.websphere.ND.v80_8.0.5.20121022_1902

com.ibm.websphere.PLG.v80_8.0.5.20121022_1902

If I remove the two JARs from the class path, either via the WAS Integrated Solutions Console, or via a modification of: -

/opt/IBM/WebSphere/AppServer/profiles/PCDMProfile/config/cells/PCCELL/resources.xml

...
    <classpath>${PUREQUERY_PATH}/pdq.jar</classpath>
    <classpath>${PUREQUERY_PATH}/pdqmgmt.jar</classpath>
...

the exceptions do not occur.

I've raised a PMR, and we'll see what happens :-)

Reducing the size of IBM installation media with the IBM Packaging Utility

This came up on Twitter earlier today: -

vanstaub How and why I reduced the #WebSpherePortal installation media from 17GB to around 3GB https://t.co/13nxxcHa #ibmportal 25/01/2013 14:33

The author, Van Staub, has written a rather neat article on IBM developWorks - Reducing the size of IBM installation media with the IBM Packaging Utility - which is definitely worth a read: -

 Recently, I received an inquiry on reducing the size of the installation media for WebSphere Portal 8.  An administrator had downloaded ~17GB of installation media consisting of:

• IBM WebSphere Portal

• IBM WebSphere Application Server ND

• IBM WAS Supplements

• IBM WAS FP005

• IBM WAS FP005 Supplements

 This was simply too much data to transmit or maintain as they conducted installations in the field.  So I created the user story below and sought a way to reduce the installation media's size as much as possible.

Check it out, it's worth your time AND disk space :-)

IBM Installation Manager - Permission to Launch

I had some fun and games with IBM Installation Manager 1.6.0 today ( on a 64-bit version of Red Hat Enterprise Linux 6.3, but that's not important right now ).

It all stemmed from when I ( stupidly ) tried to run IIM as root, having installed and configured it as a non-root user - wasadmin.

This is what I saw: -

ERROR: The Installation Manager cannot be started.The registry information does not exist or does not match with this executable.This may happen if you are trying to run Installation Manager installed by another user.

00:00.40 ERROR [main] com.ibm.cic.agent.internal.application.HeadlessApplication run
  The Installation Manager cannot be started.The registry information does not exist or does not match with this executable.This may happen if you are trying to run Installation Manager installed by another user.

However, when I then tried to run IIM as wasadmin: -

$ /opt/IBM/InstallationManager/eclipse/IBMIM -version -nosplash -silent

I got this: -

(IBMIM:18310): GLib-GObject-WARNING **: invalid (NULL) pointer instance
(IBMIM:18310): GLib-GObject-CRITICAL **: g_signal_connect_data: assertion `G_TYPE_CHECK_INSTANCE (instance)' failed
(IBMIM:18310): Gtk-CRITICAL **: gtk_settings_get_for_screen: assertion `GDK_IS_SCREEN (screen)' failed
(IBMIM:18310): GLib-GObject-CRITICAL **: g_object_get: assertion `G_IS_OBJECT (object)' failed
(IBMIM:18310): GLib-GObject-WARNING **: value "TRUE" of type `gboolean' is invalid or out of range for property `visible' of type `gboolean'
(IBMIM:18310): Gtk-CRITICAL **: gtk_settings_get_for_screen: assertion `GDK_IS_SCREEN (screen)' failed
(IBMIM:18310): GLib-GObject-CRITICAL **: g_object_get: assertion `G_IS_OBJECT (object)' failed
(IBMIM:18310): Gtk-WARNING **: Screen for GtkWindow not set; you must always set a screen for a GtkWindow before using the window
(IBMIM:18310): Gdk-CRITICAL **: gdk_pango_context_get_for_screen: assertion `GDK_IS_SCREEN (screen)' failed
(IBMIM:18310): Pango-CRITICAL **: pango_context_set_font_description: assertion `context != NULL' failed
(IBMIM:18310): Pango-CRITICAL **: pango_context_set_base_dir: assertion `context != NULL' failed
(IBMIM:18310): Pango-CRITICAL **: pango_context_set_language: assertion `context != NULL' failed
(IBMIM:18310): Pango-CRITICAL **: pango_layout_new: assertion `context != NULL' failed
(IBMIM:18310): Pango-CRITICAL **: pango_layout_set_text: assertion `layout != NULL' failed
(IBMIM:18310): Pango-CRITICAL **: pango_layout_set_attributes: assertion `layout != NULL' failed
(IBMIM:18310): Pango-CRITICAL **: pango_layout_set_alignment: assertion `layout != NULL' failed
(IBMIM:18310): Pango-CRITICAL **: pango_layout_set_ellipsize: assertion `PANGO_IS_LAYOUT (layout)' failed
(IBMIM:18310): Pango-CRITICAL **: pango_layout_set_single_paragraph_mode: assertion `PANGO_IS_LAYOUT (layout)' failed
(IBMIM:18310): Pango-CRITICAL **: pango_layout_set_width: assertion `layout != NULL' failed
(IBMIM:18310): Pango-CRITICAL **: pango_layout_get_extents: assertion `layout != NULL' failed
(IBMIM:18310): Gtk-CRITICAL **: gtk_icon_theme_get_for_screen: assertion `GDK_IS_SCREEN (screen)' failed
(IBMIM:18310): Gtk-CRITICAL **: gtk_settings_get_for_screen: assertion `GDK_IS_SCREEN (screen)' failed
(IBMIM:18310): Gtk-CRITICAL **: gtk_icon_size_lookup_for_settings: assertion `GTK_IS_SETTINGS (settings)' failed
(IBMIM:18310): Gtk-WARNING **: Invalid icon size 6

(IBMIM:18310): Gtk-CRITICAL **: gtk_icon_theme_load_icon: assertion `GTK_IS_ICON_THEME (icon_theme)' failed
(IBMIM:18310): Gtk-WARNING **: Error loading theme icon 'gtk-dialog-error' for stock: 
(IBMIM:18310): Gtk-CRITICAL **: gtk_icon_size_lookup_for_settings: assertion `GTK_IS_SETTINGS (settings)' failed
(IBMIM:18310): Gtk-WARNING **: gtkstyle.c:2318: invalid icon size '6'
(IBMIM:18310): Gtk-CRITICAL **: gtk_style_render_icon: assertion `pixbuf != NULL' failed
(IBMIM:18310): GLib-GObject-CRITICAL **: g_object_ref: assertion `G_IS_OBJECT (object)' failed
(IBMIM:18310): Gdk-CRITICAL **: gdk_pango_context_get_for_screen: assertion `GDK_IS_SCREEN (screen)' failed
(IBMIM:18310): Pango-CRITICAL **: pango_context_set_font_description: assertion `context != NULL' failed
(IBMIM:18310): Pango-CRITICAL **: pango_context_set_base_dir: assertion `context != NULL' failed
(IBMIM:18310): Pango-CRITICAL **: pango_context_set_language: assertion `context != NULL' failed
(IBMIM:18310): Pango-CRITICAL **: pango_layout_new: assertion `context != NULL' failed
(IBMIM:18310): Pango-CRITICAL **: pango_layout_set_text: assertion `layout != NULL' failed
(IBMIM:18310): Pango-CRITICAL **: pango_layout_get_attributes: assertion `PANGO_IS_LAYOUT (layout)' failed
(IBMIM:18310): Pango-CRITICAL **: pango_layout_set_alignment: assertion `layout != NULL' failed
(IBMIM:18310): Pango-CRITICAL **: pango_layout_set_ellipsize: assertion `PANGO_IS_LAYOUT (layout)' failed
(IBMIM:18310): Pango-CRITICAL **: pango_layout_set_single_paragraph_mode: assertion `PANGO_IS_LAYOUT (layout)' failed
(IBMIM:18310): Pango-CRITICAL **: pango_layout_set_wrap: assertion `PANGO_IS_LAYOUT (layout)' failed
(IBMIM:18310): Pango-CRITICAL **: pango_layout_set_width: assertion `layout != NULL' failed
(IBMIM:18310): Pango-CRITICAL **: pango_layout_get_extents: assertion `layout != NULL' failed
(IBMIM:18310): Gdk-CRITICAL **: gdk_pango_context_get_for_screen: assertion `GDK_IS_SCREEN (screen)' failed
(IBMIM:18310): Pango-CRITICAL **: pango_context_set_font_description: assertion `context != NULL' failed
(IBMIM:18310): Pango-CRITICAL **: pango_context_set_base_dir: assertion `context != NULL' failed
(IBMIM:18310): Pango-CRITICAL **: pango_context_set_language: assertion `context != NULL' failed
(IBMIM:18310): Pango-CRITICAL **: pango_layout_new: assertion `context != NULL' failed
(IBMIM:18310): Pango-CRITICAL **: pango_layout_set_text: assertion `layout != NULL' failed
(IBMIM:18310): Pango-CRITICAL **: pango_layout_get_extents: assertion `layout != NULL' failed

(IBMIM:18310): GLib-GObject-CRITICAL **: g_object_unref: assertion `G_IS_OBJECT (object)' failed
(IBMIM:18310): Gdk-CRITICAL **: gdk_screen_get_width: assertion `GDK_IS_SCREEN (screen)' failed
(IBMIM:18310): Gdk-CRITICAL **: gdk_screen_get_width: assertion `GDK_IS_SCREEN (screen)' failed
(IBMIM:18310): Pango-CRITICAL **: pango_layout_set_width: assertion `layout != NULL' failed
(IBMIM:18310): Pango-CRITICAL **: pango_layout_get_extents: assertion `layout != NULL' failed
(IBMIM:18310): Pango-CRITICAL **: pango_layout_get_line_count: assertion `layout != NULL' failed

I thought this might be a permissions issue as, having run IIM as root, I'd locked out certain crucial files, thus preventing wasadmin from accessing them.

I validated this as follows: -

$ /opt/IBM/InstallationManager/eclipse/tools/imcl -version -silent -nosplash

which returned: -

Invalid Configuration Location:
Locking is not possible in the directory "/opt/IBM/InstallationManager/eclipse/configuration/org.eclipse.osgi". A common reason is that the file system or Runtime Environment does not support file locking for that location. Please choose a different location, or disable file locking passing "-Dosgi.locking=none" as a VM argument. 
/opt/IBM/InstallationManager/eclipse/configuration/org.eclipse.osgi/.manager/.fileTableLock (Permission denied)

When I checked that file: -

$ ls -al /opt/IBM/InstallationManager/eclipse/configuration/org.eclipse.osgi/.manager/.fileTableLock 

I saw this: -

-rw-r--r-- 1 root root 0 Jan 25 09:29 /opt/IBM/InstallationManager/eclipse/configuration/org.eclipse.osgi/.manager/.fileTableLock

The same was true of other files in the same location: -

$ ls -al /opt/IBM/InstallationManager/eclipse/configuration/org.eclipse.osgi

total 552
drwxr-xr-x 3 root     root        4096 Jan 25 09:29 .
drwxr-xr-x 6 wasadmin wasadmins   4096 Jan 25 10:02 ..
-rw-r--r-- 1 root     root      201051 Jan 25 09:29 .bundledata.1
-rw-r--r-- 1 root     root      263559 Jan 25 09:29 .lazy.1
drwxr-xr-x 2 root     root        4096 Jan 25 09:29 .manager
-rw-r--r-- 1 root     root       81835 Jan 25 09:29 .state.1

$ ls -al /opt/IBM/InstallationManager/eclipse/configuration/

total 60
drwxr-xr-x  6 wasadmin wasadmins 4096 Jan 25 10:02 .
drwxr-xr-x 11 wasadmin wasadmins 4096 Jan 25 10:02 ..
-rw-r--r--  1 wasadmin wasadmins 4445 Jan 25 09:51 1359107496589.log
-rw-r--r--  1 wasadmin wasadmins 4355 Jan 25 09:53 1359107593225.log
-rw-r--r--  1 wasadmin wasadmins 4355 Jan 25 09:57 1359107865410.log
-rw-r--r--  1 wasadmin wasadmins 4369 Jan 25 10:02 1359108164852.log
-rw-r--r--  1 wasadmin wasadmins  759 Jan 24 17:24 config.ini
drwxr-xr-x  3 root     root      4096 Jan 25 09:29 org.eclipse.core.runtime
drwxr-xr-x  3 root     root      4096 Jan 25 09:29 org.eclipse.equinox.app
drwxr-xr-x  3 root     root      4096 Jan 25 09:29 org.eclipse.osgi
drwxr-xr-x  2 wasadmin wasadmins 4096 Jan 25 09:29 org.eclipse.update

Once I changed the permissions: -

$ chown -R wasadmin:wasadmins /opt/IBM/InstallationManager/

all was well: -

$ /opt/IBM/InstallationManager/eclipse/tools/imcl -version -silent -nosplash

Installation Manager (installed)
Version: 1.6.0
Internal Version: 1.6.0.20120831_1216

$ /opt/IBM/InstallationManager/eclipse/IBMIM -version -nosplash -silent

Installation Manager (installed)
Version: 1.6.0
Internal Version: 1.6.0.20120831_1216

Shiny :-)

Performance tuning considerations when IBM Business Process Manager (BPM) is running in a virtual machine

One of my colleagues, Mike, blogged this earlier today, and I thought it worth re-sharing as it's relevant to IBM BPM *and* any other WAS-based workload running on VMware, or similar.

Performance tuning considerations when IBM Business Process Manager (BPM) is running in a virtual machine

Problem(Abstract)

Running IBM Business Process Manager in a virtual machine is supported. However, there are some additional considerations when running servers in a virtual machine (VM) environment.

Resolving the problem

When you are running a Java Platform, Enterprise Edition (Java EE) application in a virtual machine (VM) there are additional considerations. These considerations are performance-related changes.

Enjoy !

PS I've blogged about VMware and WAS before, specifically this post: -

WebSphere and VMware Together - The Balloon Goes POP!!

WebSphere Portal - Switching Databases

One of my colleagues asked me how he could connect a newly installed WebSphere Portal v8 profile ( wp_profile ) to an existing set of Portal databases - JCRDB, RELEASEDB, CUSTDB etc. - without initialising the data within those DBs.

The reason for this was that he'd reinstalled WebSphere Portal, but didn't want to lose the data that he'd previously transferred ( from Derby ) into the "real" DB2 databases.

This seemed like a really straightforward question, especially given the fact this would be a common use case when, for example, upgrading from one release to another. To extend the example, one might build a new WebSphere Portal 8 cell, leaving the existing WebSphere Portal 7 cell in place, and then "connect" the new servers to the old database.

Although I couldn't come up with the solution, my colleague ( Steven ) did.

To be more specific, he found this in the WebSphere Portal Wiki: -

Product Documentation > IBM WebSphere Portal 8 Product Documentation > Installing > Installing on Linux > Setting up a stand-alone production server on Linux > Linux stand-alone: Configuring your portal to use a database > Linux stand-alone: Setting up a remote DB2 database > Linux stand-alone: Type 2 driver support > Linux stand-alone: Changing DB2 driver types

This uses the following ConfigEngine commands: -

./ConfigEngine.sh validate-database

./ConfigEngine.sh connect-database

to do the job.

Sorted :-)

Fun with IBM Business Monitor 8.0.1 and Cognos BI 10.1 on Red Hat Enterprise Linux 6.3

This is a strange one, but at least I have a solution.

Having recently ( January 17th ) installed IBM Business Monitor 8.0.1 ( aka BAM ) on a VMware image running Red Hat Enterprise Linux 6.3 x86-64, I was surprised to see the following exception: -

[1/22/13 14:48:10:787 GMT] 00000000 WsServerImpl  W   WSVR0100W: An error occurred initializing, BAM801.Support.bamNode01.0 [class com.ibm.ws.runtime.component.ServerImpl]
com.ibm.ws.exception.ConfigurationWarning: Exception caught when initializing component
…

Caused by: java.lang.NoClassDefFoundError: com.ibm.ws.process.linuxutil.ThreadUtil (initialization failure)
at java.lang.J9VMInternals.initialize(J9VMInternals.java:168)
…

when I started one of the clusters today.

This cluster ( BAM801.Support.bamNode01.0 ) hosts IBM Cognos BI 10.1, which is a core component of BAM, and is used to provide cube views etc. of business metrics and KPIs.

The strange thing is that I did not see, or perhaps notice, this exception when I last started the cluster last Friday.

Be that as it may, this is a known issue with the combination of BAM, Cognos and 64-bit RHEL, as evidenced by this IBM Technote: -

Unable to connect to IBM Business Monitor V8.0 on Red Hat Enterprise Linux 6.x

which says, in part: -

…

Cause

The libfreebl3.so file shipped with Cognos V10.1.1 is not compatible with Red Hat Enterprise Linux 6.x. The IBM Business Monitor server fails to start because Cognos fails to start.

Resolving the problem

Add the following like to the .profile, .bashrc, or appropiate login initialization script for the Cognos user account:

export LD_PRELOAD=/lib64/libfreebl3.so

…

In order to test this, without rebooting my server, I updated my local Bash shell: -

$ export LD_PRELOAD=/lib64/libfreebl3.so

and validated the change: -

$ set | grep -i PRELOAD

LD_PRELOAD=/lib64/libfreebl3.so

$ echo $LD_PRELOAD

/lib64/libfreebl3.so

I then manually started the cluster from within that shell: -

$ /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/bin/startServer.sh BAM801.Support.bamNode01.0

In this way, I could test the change without rebooting. Had I not done this, when I started the cluster from within the WAS Integrated Solutions Console, via the Deployment Manager, the LD_PRELOAD environment change would NOT have been picked up.

Lo and behold, this worked a treat, and I now have a working Cognos BI environment.

My next step was to make this change permanent for the user who "owns" the WAS processes - wasadmin - by adding the line: -

export LD_PRELOAD=/lib64/libfreebl3.so

to: -

/home/wasadmin/.bashrc 

Actually, this goes along with some WebSphere MQ and Message Broker stuff about which I have previously blogged, so I now have: -

…

source /opt/ibm/mqsi/8.0.0.1/bin/mqsiprofile
export LD_LIBRARY_PATH=/opt/mqm/java/lib:$LD_LIBRARY_PATH
export LD_PRELOAD=/lib64/libfreebl3.so

…

Now, the next time that I log in as wasadmin and spin up any WAS process, including the Deployment Manager, Node Agent etc. the LD_PRELOAD variable will be picked up.

Good stuff …..

More on the AJAX Proxy and the REST Services Gateway in IBM Business Space 8.0.1

Following on from my earlier post: -

Testing the AJAX Proxy in WebSphere Application Server 8.0

I did see some issues with the AJAX Proxy, when I tried to add and configure a KPIs widget to Business Space ( I'm using IBM Business Monitor 8.0.1 on WAS 8.0.0.5 ), including the following exceptions in SystemOut.log: -

[1/22/13 15:16:06:133 GMT] 00000026 URLConnection W   BMWPX0011W: The Ajax proxy has been enabled to support unsigned Secure Sockets Layer (SSL) certificates: mapping=/proxy, policy=null
[1/22/13 15:16:06:166 GMT] 00000026 AjaxProxyServ W   BMWPX0024W: An unexpected problem occurred while processing the response data.
                                 java.net.SocketException: Connection reset
[1/22/13 15:16:06:193 GMT] 00000026 servlet       E com.ibm.ws.webcontainer.servlet.ServletWrapper service SRVE0014E: Uncaught service() exception root cause mmOSGI: com.ibm.mm.proxy.exceptions.LocalizedProxyIOException: BMWPX0024W: An unexpected problem occurred while processing the response data.

Caused by: java.net.SocketException: Connection reset
[1/22/13 15:16:06:328 GMT] 00000026 webapp        E com.ibm.ws.webcontainer.webapp.WebApp logServletError SRVE0293E: [Servlet Error]-[mmOSGI]: com.ibm.mm.proxy.exceptions.LocalizedProxyIOException: BMWPX0024W: An unexpected problem occurred while processing the response data.
Caused by: java.net.SocketException: Connection reset

etc.

When I checked the exception that I was seeing in the widget: -

it looked like the iWidget was using the AJAX Proxy, but was going to the following URL: -

https://rhel6.uk.ibm.com:8443/mum/proxy/http/rhel6.uk.ibm.com:8443

In my environment, IHS is configured for SSL only, and is listening on port 8443.

In other words, I was accessing Business Space via the right protocol/port combination ( HTTPS to 8443 ), but the iWidget was trying to go to the same SSL port ( 8443 ) using the HTTP protocol.

Something made me think that this might be a problem with the REST endpoints.

I checked that in the WAS Integrated Solutions Console (ISC) via the following path: -

Services -> REST services -> REST services

and could see: -

Therefore, the REST endpoints were configured for the RIGHT port ( 8443 ) but the WRONG protocol ( HTTP ).

I changed this as follows: -

Services -> REST services -> REST service providers -> REST Services Gateway

and changed from this: -

to this: -

Having done this, I shut down and restarted my four clusters: -

and now I see a much more meaningful "exception": -

which is 100% accurate, as I have no KPIs defined yet :-)

Testing the AJAX Proxy in WebSphere Application Server 8.0

I'm actually using IBM Business Monitor 8.0.1 ( aka Business Activity Monitoring or IBM BM or IBM ! ), but this relates to IBM Business Process Manager 8.X, which also includes the AJAX Proxy to support the Business Space UI.

I'm still on a learning curve with this, but I just wanted to report back on some basic AJAX Proxy security/configuration testing.

So, the way I see it, the AJAX Proxy can be used to allow an iWidget rendering in Business Space to access services on other Business Space servers ….. -OR- on other web servers, regardless of whether they're in your network or not :-(

In addition, the AJAX Proxy can be used by ALL authenticated users.

Ulp !!

So, with more recent versions of IBM BAM and IBM BPM, the AJAX Proxy has been "locked down" to prevent users from going to servers outside of a small number of defaults.

This is controlled by the file - proxy-config.xml - for my environment, this is located here: -

/opt/IBM/WebSphere/AppServer/profiles/Dmgr01/BusinessSpace/BAM801.WebApp/mm.runtime.prof/config/proxy-config.xml

Within the file, I can see: -

...

        <proxy:policy url="endpoint://*" acf="none" basic-auth-support="true">
                <proxy:actions>
                        <proxy:method>GET</proxy:method>
                        <proxy:method>POST</proxy:method>
                        <proxy:method>PUT</proxy:method>
                        <proxy:method>DELETE</proxy:method>
                </proxy:actions>…

I believe this allows Business Space to access iWidgets etc. from the BAM ( or BPM ) server itself e.g. only access resources from the server on which Business Space is running.

I *MAY* be wrong, and will be checking this out.

However, I can also see: -

…

        <proxy:policy url="http://www.ibm.com/*" acf="none" basic-auth-support="true">
…

        <proxy:policy url="http://www-03.ibm.com/*" acf="none" basic-auth-support="true">…

        <proxy:policy url="http://www.redbooks.ibm.com/*" acf="none" basic-auth-support="true">…

        <proxy:policy url="http://www.google.com/ig/*"  acf="none" basic-auth-support="true">…

    <proxy:mapping contextpath="/cognosProxy/*">
           <proxy:policy url="endpoint://*" acf="none" basic-auth-support="true">…

the first four of which mean that the AJAX Proxy will allow requests to be made from Business Space to any of those four external URLs.

To "prove" this, I attempted to access another URL via the Proxy, with the following request: -

https://rhel6.uk.ibm.com:8443/mum/proxy/https/rhel6.uk.ibm.com:8443

As expected, this failed with: -

403 BMWPX0006E: The URL https://rhel6.uk.ibm.com:8443/ cannot be accessed through the proxy.

Had I added a <proxy:policy url="http://rhel6.uk.ibm.com:8443/* …/> entry to my proxy-config.xml, I would not have seen that exception.

I proved this by adding the following lines: -

..

        <proxy:policy url="https://rhel6.uk.ibm.com:8443/*"  acf="none" basic-auth-support="true">
                <proxy:actions>

                        <proxy:method>GET</proxy:method>
                </proxy:actions>
        </proxy:policy>

...

to the proxy-config.xml file.

I then needed to "check in" the updated file to the cluster within which Business Space is executing - BAM801.WebApp - as follows: -

$ /opt/IBM/WebSphere/AppServer/profiles/Dmgr01/bin/wsadmin.sh -lang jython -user wasadmin -password passw0rd

WASX7209I: Connected to process "dmgr" on node bamDM using SOAP connector;  The type of process is: DeploymentManager
WASX7031I: For help, enter: "print Help.help()"
wsadmin>

wsadmin> AdminTask.updateBlobConfig('[-clusterName "BAM801.WebApp" -propertyFileName "/opt/IBM/WebSphere/AppServer/profiles/Dmgr01/BusinessSpace/BAM801.WebApp/mm.runtime.prof/config/proxy-config.xml" -prefix "Mashups_"]')

'updateBlobConfig is executed succesfully'

wsadmin> AdminConfig.save()

''

wsadmin> AdminNodeManagement.syncActiveNodes()

---------------------------------------------------------------
 AdminNodeManagement:        Synchronize the active nodes
 Usage: AdminNodeManagement.syncActiveNodes()
 Return: If the command is successfully invoked, a value of 1 is returned.
---------------------------------------------------------------


bamNode01
1

wsadmin> exit

Now, when I attempt to have the AJAX Proxy get content directly from the web server, using the following URL: -

https://rhel6.uk.ibm.com:8443/mum/proxy/https/rhel6.uk.ibm.com:8443

I get some small element of the IBM HTTP Server default "welcome" page - index.html - as follows: -

I suspect that I need to enable more HTTP methods than merely GET, but this proves the point with regard to the AJAX Proxy configuration.

PS I should point out that this is well documented in various places, including the following IBM Technote: -

Troubleshooting IBM Business Process Manager: "403 BMWPX0006E: The URL you tried to access through the proxy is not allowed"

 

What this should show us is that (a) the AJAX Proxy is a very powerful piece of kit and (b) with great power comes great responsibility.

There are a number of IBM recommendations with regard to "hardening" the AJAX proxy, including: -

(1) Forcing SSL and disallow untrusted signers

(2) Reducing the number of endpoints to which the proxy can connect

(3) Configuring IP whitelisting to ensure that the proxy is only used with certain IP address ranges

For more detail, please contact IBM.

Of course,  IBM Software Services for WebSphere (ISSW), the team for whom I work, can certainly help with this, if required.

IBM WebSphere MQ 7.5 - Tuning up the Linux kernel

Following on from my earlier post: -

IBM WebSphere Message Broker 8.0.1 - Getting started

I've been experimenting with Linux kernel tuning, using the mqconfig tool: -

How to configure UNIX and Linux systems for WebSphere MQ

This is what mqconfig shows me ( run as user mqm ): -

$ ./mqconfig -v 7.5

mqconfig: V3.6 analyzing Red Hat Enterprise Linux Server release 6.3
          (Santiago) settings for WebSphere MQ V7.5

System V Semaphores
  semmsl     (sem:1)  250 semaphores                     IBM>=500          FAIL
  semmns     (sem:2)  17 of 256000 semaphores    (0%)    IBM>=256000       PASS
  semopm     (sem:3)  32 operations                      IBM>=250          FAIL
  semmni     (sem:4)  7 of 2048 sets             (0%)    IBM>=1024         PASS

System V Shared Memory
  shmmax              68719476736 bytes                  IBM>=268435456    PASS
  shmmni              1 of 4096 sets             (0%)    IBM>=4096         PASS
  shmall              107093 of 4294967296 pages (0%)    IBM>=2097152      PASS

System Settings
  file-max            5728 of 792955 files       (0%)    IBM>=524288       PASS
  tcp_keepalive_time  7200 seconds                       IBM<=300          FAIL

Current User Limits (mqm)
  nofile       (-Hn)  20000 files                        IBM>=10240        PASS
  nofile       (-Sn)  20000 files                        IBM>=10240        PASS
  nproc        (-Hu)  10 of 62767 processes      (0%)    IBM>=4096         PASS
  nproc        (-Su)  10 of 1024 processes       (0%)    IBM>=4096         FAIL

Following the advice here: -

Additional settings for installing on UNIX and Linux and Linux systems -> Additional settings for installing WebSphere MQ on Linux systems

this is how I turned FAIL into PASS.

(a) For the System V Semaphores, I added the following line to /etc/sysctl.conf 

…

kernel.sem = 500 256000 250 1024
…

which has increased semmsl from 250 to 500 and increased semopm from 32 to 250.

(b) For the System Settings, I added the following two lines to /etc/sysctl.conf

…

fs.file-max = 524288

net.ipv4.tcp_keepalive_time = 300

…

(c) For the Current User Limits, I added the following lines to /etc/security/limits.conf 

…

mqm hard nproc 4096
mqm soft nproc 4096
…

In order to make the sysctl.conf changes permanent, I ran the following command ( as root ): -

$ sysctl -p

which returns: -

net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
kernel.sem = 500 256000 250 1024
fs.file-max = 524288
net.ipv4.tcp_keepalive_time = 300

In order to make the limits.conf changes permanent, I logged out of the mqm and then logged back in.

This time around, I get the following: -

$ ./mqconfig -v 7.5

mqconfig: V3.6 analyzing Red Hat Enterprise Linux Server release 6.3
          (Santiago) settings for WebSphere MQ V7.5

System V Semaphores
  semmsl     (sem:1)  500 semaphores                     IBM>=500          PASS
  semmns     (sem:2)  17 of 256000 semaphores    (0%)    IBM>=256000       PASS
  semopm     (sem:3)  250 operations                     IBM>=250          PASS
  semmni     (sem:4)  7 of 1024 sets             (0%)    IBM>=1024         PASS

System V Shared Memory
  shmmax              68719476736 bytes                  IBM>=268435456    PASS
  shmmni              1 of 4096 sets             (0%)    IBM>=4096         PASS
  shmall              107093 of 4294967296 pages (0%)    IBM>=2097152      PASS

System Settings
  file-max            5792 of 524288 files       (1%)    IBM>=524288       PASS
  tcp_keepalive_time  300 seconds                        IBM<=300          PASS

Current User Limits (mqm)
  nofile       (-Hn)  20000 files                        IBM>=10240        PASS
  nofile       (-Sn)  20000 files                        IBM>=10240        PASS
  nproc        (-Hu)  10 of 4096 processes       (0%)    IBM>=4096         PASS
  nproc        (-Su)  10 of 4096 processes       (0%)    IBM>=4096         PASS