Thursday, 31 January 2013

java.lang.UnsatisfiedLinkError: JCAM_Crypto_JNI (libCCLCore.so: cannot open shared object file: No such file or directory)

This caught me out during my scripted installation of IBM Business Monitor 8.0.1 ( with a back-end DB2 LUW 9.7.0.5 database ) on Red Hat Enterprise Linux.

During the startup of my clustered environment, I saw this in the SystemOut.log: -

….
[31/01/13 15:51:27:379 GMT] 00000015 FfdcProvider  W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/logs/ffdc/PCSR011.Support_858d66b1_13.01.31_15.51.27.3254262725509703725002.txt com.ibm.ws.webcontainer.servlet.ServletInstance.init 181
[31/01/13 15:51:27:381 GMT] 00000015 servlet       E com.ibm.ws.webcontainer.servlet.ServletWrapper init Uncaught.init.exception.thrown.by.servlet
[31/01/13 15:51:27:387 GMT] 00000015 webapp        E com.ibm.ws.webcontainer.webapp.WebApp commonInitializationFinally SRVE0266E: Error occured while initializing servlets: {0}
                                 javax.servlet.ServletException: SRVE0207E: Uncaught initialization exception created by servlet
        at com.ibm.ws.webcontainer.servlet.ServletWrapper.init(ServletWrapper.java:398)
        at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.init(ServletWrapperImpl.java:168)
        at com.ibm.ws.webcontainer.servlet.ServletWrapper.loadOnStartupCheck(ServletWrapper.java:1351)
        at com.ibm.ws.webcontainer.webapp.WebApp.doLoadOnStartupActions(WebApp.java:609)
        at com.ibm.ws.webcontainer.webapp.WebApp.commonInitializationFinally(WebApp.java:578)
        at com.ibm.ws.webcontainer.webapp.WebAppImpl.initialize(WebAppImpl.java:421)
        at com.ibm.ws.webcontainer.webapp.WebGroupImpl.addWebApplication(WebGroupImpl.java:88)
        at com.ibm.ws.webcontainer.VirtualHostImpl.addWebApplication(VirtualHostImpl.java:169)
        at com.ibm.ws.webcontainer.WSWebContainer.addWebApp(WSWebContainer.java:749)
        at com.ibm.ws.webcontainer.WSWebContainer.addWebApplication(WSWebContainer.java:634)
        at com.ibm.ws.webcontainer.component.WebContainerImpl.install(WebContainerImpl.java:422)
        at com.ibm.ws.webcontainer.component.WebContainerImpl.start(WebContainerImpl.java:714)
        at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:1160)
        at com.ibm.ws.runtime.component.DeployedApplicationImpl.fireDeployedObjectStart(DeployedApplicationImpl.java:1369)
        at com.ibm.ws.runtime.component.DeployedModuleImpl.start(DeployedModuleImpl.java:639)
        at com.ibm.ws.runtime.component.DeployedApplicationImpl.start(DeployedApplicationImpl.java:967)
        at com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplication(ApplicationMgrImpl.java:766)
        at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:2153)
        at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:445)
        at com.ibm.ws.runtime.component.CompositionUnitImpl.start(CompositionUnitImpl.java:123)
        at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:388)
        at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.access$500(CompositionUnitMgrImpl.java:116)
        at com.ibm.ws.runtime.component.CompositionUnitMgrImpl$CUInitializer.run(CompositionUnitMgrImpl.java:994)
        at com.ibm.wsspi.runtime.component.WsComponentImpl$_AsynchInitializer.run(WsComponentImpl.java:349)
        at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1690)
Caused by: java.lang.UnsatisfiedLinkError: JCAM_Crypto_JNI (libCCLCore.so: cannot open shared object file: No such file or directory)
        at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1053)
        at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:1017)
        at java.lang.System.loadLibrary(System.java:509)
        at com.cognos.accman.jcam.crypto.jni.JNISystemProperties.<clinit>(JNISystemProperties.java:27)
        at java.lang.J9VMInternals.initializeImpl(Native Method)
        at java.lang.J9VMInternals.initialize(J9VMInternals.java:228)
        at com.cognos.accman.jcam.crypto.SystemProtectionSession._getRawKey(SystemProtectionSession.java:244)
        at com.cognos.accman.jcam.crypto.SystemProtectionSession.<init>(SystemProtectionSession.java:57)
        at com.cognos.accman.jcam.crypto.misc.Configuration.<init>(Configuration.java:70)
        at com.cognos.accman.jcam.crypto.misc.Configuration.getInstanceWithDefaultConfig(Configuration.java:113)
        at com.cognos.accman.jcam.crypto.misc.JVMEnvironment.isFIPSSupportedOS(JVMEnvironment.java:212)
        at com.cognos.accman.jcam.crypto.CAMFactory.checkFIPSSupport(CAMFactory.java:2814)
        at com.cognos.accman.jcam.crypto.CAMFactory.initialize(CAMFactory.java:125)
        at com.cognos.indications.LogIPFControl.initCAMCrypto(LogIPFControl.java:566)
        at com.cognos.indications.LogIPFControl.initialize(LogIPFControl.java:255)
        at com.cognos.indications.LogIPFControl.start(LogIPFControl.java:235)
        at com.cognos.p2plb.config.ConfigStoreServlet.init(ConfigStoreServlet.java:122)
        at com.cognos.pogo.isolation.ServletWrapper.initServlet(ServletWrapper.java:121)
        at com.cognos.pogo.isolation.ServletWrapper.init(ServletWrapper.java:101)
        at com.ibm.ws.webcontainer.servlet.ServletWrapper.init(ServletWrapper.java:336)
        ... 24 more

[31/01/13 15:51:27:397 GMT] 00000015 webcontainer  I com.ibm.ws.webcontainer.VirtualHostImpl addWebApplication SRVE0250I: Web Module IBM Cognos has been bound to default_host[*:8444,*:8443,*:443].
[31/01/13 15:51:27:466 GMT] 00000015 ApplicationMg A   WSVR0221I: Application started: IBM Cognos
….

Following this Technote: -


I used the ldd command against the Linux library that was failing to load - libCCLCore.so : -

$ locate libCCLCore.so

/opt/IBM/WebSphere/AppServer/cognos/bin/libCCLCore.so
/opt/IBM/WebSphere/AppServer/cognos/bin64/libCCLCore.so
/opt/IBM/WebSphere/AppServer/cognos/cgi-bin/libCCLCore.so

$ ldd /opt/IBM/WebSphere/AppServer/cognos/bin/libCCLCore.so

 linux-gate.so.1 =>  (0x00644000)
libz.so.1 => /lib/libz.so.1 (0x001c0000)
libicucogi18n.so.40 => not found
libicucoguc.so.40 => not found
 libX11.so.6 => /usr/lib/libX11.so.6 (0x00e3b000)
libdl.so.2 => /lib/libdl.so.2 (0x00ba7000)
libnsl.so.1 => /lib/libnsl.so.1 (0x00a76000)
libpthread.so.0 => /lib/libpthread.so.0 (0x006b3000)
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00395000)
libm.so.6 => /lib/libm.so.6 (0x0014d000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00177000)
libc.so.6 => /lib/libc.so.6 (0x001d4000)
libxcb.so.1 => /usr/lib/libxcb.so.1 (0x00195000)
/lib/ld-linux.so.2 (0x008ca000)
libXau.so.6 => /usr/lib/libXau.so.6 (0x001b3000)

$ ldd /opt/IBM/WebSphere/AppServer/cognos/bin64/libCCLCore.so

 linux-vdso.so.1 =>  (0x00007fff04aac000)
libz.so.1 => /lib64/libz.so.1 (0x00007ffbf50c3000)
libicucogi18n.so.40 => not found
libicucoguc.so.40 => not found
 libX11.so.6 => /usr/lib64/libX11.so.6 (0x00007ffbf4d83000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007ffbf4b7e000)
libnsl.so.1 => /lib64/libnsl.so.1 (0x00007ffbf4965000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ffbf4748000)
libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007ffbf4441000)
libm.so.6 => /lib64/libm.so.6 (0x00007ffbf41bd000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007ffbf3fa7000)
libc.so.6 => /lib64/libc.so.6 (0x00007ffbf3c13000)
libxcb.so.1 => /usr/lib64/libxcb.so.1 (0x00007ffbf39f8000)
/lib64/ld-linux-x86-64.so.2 (0x00000032d4400000)
libXau.so.6 => /usr/lib64/libXau.so.6 (0x00007ffbf37f4000)

$ ldd /opt/IBM/WebSphere/AppServer/cognos/cgi-bin/libCCLCore.so

linux-gate.so.1 =>  (0x006cb000)
libz.so.1 => /lib/libz.so.1 (0x00e73000)
libicucogi18n.so.40 => not found
libicucoguc.so.40 => not found
 libX11.so.6 => /usr/lib/libX11.so.6 (0x00952000)
libdl.so.2 => /lib/libdl.so.2 (0x00cf3000)
libnsl.so.1 => /lib/libnsl.so.1 (0x00da1000)
libpthread.so.0 => /lib/libpthread.so.0 (0x002d2000)
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x002ed000)
libm.so.6 => /lib/libm.so.6 (0x0088d000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x003d8000)
libc.so.6 => /lib/libc.so.6 (0x003f6000)
libxcb.so.1 => /usr/lib/libxcb.so.1 (0x0058c000)
/lib/ld-linux.so.2 (0x008ca000)
libXau.so.6 => /usr/lib/libXau.so.6 (0x005aa000)

I went around this loop for a while, using the locate and ldd commands to track down the libicucogi18n.so.40 and libicucoguc.so.40 libraries, but to no avail.

I then did some Googling, as one does, and found this developerWorks forum post: -


which led me to this Technote: -


Now I'm not using Oracle but the latter article said something: -

The LD_LIBRARY_PATH does not reference the 32-bit library for the Oracle Client. The 64-bit client is being referenced instead

that made me think.

I checked the IBM Business Monitor 8.0.1 Installation Guide, which said: -



Following that, I updated the LD_LIBRARY_PATH property ( Servers > Clusters > WebSphere application server clusters > PCSR01.Support > Cluster members > PCSR011.Support > Process definition > Environment Entries > LD_LIBRARY_PATH ) for the PCSR01.Support cluster member ( that which runs the Cognos BI code ) from: -

/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/cognos/PCSR011.Support/bin64

to: -

/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/cognos/PCSR011.Support/bin64:/opt/ibm/db2/V9.7/lib32/

and restarted my cluster.

As they say in the movies, Job Done !

Next step is to script the same using wsadmin and Jython.


at

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)

Note to self - use kubectl to query images in a pod or deployment

In both cases, we use JSON ... For a deployment, we can do this: - kubectl get deployment foobar --namespace snafu --output jsonpath="{...