This follows on from an earlier post: -
I saw this exception: -
10.99.131.190:9081 28605 2014-08-27 06:40:23.824 +0 na na Thread-125 DISP 5003 1 Audit.Other.dispatcher.DISP.pogo pogo com.cognos.pogo.reportservice.ProcessManager Failure <messages><message><messageString>DPR-DPR-1035 Dispatcher detected an error.</messageString></message><message><messageString>Process BIBusTKServerMain failed to start properly.</messageString></message></messages> External Report Server process BIBusTKServerMain cannot be startedProcess BIBusTKServerMain failed to start properly.java.io.IOException: Process BIBusTKServerMain failed to start properly. at com.cognos.pogo.reportservice.ReportServerProcess.getProcessOutput(ReportServerProcess.java:182) at com.cognos.pogo.reportservice.ReportServerProcess.start(ReportServerProcess.java:125) at com.cognos.pogo.reportservice.ProcessFacade.createServerProcess(ProcessFacade.java:275) at com.cognos.pogo.reportservice.ProcessFacade.<init>(ProcessFacade.java:158) at com.cognos.pogo.reportservice.ProcessFacade.<init>(ProcessFacade.java:117) at com.cognos.pogo.reportservice.RSComponentFactory.newProcessFacade(RSComponentFactory.java:76) at com.cognos.pogo.reportservice.ProcessManager.createProcessFacade(ProcessManager.java:516) at com.cognos.pogo.reportservice.ProcessManager.
in cogserver.log during the startup of the Cognos Dispatcher, included as part of IBM Business Monitor 8.0.1.2.
I also saw this: -
2014-08-28 15:09:04.276 FATAL [m.cognos.pogo.reportservice.ProcessManager] metadataServiceHandler-MDSRVProcessManager-ProcessMgrThread: External Report Server process BmtMDProviderMain cannot be started java.io.IOException: Process BmtMDProviderMain failed to start properly.
2014-08-28 15:09:04.276 FATAL [m.cognos.pogo.reportservice.ProcessManager] metadataServiceHandler-MDSRVProcessManager-ProcessMgrThread: External Report Server process BmtMDProviderMain cannot be started java.io.IOException: Process BmtMDProviderMain failed to start properly.
in the pogo_2014_08_28.log file.
I started digging into the two exceptions.
I started digging into the two exceptions.
Both relate to Cognos binaries, and the errors imply that there's something missing e.g. a dependency.
First of all, I wanted to locate the offending binary: -
cd /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/cognos/SupClusterMember1
find . -name BIBusTKServerMain
which showed one copy.
cd /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/cognos/SupClusterMember1
find . -name BIBusTKServerMain
which showed one copy.
When I try and execute this: -
cd /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/cognos/SupClusterMember1/bin
./BIBusTKServerMain
./BIBusTKServerMain
I get this: -
./BIBusTKServerMain: error while loading shared libraries: libX11.so.6: cannot open shared object file: No such file or directory
However, there is a 64-bit version: -
cd /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/cognos/SupClusterMember1/bin64
./BIBusTKServerMain
port=39164 pid=29992
I then dug further using the ldd command: -
/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/cognos/SupClusterMember1/bin64
ldd BIBusTKServerMain
linux-vdso.so.1 => (0x00007fff58759000)
/lib64/libfreebl3.so (0x0000003b25a00000)
libtcmalloc_minimal.so.0 => ./libtcmalloc_minimal.so.0 (0x00007f71de1d6000)
libBIBusTK.so => ./libBIBusTK.so (0x00007f71ddf91000)
libBIBusTKServer.so => ./libBIBusTKServer.so (0x00007f71ddd5a000)
libCCLCore.so => ./libCCLCore.so (0x00007f71dd9f8000)
libX11.so.6 => /usr/lib64/libX11.so.6 (0x0000003b17800000)
libdl.so.2 => /lib64/libdl.so.2 (0x0000003b14c00000)
libnsl.so.1 => /lib64/libnsl.so.1 (0x0000003b25000000)
libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x0000003b1bc00000)
libm.so.6 => /lib64/libm.so.6 (0x0000003b15800000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000003b1b000000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003b15400000)
libc.so.6 => /lib64/libc.so.6 (0x0000003b15000000)
libxerces-c.so.27 => ./libxerces-c.so.27 (0x00007f71dd3d1000)
librt.so.1 => /lib64/librt.so.1 (0x0000003b16000000)
libCCLCryptoShared.so => ./libCCLCryptoShared.so (0x00007f71dd1c7000)
libCCLIDOM.so => ./libCCLIDOM.so (0x00007f71dcf78000)
libCCLHttptools.so => ./libCCLHttptools.so (0x00007f71dcd54000)
libIBJStreamsDLL.so => ./libIBJStreamsDLL.so (0x00007f71dcb30000)
libCCLCFGAPI.so => ./libCCLCFGAPI.so (0x00007f71dc903000)
libcogipf2.so => ./libcogipf2.so (0x00007f71dc6c3000)
coglog4ccl.so => ./coglog4ccl.so (0x00007f71dc420000)
libz.so.1 => ./libz.so.1 (0x00007f71dc20a000)
libicui18ncognos.so.48 => ./libicui18ncognos.so.48 (0x00007f71dbdfc000)
libicuuccognos.so.48 => ./libicuuccognos.so.48 (0x00007f71dba7c000)
libicudatacognos.so.48 => ./libicudatacognos.so.48 (0x00007f71da5f0000)
libxcb.so.1 => /usr/lib64/libxcb.so.1 (0x0000003b18000000)
/lib64/ld-linux-x86-64.so.2 (0x0000003b14800000)
libXau.so.6 => /usr/lib64/libXau.so.6 (0x0000003b17c00000)
I did the same with the other binary - BmtMDProviderMain: -
/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/cognos/SupClusterMember1/bin
ldd BmtMDProviderMain
ERROR: ld.so: object '/lib64/libfreebl3.so' from LD_PRELOAD cannot be preloaded: ignored.
linux-gate.so.1 => (0x006fa000)
libCCLIDOM.so => ./libCCLIDOM.so (0x0026a000)
libxerces-c.so.27 => ./libxerces-c.so.27 (0x002b5000)
libCCLCore.so => ./libCCLCore.so (0x00c59000)
libBIBusTKServer.so => ./libBIBusTKServer.so (0x001ac000)
libBIBusTK.so => ./libBIBusTK.so (0x001e1000)
libBMTUtil.so => ./libBMTUtil.so (0x00bcc000)
libX11.so.6 => /usr/X11R6/lib/libX11.so.6 (0x006fb000)
libdl.so.2 => /lib/libdl.so.2 (0x00221000)
libnsl.so.1 => /lib/libnsl.so.1 (0x00ea7000)
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x0079f000)
libm.so.6 => /lib/libm.so.6 (0x00226000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00695000)
libpthread.so.0 => /lib/libpthread.so.0 (0x006b3000)
libc.so.6 => /lib/libc.so.6 (0x0088a000)
libIBJStreamsDLL.so => ./libIBJStreamsDLL.so (0x006ce000)
libicuuccognos.so.48 => ./libicuuccognos.so.48 (0x00a21000)
libicudatacognos.so.48 => ./libicudatacognos.so.48 (0xf64e7000)
libz.so.1 => ./libz.so.1 (0x00250000)
libicui18ncognos.so.48 => ./libicui18ncognos.so.48 (0x00ec2000)
librt.so.1 => /lib/librt.so.1 (0x006ef000)
libCCLCFGAPI.so => ./libCCLCFGAPI.so (0x00b8f000)
libCCLHttptools.so => ./libCCLHttptools.so (0x00dc8000)
libcogipf2.so => ./libcogipf2.so (0x00dea000)
coglog4ccl.so => ./coglog4ccl.so (0x0677d000)
libCCLCryptoShared.so => ./libCCLCryptoShared.so (0x00bbc000)
libIBJHelpers.so => ./libIBJHelpers.so (0x00264000)
libBmtResources.so => ./libBmtResources.so (0x00bc5000)
libCST.so => ./libCST.so (0xf53a5000)
/lib/ld-linux.so.2 (0x00161000)
libCCLVirtualMemMgmt.so => ./libCCLVirtualMemMgmt.so (0x00e29000)
libtcmalloc_virtualmm.so.0 => ./libtcmalloc_virtualmm.so.0 (0x00e8d000)
libCAM_Crypto_Interface.so => ./libCAM_Crypto_Interface.so (0x03a48000)
libcrypto.so.0.9.8 => ./libcrypto.so.0.9.8 (0x071d2000)
In both cases, I was looking for Not Found exceptions ( ironically I took the above listings AFTER I'd solved the problem - see below ).
The Not Found exception indicates a missing library.
Initially, I thought that the solution was to add: -
export LD_PRELOAD=/lib64/libfreebl3.so
into the shell ( ~/.bashrc ) for the user that actually runs WAS - wasadmin in my case.
export LD_PRELOAD=/lib64/libfreebl3.so
into the shell ( ~/.bashrc ) for the user that actually runs WAS - wasadmin in my case.
However, that was a herring rouge.
The actual solution was to add the missing dependent libraries. The main missing library, as per the exception: -
./BIBusTKServerMain: error while loading shared libraries: libX11.so.6: cannot open shared object file: No such file or directory
was libX11.so.6.
I did have the 64-bit version of this library: -
/usr/lib64/libX11.so.6
/usr/lib64/libX11.so.6.3.0
but NOT the 32-bit version: -
/usr/X11R6/lib/libX11.so.6
/usr/X11R6/lib/libX11.so.6.1
I installed XFree86-libs-3.3.6-20.i386.rpm to give me the 32-bit libraries; mainly because I didn't have access to the source Red Hat Enterprise Linux 6.4 ISO repository.
/usr/X11R6/lib/libX11.so.6.1
I installed XFree86-libs-3.3.6-20.i386.rpm to give me the 32-bit libraries; mainly because I didn't have access to the source Red Hat Enterprise Linux 6.4 ISO repository.
I'd also installed libstdc++, again 32-bit as well as 64-bit.
I am going to go back through the entire setup again, using a vanilla RHEL image, in order that I can document the end-to-end process.
Interestingly, I'm seeing the same Cognos Dispatcher ( Report Studio ) issue on AIX 7.1.
The solution appears to be the same.
On a PoC environment, where the Cognos Report Studio works OK, I do have libX11.a installed, as below: -
cd lib
ls -1 libX*
libX11.a
libXau.a
libXaw.a
libXcursor.a
libXdamage.a
libXdmcp.a
libXevie.a
libXext.a
libXfixes.a
libXfont.a
libXi.a
libXm.a
libXmu.a
libXp.a
libXpm.a
libXrender.a
libXt.a
libXtst.a
ls -al libX11.a
lrwxrwxrwx 1 bin bin 28 Apr 29 13:16 libX11.a -> /usr/lpp/X11/lib/R7/libX11.a
lslpp -w /usr/lpp/X11/lib/R7/libX11.a
File Fileset Type
----------------------------------------------------------------------------
/usr/lpp/X11/lib/R7/libX11.a X11.base.lib File
libX11.a
libXau.a
libXaw.a
libXcursor.a
libXdamage.a
libXdmcp.a
libXevie.a
libXext.a
libXfixes.a
libXfont.a
libXi.a
libXm.a
libXmu.a
libXp.a
libXpm.a
libXrender.a
libXt.a
libXtst.a
ls -al libX11.a
lrwxrwxrwx 1 bin bin 28 Apr 29 13:16 libX11.a -> /usr/lpp/X11/lib/R7/libX11.a
lslpp -w /usr/lpp/X11/lib/R7/libX11.a
File Fileset Type
----------------------------------------------------------------------------
/usr/lpp/X11/lib/R7/libX11.a X11.base.lib File
ls -al /lib/libX11.a shows sym. link to /usr/lpp/X11/lib/R7/libX11.a
lslpp –w /usr/lpp/X11/lib/R7/libX11.a
X11.base.lib
I've asked our AIX admin to install the X11.base.lib licensed program, and will see where we get to tomorrow.
lslpp –w /usr/lpp/X11/lib/R7/libX11.a
X11.base.lib
I've asked our AIX admin to install the X11.base.lib licensed program, and will see where we get to tomorrow.
*UPDATE*
Yep, once the AIX administrator installed the missing LPP, the Cognos Report Studio worked a treat, as did the BusinessSpace Report widgets.
This is what the AIX box said: -
0509-022 Cannot load module /opt/ibm/WebSphereProfiles/AppSrv01/cognos/SupClusterMember1/bin/libCgs.so.
0509-150 Dependent module libX11.a(shr4.o) could not be loaded.
0509-022 Cannot load module libX11.a(shr4.o).
0509-026 System error: A file or directory in the path name does not exist.
0509-022 Cannot load module /opt/ibm/WebSphereProfiles/AppSrv01/cognos/SupClusterMember1/./bin/RSBIBusPlugin.cpi. 0509-150 Dependent module /opt/ibm/WebSphereProfiles/AppSrv01/cognos/SupClusterMember1/bin/libCgs.so could not be loaded. (/opt/ibm/WebSphereProfiles/AppSrv01/cognos/SupClusterMember1/./bin/RSBIBusPlugin.cpi)
and this is what LDD said: -
cd /opt/ibm/WebSphere/AppServer/cognos/bin
ldd libCgs.so
libCgs.so needs:
./libCCLCore.so
./libCCLUtility.so
./libBIBusTK.so
./libIBJStreamsDLL.so
./libIBJHelpers.so
./libCCLCFGAPI.so
./libCCLIDOM.so
./libCAM_Crypto_Interface.so
./libpgsdk.so
./libxerces-c.a
./libpdfe.so
./libfnte.so
./libz.so
./libCST.so
Cannot find libX11.a(shr4.o)
/usr/lib/libC.a(shr.o)
/usr/lib/libC.a(shr2.o)
/usr/lib/libC.a(shr3.o)
/usr/lib/libC.a(ansi_32.o)
/usr/lib/libc_r.a(shr.o)
./libicucogi18n.a(libicucogi18n40.1.so)
./libicucoguc.a(libicucoguc40.1.so)
/usr/lib/libpthreads.a(shr_xpg5.o)
./libminizip.so
./libCCLCryptoShared.so
./libCCLHttptools.so
./libcrypto.so
./libcogipf2.so
./coglog4ccl.so
./libcogfreetype.a(libcogfreetype.so.6)
./libimge.so
./libpoe.so
./libicucogle.a(libicucogle40.1.so)
/usr/lib/libc.a(shr.o)
./libCCLVirtualMemMgmt.so
/usr/lib/libC.a(shrcore.o)
/usr/lib/libC.a(ansicore_32.o)
/unix
/usr/lib/libcrypt.a(shr.o)
./libicucoguc40.a(libicucoguc40.1.so)
./libicucogdata40.a(libicucogdata40.1.so)
/usr/lib/libpthread.a(shr_xpg5.o)
/usr/lib/libpthreads.a(shr_comm.o)
./libgif.so
./libtiff.so
./libtcmalloc_virtualmm.a(mem32.o)
/usr/lib/libpmapi.a(shr.o)
/usr/lib/libodm.a(shr.o)
/usr/lib/libcfg.a(shr.o)
cd /opt/ibm/WebSphere/AppServer/cognos/bin
ldd BIBusTKServerMain
BIBusTKServerMain needs:
./libBIBusTK.so
./libBIBusTKServer.so
./libCCLCore.so
/usr/lib/libC.a(shr.o)
/usr/lib/libC.a(ansi_32.o)
/usr/lib/libc_r.a(shr.o)
/usr/lib/libtli_r.a(shr.o)
/usr/lib/libpthreads.a(shr_xpg5.o)
/usr/lib/librtl.a(shr.o)
./libCCLCryptoShared.so
./libCCLIDOM.so
./libxerces-c.a
./libCCLHttptools.so
./libIBJStreamsDLL.so
./libCCLCFGAPI.so
./libcogipf2.so
./coglog4ccl.so
./libz.so
./libicucogi18n.a(libicucogi18n40.1.so)
./libicucoguc.a(libicucoguc40.1.so)
/usr/lib/libC.a(shr2.o)
/usr/lib/libc.a(shr.o)
/usr/lib/libC.a(shrcore.o)
/usr/lib/libC.a(ansicore_32.o)
/unix
/usr/lib/libcrypt.a(shr.o)
/usr/lib/libc.a(pse.o)
/usr/lib/libpthreads.a(shr_comm.o)
/usr/lib/libC.a(shr3.o)
./libicucoguc40.a(libicucoguc40.1.so)
./libicucogdata40.a(libicucogdata40.1.so)
/usr/lib/libpthread.a(shr_xpg5.o)
cd /opt/ibm/WebSphere/AppServer/cognos/bin
ldd BmtMDProviderMain
BmtMDProviderMain needs:
./libCCLIDOM.so
./libxerces-c.a
./libCCLCore.so
./libBIBusTKServer.so
./libBIBusTK.so
./libBMTUtil.so
/usr/lib/libC.a(shr.o)
/usr/lib/libC.a(ansi_32.o)
/usr/lib/libc_r.a(shr.o)
/usr/lib/libtli_r.a(shr.o)
/usr/lib/libpthreads.a(shr_xpg5.o)
/usr/lib/librtl.a(shr.o)
./libIBJStreamsDLL.so
/usr/lib/libC.a(shr3.o)
/usr/lib/libC.a(shr2.o)
./libicucoguc.a(libicucoguc40.1.so)
./libz.so
./libicucogi18n.a(libicucogi18n40.1.so)
./libCCLCFGAPI.so
./libCCLHttptools.so
./libcogipf2.so
./coglog4ccl.so
./libCCLCryptoShared.so
./libIBJHelpers.so
./libBmtResources.so
./libCST.so
/usr/lib/libc.a(shr.o)
/usr/lib/libC.a(shrcore.o)
/usr/lib/libC.a(ansicore_32.o)
/unix
/usr/lib/libcrypt.a(shr.o)
/usr/lib/libc.a(pse.o)
/usr/lib/libpthreads.a(shr_comm.o)
./libicucogdata40.a(libicucogdata40.1.so)
/usr/lib/libpthread.a(shr_xpg5.o)
./libicucoguc40.a(libicucoguc40.1.so)
./libCCLVirtualMemMgmt.so
./libCAM_Crypto_Interface.so
./libtcmalloc_virtualmm.a(mem32.o)
./libcrypto.so
/usr/lib/libpmapi.a(shr.o)
/usr/lib/libodm.a(shr.o) /usr/lib/libcfg.a(shr.o)
No comments:
Post a Comment