Saturday 30 August 2014

Cognos on Linux - Dependencies - Soup to Nuts

A follow-up: -



Replicate the problem

cd /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/cognos/SupClusterMember1/bin
./BIBusTKServerMain

./BIBusTKServerMain: error while loading shared libraries: libX11.so.6: cannot open shared object file: No such file or directory

Diagnose using LDD

cd /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/cognos/SupClusterMember1/bin
ldd BIBusTKServerMain
linux-gate.so.1 =>  (0x00917000)
libtcmalloc_minimal.so.0 => ./libtcmalloc_minimal.so.0 (0x00b27000)
libBIBusTK.so => ./libBIBusTK.so (0x00a31000)
libBIBusTKServer.so => ./libBIBusTKServer.so (0x007d5000)
libCCLCore.so => ./libCCLCore.so (0x00110000)
libX11.so.6 => not found
libdl.so.2 => /lib/libdl.so.2 (0x006e5000)
libnsl.so.1 => /lib/libnsl.so.1 (0x00663000)
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x004af000)
libm.so.6 => /lib/libm.so.6 (0x002e9000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00273000)
libpthread.so.0 => /lib/libpthread.so.0 (0x00ef6000)
libc.so.6 => /lib/libc.so.6 (0x00b63000)
libxerces-c.so.27 => ./libxerces-c.so.27 (0x00f11000)
librt.so.1 => /lib/librt.so.1 (0x00291000)
libCCLCryptoShared.so => ./libCCLCryptoShared.so (0x003f6000)
libCCLIDOM.so => ./libCCLIDOM.so (0x00d16000)
libCCLHttptools.so => ./libCCLHttptools.so (0x0096c000)
libIBJStreamsDLL.so => ./libIBJStreamsDLL.so (0x00abc000)
libX11.so.6 => not found
libCCLCFGAPI.so => ./libCCLCFGAPI.so (0x0029a000)
libcogipf2.so => ./libcogipf2.so (0x00415000)
coglog4ccl.so => ./coglog4ccl.so (0x0059a000)
libX11.so.6 => not found
libz.so.1 => ./libz.so.1 (0x002c9000)
libicucogi18n.so.40 => ./libicucogi18n.so.40 (0x07541000)
libicucoguc.so.40 => ./libicucoguc.so.40 (0x00d61000)
/lib/ld-linux.so.2 (0x003a3000)
libX11.so.6 => not found
libicucogdata.so.40 => ./libicucogdata.so.40 (0xf6ac7000)
libX11.so.6 => not found
libX11.so.6 => not found
libX11.so.6 => not found
libX11.so.6 => not found
libX11.so.6 => not found
libX11.so.6 => not found
libX11.so.6 => not found


Look at the missing library

locate libX11.so.6

/usr/lib64/libX11.so.6
/usr/lib64/libX11.so.6.3.0


See, there's a 64-bit version, but no 32-bit version :-(

Look at the RPMs that provide that library

yum list | grep -i libX11

libX11.x86_64                           1.3-2.el6                        @anaconda-RedHatEnterpriseLinux-201206132210.x86_64/6.3
libX11-common.noarch                    1.3-2.el6                        @anaconda-RedHatEnterpriseLinux-201206132210.x86_64/6.3
libX11.i686                             1.3-2.el6                        server 
libX11-devel.i686                       1.3-2.el6                        server 
libX11-devel.x86_64                     1.3-2.el6                        server 

Look at what's installed

rpm -qa | grep -i libX11

libX11-1.3-2.el6.x86_64
libX11-common-1.3-2.el6.noarch


Install the missing RPM ( the 386 version )

yum install libX11.i686

Test again using LDD

ldd BIBusTKServerMain

linux-gate.so.1 =>  (0x00f48000)
libtcmalloc_minimal.so.0 => ./libtcmalloc_minimal.so.0 (0x00110000)
libBIBusTK.so => ./libBIBusTK.so (0x004a3000)
libBIBusTKServer.so => ./libBIBusTKServer.so (0x002f7000)
libCCLCore.so => ./libCCLCore.so (0x00dcd000)
libX11.so.6 => /usr/lib/libX11.so.6 (0x0014c000)
libdl.so.2 => /lib/libdl.so.2 (0x00bab000)
libnsl.so.1 => /lib/libnsl.so.1 (0x00661000)
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00c33000)
libm.so.6 => /lib/libm.so.6 (0x0061d000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00284000)
libpthread.so.0 => /lib/libpthread.so.0 (0x00ace000)
libc.so.6 => /lib/libc.so.6 (0x0067c000)
libxerces-c.so.27 => ./libxerces-c.so.27 (0x00f49000)
librt.so.1 => /lib/librt.so.1 (0x004f1000)
libCCLCryptoShared.so => ./libCCLCryptoShared.so (0x00522000)
libCCLIDOM.so => ./libCCLIDOM.so (0x00d64000)
libCCLHttptools.so => ./libCCLHttptools.so (0x002a2000)
libIBJStreamsDLL.so => ./libIBJStreamsDLL.so (0x002c4000)
libCCLCFGAPI.so => ./libCCLCFGAPI.so (0x005e3000)
libcogipf2.so => ./libcogipf2.so (0x0032b000)
coglog4ccl.so => ./coglog4ccl.so (0x0053c000)
libz.so.1 => ./libz.so.1 (0x00419000)
libicucogi18n.so.40 => ./libicucogi18n.so.40 (0x00904000)
libicucoguc.so.40 => ./libicucoguc.so.40 (0x06517000)
libxcb.so.1 => /usr/lib/libxcb.so.1 (0x0036a000)
/lib/ld-linux.so.2 (0x003a3000)
libicucogdata.so.40 => ./libicucogdata.so.40 (0xf6b5e000)
libXau.so.6 => /usr/lib/libXau.so.6 (0x002e5000)


Now try and run the binary

./BIBusTKServerMain

port=33768 pid=40133

Sorted :-)

I love this stuff :-)

Alias command on Unix - why am I late to the party ?

So I regularly make use of this Linux command: -

history | cut -c 8-

to get the Bash history but without the numbers.

Before

  131  /opt/IBM/WebSphere/AppServer/bin/sibDDLGenerator.sh -system db2 -version 9.7 -platform unix -schema MONCM00 -statementend ";" -user db2user1 >> ~/createMESchemas.sql
  132  /opt/IBM/WebSphere/AppServer/bin/sibDDLGenerator.sh -system db2 -version 9.7 -platform unix -schema MONME00 -statementend ";" -user db2user1 >> ~/createMESchemas.sql
  133  db2 -tvf createMESchemas.sql


After

/opt/IBM/WebSphere/AppServer/bin/sibDDLGenerator.sh -system db2 -version 9.7 -platform unix -schema MONCM00 -statementend ";" -user db2user1 >> ~/createMESchemas.sql
/opt/IBM/WebSphere/AppServer/bin/sibDDLGenerator.sh -system db2 -version 9.7 -platform unix -schema MONME00 -statementend ";" -user db2user1 >> ~/createMESchemas.sql
db2 -tvf createMESchemas.sql


See, easy documentation :-)

However, I can never remember the format of the improved history command, so I end up opening a text file on my Mac to find it.

So why don't I use the alias command, I hear you say ?

Well, I do now :-)

This is the relevant alias command: -

alias hist='history | cut -c 8-'

which now gives me a shortened command ( so I actually type less ! ) which achieves my requirement.

On it's own, the alias command shows me ALL of my aliases: -

alias

alias hist='history | cut -c 8-'
alias l.='ls -d .* --color=auto'
alias ll='ls -l --color=auto'
alias ls='ls --color=auto'
alias vi='vim'
alias which='alias | /usr/bin/which --tty-only --read-alias --show-dot --show-tilde'


Guess what I've added to my .bashrc :-)

Friday 29 August 2014

Notes to self - Setting up DB2 Client connectivity

It's been ~10 years since I last did this ( DB2 on Unix to DB2 on z/OS or OS/400 ), but I now have a requirement to deliver a DB2 client for WebSphere MQ interaction ( MQ will use ODBC rather than JDBC ).

I knocked this up using a pair of VMs on my Macbook .....

Response File -  ~/db2client.rsp

PROD                      = CLIENT
FILE                      = /opt/ibm/db2/V10.1
LIC_AGREEMENT             = ACCEPT
INSTALL_TYPE              = TYPICAL

Install DB2 Client Binaries - as root

./db2setup -r ~/db2client.rsp

Create DB2 User and Group - as root

groupadd db2users
useradd -g db2users -d /home/db2user1 db2user1
passwd db2user1

Create DB2 Instance ( required for client connectivity ) - as root

/opt/ibm/db2/V10.1/instance/db2icrt -a CLIENT db2user1

Catalog Remote DB2 Server ( Node ) - as db2user1

db2 catalog tcpip node rhel6 remote rhel6 server DB2c_db2inst1

( node and remote refer to DB2 server hostname, server refers to DB2 service name in /etc/services )

Validate Node Directory

db2 list node directory

Test Connectivity - as db2user1

db2 attach to rhel6 user db2inst1 using passw0rd
db2 list applications
db2 detach

Catalog Remote Database - as db2user1

db2 catalog database foobar at node rhel6

( database refers to database alias on remote server, node refers to previously catalogued node )

Validate DB Directory - as db2user1

db2 list db directory

Create Database on Server - as db2inst1

db2 "create table snafu(surname char(16) not null,firstname char(10) not null)"

Use Database - as db2user1

db2 connect to foobar user db2inst1 using passw0rd
db2 list tables for all | more
db2 "select * from db2inst1.snafu"
db2 "describe select * from db2inst1.snafu"
db2 "insert into db2inst1.snafu(surname,firstname) values('Hay','Dave')"
db2 "select * from db2inst1.snafu"
db2 terminate

Shiny :-)

Thursday 28 August 2014

IBM Business Monitor 8.0.1.2, Cognos BI and Unix - Dependencies Again

This follows on from an earlier post: -


I saw this exception: -

10.99.131.190:9081      28605   2014-08-27 06:40:23.824 +0      na      na                      Thread-125      DISP    5003    1       Audit.Other.dispatcher.DISP.pogo        pogo    com.cognos.pogo.reportservice.ProcessManager            Failure <messages><message><messageString>DPR-DPR-1035 Dispatcher detected an error.</messageString></message><message><messageString>Process BIBusTKServerMain failed to start properly.</messageString></message></messages>  External Report Server process BIBusTKServerMain cannot be startedProcess BIBusTKServerMain failed to start properly.java.io.IOException: Process BIBusTKServerMain failed to start properly.   at com.cognos.pogo.reportservice.ReportServerProcess.getProcessOutput(ReportServerProcess.java:182)     at com.cognos.pogo.reportservice.ReportServerProcess.start(ReportServerProcess.java:125)        at com.cognos.pogo.reportservice.ProcessFacade.createServerProcess(ProcessFacade.java:275)      at com.cognos.pogo.reportservice.ProcessFacade.<init>(ProcessFacade.java:158)   at com.cognos.pogo.reportservice.ProcessFacade.<init>(ProcessFacade.java:117)   at com.cognos.pogo.reportservice.RSComponentFactory.newProcessFacade(RSComponentFactory.java:76)        at com.cognos.pogo.reportservice.ProcessManager.createProcessFacade(ProcessManager.java:516)    at com.cognos.pogo.reportservice.ProcessManager.

in cogserver.log during the startup of the Cognos Dispatcher, included as part of IBM Business Monitor 8.0.1.2.

I also saw this: -

2014-08-28 15:09:04.276 FATAL [m.cognos.pogo.reportservice.ProcessManager] metadataServiceHandler-MDSRVProcessManager-ProcessMgrThread: External Report Server process BmtMDProviderMain cannot be started java.io.IOException: Process BmtMDProviderMain failed to start properly.

in the pogo_2014_08_28.log file.

I started digging into the two exceptions.

Both relate to Cognos binaries, and the errors imply that there's something missing e.g. a dependency.

First of all, I wanted to locate the offending binary: -

cd /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/cognos/SupClusterMember1
find . -name BIBusTKServerMain


which showed one copy.

When I try and execute this: -

cd /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/cognos/SupClusterMember1/bin
./BIBusTKServerMain

I get this: -

./BIBusTKServerMain: error while loading shared libraries: libX11.so.6: cannot open shared object file: No such file or directory

However, there is a 64-bit version: -

cd /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/cognos/SupClusterMember1/bin64
./BIBusTKServerMain

port=39164 pid=29992

I then dug further using the ldd command: -

/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/cognos/SupClusterMember1/bin64
ldd BIBusTKServerMain

linux-vdso.so.1 =>  (0x00007fff58759000)
/lib64/libfreebl3.so (0x0000003b25a00000)
libtcmalloc_minimal.so.0 => ./libtcmalloc_minimal.so.0 (0x00007f71de1d6000)
libBIBusTK.so => ./libBIBusTK.so (0x00007f71ddf91000)
libBIBusTKServer.so => ./libBIBusTKServer.so (0x00007f71ddd5a000)
libCCLCore.so => ./libCCLCore.so (0x00007f71dd9f8000)
libX11.so.6 => /usr/lib64/libX11.so.6 (0x0000003b17800000)
libdl.so.2 => /lib64/libdl.so.2 (0x0000003b14c00000)
libnsl.so.1 => /lib64/libnsl.so.1 (0x0000003b25000000)
libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x0000003b1bc00000)
libm.so.6 => /lib64/libm.so.6 (0x0000003b15800000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000003b1b000000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003b15400000)
libc.so.6 => /lib64/libc.so.6 (0x0000003b15000000)
libxerces-c.so.27 => ./libxerces-c.so.27 (0x00007f71dd3d1000)
librt.so.1 => /lib64/librt.so.1 (0x0000003b16000000)
libCCLCryptoShared.so => ./libCCLCryptoShared.so (0x00007f71dd1c7000)
libCCLIDOM.so => ./libCCLIDOM.so (0x00007f71dcf78000)
libCCLHttptools.so => ./libCCLHttptools.so (0x00007f71dcd54000)
libIBJStreamsDLL.so => ./libIBJStreamsDLL.so (0x00007f71dcb30000)
libCCLCFGAPI.so => ./libCCLCFGAPI.so (0x00007f71dc903000)
libcogipf2.so => ./libcogipf2.so (0x00007f71dc6c3000)
coglog4ccl.so => ./coglog4ccl.so (0x00007f71dc420000)
libz.so.1 => ./libz.so.1 (0x00007f71dc20a000)
libicui18ncognos.so.48 => ./libicui18ncognos.so.48 (0x00007f71dbdfc000)
libicuuccognos.so.48 => ./libicuuccognos.so.48 (0x00007f71dba7c000)
libicudatacognos.so.48 => ./libicudatacognos.so.48 (0x00007f71da5f0000)
libxcb.so.1 => /usr/lib64/libxcb.so.1 (0x0000003b18000000)
/lib64/ld-linux-x86-64.so.2 (0x0000003b14800000)
libXau.so.6 => /usr/lib64/libXau.so.6 (0x0000003b17c00000)


I did the same with the other binary - BmtMDProviderMain: -

/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/cognos/SupClusterMember1/bin
ldd BmtMDProviderMain

ERROR: ld.so: object '/lib64/libfreebl3.so' from LD_PRELOAD cannot be preloaded: ignored.
linux-gate.so.1 =>  (0x006fa000)
libCCLIDOM.so => ./libCCLIDOM.so (0x0026a000)
libxerces-c.so.27 => ./libxerces-c.so.27 (0x002b5000)
libCCLCore.so => ./libCCLCore.so (0x00c59000)
libBIBusTKServer.so => ./libBIBusTKServer.so (0x001ac000)
libBIBusTK.so => ./libBIBusTK.so (0x001e1000)
libBMTUtil.so => ./libBMTUtil.so (0x00bcc000)
libX11.so.6 => /usr/X11R6/lib/libX11.so.6 (0x006fb000)
libdl.so.2 => /lib/libdl.so.2 (0x00221000)
libnsl.so.1 => /lib/libnsl.so.1 (0x00ea7000)
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x0079f000)
libm.so.6 => /lib/libm.so.6 (0x00226000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00695000)
libpthread.so.0 => /lib/libpthread.so.0 (0x006b3000)
libc.so.6 => /lib/libc.so.6 (0x0088a000)
libIBJStreamsDLL.so => ./libIBJStreamsDLL.so (0x006ce000)
libicuuccognos.so.48 => ./libicuuccognos.so.48 (0x00a21000)
libicudatacognos.so.48 => ./libicudatacognos.so.48 (0xf64e7000)
libz.so.1 => ./libz.so.1 (0x00250000)
libicui18ncognos.so.48 => ./libicui18ncognos.so.48 (0x00ec2000)
librt.so.1 => /lib/librt.so.1 (0x006ef000)
libCCLCFGAPI.so => ./libCCLCFGAPI.so (0x00b8f000)
libCCLHttptools.so => ./libCCLHttptools.so (0x00dc8000)
libcogipf2.so => ./libcogipf2.so (0x00dea000)
coglog4ccl.so => ./coglog4ccl.so (0x0677d000)
libCCLCryptoShared.so => ./libCCLCryptoShared.so (0x00bbc000)
libIBJHelpers.so => ./libIBJHelpers.so (0x00264000)
libBmtResources.so => ./libBmtResources.so (0x00bc5000)
libCST.so => ./libCST.so (0xf53a5000)
/lib/ld-linux.so.2 (0x00161000)
libCCLVirtualMemMgmt.so => ./libCCLVirtualMemMgmt.so (0x00e29000)
libtcmalloc_virtualmm.so.0 => ./libtcmalloc_virtualmm.so.0 (0x00e8d000)
libCAM_Crypto_Interface.so => ./libCAM_Crypto_Interface.so (0x03a48000)
libcrypto.so.0.9.8 => ./libcrypto.so.0.9.8 (0x071d2000)


In both cases, I was looking for Not Found exceptions ( ironically I took the above listings AFTER I'd solved the problem - see below ).

The Not Found exception indicates a missing library.

Initially, I thought that the solution was to add: -

export LD_PRELOAD=/lib64/libfreebl3.so

into the shell ( ~/.bashrc ) for the user that actually runs WAS - wasadmin in my case.

However, that was a herring rouge.

The actual solution was to add the missing dependent libraries. The main missing library, as per the exception: -

./BIBusTKServerMain: error while loading shared libraries: libX11.so.6: cannot open shared object file: No such file or directory

was libX11.so.6.

I did have the 64-bit version of this library: -

/usr/lib64/libX11.so.6
/usr/lib64/libX11.so.6.3.0

but NOT the 32-bit version: -

/usr/X11R6/lib/libX11.so.6
/usr/X11R6/lib/libX11.so.6.1

I installed XFree86-libs-3.3.6-20.i386.rpm to give me the 32-bit libraries; mainly because I didn't have access to the source Red Hat Enterprise Linux 6.4 ISO repository.

I'd also installed libstdc++, again 32-bit as well as 64-bit.

I am going to go back through the entire setup again, using a vanilla RHEL image, in order that I can document the end-to-end process.

Interestingly, I'm seeing the same Cognos Dispatcher ( Report Studio ) issue on AIX 7.1.

The solution appears to be the same.

On a PoC environment, where the Cognos Report Studio works OK, I do have libX11.a installed, as below: -

cd lib
ls -1 libX*

libX11.a
libXau.a
libXaw.a
libXcursor.a
libXdamage.a
libXdmcp.a
libXevie.a
libXext.a
libXfixes.a
libXfont.a
libXi.a
libXm.a
libXmu.a
libXp.a
libXpm.a
libXrender.a
libXt.a
libXtst.a


ls -al libX11.a

lrwxrwxrwx    1 bin      bin              28 Apr 29 13:16 libX11.a -> /usr/lpp/X11/lib/R7/libX11.a

lslpp -w /usr/lpp/X11/lib/R7/libX11.a

  File                                        Fileset               Type
  ----------------------------------------------------------------------------
  /usr/lpp/X11/lib/R7/libX11.a                X11.base.lib          File


ls -al /lib/libX11.a shows sym. link to /usr/lpp/X11/lib/R7/libX11.a

lslpp –w /usr/lpp/X11/lib/R7/libX11.a

X11.base.lib

I've asked our AIX admin to install the X11.base.lib licensed program, and will see where we get to tomorrow.

*UPDATE*

Yep, once the AIX administrator installed the missing LPP, the Cognos Report Studio worked a treat, as did the BusinessSpace Report widgets.

This is what the AIX box said: -

0509-022 Cannot load module /opt/ibm/WebSphereProfiles/AppSrv01/cognos/SupClusterMember1/bin/libCgs.so.
0509-150 Dependent module libX11.a(shr4.o) could not be loaded.
0509-022 Cannot load module libX11.a(shr4.o).
0509-026 System error: A file or directory in the path name does not exist.
0509-022 Cannot load module /opt/ibm/WebSphereProfiles/AppSrv01/cognos/SupClusterMember1/./bin/RSBIBusPlugin.cpi.       0509-150 Dependent module /opt/ibm/WebSphereProfiles/AppSrv01/cognos/SupClusterMember1/bin/libCgs.so could not be loaded. (/opt/ibm/WebSphereProfiles/AppSrv01/cognos/SupClusterMember1/./bin/RSBIBusPlugin.cpi)


 and this is what LDD said: -

cd /opt/ibm/WebSphere/AppServer/cognos/bin
ldd libCgs.so

libCgs.so needs:
         ./libCCLCore.so
         ./libCCLUtility.so
         ./libBIBusTK.so
         ./libIBJStreamsDLL.so
         ./libIBJHelpers.so
         ./libCCLCFGAPI.so
         ./libCCLIDOM.so
         ./libCAM_Crypto_Interface.so
         ./libpgsdk.so
         ./libxerces-c.a
         ./libpdfe.so
         ./libfnte.so
         ./libz.so
         ./libCST.so
Cannot find libX11.a(shr4.o)
         /usr/lib/libC.a(shr.o)
         /usr/lib/libC.a(shr2.o)
         /usr/lib/libC.a(shr3.o)
         /usr/lib/libC.a(ansi_32.o)
         /usr/lib/libc_r.a(shr.o)
         ./libicucogi18n.a(libicucogi18n40.1.so)
         ./libicucoguc.a(libicucoguc40.1.so)
         /usr/lib/libpthreads.a(shr_xpg5.o)
         ./libminizip.so
         ./libCCLCryptoShared.so
         ./libCCLHttptools.so
         ./libcrypto.so
         ./libcogipf2.so
         ./coglog4ccl.so
         ./libcogfreetype.a(libcogfreetype.so.6)
         ./libimge.so
         ./libpoe.so
         ./libicucogle.a(libicucogle40.1.so)
         /usr/lib/libc.a(shr.o)
         ./libCCLVirtualMemMgmt.so
         /usr/lib/libC.a(shrcore.o)
         /usr/lib/libC.a(ansicore_32.o)
         /unix
         /usr/lib/libcrypt.a(shr.o)
         ./libicucoguc40.a(libicucoguc40.1.so)
         ./libicucogdata40.a(libicucogdata40.1.so)
         /usr/lib/libpthread.a(shr_xpg5.o)
         /usr/lib/libpthreads.a(shr_comm.o)
         ./libgif.so
         ./libtiff.so
         ./libtcmalloc_virtualmm.a(mem32.o)
         /usr/lib/libpmapi.a(shr.o)
         /usr/lib/libodm.a(shr.o)
         /usr/lib/libcfg.a(shr.o)


cd /opt/ibm/WebSphere/AppServer/cognos/bin
ldd BIBusTKServerMain

BIBusTKServerMain needs:
         ./libBIBusTK.so
         ./libBIBusTKServer.so
         ./libCCLCore.so
         /usr/lib/libC.a(shr.o)
         /usr/lib/libC.a(ansi_32.o)
         /usr/lib/libc_r.a(shr.o)
         /usr/lib/libtli_r.a(shr.o)
         /usr/lib/libpthreads.a(shr_xpg5.o)
         /usr/lib/librtl.a(shr.o)
         ./libCCLCryptoShared.so
         ./libCCLIDOM.so
         ./libxerces-c.a
         ./libCCLHttptools.so
         ./libIBJStreamsDLL.so
         ./libCCLCFGAPI.so
         ./libcogipf2.so
         ./coglog4ccl.so
         ./libz.so
         ./libicucogi18n.a(libicucogi18n40.1.so)
         ./libicucoguc.a(libicucoguc40.1.so)
         /usr/lib/libC.a(shr2.o)
         /usr/lib/libc.a(shr.o)
         /usr/lib/libC.a(shrcore.o)
         /usr/lib/libC.a(ansicore_32.o)
         /unix
         /usr/lib/libcrypt.a(shr.o)
         /usr/lib/libc.a(pse.o)
         /usr/lib/libpthreads.a(shr_comm.o)
         /usr/lib/libC.a(shr3.o)
         ./libicucoguc40.a(libicucoguc40.1.so)
        ./libicucogdata40.a(libicucogdata40.1.so)
         /usr/lib/libpthread.a(shr_xpg5.o)


cd /opt/ibm/WebSphere/AppServer/cognos/bin
ldd BmtMDProviderMain

BmtMDProviderMain needs:
         ./libCCLIDOM.so
         ./libxerces-c.a
         ./libCCLCore.so
         ./libBIBusTKServer.so
         ./libBIBusTK.so
         ./libBMTUtil.so
         /usr/lib/libC.a(shr.o)
         /usr/lib/libC.a(ansi_32.o)
         /usr/lib/libc_r.a(shr.o)
         /usr/lib/libtli_r.a(shr.o)
         /usr/lib/libpthreads.a(shr_xpg5.o)
         /usr/lib/librtl.a(shr.o)
         ./libIBJStreamsDLL.so
         /usr/lib/libC.a(shr3.o)
         /usr/lib/libC.a(shr2.o)
         ./libicucoguc.a(libicucoguc40.1.so)
         ./libz.so
         ./libicucogi18n.a(libicucogi18n40.1.so)
         ./libCCLCFGAPI.so
         ./libCCLHttptools.so
         ./libcogipf2.so
         ./coglog4ccl.so
         ./libCCLCryptoShared.so
         ./libIBJHelpers.so
         ./libBmtResources.so
         ./libCST.so
         /usr/lib/libc.a(shr.o)
         /usr/lib/libC.a(shrcore.o)
         /usr/lib/libC.a(ansicore_32.o)
         /unix
         /usr/lib/libcrypt.a(shr.o)
         /usr/lib/libc.a(pse.o)
         /usr/lib/libpthreads.a(shr_comm.o)
         ./libicucogdata40.a(libicucogdata40.1.so)
         /usr/lib/libpthread.a(shr_xpg5.o)
         ./libicucoguc40.a(libicucoguc40.1.so)
         ./libCCLVirtualMemMgmt.so
         ./libCAM_Crypto_Interface.so
         ./libtcmalloc_virtualmm.a(mem32.o)
         ./libcrypto.so
         /usr/lib/libpmapi.a(shr.o)
         /usr/lib/libodm.a(shr.o)         /usr/lib/libcfg.a(shr.o)

Wednesday 27 August 2014

A Reminder - Which Cluster Does BAM > BPM need ?


<snip>
WebSphere® Business Monitor requires an event emitter factory to generate and send events. Use the wbmConfigureEventEmitterFactory command to install and configure the event emitter factory on a server or cluster. You must already have a common event infrastructure (CEI event service). (Otherwise, run the wbmDeployCEIEventService command to create one.) In the four-cluster topology, the event emitter factory is installed on the support cluster.
</snip>


<snip>
Use the wbmDeployCEIEventService command to install a CEI event service on a server or cluster and create the resources it requires (the service integration bus and messaging engine). You can also configure security and enable the default CEI event service data store if required. (The event service data store is not recommended for production environments.) In the core topology for high availability, the CEI event service is installed on the support cluster.
</snip>


Tuesday 26 August 2014

Gah, not again - Installation of IBM Cognos Business Intelligence has failed.

As if by magic, here it is again: -

ERROR: Error during "install" phase:

Installation of IBM Cognos Business Intelligence has failed.

Return code : -1

Building upon previous posts: -




This time I was seeing: -

<entry num='171' time='1409082967257' elapsed='00:50.44' level='ERROR' thread='Thread-23'>
 <logger>global</logger>
 <class>com.ibm.cognos.exec.Install</class>
 <method>exec</method>
 <message>com.ibm.cognos.exec.Install
Command: [Ljava.lang.String;@67a467a4
Return code : -1
StdOut :
StdErr : Cannot run program "/opt/IBM/WebSphere/AppServer/CognosImage/linuxi38664h/issetupnx": java.io.IOException: error=2, No such file or directory</message>
</entry>


in: -

/home/wasadmin/var/ibm/InstallationManager/logs/20140826_2055.xml

The ERROR: Error during "install" phase exception was a recurring theme for ~2 hours.

The first time out of the gate, I also saw: -

WARNING: Current system has detected a lower level of ulimit open files setting than the recommended value of 8192. Increase the ulimit open files value to minimum value of 8192 and re-start the installation.

Shutdown your installer. If you are a root user open a command prompt and issue ulimit -n 8192 and then restart the installer. If you are a non-root user, work with your system administrator to increase your ulimit -n 8192 and then restart the installer.


which was a new one on me, but a herring rouge.

I fixed that particular problem by checking the installing user's ulimits: -

ulimit -n

1024

/etc/security/limits.conf

Already has: -

# - nofile - max number of open files
root soft nofile 65536
root hard nofile 65536


*BUT*

/etc/security/limits.d/90-nproc.conf

needed: -

# - nofile - max number of open files
wasadmin   soft    nofile  65536
wasadmin   hard    nofile  65536


as it overrides limits.conf in RHEL6.

This gives me: -

ulimit -n

65536

but, as I say, it was a herring rouge.

I eventually found the solution by expanding

/tmp/Repo/BAM855/repository/COGNOS_64/native/cognos-linuxia64_10.2.1.zip

into a temporary directory: -

/tmp/foobar/

and then tested the failing binary - issetupnx : -

chmod +x issetupnx
./issetupnx

bash: ./issetupnx: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory

ld issetupnx

ld: i386 architecture of input file `issetupnx' is incompatible with i386:x86-64 output

After much faffing about, I worked out that I was missing certain RPMs, including GLIBC and GCC.

I ended up installing the following list: -

glibc-2.12-1.107.el6.i686.rpm
libXp-1.0.0-15.1.el6.x86_64.rpm
openmotif-2.3.3-4.el6.i686.rpm
glibc-2.12-1.47.el6.i686.rpm
nss-softokn-freebl-3.12.9-11.el6.i686.rpm
openmotif-2.3.3-4.el6.x86_64.rpm
libgcc-4.4.7-3.el6.i686.rpm
openmotif22-2.2.3-19.el6.x86_64.rpm

although I have highlighted those that I believe to be actually relevant :-)

I now get: -

./issetupnx

Licensed Materials - Property of IBM, BI and PM: is,
(C) Copyright IBM Corp. 2004, 2013. US Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
UNIX IBM Cognos InstallStream Version 10.2.1


The executable "issetupnx" may only be run in unattended mode, Please use the "-s" option and try again. 

which makes more sense.

As you can imagine, when I re-ran the installation process: -

/opt/IBM/InstallationManager/eclipse/tools/imcl -input installCognos.rsp -acceptLicense

it worked like a dram: -

Installed com.ibm.ws.cognos.v1021.linuxia64_10.2.1.20140530_2310 to the /opt/IBM/WebSphere/AppServer directory.

Guidelines for setting ulimits (WebSphere Application Server) - includes RHEL 6 updates


This document describes various ways to set ulimit values on UNIX and Linux operating systems as well as information that pertains to WebSphere Application Server

...
Global settings are set in the file /etc/security/limits.conf.
...

...
On Red Hat Enterprise Linux 6, why are my settings for nproc (ulimit -u) not being honored. Red Hat Enterprise Linux 6 has introduced a new configuration file, addressed in a bug report, ( /etc/security/limits.d/90-nproc.conf ) that overrides the nproc setting in the limits.conf. This file contains a line for nproc configured with a soft limit of 1024. 

*          soft    nproc     1024 

The rationale for this was to prevent fork bombs from occurring, To keep using the limits.conf for your nproc ulimit settings, you would need to comment this line out by making the first character on the line a hash/pound symbol. 

#*          soft    nproc     1024
...

Guidelines for setting ulimits (WebSphere Application Server)

For the record, this was the exception that led me to this particular document: -

WARNING: Current system has detected a lower level of ulimit open files setting than the recommended value of 8192. Increase the ulimit open files value to minimum value of 8192 and re-start the installation.

Shutdown your installer. If you are a root user open a command prompt and issue ulimit -n 8192 and then restart the installer. If you are a non-root user, work with your system administrator to increase your ulimit -n 8192 and then restart the installer.
 

IBM Business Monitor and DB2 users, not instances

So I have been wrestling with IBM Business Monitor and, more specifically, Cognos, which was failing to start up with errors such as DPR-DPR-1035, DPR-CMI-4007CM-CFG-5063 and CM-CFG-5137.

Various Technotes etc. suggested that the problem might be with the DB2 database.

I did some digging: -

db2 list tables for schema db2user2

db2 connect to cognoscs

   Database Connection Information

 Database server        = DB2/AIX64 10.1.0
 SQL authorization ID   = DB2INST3
 Local database alias   = COGNOSCS


db2 list tables for schema DB2USER2 

Table/View                      Schema          Type  Creation time
------------------------------- --------------- ----- --------------------------
CMCAPACITY                      DB2USER2        T     2014-08-26-15.26.49.093007
CMSYSPROPS                      DB2USER2        T     2014-08-26-15.26.48.688417

  2 record(s) selected.

whereas this is what I should be seeing: -

db2 list tables for schema DB2INST3

...
Table/View                      Schema          Type  Creation time             
------------------------------- --------------- ----- --------------------------
CMARCHIVEQUEUE                  DB2INST3        T     2014-05-02-19.01.41.736857
CMARCHIVESTATUS                 DB2INST3        T     2014-05-02-18.59.53.722937
CMCAKEYS                        DB2INST3        T     2014-05-02-18.59.47.494415
CMCAPACITY                      DB2INST3        T     2014-05-02-18.59.43.961958
CMCAPROPS                       DB2INST3        T     2014-05-02-18.59.49.009567
CMCLASSES                       DB2INST3        T     2014-05-02-19.01.30.263672
CMCRYPTOKEYS                    DB2INST3        T     2014-05-02-18.59.45.716151
CMDATA                          DB2INST3        T     2014-05-02-18.59.52.343365
CMDATAUPGRADE                   DB2INST3        T     2014-05-02-19.01.39.248572
CMDELETEQUEUE                   DB2INST3        T     2014-05-02-19.01.42.466589
CMGUIDS                         DB2INST3        T     2014-05-02-19.01.32.947199
CMLOCALES                       DB2INST3        T     2014-05-02-19.01.31.703091
CMOBJECTS                       DB2INST3        T     2014-05-02-18.59.50.444984
CMOBJNAMES                      DB2INST3        T     2014-05-02-18.59.51.545778
CMOBJPROPS1                     DB2INST3        T     2014-05-02-18.59.55.174490
CMOBJPROPS10                    DB2INST3        T     2014-05-02-18.59.55.916177
CMOBJPROPS11                    DB2INST3        T     2014-05-02-18.59.57.279869
CMOBJPROPS13                    DB2INST3        T     2014-05-02-18.59.58.081353
CMOBJPROPS14                    DB2INST3        T     2014-05-02-18.59.59.524858
CMOBJPROPS15                    DB2INST3        T     2014-05-02-19.00.00.901173
CMOBJPROPS16                    DB2INST3        T     2014-05-02-19.00.02.495462
CMOBJPROPS17                    DB2INST3        T     2014-05-02-19.00.04.026643
CMOBJPROPS18                    DB2INST3        T     2014-05-02-19.00.04.740375
CMOBJPROPS2                     DB2INST3        T     2014-05-02-19.00.06.155965
CMOBJPROPS20                    DB2INST3        T     2014-05-02-19.00.06.961562

...

I figured out that the problem is  :-)

I'm connecting from WAS to DB2 as a user other than the DB2 instance account, and ( therefore ) I need to change my SQL that's used to create the database objects etc.

Here's what worked for me: - 

db2 "CHANGE DATABASE COGNOSCS COMMENT WITH 'IBM Cognos Content Store'"
db2 CONNECT TO COGNOSCS

db2 GRANT CREATETAB,BINDADD,CONNECT,IMPLICIT_SCHEMA ON DATABASE  TO USER db2user2

db2 UPDATE DATABASE CONFIGURATION USING LOCKTIMEOUT 240 DEFERRED

db2 CREATE BUFFERPOOL CCS_08KBP IMMEDIATE SIZE 1000 PAGESIZE 8K
db2 CREATE BUFFERPOOL CCS_32KBP IMMEDIATE SIZE 1000 PAGESIZE 32K

db2 CREATE SYSTEM TEMPORARY TABLESPACE TSN_SYS_CCS IN DATABASE PARTITION GROUP IBMTEMPGROUP PAGESIZE 32K BUFFERPOOL CCS_32KBP

db2 CREATE USER TEMPORARY TABLESPACE TSN_USR_CCS IN DATABASE PARTITION GROUP IBMDEFAULTGROUP PAGESIZE 8K BUFFERPOOL CCS_08KBP
db2 CREATE REGULAR TABLESPACE TSN_REG_CCS IN DATABASE PARTITION GROUP IBMDEFAULTGROUP PAGESIZE 8K BUFFERPOOL CCS_08KBP

db2 GRANT USE OF TABLESPACE TSN_USR_CCS TO USER db2user2
db2 GRANT USE OF TABLESPACE TSN_REG_CCS TO USER db2user2

db2 terminate

Once I rebuilt the COGNOSCS database, and restarted the Support cluster, all was well :-)

Background Reading - IBM BPM - Process Designer and SSL Goodness

Not sure whether this will help me resolve a PD to PC problem on a newly minted BPM Advanced 8.5.0.1 environment, but I'm book-marking these for future reading: -




For the record, Process Designer has a number of SSL object stores: -

/etc/trust.p12
/etc/key.p12
cacerts

which is nice :-)

IBM BPM 8.5.5 - Process Center to Process Server - and vice versa - beware of the quotes


I'm just completing the build of an IBM BPM Advanced 8.5.5 environment, setting up Process Server on one VM to be available for online deployments from a Process Center on another VM.

All was looking good until I tried to start the third cluster, AppCluster, which contains the Business Process Modelling Notation (BPNM) run-time.

The start process stalled, and I found: -

...
[25/08/14 18:41:30:302 BST] 0000011b wle_repocore_ W   CWLLG4010W: The repository contact failed due to untrusted SSL certificate. Process Center's SSL certificate needs to be trusted by Process Server, please see: http://pic.dhe.ibm.com/infocenter/dmndhelp/v8r5m0/topic/com.ibm.wbpm.admin.doc/topics/tins_cnfg_ssl_nd.html
[25/08/14 18:41:40:350 BST] 0000011b wle_repocore_ W   CWLLG4010W: The repository contact failed due to untrusted SSL certificate. Process Center's SSL certificate needs to be trusted by Process Server, please see: http://pic.dhe.ibm.com/infocenter/dmndhelp/v8r5m0/topic/com.ibm.wbpm.admin.doc/topics/tins_cnfg_ssl_nd.html
[25/08/14 18:41:50:378 BST] 0000011b wle_repocore_ W   CWLLG4010W: The repository contact failed due to untrusted SSL certificate. Process Center's SSL certificate needs to be trusted by Process Server, please see: http://pic.dhe.ibm.com/infocenter/dmndhelp/v8r5m0/topic/com.ibm.wbpm.admin.doc/topics/tins_cnfg_ssl_nd.html

...

in SystemOut.log.

The exception refers one here: -


which really tells me what I already knew e.g. that I need to retrieve the endpoint certificate from Process Center ( actually from IBM HTTP Server fronting PC ) into the Process Server's cell default trust-store.

This I did via a Jython script: -

/opt/IBM/WebSphere/AppServer/profiles/Dmgr01/bin/wsadmin.sh -lang jython -host `hostname` -port 8879

cellID=AdminControl.getCell() 
AdminTask.retrieveSignerFromPort('[-keyStoreName CellDefaultTrustStore -keyStoreScope (cell):'+cellID+' -host rhel6.uk.ibm.com -port 8443 -certificateAlias ProcessCenter -sslConfigName CellDefaultSSLSettings -sslConfigScopeName (cell):'+cellID+' ]') 
AdminConfig.save() 
AdminNodeManagement.syncActiveNodes() 
quit 

( port 8443 is the HTTPS port on which IHS listens )

Once I did this, and restarted the cluster, it started up without problems but .....

I now see this: -

...
[25/08/14 18:56:57:511 BST] 000000ee wle_repocore_ W   CWLLG0098W: Unable to connect to Process Center.
[25/08/14 18:57:07:536 BST] 000000ee wle_repocore_ W   CWLLG0098W: Unable to connect to Process Center.
[25/08/14 18:57:17:554 BST] 000000ee wle_repocore_ W   CWLLG0098W: Unable to connect to Process Center.

...

in SystemOut.log and this: -

...
[25/08/14 19:05:08:605 BST]     FFDC Exception:java.net.ConnectException SourceId:com.lombardisoftware.servlet.heartbeat.RepositoryHeartbeat.callProcessCenter ProbeId:518 Reporter:com.lombardisoftware.servlet.heartbeat.RepositoryHeartbeat@b975d2a1
java.net.ConnectException: Connection refused
....

in the node's FFDC logs.

When I looked at the error_log for the IHS instance fronting Process Center, I can see this: -

...
[Mon Aug 25 18:46:57 2014] [error] [client 10.99.79.100] [7f1a780147b0] [7824] SSL0279E: SSL Handshake Failed due to fatal alert from client. Client sent fatal alert [level 2 (fatal), description 46 (certificate_unknown)]  [10.99.79.100:44309 -> 10.99.79.101:8443] [18:46:57.000038538] 0ms
[Mon Aug 25 18:47:00 2014] [error] [client 10.99.79.100] [7f1a58004180] [6406] SSL0279E: SSL Handshake Failed due to fatal alert from client. Client sent fatal alert [level 2 (fatal), description 46 (certificate_unknown)]  [10.99.79.100:40941 -> 10.99.79.101:8443] [18:47:00.000955120] 0ms
[Mon Aug 25 18:52:37 2014] [error] [client 10.99.79.100] [7f1a700147b0] [7824] SSL0279E: SSL Handshake Failed due to fatal alert from client. Client sent fatal alert [level 2 (fatal), description 46 (certificate_unknown)]  [10.99.79.100:41584 -> 10.99.79.101:8443] [18:52:37.000689637] 0ms
...

whereas I see nothing in the corresponding SystemOut.log for the AppCluster cluster member on the Process Center box.

This suggests that IHS ( Process Center ) is receiving a HTTPS connection from the AppCluster JVM ( Process Server ), but doesn't have a corresponding SSL certificate to decrypt the connection.

I fixed this by exporting the signer certificate from the cell-default trust store ( Process Server ): -

/opt/IBM/WebSphere/AppServer/profiles/Dmgr01/bin/wsadmin.sh -lang jython -host `hostname` -port 8879

cellID=AdminControl.getCell() 
AdminTask.extractSignerCertificate('[-keyStoreName CellDefaultTrustStore -keyStoreScope (cell):'+cellID+' -certificateFilePath /tmp/rhel6.uk.ibm.com -base64Encoded false -certificateAlias root ]')

'/tmp/rhel6.uk.ibm.com'

quit

and imported it into the IHS key store: -

/opt/IBM/HTTPServer/bin/gskcapicmd -cert -add -db /opt/IBM/HTTPServer/ssl/keystore.kdb -pw passw0rd -file ~/rhel6.uk.ibm.com 

and validated it as follows: -

/opt/IBM/HTTPServer/bin/gskcapicmd -cert -list -db /opt/IBM/HTTPServer/ssl/keystore.kdb -pw passw0rd

Certificates found
* default, - personal, ! trusted, # secret key
! "CN=rhel6.uk.ibm.com,OU=Root Certificate,OU=uk,O=ibm,C=com"
*- rhel6.uk.ibm.com


Whilst this resolved the IHS exception, I'm still seeing: -

...
[25/08/14 18:56:57:511 BST] 000000ee wle_repocore_ W   CWLLG0098W: Unable to connect to Process Center.
...

This led me to check the Process Server configuration, in terms of what it knows about the Process Center box: -

/opt/IBM/WebSphere/AppServer/profiles/Dmgr01/bin/wsadmin.sh -lang jython -host `hostname` -port 8879

ps = AdminConfig.getid("/Cell:/ServerCluster:AppCluster/BPMClusterConfigExtension:/BPMProcessServer:/")
print AdminConfig.show(ps)


[authoringEnvironmentPortalPrefix portal]
[baseUrl teamworks/webservices]
[bpdTrackingEnabledDefault false]
[clientLink teamworks]
[coachDesignerXslUrl teamworks/coachdesigner/transform/CoachDesigner.xsl]
[commonPortalPrefix portal]
[consoleSections [root(cells/bpm855Cell/clusters/AppCluster|cluster-bpm.xml#BPMConsoleSection_1408983776561) console.lombardi.admin(cells/bpm855Cell/clusters/AppCluster|cluster-bpm.xml#BPMConsoleSection_1408983776562) console.user.management(cells/bpm855Cell/clusters/AppCluster|cluster-bpm.xml#BPMConsoleSection_1408983776563) console.monitoring(cells/bpm855Cell/clusters/AppCluster|cluster-bpm.xml#BPMConsoleSection_1408983776564) console.event.manager(cells/bpm855Cell/clusters/AppCluster|cluster-bpm.xml#BPMConsoleSection_1408983776565) console.admin.tools(cells/bpm855Cell/clusters/AppCluster|cluster-bpm.xml#BPMConsoleSection_1408983776566)]]
[defaultNamespaceUri schema/]
[heartBeatInterval 10]
[httpProtocolOnly true]
[imagePrefix teamworks]
[processAdminPrefix ProcessAdmin]
[processCenterInternalUrl https://rhel6.uk.ibm.com:8443/ProcessCenterInternal]
[processCenterUrl https://rhel6.uk.ibm.com:8443/ProcessCenter]

[processHelpWikiUrlEdit processhelp/en/Special:Edit?topic=%TITLE%&teamworksTitle=%TEAMWORKS_TITLE%]
[processHelpWikiUrlView processhelp/en/%TITLE%?teamworksTitle=%TEAMWORKS_TITLE%]
[repositoryPrefix ProcessCenter]
[security (cells/bpm855Cell/clusters/AppCluster|cluster-bpm.xml#BPMServerSecurity_1408983776561)]
[servletPrefix teamworks]
[teamworksWebappPrefix teamworks]
[useHTTPSURLPrefixes true]
[virtualHost (cells/bpm855Cell/clusters/AppCluster|cluster-bpm.xml#BPMVirtualHostInfo_1408983776561)]
[webapiPrefix webapi]


Whilst the processCenterInternalUrl and processCenterUrl entries were quite correct, the parameter httpProtocolOnly should've been set to FALSE.

I changed this during the same wsadmin session: -

AdminConfig.modify(ps, [['httpProtocolOnly','false']])
AdminConfig.save() 
AdminNodeManagement.syncActiveNodes() 


and validated the change: -

ps = AdminConfig.getid("/Cell:/ServerCluster:AppCluster/BPMClusterConfigExtension:/BPMProcessServer:/")
print AdminConfig.show(ps)


...
[httpProtocolOnly false]
...

This got me further forward, but I'm still seeing: -

...
[25/08/14 20:52:33:059 BST] 000000ed wle_repocore_ W   CWLLG0098W: Unable to connect to Process Center.
[25/08/14 20:52:43:078 BST] 000000ed wle_repocore_ W   CWLLG0098W: Unable to connect to Process Center.
[25/08/14 20:52:53:107 BST] 000000ed wle_repocore_ W   CWLLG0098W: Unable to connect to Process Center.

....

This led me to the FFC logs ( /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/logs/ffdc ): -

[25/08/14 20:52:23:026 BST]     FFDC Exception:javax.net.ssl.SSLException SourceId:com.lombardisoftware.servlet.heartbeat.RepositoryHeartbeat.callProcessCenter ProbeId:518 Reporter:com.lombardisoftware.servlet.heartbeat.RepositoryHeartbeat@4ebf502e
javax.net.ssl.SSLException: hostname in certificate didn't match: <rhel6.uk.ibm.com> != <"rhel6.uk.ibm.com>
at org.apache.commons.httpclient.protocol.AbstractVerifier.verify(Unknown Source)
at org.apache.commons.httpclient.protocol.BrowserCompatHostnameVerifier.verify(Unknown Source)
at org.apache.commons.httpclient.protocol.AbstractVerifier.verify(Unknown Source)
at org.apache.commons.httpclient.protocol.AbstractVerifier.verify(Unknown Source)
at org.apache.commons.httpclient.protocol.SSLProtocolSocketFactory.createSocket(Unknown Source)
at org.apache.commons.httpclient.protocol.SSLProtocolSocketFactory.createSocket(Unknown Source)
at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
at com.lombardisoftware.server.util.HttpUtils.execute(HttpUtils.java:123)
at com.lombardisoftware.server.util.HttpUtils.executeWithBasicAuthenticationInternal(HttpUtils.java:109)
at com.lombardisoftware.server.util.HttpUtils.executeWithBasicAuthentication(HttpUtils.java:63)
at com.lombardisoftware.servlet.heartbeat.RepositoryHeartbeat.contactProcessCenterInternal(RepositoryHeartbeat.java:609)
at com.lombardisoftware.servlet.heartbeat.RepositoryHeartbeat.callProcessCenter(RepositoryHeartbeat.java:490)
at com.lombardisoftware.servlet.heartbeat.RepositoryHeartbeat.registerWithRepository(RepositoryHeartbeat.java:433)
at com.lombardisoftware.servlet.heartbeat.RepositoryHeartbeat.beat(RepositoryHeartbeat.java:366)
at com.lombardisoftware.servlet.heartbeat.RepositoryHeartbeat.run(RepositoryHeartbeat.java:169)


Interestingly, when I used "Retrieve from port" in the WAS Integrated Solutions Console (ISC), I also noticed that the Distinguished Name (DN) of the certificate was: -


rather than: -


In other words, the quotes ( " ) were creeping in somewhere .....

Given that the "failing" SSL certificate was coming from IHS ( remember, Process Server "talks" to Process Center via IHS, rather than the WAS Web Container port ), I looked at the process that I had used to generate SSL certificates in IHS.

Lo and behold .....

This is the command that I was using to generate the self-signed certificate in IHS: -

/opt/IBM/HTTPServer/bin/gskcapicmd -cert -create -db /opt/IBM/HTTPServer/ssl/keystore.kdb -pw passw0rd -size 2048 -dn cn=rhel6.uk.ibm.com\\,o=ibm\\,c=uk -label "rhel6.uk.ibm.com" -default_cert yes

I strongly suspect that the addition of the \\ characters was somehow causing the problem.

I removed the certificate and then recreated it WITHOUT the \\ characters: -

/opt/IBM/HTTPServer/bin/gskcapicmd -cert -create -db /opt/IBM/HTTPServer/ssl/keystore.kdb -pw passw0rd -size 2048 -dn cn=rhel6.uk.ibm.com\\,o=ibm\\,c=uk -label "rhel6.uk.ibm.com" -default_cert yes

Having restarted IHS and re-imported the IHS certificate into the cell-default trust store for Process Server, things then burst into life.

The moral of the story ? SSL is wonderful and powerful and can really mix you up :-)

PS In conclusion, this is the way that the Cell Default Trust Stores look on Process Center and Process Server: -

Process Center has the root signer certificate for the Process Server cell
Process Server has the self-signed certificate from IHS on the Process Center cell

PPS The whole business with importing the Process Server root certificate into the Process Center IHS keystore .... probably best to forget that :-)

Monday 25 August 2014

Now that I did not know - Amazon and iTunes together

So I was recently shopping for a copy of an old-but-good Kate Bush album, The Whole Story. Apple's iTunes had it for £4.99 whereas Amazon had the same MP3 version for £4.59. Now I'm not saying I'm mean but ..... well, it is 50 pence :-)

I bought the album via Amazon, and then thought .... OK, how do I get the album downloaded AND added into my iTunes library in order that I can play it on my iPhone, iPad etc.

Well, Amazon came to the rescue ....


Once I'd downloaded and installed this OS X app, I logged into my Amazon account from within the app. and was given the opportunity to download the new LP.

Post-download, the app then prompted me to "export" the music into iTunes.

I went into my iTunes library and .... voila.

Colour me impressed.

For further information: 



RAM Raiders - Where has my RAM gone ?

Looking at my Linux box ( I am running Red Hat Enterprise Linux Server release 6.4 ), I was wondering where my RAM had gone .....

This is what I have: -

free -m

             total       used       free     shared    buffers     cached
Mem:         16081       8876       7205          0        378       7036
-/+ buffers/cache:       1462      14619
Swap:         8015        153       7862



Don't worry, this site had the answer: -



<snip>
How do I see how much free ram I really have?

To see how much ram is free to use for your applications, run free -m and look at the row that says "-/+ buffers/cache" in the column that says "free". That is your answer in megabytes:
$ free -m
             total       used       free     shared    buffers     cached
Mem:          1504       1491         13          0         91        764
-/+ buffers/cache:        635        
869

Swap:         2047          6       2041
$

If you don't know how to read the numbers, you'll think the ram is 99% full when it's really just 42%.
</snip>

In other words, I have most of my 16 GB RAM available to me ( ~89% ) which is nice.

Sunday 24 August 2014

IBM BPM Advanced 8.5.5 - More fun and games with BPMDB Boot Strapping ...

This is definitely a case of "Look for the simple solution, before looking for the complex answer"

So I'm trying to bootstrap my BPM database ( the BPMDB used by the AppCluster ): -

/opt/IBM/WebSphere/AppServer/profiles/Dmgr01/bin/bootstrapProcessServerData.sh -clusterName AppCluster

and I see this: -

Bootstraping data into cluster AppCluster

WASX7357I: By request, this scripting client is not connected to any server process. Certain configuration and application operations will be available in local mode.
java.lang.Exception: java.lang.reflect.InvocationTargetException
at com.ibm.bpm.bootstrap.BootstrapProcessServerDataHelper.loadInstallationGuid(BootstrapProcessServerDataHelper.java:534)
at com.ibm.bpm.bootstrap.BootstrapProcessServerDataHelper.bootstrapData(BootstrapProcessServerDataHelper.java:248)
at com.ibm.bpm.bootstrap.BootstrapHelper.execute(BootstrapHelper.java:125)
at com.ibm.websphere.management.async.client.AsyncCommandClient.execute(AsyncCommandClient.java:177)
at com.ibm.ws.scripting.adminCommand.AdminCmdController.executeCmd(AdminCmdController.java:1877)
at com.ibm.ws.scripting.adminCommand.AdminCmdController.processCmd(AdminCmdController.java:318)
at com.ibm.ws.scripting.adminCommand.AdminCmdProxyHandler.invoke(AdminCmdProxyHandler.java:92)
at com.ibm.ws.scripting.adminCommand.AdminCmdProxyBase.__method4(AdminCmdProxyBase.java:58)
at com.ibm.ws.scripting.adminCommand.AdminTask.bootstrapProcessServerData(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java)
at org.python.core.PyMethod.__call__(PyMethod.java)
at org.python.core.PyObject.__call__(PyObject.java)
at org.python.core.PyInstance.invoke(PyInstance.java)
at org.python.pycode._pyx50.f$0(<input>:1)
at org.python.pycode._pyx50.call_function(<input>)
at org.python.core.PyTableCode.call(PyTableCode.java)
at org.python.core.PyCode.call(PyCode.java)
at org.python.core.Py.runCode(Py.java)
at org.python.core.Py.exec(Py.java)
at org.python.util.PythonInterpreter.exec(PythonInterpreter.java)
at com.ibm.bsf.engines.jython.JythonEngine$BSFPythonInterpreter.runcode(Unknown Source)
at org.python.util.InteractiveInterpreter.runsource(InteractiveInterpreter.java)
at org.python.util.InteractiveInterpreter.runsource(InteractiveInterpreter.java)
at com.ibm.bsf.engines.jython.JythonEngine.iexec(Unknown Source)
at com.ibm.bsf.BSFManager$7.run(Unknown Source)
at java.security.AccessController.doPrivileged(AccessController.java:327)
at com.ibm.bsf.BSFManager.iexec(Unknown Source)
at com.ibm.ws.scripting.AbstractShell.executeCommands(AbstractShell.java:1117)
at com.ibm.ws.scripting.AbstractShell.run(AbstractShell.java:2265)
at com.ibm.ws.scripting.WasxShell.main(WasxShell.java:1108)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at com.ibm.wsspi.bootstrap.WSLauncher.launchMain(WSLauncher.java:234)
at com.ibm.wsspi.bootstrap.WSLauncher.main(WSLauncher.java:96)
at com.ibm.wsspi.bootstrap.WSLauncher.run(WSLauncher.java:77)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at org.eclipse.equinox.internal.app.EclipseAppContainer.callMethodWithException(EclipseAppContainer.java:587)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:198)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:369)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:179)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at org.eclipse.core.launcher.Main.invokeFramework(Main.java:340)
at org.eclipse.core.launcher.Main.basicRun(Main.java:282)
at org.eclipse.core.launcher.Main.run(Main.java:981)
at com.ibm.wsspi.bootstrap.WSPreLauncher.launchEclipse(WSPreLauncher.java:380)
at com.ibm.wsspi.bootstrap.WSPreLauncher.main(WSPreLauncher.java:151)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at com.ibm.bpm.bootstrap.BootstrapProcessServerDataHelper.loadInstallationGuid(BootstrapProcessServerDataHelper.java:519)
... 59 more
Caused by: java.lang.ExceptionInInitializerError
at java.lang.J9VMInternals.initialize(J9VMInternals.java:259)
at com.ibm.bpm.bootstrap.InstallationGuidHelper.loadInstallationGuid(InstallationGuidHelper.java:41)
... 64 more
Caused by: java.lang.IllegalStateException: Failed to initialize registry
at com.lombardisoftware.core.Registry.getInstance(Registry.java:116)
at com.lombardisoftware.utility.spring.ProgrammaticTransactionSupport.<clinit>(ProgrammaticTransactionSupport.java:112)
at java.lang.J9VMInternals.initializeImpl(Native Method)
at java.lang.J9VMInternals.initialize(J9VMInternals.java:237)
... 65 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'handlersMap': Cannot create inner bean 'com.lombardisoftware.server.ejb.persistence.PSDefaultHandler#d32d0670' of type [com.lombardisoftware.server.ejb.persistence.PSDefaultHandler] while setting bean property 'sourceMap' with key [TypedStringValue: value [Task], target type [null]]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.lombardisoftware.server.ejb.persistence.PSDefaultHandler#d32d0670' defined in class path resource [registry.persistence.xml]: Cannot resolve reference to bean 'dao.task' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dao.task' defined in class path resource [registry.persistence.xml]: Instantiation of bean failed; nested exception is java.lang.ExceptionInInitializerError
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:230)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:117)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveManagedMap(BeanDefinitionValueResolver.java:320)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:134)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1245)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1010)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:472)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory$1.run(AbstractAutowireCapableBeanFactory.java:409)
at java.security.AccessController.doPrivileged(AccessController.java:298)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:380)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:264)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:221)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:261)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:185)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:423)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:729)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:381)
at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139)
at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:83)
at com.lombardisoftware.core.Registry.loadFromClasspath(Registry.java:73)
at com.lombardisoftware.core.Registry.load(Registry.java:65)
at com.lombardisoftware.core.Registry.getInstance(Registry.java:111)
... 68 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.lombardisoftware.server.ejb.persistence.PSDefaultHandler#d32d0670' defined in class path resource [registry.persistence.xml]: Cannot resolve reference to bean 'dao.task' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dao.task' defined in class path resource [registry.persistence.xml]: Instantiation of bean failed; nested exception is java.lang.ExceptionInInitializerError
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:275)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:104)
at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:479)
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:162)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:925)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:835)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:440)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory$1.run(AbstractAutowireCapableBeanFactory.java:409)
at java.security.AccessController.doPrivileged(AccessController.java:298)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:380)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:219)
... 90 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dao.task' defined in class path resource [registry.persistence.xml]: Instantiation of bean failed; nested exception is java.lang.ExceptionInInitializerError
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:254)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:925)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:835)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:440)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory$1.run(AbstractAutowireCapableBeanFactory.java:409)
at java.security.AccessController.doPrivileged(AccessController.java:298)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:380)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:264)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:221)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:261)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:185)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:269)
... 100 more
Caused by: java.lang.ExceptionInInitializerError
at java.lang.J9VMInternals.initialize(J9VMInternals.java:259)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:56)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:39)
at java.lang.reflect.Constructor.newInstance(Constructor.java:527)
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:100)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:87)
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:248)
... 112 more
Caused by: java.lang.IllegalStateException: Failed to initialize registry
at com.lombardisoftware.core.Registry.getInstance(Registry.java:116)
at com.lombardisoftware.server.ejb.persistence.dao.TaskDAO.<clinit>(TaskDAO.java:73)
at java.lang.J9VMInternals.initializeImpl(Native Method)
at java.lang.J9VMInternals.initialize(J9VMInternals.java:237)
... 119 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'handlersMap': Cannot create inner bean 'com.lombardisoftware.server.ejb.persistence.UserGroupPersistenceHandler#49493cf7' of type [com.lombardisoftware.server.ejb.persistence.UserGroupPersistenceHandler] while setting bean property 'sourceMap' with key [TypedStringValue: value [UserGroup], target type [null]]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.lombardisoftware.server.ejb.persistence.UserGroupPersistenceHandler#49493cf7' defined in class path resource [registry.persistence.xml]: Instantiation of bean failed; nested exception is java.lang.ExceptionInInitializerError
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:230)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:117)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveManagedMap(BeanDefinitionValueResolver.java:320)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:134)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1245)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1010)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:472)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory$1.run(AbstractAutowireCapableBeanFactory.java:409)
at java.security.AccessController.doPrivileged(AccessController.java:298)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:380)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:264)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:221)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:261)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:185)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:423)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:729)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:381)
at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139)
at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:83)
at com.lombardisoftware.core.Registry.loadFromClasspath(Registry.java:73)
at com.lombardisoftware.core.Registry.load(Registry.java:65)
at com.lombardisoftware.core.Registry.getInstance(Registry.java:111)
... 122 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.lombardisoftware.server.ejb.persistence.UserGroupPersistenceHandler#49493cf7' defined in class path resource [registry.persistence.xml]: Instantiation of bean failed; nested exception is java.lang.ExceptionInInitializerError
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:254)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:925)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:835)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:440)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory$1.run(AbstractAutowireCapableBeanFactory.java:409)
at java.security.AccessController.doPrivileged(AccessController.java:298)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:380)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:219)
... 144 more
Caused by: java.lang.ExceptionInInitializerError
at java.lang.J9VMInternals.initialize(J9VMInternals.java:259)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:56)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:39)
at java.lang.reflect.Constructor.newInstance(Constructor.java:527)
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:100)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:87)
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:248)
... 151 more
Caused by: java.lang.IllegalStateException: Failed to initialize registry
at com.lombardisoftware.core.Registry.getInstance(Registry.java:116)
at com.lombardisoftware.server.ejb.persistence.UserGroupPersistenceHandler.<clinit>(UserGroupPersistenceHandler.java:34)
at java.lang.J9VMInternals.initializeImpl(Native Method)
at java.lang.J9VMInternals.initialize(J9VMInternals.java:237)
... 158 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'message.routingCache' defined in class path resource [registry.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.lombardisoftware.bpd.runtime.engine.message.DefaultMessageRoutingCache]: Constructor threw exception; nested exception is java.lang.NullPointerException
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateBean(AbstractAutowireCapableBeanFactory.java:883)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:839)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:440)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory$1.run(AbstractAutowireCapableBeanFactory.java:409)
at java.security.AccessController.doPrivileged(AccessController.java:298)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:380)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:264)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:221)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:261)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:185)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:429)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:729)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:381)
at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139)
at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:83)
at com.lombardisoftware.core.Registry.loadFromClasspath(Registry.java:73)
at com.lombardisoftware.core.Registry.load(Registry.java:65)
at com.lombardisoftware.core.Registry.getInstance(Registry.java:111)
... 161 more
Caused by: org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.lombardisoftware.bpd.runtime.engine.message.DefaultMessageRoutingCache]: Constructor threw exception; nested exception is java.lang.NullPointerException
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:115)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:61)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateBean(AbstractAutowireCapableBeanFactory.java:877)
... 179 more
Caused by: java.lang.NullPointerException
at com.lombardisoftware.client.security.ServerTypeUtil.isRepository(ServerTypeUtil.java:35)
at com.lombardisoftware.server.ejb.environment.EnvironmentServicesCore.isRepository(EnvironmentServicesCore.java:460)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
at com.lombardisoftware.utility.spring.TransactionInterceptor$2.call(TransactionInterceptor.java:67)
at com.lombardisoftware.utility.spring.ProgrammaticTransactionSupport.executeWithNoTransaction(ProgrammaticTransactionSupport.java:626)
at com.lombardisoftware.utility.spring.ProgrammaticTransactionSupport.execute(ProgrammaticTransactionSupport.java:327)
at com.lombardisoftware.utility.spring.TransactionInterceptor.invoke(TransactionInterceptor.java:65)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at com.lombardisoftware.utility.spring.CoreEntryInterceptor.invoke(CoreEntryInterceptor.java:44)
at com.lombardisoftware.utility.spring.PSCoreEntryInterceptor.invoke(PSCoreEntryInterceptor.java:14)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy47.isRepository(Unknown Source)
at com.lombardisoftware.bpd.runtime.engine.message.DefaultMessageRoutingCache.doInitCache(DefaultMessageRoutingCache.java:404)
at com.lombardisoftware.core.cache.LocalCache.initCache(LocalCache.java:254)
at com.lombardisoftware.server.core.cache.ServerCoreCache.<init>(ServerCoreCache.java:46)
at com.lombardisoftware.bpd.runtime.engine.message.DefaultMessageRoutingCache.<init>(DefaultMessageRoutingCache.java:78)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:56)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:39)
at java.lang.reflect.Constructor.newInstance(Constructor.java:527)
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:100)
... 181 more
WASX7015E: Exception running command: "AdminTask.bootstrapProcessServerData(['-clusterName','AppCluster'])"; exception information:
java.lang.Exception: java.lang.Exception: java.lang.reflect.InvocationTargetException

Bootstrap failed


I checked to see whether WAS could "see" DB2 by testing one of the JDBC datasources - jdbc/mashupDS

The test connection operation failed for data source BPM Business Space data source on server nodeagent at node AppSrv01Node with the following exception: java.sql.SQLNonTransientException: [jcc][t4][2043][11550][4.11.69] Exception java.net.ConnectException: Error opening socket to server rhel6.uk.ibm.com/10.99.79.101 on port 60,011 with message: Connection refused. ERRORCODE=-4499, SQLSTATE=08001 DSRA0010E: SQL State = 08001, Error Code = -4,499. View JVM logs for further details.

and: -

telnet rhel6.uk.ibm.com 60011

Trying 10.99.79.101...
telnet: connect to address 10.99.79.101: Connection refused
telnet: Unable to connect to remote host

I checked /etc/services: -

...
DB2_db2inst1 60006/tcp
DB2_db2inst1_1 60007/tcp
DB2_db2inst1_2 60008/tcp
DB2_db2inst1_3 60009/tcp
DB2_db2inst1_4 60010/tcp
DB2_db2inst1_END 60011/tcp

...

and confirmed that the database manager configuration was correctly setup: -

db2 get dbm config | grep -i svcename

 TCP/IP Service name                          (SVCENAME) = db2c_db2inst1
 SSL service name                         (SSL_SVCENAME) = 


db2set

DB2COMM=TCPIP
DB2AUTOSTART=NO


db2start

...
08/24/2014 20:50:30     0   0   SQL5043N  Support for one or more communications protocols specified in the DB2COMM environment variable failed to start successfully. However, core database manager functionality started successfully.
SQL1063N  DB2START processing was successful.
...

On the off-chance, I disabled my old friend, iptables: -

chkconfig --list| grep -i iptables

iptables        0:off 1:off 2:on 3:on 4:on 5:on 6:off

chkconfig --levels 345 iptables off

chkconfig --list| grep -i iptables

iptables        0:off 1:off 2:on 3:off 4:off 5:off 6:off

but to no avail.

I then looked deeper into the DB2 logs, specifically: -

/home/db2inst1/sqllib/db2dump/db2diag.log

...
2014-08-24-21.05.44.508779+060 I32445E505            LEVEL: Error
PID     : 27818                TID : 140205466052352 PROC : db2sysc 0
INSTANCE: db2inst1             NODE : 000
HOSTNAME: rhel6.uk.ibm.com
EDUID   : 1                    EDUNAME: db2sysc 0
FUNCTION: DB2 UDB, common communication, sqlcctcpconnmgr, probe:5
MESSAGE : DIA3201E The service name "db2c_db2inst1" specified in the database
          manager configuration file cannot be found in the TCP/IP services
          file.

...

Can you spot where I've gone wrong ??

Let's look at /etc/services again: -

DB2_db2inst1 60006/tcp
DB2_db2inst1_1 60007/tcp
DB2_db2inst1_2 60008/tcp
DB2_db2inst1_3 60009/tcp
DB2_db2inst1_4 60010/tcp
DB2_db2inst1_END 60011/tcp

Spot anything missing ?

Yep, I've referenced a Service Name - db2c_db2inst1 - that does NOT actually exist :-)

Once I added this line to /etc/services: -

db2c_db2inst1   60012/tcp

I was able to start DB2 again: -

08/24/2014 21:12:00     0   0   SQL1063N  DB2START processing was successful.
SQL1063N  DB2START processing was successful.


and test the connection via Telnet: -

telnet rhel6.uk.ibm.com 60012

Trying 10.99.79.101...
Connected to bpm85dev.opentec.tec.hur.cdn.
Escape character is '^]'.

^]
telnet> quit
Connection closed.


I now need to go and update all of my JDBC data sources to point at 60012 rather than 60011, but c'est la vie :-)

Guess what, my JDBC Data Sources now test OK: -

...
The test connection operation for data source BPM Business Space data source on server nodeagent at node AppSrv01Node was successful.
...

and, even better, the bootstrap process works OK: -

...
importFile: /opt/IBM/WebSphere/AppServer/BPM/Lombardi/imports/advanced-hiring-sample.twx
deploy: false
latestSystemToolkit: false
activateSnapshot: false
importFile: /opt/IBM/WebSphere/AppServer/BPM/Lombardi/imports/procurement-sample.twx
deploy: false
latestSystemToolkit: false
activateSnapshot: false
'BootstrapProcessServerData admin command completed successfully.....'

...


Visual Studio Code - Wow 🙀

Why did I not know that I can merely hit [cmd] [p]  to bring up a search box allowing me to search my project e.g. a repo cloned from GitHub...