IBM Business Monitor and IBM Cognos - DPR-DPR-1035 Dispatcher detected an error

So I noticed an interesting quirk with my IBM Business Monitor 8.0.1.2 environment today, whilst monitoring (!) the logs during a startup.

I saw this: -

35      1       Audit.Other.dispatcher.DISP.pogo        pogo    com.cognos.pogo.reportservice.ProcessManager            Failure <m
essages><message><messageString>DPR-DPR-1035 Dispatcher detected an error.</messageString></message><message><messageString>Proces
s BIBusTKServerMain failed to start properly.</messageString></message></messages>      External Report Server process BIBusTKServ
erMain cannot be startedProcess BIBusTKServerMain failed to start properly.java.io.IOException: Process BIBusTKServerMain failed t
o start properly.       at com.cognos.pogo.reportservice.ReportServerProcess.getProcessOutput(ReportServerProcess.java:154)     at
 com.cognos.pogo.reportservice.ReportServerProcess.start(ReportServerProcess.java:117)  at com.cognos.pogo.reportservice.ProcessFa
cade.createServerProcess(ProcessFacade.java:219)        at com.cognos.pogo.reportservice.ProcessFacade.<init>(ProcessFacade.java:1
20)     at com.cognos.pogo.reportservice.RSComponentFactory.newProcessFacade(RSComponentFactory.java:67)        at com.cognos.pogo
.reportservice.ProcessManager.createProcessFacade(ProcessManager.java:514)      at com.cognos.pogo.reportservice.ProcessManager.st
artProcess(ProcessManager.java:490)     at com.cognos.pogo.reportservice.ProcessManager.startInitialProcesses(ProcessManager.java:
364)    at com.cognos.pogo.reportservice.ProcessManager.start(ProcessManager.java:295)  at com.cognos.pogo.reportservice.ReportSer
verHandler.start(ReportServerHandler.java:737)  at com.cognos.pogo.services.DefaultHandlerService.start(DefaultHandlerService.java
:94)    at com.cognos.pogo.services.DispatcherServices.start(DispatcherServices.java:189)       at com.cognos.pogo.services.Dispat
cherServices.continueStartup(DispatcherServices.java:417)       at com.cognos.pogo.services.DispatcherServices.configure(Dispatche
rServices.java:137)     at com.cognos.pogo.contentmanager.coordinator.RefreshController.composeAndConfigureServices(RefreshControl
ler.java:120)   at com.cognos.pogo.contentmanager.coordinator.RefreshController.run(RefreshController.java:80)  at com.cognos.pogo
.contentmanager.coordinator.BootstrapConfigurePublish.startConfiguration(BootstrapConfigurePublish.java:154)    at com.cognos.pogo
.contentmanager.coordinator.BootstrapConfigurePublish.checkConfiguration(BootstrapConfigurePublish.java:127)    at com.cognos.pogo
.contentmanager.coordinator.BootstrapConfigurePublish$ConfigurationCheckTask.safeRun(BootstrapConfigurePublish.java:120)        at
 com.cognos.pogo.util.threads.SafeTimerTask.run(SafeTimerTask.java:32)  at java.util.Timer$TimerImpl.run(Timer.java:296)

in: -

/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/cognos/BAM8012.AppTarget.AppSrv01Node.0/logs/cogserver.log

I asked Mr Google, and found this IBM Technote: -

Since applying AIX updater, unable to start Cognos service

which said, in part: -

...

Known defect with AIX 6.1 TL09 (refer to APAR#IV52684 for AIX6.1, APAR#52745 for AIX7. Thread memory allocation will cause IBM Cognos BIBus to crash

...

and directs one to obtain an iFix from IBM.

However, there's also a circumvention, which appears to have worked for me.

It effectively involves creating a shell script that wraps around the BIBusTKServerMain binary, and 

This is what I did: -

(1) cd /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/cognos/BAM8012.AppTarget.AppSrv01Node.0/bin64

(2) mv BIBusTKServerMain BIBusTKServerMain.exe

(3) vi BIBusTKServerMain

#!/bin/sh 

MALLOCTYPE=watson 
export MALLOCTYPE 

`pwd`/BIBusTKServerMain.exe $* <&0 >&1 &

(4) chmod +x BIBusTKServerMain

(5) cd ../bin64

(6) mv BIBusTKServerMain BIBusTKServerMain.exe

(7) cp ../bin/BIBusTKServerMain .

Once I put this circumvention in place, and restarted BAM, the exception appears to have gone away.

Which is nice :-)

Of course, I will be calling IBM Support to get the AIX fix for this .....

IBM HTTP Server - Serving multiple directories via Alias and DocumentRoot and Directory directives

I needed to deliver a simple mechanism for log file access to my developers, so chose to use IBM HTTP Server to provide a simple web-based access to the files, rather than insisting that they log into WebSphere Application Server or use a terminal emulator such as PuTTY ( the servers are hosted on AIX ).

I've done this before, and it's a simple solution.

Here's the relevant directives from httpd.conf for IBM Business Process Manager: -

...

DocumentRoot /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/logs/ProcessCenterJVM

<Directory />
    Options FollowSymLinks
    AllowOverride None
    FileETag All -INode
</Directory>

<Directory "/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/logs/ProcessCenterJVM">

    Options All
    AllowOverride None
    Order allow,deny
    Allow from all
</Directory>
...

I've highlighted the things that I changed, and that are particularly relevant to the IBM BPM.

Please note that I left the <Directory /> </Directory> stanza alone - this means that my users cannot browse the root file-system served by IHS, only the directories that I specify :-)

This is the URL that I've given my users: -

https://p71004lpar1.static1.tec.hur.cdn:8443/

However, for IBM Operational Decision Manager, I have not one, but two directories to serve up.

The solution is to use the Alias directive, thanks to this StackOverflow article: -

multiple document root for httpd conf file

...

DocumentRoot "/opt/IBM/WebSphere/AppServer/profiles/DecisionCenterNode01/logs"
Alias /DecisionCenterLogs "/opt/IBM/WebSphere/AppServer/profiles/DecisionCenterNode01/logs"
Alias /DecisionServerLogs "/opt/IBM/WebSphere/AppServer/profiles/DecisionServerNode01/logs"

<Directory />
    Options FollowSymLinks
    AllowOverride None
    FileETag All -INode
</Directory>

<Directory "/opt/IBM/WebSphere/AppServer/profiles/DecisionCenterNode01/logs">
    Options All
    AllowOverride None
    Order allow,deny
    Allow from all

</Directory>

<Directory "/opt/IBM/WebSphere/AppServer/profiles/DecisionServerNode01/logs">
        Options All
        AllowOverride None
        Allow from all
</Directory>
...

This allows me to specify two URLs: -

https://p71005lpar1.static1.tec.hur.cdn:8443/DecisionServerLogs/

https://p71005lpar1.static1.tec.hur.cdn:8443/DecisionCenterLogs/

Nice :-)

IBM Operational Decision Manager - Problems connecting Rule Designer to Decision Center and Decision Server

I installed Rule Designer into my W2K8 VM, and immediately saw issues connecting to Decision Server (RES) and Decision Center (RTS), including: -

Unexpected error: ilog.rules.res.util.http.IlrConnectionException

IO error when contacting "/res/repositoryService"

This is the same problem that I saw at a client back in late 2012, and occurs because, unlike Process Designer, Rule Designer doesn't seem "smart" enough to retrieve the SSL certificates from the JVMs on which RES and RTS are running.

The solution is to grab the endpoint certificates from the RES and RTS URLs - I chose to use Internet Explorer to do this, and save the certificates, in DER encoded binary X.509 (.CER) format, to my hard drive ( C:\temp\rts.cer and C:\temp\res.cer respectively ).

I then imported each into the Rule Designer's local SSL key/trust store ( CACerts ), as follows: -

"C:\Program Files\IBM\ODM851\jdk\bin\keytool.exe" -import -file c:\temp\odm.cer -alias RES-P71005LPAR1.static1.tec.hur.cdn -keystore "c:\Program Files\IBM\ODM851\jdk\jre\lib\security\cacerts" -storepass changeit

Owner: CN=P71005LPAR1, OU=P71005LPAR1Node01Cell, OU=DecisionServerNode01, O=IBM,
 C=US
Issuer: CN=P71005LPAR1.static1.tec.hur.cdn, OU=Root Certificate, OU=odm85Cell, O
U=odm85Node1, O=IBM, C=US
Serial number: 912cd201ddc8
Valid from: 4/30/14 6:00 PM until: 4/30/15 6:00 PM
Certificate fingerprints:
         MD5:  9A:7B:E3:1B:B1:02:D3:38:08:A4:4B:24:6D:04:CB:1B
         SHA1: 47:42:81:7F:CC:A2:25:D3:5F:BE:47:6F:92:91:A8:74:F9:6C:ED:7B
         SHA256: 86:3A:1D:49:EC:5B:08:E7:CE:03:9B:FD:59:13:B1:12:90:A9:5B:EE:45:
65:BE:5F:DA:19:B3:F9:54:8A:D2:4A
         Signature algorithm name: SHA1withRSA
         Version: 3

Extensions:

#1: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
[RFC822Name: ProfileUUID:Dmgr01-DEPLOYMENT_MANAGER-1d2ca68d-5864-4176-a6ed-63293
baa9766]]

#2: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 4e 7e a8 9c 7b fa f8 eb                           N.......
]
]

Trust this certificate? [no]:  yes
Certificate was added to keystore

"C:\Program Files\IBM\ODM851\jdk\bin\keytool.exe" -import -file c:\temp\rts.cer -alias RTS-P71005LPAR1.static1.tec.hur.cdn -keystore "c:\Program Files\IBM\ODM851\jdk\jre\lib\security\cacerts" -storepass changeit

Owner: CN=P71005LPAR1, OU=P71005LPAR1Node01Cell, OU=DecisionCenterNode01, O=IBM,
 C=US
Issuer: CN=P71005LPAR1.static1.tec.hur.cdn, OU=Root Certificate, OU=odm85Cell, O
U=odm85Node1, O=IBM, C=US
Serial number: d8b5af263526
Valid from: 5/1/14 3:51 PM until: 5/1/15 3:51 PM
Certificate fingerprints:
         MD5:  E5:BB:A2:FA:81:D1:2E:7C:23:50:9D:68:E7:E8:AA:71
         SHA1: 77:33:BE:8C:14:AA:1B:CF:40:15:D8:A8:C9:3B:0F:7B:BB:0B:E3:94
         SHA256: 4A:83:6E:61:1E:A1:65:D3:42:1A:79:F4:74:9E:2E:41:0A:B9:EE:7C:65:
C0:5F:DB:7A:01:36:03:29:E9:35:A4
         Signature algorithm name: SHA1withRSA
         Version: 3

Extensions:

#1: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
[RFC822Name: ProfileUUID:Dmgr01-DEPLOYMENT_MANAGER-1d2ca68d-5864-4176-a6ed-63293
baa9766]]

#2: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 40 4d 83 cb f4 e0 56 b4                           .M....V.
]
]

Trust this certificate? [no]:  yes
Certificate was added to keystore

Note that the command will prompt you to Trust this certificate to which you need to respond yes :-)

Once both certificates are added to CACerts, I simply restarted Rule Designer, and was then able to publish projects to Decision Center (RTS) and rulesets to Decision Server (RES).

PS Two useful links on the same subject here: -

Executing a scenario suite with a Scenario Service Provider (SSP) over HTTPS

"CertPathBuilderException: unable to find valid certification path to requested target" when connecting to Rule Team Server/Decision Center or Rule Execution Server over HTTPS

Five Things to Know About Systematically Deploying IBM Business Process Manager

From IBM developerWorks: -

5 Things to Know About Systematically Deploying IBM Business Process Manager

Want to get a leg up on deploying the new IBM Business Process Manager (BPM) Version 8.5?  A recent IBM Redbooks publication describes the latest features of this popular BPM software and includes step-by-step tips to help ensure your set-up goes smoothly.

IBM BPM provides a common software platform for business process improvement and lifecycle governance. Built-in analytics and search capabilities enable individuals throughout an enterprise to contribute to improving and optimizing critical processes, quickly and collaboratively.

BPM V8.5 includes several new capabilities and improvements compared to earlier versions:

• Changes in security, with fewer default system users and a new ability to alter the security configuration using the wsadamin tool instead of XML files
  
   
• Improved web services, including a new type of server configuration that allows web service endpoint connection details to be defined separately for each environment
   
• Simplified installation and configuration, featuring consolidated deployment patterns and clustering topologies

 

Speaking of installation and configuration, the new IBM Rebooks publication, Business Process Management Deployment Guide: Using IBM Business Process Manager V8.5, describes an approach with step-by-step instructions for building a new IBM BPM environment successfully.

And, of course, remember that my team, IBM Software Services for WebSphere (ISSW), can certainly help with this work.

More about GNU tools on AIX

So I've got a script that creates and updates the httpd.conf configuration file used by IBM HTTP Server.

This script makes use of the sed command, and works perfectly on Linux, specifically Red Hat Enterprise Linux 6, both on x86-64 and z/Linux.

As an example, here's a portion of the script: -

...

sed -i'' 's/Listen 8080/#Listen 8080/g' httpd.conf
sed -i'' 's/#LoadModule ibm_ssl_module/LoadModule ibm_ssl_module/g' httpd.conf
sed -i'' 's/#Listen 443/Listen '${httpsPort}'/g' httpd.conf
sed -i'' 's/#<VirtualHost \*:443>/<VirtualHost \*:'${httpsPort}'>/g' httpd.conf
sed -i'' 's/#SSLEnable/SSLEnable/g' httpd.conf
...

Sadly, when I run this on AIX 7.1, it fails with: -

...

sed: Not a recognized flag: i
Usage:  sed [-n] [-u] Script [File ...]
        sed [-n] [-u] [-e Script] ... [-f Script_file] ... [File ...]

...

This is because, apparently: -

...

The -i option is a GNU (non-standard) extension to the sed command. It was not part of the classic interface to sed.
...

Source: Sed on AIX does not recognize -i flag

The solution ?

Of course, I needed to install the GNU version of sed from here: -

AIX Toolbox for Linux Applications

Once I did this, and updated the symbolic link for the sed binary to point at /opt/freeware/bin/sed, my script worked like a dream.

For the record, this is what I now have for sed : -

$ ls -al `which see`

lrwxrwxrwx    1 root     system           21 May 03 07:22 /usr/bin/sed -> /opt/freeware/bin/sed

What's my AIX box doing ?

Let's ask Nigel's Monitor - aka NMON - written by Nigel Griffiths from IBM

Here's the website from 2006: -

nmon performance: A free tool to analyze AIX and Linux performance

This free tool gives you a huge amount of information all on one screen. Even though IBM doesn't officially support the tool and you must use it at your own risk, you can get a wealth of performance statistics. Why use five or six tools when one free tool can give you everything you need?

or topas

          

Happily nmon is now built into AIX, which is nice :-)

Foiled Again - Installation of IBM Cognos Business Intelligence has failed. Return code : 1

So I hit this issue last year: -

Installation of IBM Cognos Business Intelligence has failed. Return code : 1

during the installation of an IBM Business Monitor 8.0.1.2 environment.

Then I was using Red Hat Enterprise Linuux 6 on VMware.

I blogged about it at the time: -

Installation of IBM Cognos Business Intelligence has failed. Return code : 1

This time around I've hit the same problem again, but using AIX 7.1 on IBM Power7.

Believe it or not, I'm almost the only person in the world who's ever hit this problem - or, to be more realistic, who's written about it :-)

It took me a fair amount of time to crack the problem. To start with, I tried installing the IBM Business Monitor stack one product at  a time; WAS 8.0.0.8, BAM 8.0.1.2 and then Cognos 10.1.1.

However, Cognos kept on failing with the same exception.

I dug around, and eventually found a log file that has some useful insight: -

/opt/IBM/WebSphere/AppServer/cognos/instlog/tl-BISRVR-10.1-6235.144-20140502_0341.txt

When I dug into this log file using grep to pull out all instances of the word fail, I saw this: -

Warning: Failed to get file list for directory "/opt/IBM/WebSphere/AppServer/cognos/maps".
Warning: Failed to retrieve environment variable "LC_ALL".
Failed to extract file "." from tar file "/opt/IBM/WebSphere/AppServer/CognosImage/zipfiles/aix64h/docs8qckstrt-aix64h-gate-10.1.146.0-inst.tar.gz".
Failed to extract file "webcontent/documentation/en/common/help_toolbar_bg.gif" from tar file "/opt/IBM/WebSphere/AppServer/CognosImage/zipfiles/aix64h/docs8qckstrt-aix64h-gate-10.1.146.0-inst.tar.gz".
Failed to extract file "." from tar file "/opt/IBM/WebSphere/AppServer/CognosImage/zipfiles/aix64h/docmigmfdm-aix64h-gate-10.1.82.0-inst.tar.gz".
Failed to extract file "webcontent/documentation/en/images/ug_mfdm_chkmrk.jpg" from tar file "/opt/IBM/WebSphere/AppServer/CognosImage/zipfiles/aix64h/docmigmfdm-aix64h-gate-10.1.82.0-inst.tar.gz".
Failed to extract file "." from tar file "/opt/IBM/WebSphere/AppServer/CognosImage/zipfiles/aix64h/docdashboard-aix64h-gate-10.1.127.0-inst.tar.gz".
Failed to extract file "webcontent/documentation/en/common/chkbx.gif" from tar file "/opt/IBM/WebSphere/AppServer/CognosImage/zipfiles/aix64h/docdashboard-aix64h-gate-10.1.127.0-inst.tar.gz".
Failed to extract file "." from tar file "/opt/IBM/WebSphere/AppServer/CognosImage/zipfiles/aix64h/docbitrblshtngde-aix64h-gate-10.1.44.0-inst.tar.gz".
Failed to extract file "webcontent/documentation/de/images/tg_bitshoot_action_import_portlet.jpg" from tar file "/opt/IBM/WebSphere/AppServer/CognosImage/zipfiles/aix64h/docbitrblshtngde-aix64h-gate-10.1.44.0-inst.tar.gz".
Failed to extract file "." from tar file "/opt/IBM/WebSphere/AppServer/CognosImage/zipfiles/aix64h/docbitrblshtngen-aix64h-gate-10.1.44.0-inst.tar.gz".
Failed to extract file "webcontent/documentation/en/images/tg_bitshoot_action_import_portlet.jpg" from tar file "/opt/IBM/WebSphere/AppServer/CognosImage/zipfiles/aix64h/docbitrblshtngen-aix64h-gate-10.1.44.0-inst.tar.gz".
Failed to extract file "." from tar file "/opt/IBM/WebSphere/AppServer/CognosImage/zipfiles/aix64h/docbitrblshtnges-aix64h-gate-10.1.44.0-inst.tar.gz".
Failed to extract file "webcontent/documentation/es/images/tg_bitshoot_action_import_portlet.jpg" from tar file "/opt/IBM/WebSphere/AppServer/CognosImage/zipfiles/aix64h/docbitrblshtnges-aix64h-gate-10.1.44.0-inst.tar.gz".
Failed to extract file "." from tar file "/opt/IBM/WebSphere/AppServer/CognosImage/zipfiles/aix64h/docbitrblshtngfr-aix64h-gate-10.1.44.0-inst.tar.gz".
Failed to extract file "webcontent/documentation/fr/images/tg_bitshoot_action_import_portlet.jpg" from tar file "/opt/IBM/WebSphere/AppServer/CognosImage/zipfiles/aix64h/docbitrblshtngfr-aix64h-gate-10.1.44.0-inst.tar.gz".
Failed to extract file "." from tar file "/opt/IBM/WebSphere/AppServer/CognosImage/zipfiles/aix64h/docbitrblshtngja-aix64h-gate-10.1.44.0-inst.tar.gz".
Failed to extract file "webcontent/documentation/ja/images/tg_bitshoot_action_import_portlet.jpg" from tar file "/opt/IBM/WebSphere/AppServer/CognosImage/zipfiles/aix64h/docbitrblshtngja-aix64h-gate-10.1.44.0-inst.tar.gz".

For me, there were two possible solutions: -

(1) I was running out of disk space

(2) There was an issue with the AIX tar command

I tried both solutions and, quelle surprise, I think disk space was the issue, even though I didn't see exceptions similar to the previous time e.g. Error: Call to mkdir() failed for directory "/opt/IBM/WebSphere/AppServer/cognos/war/gateway/lib/": No space left on device.

Still, once I increased the space available to /opt/IBM/WebSphere ( the file-system that hosts WAS, BAM, Cognos ) and did a clean uninstall / reinstall of the whole BAM stack, it just worked, which is nice :-)

Bottom line, for me, the exception has always been related to disk space issues :-)