CWPKI0672E: Alias "" is not a personal certificate in key store "CellDefaultKeyStore".

If you see: -

CWPKI0672E: Alias "" is not a personal certificate in key store "CellDefaultKeyStore".

when running a Jython script such as: -

cellID=AdminControl.getCell()
configAlias="WAS_to_WMQ"
cipher="SSL_RSA_WITH_AES_128_CBC_SHA256"

AdminTask.createSSLConfig('[-alias '+configAlias+' -type JSSE -scopeName (cell):'+cellID+' -keyStoreName CellDefaultKeyStore -keyStoreScopeName (cell):'+cellID+' -trustStoreName CellDefaultTrustStore -trustStoreScopeName (cell):'+cellID+' -serverKeyAlias -clientKeyAlias -jsseProvider IBMJSSE2 -sslProtocol TLSv1.2 -clientAuthentication false -clientAuthenticationSupported false -securityLevel HIGH -enabledCiphers '+cipher+' ]')

which returned: -

WASX7015E: Exception running command: "AdminTask.createSSLConfig('[-alias '+configAlias+' -type JSSE -scopeName (cell):'+cellID+' -keyStoreName CellDefaultKeyStore -keyStoreScopeName (cell):'+cellID+' -trustStoreName CellDefaultTrustStore -trustStoreScopeName (cell):'+cellID+' -serverKeyAlias -clientKeyAlias -jsseProvider IBMJSSE2 -sslProtocol TLSv1.2 -clientAuthentication false -clientAuthenticationSupported false -securityLevel HIGH -enabledCiphers '+cipher+' ]')"; exception information:
com.ibm.websphere.management.cmdframework.CommandValidationException: CWPKI0672E: Alias "" is not a personal certificate in key store "CellDefaultKeyStore".

consider stopping/restarting the wsadmin shell in order to create a nice clean new shell.

**UPDATE**

Or simply avoid trying to specify null aliases - instead of the above, use this: -

AdminTask.createSSLConfig('[-alias '+configAlias+' -type JSSE -scopeName (cell):'+cellID+' -keyStoreName CellDefaultKeyStore -keyStoreScopeName (cell):'+cellID+' -trustStoreName CellDefaultTrustStore -trustStoreScopeName (cell):'+cellID+'  -jsseProvider IBMJSSE2 -sslProtocol TLSv1.2 -clientAuthentication false -clientAuthenticationSupported false -securityLevel HIGH -enabledCiphers '+cipher+' ]')

thus avoiding the use of -serverKeyAlias and -clientKeyAlias altogether ( the objective is to ensure that both are NULL in order to avoid WAS > MQ connectivity problems over TLS ).

**UPDATE**

This old, but good, IBM document: -

PK86031: ADMINTASK.MODIFYSSLCONFIG UNABLE TO MODIFY NEWLY ADDED CERT DATA IN WORKSPACE TEMP, ERROR CWPKI0672E

led me to this solution, which is nice :-)

The text makes it more clear: -

...

Certificates added using AdminTask can not be modified with AdminTask.modifySSLConfig unless an AdminConfig.save() is performed before the modifySSLConfig.

Example error : CWPKI0672E: Alias "test_alias" is not a personal certificate in key store "CellDefaultKeyStore".

The modifySSLConfig command should read the key.p12 file from the workspace temp  - if there is any - and not just from the config repository.

The problem with executing the save operation is as follows:

For stable automation it is important that one script is executed as a unit.

If one command fails in this script, the whole commands in this script could be "rolled back". This is perfectly possible with the design of wsadmin. You execute your wsadmin commands and they do the changes only in your workspace temp. If one of your commands fail and you exit wsadmin, none of your previously executed commands will get saved.  Only at the end of your procedure you call the save operation and all changes get saved together.

However, if I have to do a save operation between these commands the first changes are already commited to the config repository although it is not assured that the next commands will succeed.

This could end in an inconsistent configuration state.

...

Whilst it's not 100% the same as my situation, I had had the wsadmin "shell" open for some time, and had also been deleting stuff from within the Integrated Solutions Console, meaning that the temporary configuration repository within the shel was likely to be out-of-sync with the cell configuration.

Bottom line, restarting the wsadmin client did the trick :-)

WAS Scripting and Syntax Errors - Tear your hair out

I've just posted a hair-tearing post to the Global WebSphere Community here: -

WAS Scripting and Syntax Errors - Tear your hair out

detailing my fun with a Jython script, a syntax error and WASX7122E.

As ever, it was a PEBCAK :-)

Enjoy :-)

Using OpenSSL to connect via a specific SSL/TLS cipher

We're busy setting up TLS 1.2 encryption for WebSphere MQ 8, forcing all connections ( from WebSphere Application Server, IBM Integration Bus etc. ) to be encrypted, via a dedicated SVRCONN Channel.

The MQ setup script includes the following: -

...

QMGR=TESTQM
QMGRPORT=1420
MQCIPHERSPEC=TLS_RSA_WITH_AES_128_CBC_SHA256

echo "DEFINE CHANNEL(TEST.QMGR.SVRCONN) CHLTYPE(SVRCONN) SSLCIPH("$MQCIPHERSPEC") REPLACE" | runmqsc $QMGR
echo "ALTER CHANNEL(TEST.QMGR.SVRCONN) CHLTYPE(SVRCONN) SSLCAUTH(OPTIONAL)" | runmqsc $QMGR
echo "DIS CHANNEL(TEST.QMGR.SVRCONN) CHLTYPE(SVRCONN)" | runmqsc $QMGR
...

meaning that connectivity to this specific Channel will use the TLS_RSA_WITH_AES_128_CBC_SHA256 cipher.

Having completed the configuration, I wanted to validate the connectivity, using OpenSSL, which is built into my server's OS ( Red Hat Enterprise Linux ).

openssl s_client -tls1_2 -connect `hostname`:1420

CONNECTED(00000003)
depth=0 CN = bpm856.uk.ibm.com
verify error:num=18:self signed certificate
verify return:1
depth=0 CN = bpm856.uk.ibm.com
verify return:1
---
Certificate chain
 0 s:/CN=bpm856.uk.ibm.com
   i:/CN=bpm856.uk.ibm.com
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIB2zCCAUSgAwIBAgIEVfA5qjANBgkqhkiG9w0BAQUFADAcMRowGAYDVQQDExFi
cG04NTYudWsuaWJtLmNvbTAeFw0xNTA5MDkxMzUyNDJaFw0zNTA5MDQxMzUyNDJa
MBwxGjAYBgNVBAMTEWJwbTg1Ni51ay5pYm0uY29tMIGfMA0GCSqGSIb3DQEBAQUA
A4GNADCBiQKBgQCjEQKO/DA4y1gIv0wosNiAQx/V5vqaQHW8pEgy+mzJ/wcCdDRm
lqtm/E4JtAlbway70+OPdunKRMF54zizttXSWiDxrjFM8MRXJKGEsQfo2RpsKpUf
1YC0TRE8FUiUUCLIQmpLJ1aeW5RoxOxTQQwn97Rp0ULj7nq95anxgFuzDQIDAQAB
oyowKDATBgNVHSMEDDAKgAgPevSoHR450jARBgNVHQ4ECgQID3r0qB0eOdIwDQYJ
KoZIhvcNAQEFBQADgYEALGop6W2G8W/vlyf/0zBzZ0SupScUVQ875CIqE8Dm3wW6
pBn7HTDBGLIESJ4oQ0CdseiJEStBxmBtW4klotgL0A9C3nh2WNnSOxs1W1UqgCBB
fPQdWHU7iwogplgdfyb0xgWbwiobRoej8aOxMgL2yOyoojxFgspHrmMySEAjQvw=
-----END CERTIFICATE-----
subject=/CN=bpm856.uk.ibm.com
issuer=/CN=bpm856.uk.ibm.com
---
Acceptable client certificate CA names
/CN=bpm856.uk.ibm.com
---
SSL handshake has read 742 bytes and written 491 bytes
---
New, TLSv1/SSLv3, Cipher is AES128-SHA256
Server public key is 1024 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : AES128-SHA256
    Session-ID: 835400002792AB6982E40F6F59B50F396703953B58585858D875F0550000000A
    Session-ID-ctx: 
    Master-Key: D20745FE1E201620E7EC9B209D2858059E5CC7D2A68AE7D8B40CACAFED2767B891A9330156EAB5F9E46E151D3BCA3B27
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1441822168
    Timeout   : 7200 (sec)
    Verify return code: 18 (self signed certificate)
---

To further ensure that I could only connect with one cipher, I narrowed down my OpenSSL command: -

openssl s_client -tls1_2 -connect `hostname`:1420 -cipher 'TLS_RSA_WITH_AES_128_CBC_SHA256'

which returned: -

error setting cipher list
140245232068424:error:1410D0B9:SSL routines:SSL_CTX_set_cipher_list:no cipher match:ssl_lib.c:1314:

( Note that I'd specified the actual cipher - TLS_RSA_WITH_AES_128_CBC_SHA256 - in the command )

I then checked the online man page for the -ciphers option: -

https://www.openssl.org/docs/man1.0.1/apps/ciphers.html

*UPDATED 24/10/2020*

https://www.openssl.org/docs/manmaster/man1/ciphers.html

*UPDATED 24/10/2020*

which did the trick: -

openssl s_client -tls1_2 -connect `hostname`:1420 -cipher 'AES128-SHA256'

returns: -

...
CONNECTED(00000003)
...

SSL handshake has read 736 bytes and written 343 bytes

...

New, TLSv1/SSLv3, Cipher is AES128-SHA256

...

    Protocol  : TLSv1.2
    Cipher    : AES128-SHA256

...

DB2 HADR - Updating Host Names

Following my earlier posts, I've now gone to another level, cloning my DB2 HADR server ( the original standby server ) and thus creating TWO DB2 servers, one dedicated as primary and one dedicated as standby.

Of course, the delights of HADR mean that I can switch back and forth, and that's the objective - I want to have a single VM hosting WebSphere Application Server ( IBM Business Monitor ) and TWO VMs hosting DB2, so that I can properly test failover by physically turning off a VM ( shutdown -h now ).

Having made all the necessary OS and DB2 / HADR changes, the only thing that I noticed was the the DB2 catalog was out-of-date.

This is what I had: -

db2 list db directory

 System Database Directory

 Number of entries in the directory = 2

Database 1 entry:

 Database alias                       = COGNOS
 Database name                        = COGNOS
 Local database directory             = /home/db2inst1
 Database release level               = 10.00
 Comment                              = IBM Cognos Content Store
 Directory entry type                 = Indirect
 Catalog database partition number    = 0
 Alternate server hostname            = bpm856
 Alternate server port number         = 60006

Database 2 entry:

 Database alias                       = MONITOR
 Database name                        = MONITOR
 Local database directory             = /home/db2inst1
 Database release level               = 10.00
 Comment                              =
 Directory entry type                 = Indirect
 Catalog database partition number    = 0
 Alternate server hostname            = bpm856
 Alternate server port number         = 60006

whereas I expected the Alternate server hostname to be either db2one or db2two, depending upon at which box I looked.

How do I fix this ??

I ask Google :-)

How to change the hostname in DB2 HADR and TSA environment system

which led me to this: -

Primary

db2 "update alternate server for database cognos using hostname db2one port 60006"
db2 "update alternate server for database monitor using hostname db2one port 60006"

Standby

db2 "update alternate server for database cognos using hostname db2two port 60006"
db2 "update alternate server for database monitor using hostname db2two port 60006"

Now the catalog looks rosy :-)

Primary

 System Database Directory

 Number of entries in the directory = 2

Database 1 entry:

 Database alias                       = COGNOS
 Database name                        = COGNOS
 Local database directory             = /home/db2inst1
 Database release level               = 10.00
 Comment                              = IBM Cognos Content Store
 Directory entry type                 = Indirect
 Catalog database partition number    = 0
 Alternate server hostname            = db2two
 Alternate server port number         = 60006

Database 2 entry:

 Database alias                       = MONITOR
 Database name                        = MONITOR
 Local database directory             = /home/db2inst1
 Database release level               = 10.00
 Comment                              =
 Directory entry type                 = Indirect
 Catalog database partition number    = 0
 Alternate server hostname            = db2two
 Alternate server port number         = 60006

Standby

 System Database Directory

 Number of entries in the directory = 2

Database 1 entry:

 Database alias                       = COGNOS
 Database name                        = COGNOS
 Local database directory             = /home/db2inst1
 Database release level               = 10.00
 Comment                              = IBM Cognos Content Store
 Directory entry type                 = Indirect
 Catalog database partition number    = 0
 Alternate server hostname            = db2one
 Alternate server port number         = 60006

Database 2 entry:

 Database alias                       = MONITOR
 Database name                        = MONITOR
 Local database directory             = /home/db2inst1
 Database release level               = 10.00
 Comment                              =
 Directory entry type                 = Indirect
 Catalog database partition number    = 0
 Alternate server hostname            = db2one
 Alternate server port number         = 60006

And DB2 HADR is all clean and green.

Now to update WAS to use the correct hostnames via JDBC data sources :-)

DB2 HADR - A Clash of Primaries

I had fun with DB2 this afternoon, following an unplanned restart of BOTH of my DB2 servers.

In essence, neither the primary nor the standby wanted to be ... the primary.

This is what I saw for one of the two databases: -

On original primary

db2pd -hadr -db cognos

Database Member 0 -- Database COGNOS -- Standby -- Up 0 days 00:11:28 -- Date 2015-09-02-18.53.28.860276

                            HADR_ROLE = STANDBY
                          REPLAY_TYPE = PHYSICAL
                        HADR_SYNCMODE = 
                           STANDBY_ID = 0
                        LOG_STREAM_ID = 0
                           HADR_STATE = REMOTE_CATCHUP_PENDING
                           HADR_FLAGS = 
                  PRIMARY_MEMBER_HOST = NULL
                     PRIMARY_INSTANCE = NULL
                       PRIMARY_MEMBER = NULL
                  STANDBY_MEMBER_HOST = bpm856
                     STANDBY_INSTANCE = db2inst1
                       STANDBY_MEMBER = 0
                  HADR_CONNECT_STATUS = DISCONNECTED
             HADR_CONNECT_STATUS_TIME = 02/09/2015 18:53:27.015115 (1441216407)
          HEARTBEAT_INTERVAL(seconds) = 15
                     HEARTBEAT_MISSED = 0
                   HEARTBEAT_EXPECTED = 0
                HADR_TIMEOUT(seconds) = 60
        TIME_SINCE_LAST_RECV(seconds) = 0
             PEER_WAIT_LIMIT(seconds) = 0
           LOG_HADR_WAIT_CUR(seconds) = 0.000
    LOG_HADR_WAIT_RECENT_AVG(seconds) = 0.000000
   LOG_HADR_WAIT_ACCUMULATED(seconds) = 0.000
                  LOG_HADR_WAIT_COUNT = 0
SOCK_SEND_BUF_REQUESTED,ACTUAL(bytes) = 0, 16384
SOCK_RECV_BUF_REQUESTED,ACTUAL(bytes) = 0, 87380
            PRIMARY_LOG_FILE,PAGE,POS = S0000000.LOG, 0, 0
            STANDBY_LOG_FILE,PAGE,POS = S0000004.LOG, 446, 80142794
                  HADR_LOG_GAP(bytes) = 0
     STANDBY_REPLAY_LOG_FILE,PAGE,POS = S0000004.LOG, 446, 80142794
       STANDBY_RECV_REPLAY_GAP(bytes) = 0
                     PRIMARY_LOG_TIME = NULL
                     STANDBY_LOG_TIME = 02/09/2015 09:00:32.000000 (1441180832)
              STANDBY_REPLAY_LOG_TIME = 02/09/2015 09:00:32.000000 (1441180832)
         STANDBY_RECV_BUF_SIZE(pages) = 16
             STANDBY_RECV_BUF_PERCENT = 0
           STANDBY_SPOOL_LIMIT(pages) = 25600
                STANDBY_SPOOL_PERCENT = 0
                   STANDBY_ERROR_TIME = NULL
                 PEER_WINDOW(seconds) = 0
             READS_ON_STANDBY_ENABLED = N

On original standby

db2pd -hadr -db cognos

Database Member 0 -- Database COGNOS -- Standby -- Up 0 days 00:12:52 -- Date 2015-09-02-18.55.07.690822

                            HADR_ROLE = STANDBY
                          REPLAY_TYPE = PHYSICAL
                        HADR_SYNCMODE = 
                           STANDBY_ID = 0
                        LOG_STREAM_ID = 0
                           HADR_STATE = LOCAL_CATCHUP
                           HADR_FLAGS = 
                  PRIMARY_MEMBER_HOST = NULL
                     PRIMARY_INSTANCE = NULL
                       PRIMARY_MEMBER = NULL
                  STANDBY_MEMBER_HOST = db2hadr
                     STANDBY_INSTANCE = db2inst1
                       STANDBY_MEMBER = 0
                  HADR_CONNECT_STATUS = DISCONNECTED
             HADR_CONNECT_STATUS_TIME = 02/09/2015 18:55:06.448385 (1441216506)
          HEARTBEAT_INTERVAL(seconds) = 15
                     HEARTBEAT_MISSED = 0
                   HEARTBEAT_EXPECTED = 0
                HADR_TIMEOUT(seconds) = 60
        TIME_SINCE_LAST_RECV(seconds) = 0
             PEER_WAIT_LIMIT(seconds) = 0
           LOG_HADR_WAIT_CUR(seconds) = 0.000
    LOG_HADR_WAIT_RECENT_AVG(seconds) = 0.000000
   LOG_HADR_WAIT_ACCUMULATED(seconds) = 0.000
                  LOG_HADR_WAIT_COUNT = 0
SOCK_SEND_BUF_REQUESTED,ACTUAL(bytes) = 0, 16384
SOCK_RECV_BUF_REQUESTED,ACTUAL(bytes) = 0, 87380
            PRIMARY_LOG_FILE,PAGE,POS = S0000000.LOG, 0, 0
            STANDBY_LOG_FILE,PAGE,POS = S0000006.LOG, 18, 86748034
                  HADR_LOG_GAP(bytes) = 0
     STANDBY_REPLAY_LOG_FILE,PAGE,POS = S0000006.LOG, 18, 86748034
       STANDBY_RECV_REPLAY_GAP(bytes) = 0
                     PRIMARY_LOG_TIME = NULL
                     STANDBY_LOG_TIME = 02/09/2015 16:22:27.000000 (1441207347)
              STANDBY_REPLAY_LOG_TIME = 02/09/2015 16:22:27.000000 (1441207347)
         STANDBY_RECV_BUF_SIZE(pages) = 16
             STANDBY_RECV_BUF_PERCENT = 0
           STANDBY_SPOOL_LIMIT(pages) = 25600
                STANDBY_SPOOL_PERCENT = 0
                   STANDBY_ERROR_TIME = NULL
                 PEER_WINDOW(seconds) = 0
             READS_ON_STANDBY_ENABLED = N

with messages such as: -

2015-09-02-18.53.02.766411+060 I15074291E494         LEVEL: Info
PID     : 2497                 TID : 139896769472256 PROC : db2sysc 0
INSTANCE: db2inst1             NODE : 000            DB   : COGNOS  
HOSTNAME: bpm856.uk.ibm.com
EDUID   : 45                   EDUNAME: db2hadrs.0.0 (COGNOS) 0
FUNCTION: DB2 UDB, High Availability Disaster Recovery, hdrSendHsMsgNoDefer, probe:30539
DATA #1 : <preformatted>
A HDR_MSG_NOTPRIMARY message was sent to db2hadr:cognos_hadr (192.168.33.100:60009)

2015-09-02-18.53.02.769628+060 I15074786E502         LEVEL: Info
PID     : 2497                 TID : 139896769472256 PROC : db2sysc 0
INSTANCE: db2inst1             NODE : 000            DB   : COGNOS  
HOSTNAME: bpm856.uk.ibm.com
EDUID   : 45                   EDUNAME: db2hadrs.0.0 (COGNOS) 0
FUNCTION: DB2 UDB, High Availability Disaster Recovery, hdrHandleHsAck, probe:43900
DATA #1 : <preformatted>
Handshake HDR_MSG_NOTPRIMARY message is received from db2hadr:cognos_hadr (192.168.33.100:60009)

2015-09-02-18.53.02.769830+060 I15075289E603         LEVEL: Error
PID     : 2497                 TID : 139896769472256 PROC : db2sysc 0
INSTANCE: db2inst1             NODE : 000            DB   : COGNOS  
HOSTNAME: bpm856.uk.ibm.com
EDUID   : 45                   EDUNAME: db2hadrs.0.0 (COGNOS) 0
FUNCTION: DB2 UDB, High Availability Disaster Recovery, hdrHandleHsAck, probe:43970
MESSAGE : ZRC=0x878001BA=-2021654086=HDR_ZRC_NOT_PRIMARY
          "The database being contacted as primary is not a primary"
DATA #1 : <preformatted>
HADR handshake with db2hadr:cognos_hadr (192.168.33.100:60009) failed.

in db2diag.log.

On "new" primary

db2 deactivate db cognos

DB20000I  The DEACTIVATE DATABASE command completed successfully.

db2 stop hadr on db cognos

DB20000I  The STOP HADR ON DATABASE command completed successfully.

db2 start hadr on db cognos as primary

SQL1117N  A connection to or activation of database "COGNOS" cannot be made because of ROLL-FORWARD PENDING.  SQLSTATE=57019

db2 rollforward database cognos to end of logs and complete

                                 Rollforward Status

 Input database alias                   = cognos
 Number of members have returned status = 1

 Member ID                              = 0
 Rollforward status                     = DB  working
 Next log file to be read               = S0000007.LOG
 Log files processed                    = S0000005.LOG - S0000006.LOG
 Last committed transaction             = 2015-09-02-16.39.40.000000 UTC

DB20000I  The ROLLFORWARD command completed successfully.

db2 start hadr on db cognos as primary

DB20000I  The START HADR ON DATABASE command completed successfully.

On "new" standby

db2 deactivate db cognos

DB20000I  The DEACTIVATE DATABASE command completed successfully.

db2 stop hadr on db cognos

DB20000I  The STOP HADR ON DATABASE command completed successfully.

db2 start hadr on db cognos as standby

DB20000I  The START HADR ON DATABASE command completed successfully.

Now things look clean and green .....

New "primary"

db2pd -hadr -db cognos

Database Member 0 -- Database COGNOS -- Active -- Up 0 days 00:00:54 -- Date 2015-09-02-19.25.29.081627

                            HADR_ROLE = PRIMARY
                          REPLAY_TYPE = PHYSICAL
                        HADR_SYNCMODE = SYNC
                           STANDBY_ID = 1
                        LOG_STREAM_ID = 0
                           HADR_STATE = PEER
                           HADR_FLAGS = 
                  PRIMARY_MEMBER_HOST = db2hadr
                     PRIMARY_INSTANCE = db2inst1
                       PRIMARY_MEMBER = 0
                  STANDBY_MEMBER_HOST = bpm856
                     STANDBY_INSTANCE = db2inst1
                       STANDBY_MEMBER = 0
                  HADR_CONNECT_STATUS = CONNECTED
             HADR_CONNECT_STATUS_TIME = 02/09/2015 19:24:36.897314 (1441218276)
          HEARTBEAT_INTERVAL(seconds) = 15
                     HEARTBEAT_MISSED = 0
                   HEARTBEAT_EXPECTED = 3
                HADR_TIMEOUT(seconds) = 60
        TIME_SINCE_LAST_RECV(seconds) = 8
             PEER_WAIT_LIMIT(seconds) = 0
           LOG_HADR_WAIT_CUR(seconds) = 0.000
    LOG_HADR_WAIT_RECENT_AVG(seconds) = 0.000000
   LOG_HADR_WAIT_ACCUMULATED(seconds) = 0.000
                  LOG_HADR_WAIT_COUNT = 0
SOCK_SEND_BUF_REQUESTED,ACTUAL(bytes) = 0, 19800
SOCK_RECV_BUF_REQUESTED,ACTUAL(bytes) = 0, 87380
            PRIMARY_LOG_FILE,PAGE,POS = S0000007.LOG, 0, 90845934
            STANDBY_LOG_FILE,PAGE,POS = S0000007.LOG, 0, 90845934
                  HADR_LOG_GAP(bytes) = 0
     STANDBY_REPLAY_LOG_FILE,PAGE,POS = S0000007.LOG, 0, 90845934
       STANDBY_RECV_REPLAY_GAP(bytes) = 0
                     PRIMARY_LOG_TIME = 02/09/2015 17:39:40.000000 (1441211980)
                     STANDBY_LOG_TIME = 02/09/2015 17:39:40.000000 (1441211980)
              STANDBY_REPLAY_LOG_TIME = 02/09/2015 17:39:40.000000 (1441211980)
         STANDBY_RECV_BUF_SIZE(pages) = 4298
             STANDBY_RECV_BUF_PERCENT = 0
           STANDBY_SPOOL_LIMIT(pages) = 25600
                STANDBY_SPOOL_PERCENT = 0
                   STANDBY_ERROR_TIME = NULL
                 PEER_WINDOW(seconds) = 120
                      PEER_WINDOW_END = 02/09/2015 19:27:22.000000 (1441218442)
             READS_ON_STANDBY_ENABLED = N

New "standby"

db2pd -hadr -db cognos

Database Member 0 -- Database COGNOS -- Standby -- Up 0 days 00:13:07 -- Date 2015-09-02-19.26.59.577531

                            HADR_ROLE = STANDBY
                          REPLAY_TYPE = PHYSICAL
                        HADR_SYNCMODE = SYNC
                           STANDBY_ID = 0
                        LOG_STREAM_ID = 0
                           HADR_STATE = PEER
                           HADR_FLAGS = 
                  PRIMARY_MEMBER_HOST = db2hadr
                     PRIMARY_INSTANCE = db2inst1
                       PRIMARY_MEMBER = 0
                  STANDBY_MEMBER_HOST = bpm856
                     STANDBY_INSTANCE = db2inst1
                       STANDBY_MEMBER = 0
                  HADR_CONNECT_STATUS = CONNECTED
             HADR_CONNECT_STATUS_TIME = 02/09/2015 19:24:36.898265 (1441218276)
          HEARTBEAT_INTERVAL(seconds) = 15
                     HEARTBEAT_MISSED = 0
                   HEARTBEAT_EXPECTED = 9
                HADR_TIMEOUT(seconds) = 60
        TIME_SINCE_LAST_RECV(seconds) = 8
             PEER_WAIT_LIMIT(seconds) = 0
           LOG_HADR_WAIT_CUR(seconds) = 0.000
    LOG_HADR_WAIT_RECENT_AVG(seconds) = 0.000000
   LOG_HADR_WAIT_ACCUMULATED(seconds) = 0.000
                  LOG_HADR_WAIT_COUNT = 0
SOCK_SEND_BUF_REQUESTED,ACTUAL(bytes) = 0, 19800
SOCK_RECV_BUF_REQUESTED,ACTUAL(bytes) = 0, 87380
            PRIMARY_LOG_FILE,PAGE,POS = S0000007.LOG, 0, 90845934
            STANDBY_LOG_FILE,PAGE,POS = S0000007.LOG, 0, 90845934
                  HADR_LOG_GAP(bytes) = 0
     STANDBY_REPLAY_LOG_FILE,PAGE,POS = S0000007.LOG, 0, 90845934
       STANDBY_RECV_REPLAY_GAP(bytes) = 0
                     PRIMARY_LOG_TIME = 02/09/2015 17:39:40.000000 (1441211980)
                     STANDBY_LOG_TIME = 02/09/2015 17:39:40.000000 (1441211980)
              STANDBY_REPLAY_LOG_TIME = 02/09/2015 17:39:40.000000 (1441211980)
         STANDBY_RECV_BUF_SIZE(pages) = 4298
             STANDBY_RECV_BUF_PERCENT = 0
           STANDBY_SPOOL_LIMIT(pages) = 25600
                STANDBY_SPOOL_PERCENT = 0
                   STANDBY_ERROR_TIME = NULL
                 PEER_WINDOW(seconds) = 120
                      PEER_WINDOW_END = 02/09/2015 19:28:52.000000 (1441218532)
             READS_ON_STANDBY_ENABLED = N

WebSphere Application Server 8.5.5 and DB2 10.5 - High Availability for Disaster Recovery - Like Manuel, I learn ....

I've been testing, experimenting and documenting my experiences with IBM DB2 10.5, specifically with the High Availability for Disaster Recovery (HADR) configuration, in the context of making a WebSphere Application Server (WAS) configuration more resilient.

To that end, I've been scripting the necessary configuration to allow WAS to use the DB2 Automatic Client Reroute Options: -

in order that WAS can connect to another DB2 server within the "cluster" in the situation where the primary server, to which the JDBC Data Source normally points: -

Now, in the first screenshot above, you'll notice that the Client reroute server list JNDI name is left blank.

This is by design :-)

Initially, I was populating that particular property, being unaware as to what it did :-)

That was a BAD move.

Basically, the property is ONLY used if one creates a second JDBC Data Source to point at the second, standby database server.

However, the use of the other parameters, Alternate Server Names and Alternate Port Numbers makes this unnecessary.

This IBM Technote says it far better than I: -

How to configure DB2 HADR using Data Sources

<snip>

Note 1: If you choose to use Client reroute server list JNDI name, you do not need to supply Alternate server names and Alternate port numbers (and vice versa). 
Note 2: The Client reroute server list JNDI name should be a unique JNDI name, different from the JNDI Name used in the previous step.

</snip>

So, back to me, and this is what I was seeing when I attempted to test a JDBC Data Source whilst the primary ( configured ) DB2 server was down down down: -

The test connection operation failed for data source Monitor_Database on server dmgr at node Dmgr with the following exception: java.sql.SQLNonTransientException: [jcc][t4][2043][11550][4.11.69] Exception java.net.ConnectException: Error opening socket to server bpm856.uk.ibm.com/192.168.33.200 on port 60,006 with message: Connection refused. ERRORCODE=-4499, SQLSTATE=08001 DSRA0010E: SQL State = 08001, Error Code = -4,499. View JVM logs for further details.

with this in SystemErr.log: -

...

[02/09/15 15:18:41:143 BST] 00000107 SystemErr     R java.sql.SQLNonTransientException: [jcc][t4][2043][11550][4.11.69] Exception java.net.ConnectException: Error opening socket to server bpm856.uk.ibm.com/192.168.33.200 on port 60,006 with message: Connection refused. ERRORCODE=-4499, SQLSTATE=08001 DSRA0010E: SQL State = 08001, Error Code = -4,499
...

[02/09/15 15:18:41:150 BST] 00000107 SystemErr     R Caused by: java.net.ConnectException: Connection refused
...

which, when you think about it, makes PERFECT sense.

In other words, I've told WAS to, in the eventuality of losing a DB2 server, use an alternative JDBC Data Source which .... DOES NOT EXIST :-)

Once I changed this - and, more importantly, fixed my Jython script, it all became shiny again :-)

CRIMA1062E ERROR: Installing IBM® Business Monitor 8.5.5.0 is not allowed ....

I saw this issue today: -

ERROR: The following errors were generated while installing.
  CRIMA1062E ERROR: Installing IBM® Business Monitor 8.5.5.0 is not allowed; it is not compatible with IBM® Cognos Business Intelligence 64 bit 10.2.1.2.
    ERROR: In root installation context:
      ERROR: Installation context "com.ibm.websphere.appserver.installcontext" was not resolved

whilst trying to install IBM Business Monitor 8.5.5.0 ( in order to configure/test DB2 HADR ).

As is often the case, it was a PEBCAK i.e. a user error.

The last line: -

      ERROR: Installation context "com.ibm.websphere.appserver.installcontext" was not resolved

was the most meaningful.

In my response file: -

<?xml version="1.0" encoding="UTF-8"?>
<agent-input acceptLicense='true'>
<server>
<repository location='/tmp/Repository/BAM/repository/'/>
</server>
<profile id='IBM WebSphere Application Server V8.5' installLocation='/opt/IBM/WebSphere/AppServer'>
<data key='eclipseLocation' value='/opt/IBM/WebSphere/AppServer'/>
<data key='user.import.profile' value='false'/>
<data key='cic.selector.os' value='linux'/>
<data key='cic.selector.arch' value='x86'/>
<data key='cic.selector.ws' value='gtk'/>
<data key='cic.selector.nl' value='en'/>
</profile>
<install modify='false'>
<offering id='com.ibm.websphere.MON.v85' version='8.5.5000.20140530_1037' profile='IBM WebSphere Application Server V8.5' features='Monitor.NonProduction' installFixes='none'/>
<offering profile='IBM WebSphere Application Server V8.5' id='com.ibm.ws.cognos.v1021.linuxia64' version='10.2.1.20140530_2310' features='com.ibm.cognos.feature' installFixes='none'/>
</install>
<preference name='com.ibm.cic.common.core.preferences.eclipseCache' value='/opt/IBM/IMShared'/>
<preference name='com.ibm.cic.common.core.preferences.connectTimeout' value='30'/>
<preference name='com.ibm.cic.common.core.preferences.readTimeout' value='45'/>
<preference name='com.ibm.cic.common.core.preferences.downloadAutoRetryCount' value='0'/>
<preference name='offering.service.repositories.areUsed' value='true'/>
<preference name='com.ibm.cic.common.core.preferences.ssl.nonsecureMode' value='false'/>
<preference name='com.ibm.cic.common.core.preferences.http.disablePreemptiveAuthentication' value='false'/>
<preference name='http.ntlm.auth.kind' value='NTLM'/>
<preference name='http.ntlm.auth.enableIntegrated.win32' value='true'/>
<preference name='com.ibm.cic.common.core.preferences.preserveDownloadedArtifacts' value='true'/>
<preference name='com.ibm.cic.common.core.preferences.keepFetchedFiles' value='false'/>
<preference name='PassportAdvantageIsEnabled' value='false'/>
<preference name='com.ibm.cic.common.core.preferences.searchForUpdates' value='false'/>
<preference name='com.ibm.cic.agent.ui.displayInternalVersion' value='false'/>
<preference name='com.ibm.cic.common.sharedUI.showErrorLog' value='true'/>
<preference name='com.ibm.cic.common.sharedUI.showWarningLog' value='true'/>
<preference name='com.ibm.cic.common.sharedUI.showNoteLog' value='true'/>
</agent-input>

I'd forgotten to actually install WAS :-)

Once I amended my response file to: -

<?xml version="1.0" encoding="UTF-8"?>
<agent-input acceptLicense='true'>
<server>
<repository location='/tmp/Repository/BAM/repository/'/>
</server>
<profile id='IBM WebSphere Application Server V8.5' installLocation='/opt/IBM/WebSphere/AppServer'>
<data key='eclipseLocation' value='/opt/IBM/WebSphere/AppServer'/>
<data key='user.import.profile' value='false'/>
<data key='cic.selector.os' value='linux'/>
<data key='cic.selector.arch' value='x86'/>
<data key='cic.selector.ws' value='gtk'/>
<data key='cic.selector.nl' value='en'/>
</profile>
<install modify='false'>
<offering id='com.ibm.websphere.ND.v85' version='8.5.5002.20140408_1947' profile='IBM WebSphere Application Server V8.5' features='core.feature,ejbdeploy,thinclient,embeddablecontainer,com.ibm.sdk.6_64bit,samples' installFixes='none'/>
<offering id='com.ibm.websphere.MON.v85' version='8.5.5000.20140530_1037' profile='IBM WebSphere Application Server V8.5' features='Monitor.NonProduction' installFixes='none'/>
<offering profile='IBM WebSphere Application Server V8.5' id='com.ibm.ws.cognos.v1021.linuxia64' version='10.2.1.20140530_2310' features='com.ibm.cognos.feature' installFixes='none'/>
</install>
<preference name='com.ibm.cic.common.core.preferences.eclipseCache' value='/opt/IBM/IMShared'/>
<preference name='com.ibm.cic.common.core.preferences.connectTimeout' value='30'/>
<preference name='com.ibm.cic.common.core.preferences.readTimeout' value='45'/>
<preference name='com.ibm.cic.common.core.preferences.downloadAutoRetryCount' value='0'/>
<preference name='offering.service.repositories.areUsed' value='true'/>
<preference name='com.ibm.cic.common.core.preferences.ssl.nonsecureMode' value='false'/>
<preference name='com.ibm.cic.common.core.preferences.http.disablePreemptiveAuthentication' value='false'/>
<preference name='http.ntlm.auth.kind' value='NTLM'/>
<preference name='http.ntlm.auth.enableIntegrated.win32' value='true'/>
<preference name='com.ibm.cic.common.core.preferences.preserveDownloadedArtifacts' value='true'/>
<preference name='com.ibm.cic.common.core.preferences.keepFetchedFiles' value='false'/>
<preference name='PassportAdvantageIsEnabled' value='false'/>
<preference name='com.ibm.cic.common.core.preferences.searchForUpdates' value='false'/>
<preference name='com.ibm.cic.agent.ui.displayInternalVersion' value='false'/>
<preference name='com.ibm.cic.common.sharedUI.showErrorLog' value='true'/>
<preference name='com.ibm.cic.common.sharedUI.showWarningLog' value='true'/>
<preference name='com.ibm.cic.common.sharedUI.showNoteLog' value='true'/>
</agent-input>

it all became wonderful and, more importantly, actually worked :-)

/opt/IBM/InstallationManager/eclipse/tools/imcl -input installBAM855.rsp -acceptLicense

Installed com.ibm.websphere.ND.v85_8.5.5002.20140408_1947 to the /opt/IBM/WebSphere/AppServer directory.
Installed com.ibm.websphere.MON.v85_8.5.5000.20140530_1037 to the /opt/IBM/WebSphere/AppServer directory.
Installed com.ibm.ws.cognos.v1021.linuxia64_10.2.1.20140530_2310 to the /opt/IBM/WebSphere/AppServer directory.

/opt/IBM/WebSphere/AppServer/bin/versionInfo.sh

 
WVER0010I: Copyright (c) IBM Corporation 2002, 2012; All rights reserved.
WVER0012I: VersionInfo reporter version 1.15.1.48, dated 2/8/12

--------------------------------------------------------------------------------
IBM WebSphere Product Installation Status Report
--------------------------------------------------------------------------------

Report at date and time 01 September 2015 14:23:00 BST

Installation
--------------------------------------------------------------------------------
Product Directory        /opt/IBM/WebSphere/AppServer
Version Directory        /opt/IBM/WebSphere/AppServer/properties/version
DTD Directory            /opt/IBM/WebSphere/AppServer/properties/version/dtd
Log Directory            /home/wasadmin/var/ibm/InstallationManager/logs

Product List
--------------------------------------------------------------------------------
WBM                      installed
ND                       installed

Installed Product
--------------------------------------------------------------------------------
Name                  IBM Business Monitor
Version               8.5.5.0
ID                    WBM
Build Level           20140530-093022
Build Date            5/30/14
Package               com.ibm.websphere.MON.v85_8.5.5000.20140530_1037
Architecture          x86-64 (64 bit)
Installed Features    Business Monitor Server Non-production

Installed Product
--------------------------------------------------------------------------------
Name                  IBM WebSphere Application Server Network Deployment
Version               8.5.5.2
ID                    ND
Build Level           cf021414.01
Build Date            4/8/14
Package               com.ibm.websphere.ND.v85_8.5.5002.20140408_1947
Architecture          x86-64 (64 bit)
Installed Features    IBM 64-bit WebSphere SDK for Java
                      WebSphere Application Server Full Profile
                      EJBDeploy tool for pre-EJB 3.0 modules
                      Embeddable EJB container
                      Sample applications
                      Stand-alone thin clients and resource adapters

--------------------------------------------------------------------------------
End Installation Status Report
--------------------------------------------------------------------------------