IBM UrbanCode Deploy - Adding Agents - Or, Waiting for provisioned node

Continuing on my voyage of discovery with IBM UrbanCode Deploy (UCD), I was having all sorts of fun and games adding a new agent to a "clean" installation of Red Hat Enterprise Linux 6.6.

I chose to add the agent via Resources > Agents > Add New Agent, manually entering details such as hostname, SSH port, user ID and password: -

validating the Agent Installation Properties: -

Note that I'd previously installed IBM Java 1.7 as validated here: -

 /opt/IBM/Java/jre/bin/java -version

java version "1.7.0"
Java(TM) SE Runtime Environment (build pxa6470_27sr2-20141026_01(SR2))
IBM J9 VM (build 2.7, JRE 1.7.0 Linux amd64-64 Compressed References 20141017_217728 (JIT enabled, AOT enabled)
J9VM - R27_Java727_SR2_20141017_1632_B217728
JIT  - tr.r13.java_20141003_74587.01
GC   - R27_Java727_SR2_20141017_1632_B217728_CMPRSS
J9CL - 20141017_217728)
JCL - 20141004_01 based on Oracle 7u71-b13

I verified that the installation had completed successfully by checking the appropriate log on the UCD server: -

view /opt/ibm-ucd/server/var/log/agent-install.out

...

install-non-interactive:
    [unzip] Expanding: /tmp/agent-upgrade/overlay.zip into /tmp/agent-install-4316336539585453701.tmp
     [echo]
     [echo] Installing IBM UrbanCode Deploy Agent (non-interactive)
     [echo] The specified directory does not exist. Do you want to create it? Y,n [Default: Y]
     [echo] Warning: Installer prompting for input in non-interactive mode.  Returning default: Y
     [echo]
     [echo] Installing Agent to: /opt/ibm-ucd/agent
     [echo] JAVA_HOME: /opt/IBM/Java/jre
     [echo] Enter the hostname or address of the server the agent will connect to. [Default: localhost]
     [echo] Warning: Installer prompting for input in non-interactive mode.  Returning default: localhost
     [echo] Enter the agent communication port for the server. [Default: 7918]
     [echo] Warning: Installer prompting for input in non-interactive mode.  Returning default: 7918
     [echo] Do you want to configure another failover server connection? y,N [Default: N]
     [echo] Warning: Installer prompting for input in non-interactive mode.  Returning default: N
     [echo] The agent can be added to one or more teams when it first connects to the server. Changing this setting after initial connection to the server will not have any effect.
     [echo] Enter teams to add this agent to, separated by commas. [Default: None]
     [echo] Warning: Installer prompting for input in non-interactive mode.
     [copy] Copying 2 files to /opt/ibm-ucd/agent/conf
     [copy] Copying 1 file to /opt/ibm-ucd/agent/properties
[propertyfile] Creating new property file: /opt/ibm-ucd/agent/conf/agent/installed.properties
[propertyfile] Updating property file: /opt/ibm-ucd/agent/conf/agent/installed.properties
     [copy] Copying 56 files to /opt/ibm-ucd/agent/lib
     [copy] Copying 4 files to /opt/ibm-ucd/agent/monitor
     [copy] Copying 1621 files to /opt/ibm-ucd/agent/opt
     [copy] Copying 2 files to /opt/ibm-ucd/agent/opt/udclient
    [mkdir] Created dir: /opt/ibm-ucd/agent/bin/init
     [copy] Copying 1 file to /opt/ibm-ucd/agent/bin
     [copy] Copying 3 files to /opt/ibm-ucd/agent/bin
     [copy] Copying 2 files to /opt/ibm-ucd/agent/bin
     [echo] Installed version 6.1.1.0.608443
     [echo] Installer Complete. (press return to exit installer)
     [echo] Warning: Installer prompting for input in non-interactive mode.
   [delete] Deleting directory /tmp/agent-install-4316336539585453701.tmp

BUILD SUCCESSFUL
Total time: 7 seconds

...

Having installed the agent, I was expecting to find the agent up and running, and ready to receive.

Instead, I saw this: -

( Waiting for provisioned node )

When I checked, the agent appeared to be running: -

cat /opt/ibm-ucd/agent/var/log/agent.out 

2014-12-30 15:31:00,943 INFO  com.urbancode.air.agent.AgentWorker - Logging configured
2014-12-30 15:31:00,990 INFO  com.urbancode.air.agent.AgentWorker - Agent version: 6.1.1.0.608443

ps auxw | grep -i java

wasadmin  23880  0.0  1.5 1729324 28996 ?       Sl   15:30   0:00 /opt/IBM/Java/jre/bin/java -Dfile.encoding=UTF-8 -jar /opt/ibm-ucd/agent/monitor/air-monitor.jar /opt/ibm-ucd/agent /opt/ibm-ucd/agent/bin/worker-args.conf 7000 -Djava.io.tmpdir=/opt/ibm-ucd/agent/var/temp
wasadmin  23897  0.5  2.2 1461488 42272 ?       Sl   15:30   0:03 /opt/IBM/Java/jre/bin/java -Dcom.urbancode.air.mw.common.Monitor.port=46470 -Djava.io.tmpdir=/opt/ibm-ucd/agent/var/temp -Xmx256m -Dfile.encoding=UTF-8 -Dconsole.encoding=UTF-8 -Djava.security.properties=/opt/ibm-ucd/agent/conf/agent/java.security -Djava.io.tmpdir=/opt/ibm-ucd/agent/var/temp -jar /opt/ibm-ucd/agent/monitor/air-worker.jar /opt/ibm-ucd/agent/bin/classpath.conf 5000 com.urbancode.air.agent.AgentWorker

and yet nothing is listening on port 7918 : -

netstat -aon | grep 7918

<NOTHING RETURNED>

I dug further into the UCD agent configuration: -

cat /opt/ibm-ucd/agent/conf/agent/installed.properties

#Tue Dec 30 15:31:01 GMT 2014
IBM\ UrbanCode\ Deploy/java.home=/opt/IBM/Java/jre
agent.HttpFailoverHandler.disabled=null
locked/agent.brokerUrl=failover\:(ah3\://was855.uk.ibm.com\:7918,ah3\://localhost\:7918)
locked/agent.home=/opt/ibm-ucd/agent
locked/agent.http.proxy.host=
locked/agent.http.proxy.port=
locked/agent.id=ShbCxatf4gd13mrtGL7P
locked/agent.jms.remote.host=was855.uk.ibm.com
locked/agent.jms.remote.port=7918
locked/agent.keystore=../conf/agent.keystore
locked/agent.keystore.pwd=pbe{Y1NGSlNWQFuHSkFNYFwpTGmxmeT6djKzSAp/m8zIHSI\=}
locked/agent.mutual_auth=false
locked/agent.name=was855.uk.ibm.com
system.default.encoding=UTF-8

and compared this to the configuration of the agent on the actual UCD server: -

cat /opt/ibm-ucd/agent/conf/agent/installed.properties

 
#Tue Dec 23 21:00:27 GMT 2014
IBM\ UrbanCode\ Deploy/java.home=/opt/IBM/Java/jre
agent.HttpFailoverHandler.disabled=null
locked/agent.brokerUrl=failover\:(ah3\://ucd61.uk.ibm.com\:7918,ah3\://localhost\:7918)
locked/agent.home=/opt/ibm-ucd/agent
locked/agent.http.proxy.host=
locked/agent.http.proxy.port=
locked/agent.id=ZzloWWuEcF3FGHkfe6HJ
locked/agent.jms.remote.host=ucd61.uk.ibm.com
locked/agent.jms.remote.port=7918
locked/agent.keystore=../conf/agent.keystore
locked/agent.keystore.pwd=pbe{rGO6c95brUqvVAJ6/myFWKT3M/a3fLNUSTgUNeAQLUI\=}
locked/agent.mutual_auth=false
locked/agent.name=ucd61
system.default.encoding=UTF-8

Can you spot the problem ?

The properties agent.brokerURL and agent.ms.remote.host are, on the UCD server, pointing at  ..... the UCD server, as the agent AND server are on the same host-name.

However, on the standalone agent, they're also pointing at .... the hostname of the agent itself :-(

I changed the properties of the standalone agent: -

locked/agent.brokerUrl=failover\:(ah3\://ucd61.uk.ibm.com\:7918)
locked/agent.jms.remote.host=ucd61.uk.ibm.com

and restarted it: -

/opt/ibm-ucd/agent/bin/agent stop
/opt/ibm-ucd/agent/bin/agent start

and checked the logs: -

cat /opt/ibm-ucd/agent/var/log/agent.out

2014-12-30 16:16:46,105 INFO  com.urbancode.air.agent.AgentWorker - Logging configured
2014-12-30 16:16:46,134 INFO  com.urbancode.air.agent.AgentWorker - Agent version: 6.1.1.0.608443
2014-12-30 16:16:51,608 INFO  com.urbancode.air.devilfish.common.SaContainer - Starting SaContainer...
2014-12-30 16:16:51,727 INFO  com.urbancode.air.devilfish.apps.plugin.PluginRuntimeServer - Plugin Recovery ...
2014-12-30 16:16:51,728 INFO  com.urbancode.air.devilfish.common.SaContainer - Started SaContainer
2014-12-30 16:16:51,728 INFO  com.urbancode.air.agent.AgentWorker - Agent started
2014-12-30 16:16:52,542 INFO  com.urbancode.air.devilfish.apps.agentcontrol.PropertiesCollector - Checking connection to ucd61.uk.ibm.com:8443...
2014-12-30 16:16:52,591 INFO  com.urbancode.air.devilfish.apps.agentcontrol.PropertiesCollector - HTTP connection successful.

and checked that port 7918 is now listening: -

netstat -aon | grep 7918

tcp        0      0 ::ffff:192.168.1.74:38701   ::ffff:192.168.1.71:7918    ESTABLISHED off (0.00/0/0)

and finally verified by the UCD server itself: -

Hmm, interesting or what ?

IBM UrbanCode Deploy - More into which to get your teeth stuck

From the UCD 6.1.1 Knowledge Centre

Create a simple helloWorld deployment

In this tutorial, you create a simple application with IBM® UrbanCode Deploy. You use standard plug-ins to create a component and define an application that deploys the component to an environment.

Deploying a simple web application

In this tutorial, you deploy a simple web application with IBM UrbanCode Deploy. You create components, create an application that contains those components, and then deploy the components to an environment.

Plus: -

Deploying (to) WebSphere Liberty

The WASDev team has a pair of articles looking at using IBM UrbanCode Deploy to deploy WebSphere Liberty to a server, and then deploying a web app to that WAS Liberty server.

and ....

The UCD Tour

IBM UrbanCode Deploy is a tool for automating application deployments through your environments. It is designed to facilitate rapid feedback and continuous delivery in agile development while providing the audit trails, versioning and approvals needed in production.

UrbanCode Deploy provides

• Automated, consistent deployments and rollbacks of applications

• Orchestration of changes across servers, tiers and components

• Configuration and security differences across environments

• Clear visibility: what is deployed where and who changed what

• Integrated with middleware, provisioning and service virtualization

Typical Uses

• Continuous Delivery: Integrate with build and test tools to automatically deploy, test and promote new builds

• Production Deployments: Orchestrate a complex production deployments of applications and configuration

• Self-Service: Grant different teams rights to "push the go button" for different applications and environments

• Incremental Updates: Deploy only the changes components or missing incremental (patch) versions

IBM UrbanCode Deploy to WebSphere Application Server - "peer not authenticated"

So I have spent quite literally hours over the past two days working ^H^H^H^H^H^H^H playing with this.

I've got UrbanCode Deploy (UCD) 6.1.0 installed on my VM ( running Red Hat Enterprise Linux 6.4 ) and all is working nicely. However, I was trying, and frequently failing, to get UCD to inspect a WebSphere Application Server (WAS) 8.5.5.3 environment using the Configure using WebSphere Topology Discovery workflow: -

I'd already installed the Application Deployment for WebSphere plugin.

I also had a WAS profile created and running happily, and was able to log into WAS using the Integrated Solutions Console (ISC) and wsadmin.sh via SOAP on port 8880.

However, the Topology Discovery workflow kept failing with: -

Caught: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
at com.ibm.jsse2.bc.getPeerCertificates(bc.java:107)
at org.apache.http.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:128)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:572)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:640)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:479)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
at com.urbancode.ud.client.UDRestClient.invokeMethod(UDRestClient.java:134)
at com.urbancode.ud.client.ResourceClient.getResourceByPath(ResourceClient.java:214)
at com.urbancode.ud.client.ResourceClient$getResourceByPath.call(Unknown Source)
at wasConfig$_run_closure5.doCall(wasConfig.groovy:246)
at wasConfig.run(wasConfig.groovy:447)
 

As I say, I spent hours and hours and hours hacking around with this.

My efforts included: -

Enabling Java SSL Debugging

This I achieved in two ways: -

Setting: -

-Djavax.net.debug=ssl

in server.xml: -

/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/ucd61Node01Cell/nodes/ucd61Node01/servers/server1/server.xml 

( Of course, I would/could/should have done this via Jython but not today, today is a day for celebrations )

Setting: -

-Djavax.net.debug=ssl,handshake

in plugin-javaopts.conf: -

/opt/ibm-ucd/agent/conf/plugin-javaopts.conf

This gave me a slew of interesting, but not totally helpful, debug information: -

WAS

...
[24/12/14 18:04:27:327 GMT] 0000005d SystemOut     O *** ServerHello, TLSv1
[24/12/14 18:04:27:327 GMT] 0000005d SystemOut     O RandomCookie:  GMT: 1402667051 bytes = { 125, 232, 230, 27, 197, 137, 249, 
81, 163, 103, 70, 81, 109, 233, 143, 148, 173, 87, 226, 89, 63, 200, 211, 72, 187, 228, 241, 81 }[24/12/14 18:04:27:327 GMT] 0000005d SystemOut     O Session ID:  {84, 155, 0, 36, 45, 184, 111, 190, 179, 72, 26, 92, 2, 182, 6
4, 128, 247, 137, 170, 40, 221, 74, 75, 59, 0, 110, 53, 213, 115, 217, 13, 27}
[24/12/14 18:04:27:327 GMT] 0000005d SystemOut     O Cipher Suite: SSL_RSA_WITH_RC4_128_MD5
[24/12/14 18:04:27:327 GMT] 0000005d SystemOut     O Compression Method: 0
[24/12/14 18:04:27:327 GMT] 0000005d SystemOut     O Extension renegotiation_info, ri_length: 0, ri_connection_data: { null }
[24/12/14 18:04:27:327 GMT] 0000005d SystemOut     O ***
[24/12/14 18:04:27:327 GMT] 0000005d SystemOut     O Cipher suite:  SSL_RSA_WITH_RC4_128_MD5
[24/12/14 18:04:27:328 GMT] 0000005d SystemOut     O JsseJCE:  Using KeyGenerator IbmTlsKeyMaterial from provider TBD via init
[24/12/14 18:04:27:328 GMT] 0000005d SystemOut     O CONNECTION KEYGEN:
[24/12/14 18:04:27:328 GMT] 0000005d SystemOut     O Client Nonce:
[24/12/14 18:04:27:328 GMT] 0000005d SystemOut     O 0000: 54 9b 00 2b 71 e2 2b aa  ca a4 31 d2 f0 53 14 52  T...q.....1..S.R
0010: 15 53 f2 d2 6b 27 55 90  94 58 dd 30 0a 02 56 38  .S..k.U..X.0..V8

[24/12/14 18:04:27:328 GMT] 0000005d SystemOut     O Server Nonce:
[24/12/14 18:04:27:328 GMT] 0000005d SystemOut     O 0000: 54 9b 00 2b 7d e8 e6 1b  c5 89 f9 51 a3 67 46 51  T..........Q.gFQ
0010: 6d e9 8f 94 ad 57 e2 59  3f c8 d3 48 bb e4 f1 51  m....W.Y...H...Q
[24/12/14 18:04:27:328 GMT] 0000005d SystemOut     O Master Secret:
[24/12/14 18:04:27:328 GMT] 0000005d SystemOut     O 0000: 57 d2 31 ca c5 03 13 84  c3 1f 0a 6e ec ce a7 f1  W.1........n....
0010: e7 4c a8 7f 3c 59 52 32  36 4f ce 88 fa 01 18 41  .L...YR26O.....A
0020: da 62 f0 85 55 ac 96 36  b1 f0 d3 87 3f 48 82 65  .b..U..6.....H.e

[24/12/14 18:04:27:328 GMT] 0000005d SystemOut     O Client MAC write Secret:
[24/12/14 18:04:27:328 GMT] 0000005d SystemOut     O 0000: ad 91 42 34 35 4c 2f d7  ad bf 01 0a 24 db 03 d3  ..B45L..........

[24/12/14 18:04:27:328 GMT] 0000005d SystemOut     O Server MAC write Secret:
[24/12/14 18:04:27:328 GMT] 0000005d SystemOut     O 0000: 95 8d 17 be b1 6d 98 89  bc ab 93 e2 d8 55 82 3e  .....m.......U..

[24/12/14 18:04:27:328 GMT] 0000005d SystemOut     O Client write key:
[24/12/14 18:04:27:328 GMT] 0000005d SystemOut     O 0000: 65 8e 3c 03 35 d2 32 29  ca 24 3b 92 ba 19 d7 0b  e...5.2.........

[24/12/14 18:04:27:328 GMT] 0000005d SystemOut     O Server write key:
[24/12/14 18:04:27:328 GMT] 0000005d SystemOut     O 0000: ab 79 bb 21 38 4d dd bd  e5 90 9e 1c 53 34 76 50  .y..8M......S4vP

[24/12/14 18:04:27:328 GMT] 0000005d SystemOut     O ... no IV used for this cipher
[24/12/14 18:04:27:328 GMT] 0000005d SystemOut     O SoapConnectorThreadPool : 1, WRITE: TLSv1 Handshake, length = 81
[24/12/14 18:04:27:329 GMT] 0000005d SystemOut     O JsseJCE:  Using KeyGenerator IbmTlsPrf from provider TBD via init
[24/12/14 18:04:27:329 GMT] 0000005d SystemOut     O HandshakeMessage:  TLS Keygenerator IbmTlsPrf  from provider from init IBMJCE version 1.2
[24/12/14 18:04:27:330 GMT] 0000005d SystemOut     O SoapConnectorThreadPool : 1, WRITE: TLSv1 Change Cipher Spec, length = 1
[24/12/14 18:04:27:331 GMT] 0000005d SystemOut     O JsseJCE:  Using cipher RC4 from provider TBD via init
[24/12/14 18:04:27:331 GMT] 0000005d SystemOut     O CipherBox:  Using cipher RC4 from provider from init IBMJCE version 1.2
[24/12/14 18:04:27:331 GMT] 0000005d SystemOut     O JsseJCE:  Using MAC HmacMD5 from provider TBD via init
[24/12/14 18:04:27:331 GMT] 0000005d SystemOut     O MAC:  Using MessageDigest HmacMD5 from provider IBMJCE version 1.2
[24/12/14 18:04:27:331 GMT] 0000005d SystemOut     O *** Finished
[24/12/14 18:04:27:331 GMT] 0000005d SystemOut     O verify_data:  { 207, 227, 130, 132, 11, 191, 247, 248, 179, 164, 79, 92 }
[24/12/14 18:04:27:331 GMT] 0000005d SystemOut     O ***
[24/12/14 18:04:27:331 GMT] 0000005d SystemOut     O SoapConnectorThreadPool : 1, WRITE: TLSv1 Handshake, length = 32
[24/12/14 18:04:27:332 GMT] 0000005d SystemOut     O SoapConnectorThreadPool : 1, READ: TLSv1 Change Cipher Spec, length = 1
[24/12/14 18:04:27:332 GMT] 0000005d SystemOut     O JsseJCE:  Using cipher RC4 from provider TBD via init
[24/12/14 18:04:27:333 GMT] 0000005d SystemOut     O CipherBox:  Using cipher RC4 from provider from init IBMJCE version 1.2
[24/12/14 18:04:27:333 GMT] 0000005d SystemOut     O JsseJCE:  Using MAC HmacMD5 from provider TBD via init
[24/12/14 18:04:27:333 GMT] 0000005d SystemOut     O MAC:  Using MessageDigest HmacMD5 from provider IBMJCE version 1.2
[24/12/14 18:04:27:333 GMT] 0000005d SystemOut     O SoapConnectorThreadPool : 1, READ: TLSv1 Handshake, length = 32

...

[24/12/14 19:25:24:539 GMT] 0000005b SystemOut     O SoapConnectorThreadPool : 0, received EOFException: error
[24/12/14 19:25:24:539 GMT] 0000005b SystemOut     O SoapConnectorThreadPool : 0, handling exception: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
[24/12/14 19:25:24:539 GMT] 0000005b SystemOut     O SoapConnectorThreadPool : 0, SEND TLSv1 ALERT:  fatal, description = handshake_failure
...

UCD

...

IBMJSSE2 will not enable CBC protection
JsseJCE:  Using SecureRandom IBMSecureRandom from provider IBMJCE version 1.7
JsseJCE:  Using KeyAgreement ECDH from provider IBMJCE version 1.7
JsseJCE:  Using signature SHA1withECDSA from provider TBD via init 
JsseJCE:  Using signature NONEwithECDSA from provider TBD via init 
JsseJCE:  Using KeyFactory EC from provider IBMJCE version 1.7
JsseJCE:  Using KeyPairGenerator EC from provider TBD via init 
JsseJce:  EC is available
main, setSoTimeout(0) called
IBMJSSE2 will allow RFC 5746 renegotiation per com.ibm.jsse2.renegotiate set to none or default
IBMJSSE2 will not require renegotiation indicator during initial handshake per com.ibm.jsse2.renegotiation.indicator set to OPTIONAL or default taken
IBMJSSE2 will not perform identity checking against the peer cert check during renegotiation per com.ibm.jsse2.renegotiation.peer.cert.check set to OFF or default
 
Is initial handshake: true
Ignoring unsupported cipher suite: SSL_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
Ignoring unsupported cipher suite: SSL_ECDHE_RSA_WITH_AES_128_CBC_SHA256
Ignoring unsupported cipher suite: SSL_RSA_WITH_AES_128_CBC_SHA256
Ignoring unsupported cipher suite: SSL_ECDH_ECDSA_WITH_AES_128_CBC_SHA256
Ignoring unsupported cipher suite: SSL_ECDH_RSA_WITH_AES_128_CBC_SHA256
Ignoring unsupported cipher suite: SSL_DHE_RSA_WITH_AES_128_CBC_SHA256
Ignoring unsupported cipher suite: SSL_DHE_DSS_WITH_AES_128_CBC_SHA256
Ignoring unsupported cipher suite: SSL_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
Ignoring unsupported cipher suite: SSL_ECDHE_RSA_WITH_AES_128_CBC_SHA
Ignoring unsupported cipher suite: SSL_RSA_WITH_AES_128_CBC_SHA
Ignoring unsupported cipher suite: SSL_ECDH_ECDSA_WITH_AES_128_CBC_SHA
Ignoring unsupported cipher suite: SSL_ECDH_RSA_WITH_AES_128_CBC_SHA
Ignoring unsupported cipher suite: SSL_DHE_RSA_WITH_AES_128_CBC_SHA
Ignoring unsupported cipher suite: SSL_DHE_DSS_WITH_AES_128_CBC_SHA
Ignoring unsupported cipher suite: SSL_ECDHE_ECDSA_WITH_RC4_128_SHA
Ignoring unsupported cipher suite: SSL_ECDHE_RSA_WITH_RC4_128_SHA
Ignoring unsupported cipher suite: SSL_ECDH_ECDSA_WITH_RC4_128_SHA
Ignoring unsupported cipher suite: SSL_ECDH_RSA_WITH_RC4_128_SHA
Ignoring unsupported cipher suite: SSL_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA
Ignoring unsupported cipher suite: SSL_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA
Ignoring unsupported cipher suite: SSL_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA
Ignoring unsupported cipher suite: SSL_ECDH_RSA_WITH_3DES_EDE_CBC_SHA
%% No cached client session
*** ClientHello, SSLv3
RandomCookie:  GMT: 1402593669 bytes = { 215, 60, 79, 216, 73, 171, 239, 0, 81, 69, 93, 98, 89, 131, 202, 26, 159, 74, 101, 239, 235, 105, 218, 190, 41, 139, 196, 23 }
Session ID:  {}
Cipher Suites: [TLS_EMPTY_RENEGOTIATION_INFO_SCSV, SSL_RSA_WITH_RC4_128_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_RC4_128_MD5]
Compression Methods:  { 0 }
***
main, WRITE: SSLv3 Handshake, length = 55
main, READ: TLSv1 Alert, length = 2
main, RECV TLSv1 ALERT:  fatal, handshake_failure
main, called closeSocket()
main, handling exception: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
main, IOException in getSession():  javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
main, called close()
main, called closeInternal(true)
main, called close()
main, called closeInternal(true)
Caught: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
at com.ibm.jsse2.bc.getPeerCertificates(bc.java:107)
at org.apache.http.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:128)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:572)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:640)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:479)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
at com.urbancode.ud.client.UDRestClient.invokeMethod(UDRestClient.java:134)
at com.urbancode.ud.client.ResourceClient.getResourceByPath(ResourceClient.java:214)
at com.urbancode.ud.client.ResourceClient$getResourceByPath.call(Unknown Source)
at wasConfig$_run_closure5.doCall(wasConfig.groovy:246)
at wasConfig.run(wasConfig.groovy:447)
...

This led me up and down all sorts of lovely lovely garden paths, including: -

• Fiddling with SSL v3 and TLS v1 protocols, via -Dhttps.protocols=SSLv3 etc.
• Disabling and enabling session renegotiation via -Dcom.ibm.jsse2.renegotiate
• Disabling the Server Name Indication (SNI) Extension via -Djsse.enableSNIExtension=false

I also spent an absolute age adding SSL certificates to the JRE keystore: -

export ADDRESS=ucd61.uk.ibm.com
export PORT=8880
echo -n | openssl s_client -connect $ADDRESS:$PORT | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /tmp/$ADDRESS.cert

/opt/IBM/Java/jre/bin/keytool -importcert -trustcacerts -alias was855_3 -keystore /opt/IBM/Java/jre/lib/security/cacerts -storepass changeit -file /tmp/ucd61.uk.ibm.com.cert

but to no avail.

The solution ?

Well, I'd wondered about the Java client version underlying UCD, and checked this: -

/opt/IBM/Java/jre/bin/java -version

java version "1.7.0"
Java(TM) SE Runtime Environment (build pxa6470sr5-20130619_01(SR5))
IBM J9 VM (build 2.6, JRE 1.7.0 Linux amd64-64 Compressed References 20130617_152572 (JIT enabled, AOT enabled)
J9VM - R26_Java726_SR5_20130617_1436_B152572
JIT  - r11.b04_20130528_38954ifx1
GC   - R26_Java726_SR5_20130617_1436_B152572_CMPRSS
J9CL - 20130617_152572)
JCL - 20130616_01 based on Oracle 7u25-b12

and wondered whether, post-POODLE, this was part of the problem.

I downloaded, and installed, the most recent IBM JRE: -

IBM SDK, Java Technology Edition, Version 7 Release 1, Service Refresh 2 Fix pack 0

/opt/IBM/Java/jre/bin/java -version

java version "1.7.0"
Java(TM) SE Runtime Environment (build pxa6470_27sr2-20141026_01(SR2))
IBM J9 VM (build 2.7, JRE 1.7.0 Linux amd64-64 Compressed References 20141017_217728 (JIT enabled, AOT enabled)
J9VM - R27_Java727_SR2_20141017_1632_B217728
JIT  - tr.r13.java_20141003_74587.01
GC   - R27_Java727_SR2_20141017_1632_B217728_CMPRSS
J9CL - 20141017_217728)
JCL - 20141004_01 based on Oracle 7u71-b13

and retried the Configure using WebSphere Topology Discovery workflow .... and it worked :-)

The moral of the story ? Check your JRE version.

Now I need to go and undo all the hacks I've made to my environment, but at least I've learned a valuable lesson.

Sources of Inspiration

How to Fix 'SSLPeerUnverifiedException: peer not authenticated' Exception in Java / Groovy

Transport Layer Security

How to analyze Java SSL errors

Diagnosing TLS, SSL, and HTTPS

IBM Java 1.7 - "The installer cannot run on your configuration. It will now quit. "

Notes on a scandal ....

Specifically, notes seen when trying to install IBM Java 1.7 onto RHEL 6.4

cat /etc/redhat-release

Red Hat Enterprise Linux Server release 6.4 (Santiago)

uname -a

Linux ucd61.uk.ibm.com 2.6.32-358.51.1.el6.x86_64 #1 SMP Sun Oct 26 14:30:34 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux

A quick debug ( thanks to this )

export _JAVA_OPTIONS="-Dlax.debug.level=3 -Dlax.debug.all=true"
export LAX_DEBUG=1

~/ibm-java-x86_64-jre-7.1-2.0.bin 

Preparing to install...
Checking for POSIX df.
Found POSIX df.
Checking tail options...
Using tail -n 1.
True location of the self extractor: /home/wasadmin/ibm-java-x86_64-jre-7.1-2.0.bin
Creating installer data directory: /tmp/install.dir.68246
Creating installer data directory: /tmp/install.dir.68246/InstallerData
Gathering free-space information...
Space needed to complete the self-extraction: 589664 blocks
Available space: 64802224 blocks
Available blocks: 64802224    Needed blocks: 589664 (block = 512 bytes)
Computed number of blocks to extract: 2934
Extracting the JRE from the installer archive...
Extracting JRE from /home/wasadmin/ibm-java-x86_64-jre-7.1-2.0.bin to /tmp/install.dir.68246/Linux/resource/jre_padded ...
Extracting done, exit code = 0
Extracting JRE from /tmp/install.dir.68246/Linux/resource/jre_padded to /tmp/install.dir.68246/Linux/resource/vm.tar.Z ...
 Extracting done, exit code = 0
Unpacking the JRE...
Unpacking the JRE...
gzip is /bin/gzip
 GZIP done.
 TAR done.
Extracting the installation resources from the installer archive...
Extracting install.zip from /home/wasadmin/ibm-java-x86_64-jre-7.1-2.0.bin to /tmp/install.dir.68246/InstallerData/installer.padded ...
Extracting to padded done, exit code = 0
Extracting from padded to zip done, exit code = 0
Creating disk1 data directory: /tmp/install.dir.68246/InstallerData/Disk1
Creating instdata data directory: /tmp/install.dir.68246/InstallerData/Disk1/InstData
Extracting resources from /home/wasadmin/ibm-java-x86_64-jre-7.1-2.0.bin to /tmp/install.dir.68246/InstallerData/Disk1/InstData/Resource1.zip ...
Extracting done, exit code = 0
Configuring the installer for this system's environment...

========= Analyzing UNIX Environment =================================
Setting UNIX (linux) flavor specifics.
Importing UNIX environment into LAX properties.
Checking for POSIX awk.

========= Analyzing LAX ==============================================
LAX found............................ OK.
LAX properties read.................. OK.

========= Finding VM =================================================
Valid VM types.......................... 1.4+
Absolute LAX_VM path.................... /tmp/install.dir.68246/Linux/resource/jre/bin/java
Expanded Valid VM types.................  1.4+ 
* Using VM.....(lax.nl.current.vm)...... /tmp/install.dir.68246/Linux/resource/jre/bin/java
checking for NPTL + JVM vulernability...
NPTL detected! checking for vulnerable JVM....

========= Virtual Machine Options ====================================
LAX properties incorporated............. OK.
classpath............................... "/tmp/install.dir.68246/InstallerData:/tmp/install.dir.68246/InstallerData/installer.zip"
main class.............................. "com.zerog.ia.installer.Main"
.lax file path.......................... "/tmp/install.dir.68246/temp.lax"
user directory.......................... "/tmp/install.dir.68246"
stdout to............................... "console"
sterr to................................ "console"
install directory....................... ""
JIT..................................... none
option (verify)......................... off
option (verbosity)...................... none
option (garbage collection extent)...... none
option (garbage collection thread)...... none
option (native stack max size).......... none
option (java stack max size)............ none
option (java heap max size)............. 50331648
option (java heap initial size)......... 16777216
option (lax.nl.java.option.additional).. none

========= Display settings ===========================================
X display............................... remote
WARNING:  The name  of  this  host (ucd61.uk.ibm.com) and  the setting
of this  shell's DISPLAY (localhost:10.0) variable do not match.
If this launcher is being displayed to a Microsoft Windows desktop
through X Windows the Java Virtual Machine might abort. Try running
this installer locally on the target system or through X Windows to
another UNIX host if the installer unexpectedly fails.
UI mode................................. gui

Launching installer...

========= VM Command Line ============================================
options: -Djava.compiler=NONE  -Xmx50331648 -Xms16777216 
CLASSPATH:/tmp/install.dir.68246/InstallerData:/tmp/install.dir.68246/InstallerData/installer.zip:

========= Forking JAVA =============================================
LAX Version = 11.5

Graphical installers are not supported by the VM. The console mode will be used instead...

__________________________________________________________________________

InstallAnywhere 2010
Version: 11.5
__________________________________________________________________________

Wed Dec 24 19:36:18 GMT 2014

Free Memory: 12125 kB
Total Memory: 16384 kB

No Arguments

java.class.path:
    /tmp/install.dir.68246/InstallerData
    /tmp/install.dir.68246/InstallerData/installer.zip

ZGUtil.CLASS_PATH:
    /tmp/install.dir.68246/InstallerData
    /tmp/install.dir.68246/InstallerData/installer.zip

sun.boot.class.path:
    /tmp/install.dir.68246/Linux/resource/jre/lib/amd64/compressedrefs/jclSC170/vm.jar
    /tmp/install.dir.68246/Linux/resource/jre/lib/se-service.jar
    /tmp/install.dir.68246/Linux/resource/jre/lib/math.jar
    /tmp/install.dir.68246/Linux/resource/jre/lib/jlm.jar
    /tmp/install.dir.68246/Linux/resource/jre/lib/ibmorb.jar
    /tmp/install.dir.68246/Linux/resource/jre/lib/ibmorbapi.jar
    /tmp/install.dir.68246/Linux/resource/jre/lib/ibmcfw.jar
    /tmp/install.dir.68246/Linux/resource/jre/lib/ibmpkcs.jar
    /tmp/install.dir.68246/Linux/resource/jre/lib/ibmcertpathfw.jar
    /tmp/install.dir.68246/Linux/resource/jre/lib/ibmjgssfw.jar
    /tmp/install.dir.68246/Linux/resource/jre/lib/ibmjssefw.jar
    /tmp/install.dir.68246/Linux/resource/jre/lib/ibmsaslfw.jar
    /tmp/install.dir.68246/Linux/resource/jre/lib/ibmjcefw.jar
    /tmp/install.dir.68246/Linux/resource/jre/lib/ibmjgssprovider.jar
    /tmp/install.dir.68246/Linux/resource/jre/lib/ibmjsseprovider2.jar
    /tmp/install.dir.68246/Linux/resource/jre/lib/ibmcertpathprovider.jar
    /tmp/install.dir.68246/Linux/resource/jre/lib/xmldsigfw.jar
    /tmp/install.dir.68246/Linux/resource/jre/lib/xml.jar
    /tmp/install.dir.68246/Linux/resource/jre/lib/charsets.jar
    /tmp/install.dir.68246/Linux/resource/jre/lib/resources.jar
    /tmp/install.dir.68246/Linux/resource/jre/lib/rt.jar
    /tmp/install.dir.68246/Linux/resource/jre/lib/dataaccess.jar
    /tmp/install.dir.68246/Linux/resource/jre/lib/ibmgpu.jar

java.ext.dirs:
    /tmp/install.dir.68246/Linux/resource/jre/lib/ext

java.version                  == 1.7.0 (Java 1)
java.vm.name                  == IBM J9 VM
java.vm.vendor                == IBM Corporation
java.vm.version               == 2.7
java.vm.specification.name    == Java Virtual Machine Specification
java.vm.specification.vendor  == Sun Microsystems Inc.
java.vm.specification.version == 1.0
java.specification.name       == Java Platform API Specification
java.specification.vendor     == Sun Microsystems Inc.
java.specification.version    == 1.7
java.vendor                   == IBM Corporation
java.vendor.url               == http://www.ibm.com/
java.class.version            == 51.0
java.compiler                 == NONE
java.home                     == /tmp/install.dir.68246/Linux/resource/jre
java.io.tmpdir                == /tmp
os.name                       == Linux
os.arch                       == amd64
os.version                    == 2.6.32-358.51.1.el6.x86_64
path.separator                == :
file.separator                == /
file.encoding                 == UTF-8
user.name                     == wasadmin
user.home                     == /home/wasadmin
user.dir                      == /tmp/install.dir.68246
user.language                 == en
user.region                   == null
__________________________________________________________________________

===============================================================================
Choose Locale...
----------------

    1- Bahasa Indonesia
    2- Català
    3- Deutsch
  ->4- English
    5- Español
    6- Français
    7- Italiano
    8- Português  (Brasil)

CHOOSE LOCALE BY NUMBER: 4
Selected Locale = English

ChooseBundledVMs: Unable to locate the VMPack Directory
RepositoryManager: Trying fallback repository location...
Loading externalized properties

===============================================================================
IBM 64-bit Runtime Environment for Linux, Java Technology Edition, Version 7.1(created with InstallAnywhere)
------------------------------------------------------------------------------------------------------------

Preparing CONSOLE Mode Installation...


The installer cannot run on your configuration. It will now quit.
System's temporary directory = /tmp
SHUTDOWN REQUESTED
(X) commiting registry
(X) shutting down service manager
(X) cleaning up temporary directories
Exiting with exit code: 0
cleanUp()
calling cleanUpUNIX()

So what was I doing wrong ?

I was trying to install as a non-root user - wasadmin.

Once I switched to root: -

su -

and tried again, all was well.

IBM UrbanCode Deploy - Working with Apache Tomcat SSL Key Store

One of my friends asked me how one can add SSL certificates to the Apache Tomcat SSL trust store underlying the IBM UrbanCode Deploy automation solution.

In this scenario, he needed to retrieve a certificate from IBM Rational Asset Manager (IRAM) into the UCD key store, in order that a UCD process can access IRAM.

I've done this for IBM HTTP Server and IBM WebSphere Application Server in the past, using the IBM Global Security Toolkit ( GSK ), but Tomcat uses something slightly different.

This blog post gave me the pointer: -

How to import IBM UrbanCode Deploy self-signed certificate in the client JVM for use with the REST API

and this is what I did: -

List Current Certificates in Key Store

/opt/IBM/Java/jre/bin/keytool -list -keystore /opt/ibm-ucd/server/opt/tomcat/conf/tomcat.keystore -storepass changeit 

Keystore type: jks
Keystore provider: IBMJCE

Your keystore contains 2 entries

server, 14-Dec-2014, keyEntry,
Certificate fingerprint (SHA1): 65:22:8A:B7:B8:EA:53:36:0D:75:E9:74:DF:20:90:DB:BB:C1:AC:4A

Get IRAM Certificate

openssl s_client -showcerts -connect ucd61.uk.ibm.com:9443 </dev/null  > ~/iram.cer

depth=1 C = US, O = IBM, OU = ucd61Node01, OU = ucd61Node01Cell, OU = Root Certificate, CN = ucd61.uk.ibm.com
verify error:num=19:self signed certificate in certificate chain
verify return:0
DONE

( In my case, I'm using WAS 8.5.5 on port 9443 in lieu of IRAM )

Note, I needed to manually edit the retrieved certificate to reduce superfluous tags, possible because the WAS certificate is self-signed e.g.: -

-----BEGIN CERTIFICATE-----
MIIDyDCCArCgAwIBAgIGH9jdPEHdMA0GCSqGSIb3DQEBBQUAMIGBMQswCQYDVQQG
EwJVUzEMMAoGA1UEChMDSUJNMRQwEgYDVQQLEwt1Y2Q2MU5vZGUwMTEYMBYGA1UE
CxMPdWNkNjFOb2RlMDFDZWxsMRkwFwYDVQQLExBSb290IENlcnRpZmljYXRlMRkw
FwYDVQQDExB1Y2Q2MS51ay5pYm0uY29tMB4XDTE0MTIxMTIxMjgyMloXDTE1MTIx
MTIxMjgyMlowZjELMAkGA1UEBhMCVVMxDDAKBgNVBAoTA0lCTTEUMBIGA1UECxML
dWNkNjFOb2RlMDExGDAWBgNVBAsTD3VjZDYxTm9kZTAxQ2VsbDEZMBcGA1UEAxMQ
dWNkNjEudWsuaWJtLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AKlFEK54+pN68kp2V1//jU7hiYDjoOvRzECnn6D4CNWAuQ0VOtzaRbOmUmxmPyIb
44kPmxRz34yQj5mjA/0cNGS176EDeGikBlFvMZapJyf2wgEissuliDuaHrBRWrRe
0SWu3rk0vEPZjBAbl7V26vUOMDpFHoHbh3/Um7Qx7Ur9PdV3Qd/9ANhOeyw1G59D
acIhDKuLt3cHWTqLWbozOp0y3Z90sK8dwcOj+z0CH7e4/PHvO1mwlE5I8z9k378e
1FYwtuo/FD+NQMVEpZImTJMrFZVNPGIVp1EOTBC/XMp/cuvVAqcZMoG59bjpjyRd
mWH42L05CH/X1sNw3CI5bWMCAwEAAaNgMF4wSQYDVR0RBEIwQIE+UHJvZmlsZVVV
SUQ6QXBwU3J2MDEtQkFTRS1lMzAzNjNkZi01Y2I1LTQ2MmEtYmM0ZC02Yjg3NTA5
YzRiNTQwEQYDVR0OBAoECEyME/33gIvbMA0GCSqGSIb3DQEBBQUAA4IBAQBeWvNX
g1gOmJLkqIR9agSOxiTvWKAv+nzRIb9sQ9cQX/B0ucFx+Xh58tSO+bfJFDI42QKU
aiUXwtq8pAf+RWvf3kHagpN4GZFKByhp19lBwoceG/3gk/5IGALT+9useJU3N2+T
hNcqJERyAuCH5oBR+goml28yGjWOUAf/6fWf08heGEnfbLMl2S5IiVAoyaRY+byg
KITMlxj9x3YHxME8g7/hL0RyYfRe/Etb0AV4YL2YkAvaPcjFwjOVR35Bv1C1S3zP
q3FJaizXK2M8Albz8UJxnHj+Li3Qnps4wxLci/n40olEbLH6+Lfd1KOc1Nq6817M
6R4QGcCNIClZdIg8
-----END CERTIFICATE-----

otherwise, I end up with: -

keytool error: java.lang.Exception: Input not an X.509 certificate
Add IRAM Certificate to Key Store

/opt/IBM/Java/jre/bin/keytool -importcert -alias iram -file ~/iram.cer -keystore /opt/ibm-ucd/server/opt/tomcat/conf/tomcat.keystore -storepass changeit

Owner: CN=ucd61.uk.ibm.com, OU=ucd61Node01Cell, OU=ucd61Node01, O=IBM, C=US
Issuer: CN=ucd61.uk.ibm.com, OU=Root Certificate, OU=ucd61Node01Cell, OU=ucd61Node01, O=IBM, C=US
Serial number: 1fd8dd3c41dd
Valid from: 11/12/14 21:28 until: 11/12/15 21:28
Certificate fingerprints:
 MD5:  0F:E7:18:C1:69:1B:ED:FC:47:D7:B7:25:7A:5F:E5:8B
 SHA1: 7B:27:67:B7:DC:12:02:15:0C:90:2F:71:7D:F8:CB:59:5F:3D:34:72
 SHA256: 4F:F0:ED:7B:BA:E1:74:2A:20:E2:ED:B6:E8:6B:50:DD:6E:37:3B:0D:19:DB:8B:3C:A4:71:A6:69:44:56:FD:2C
 Signature algorithm name: SHA1withRSA
 Version: 3

Extensions: 

#1: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
[RFC822Name: ProfileUUID:AppSrv01-BASE-e30363df-5cb5-462a-bc4d-6b87509c4b54]]

#2: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 4c 8c 13 fd f7 80 8b db                           L.......
]
]

Trust this certificate? [no]:  y
Certificate was added to keystore

List Current Certificates in Key Store

/opt/IBM/Java/jre/bin/keytool -list -keystore /opt/ibm-ucd/server/opt/tomcat/conf/tomcat.keystore -storepass changeit 

Keystore type: jks
Keystore provider: IBMJCE

Your keystore contains 2 entries

iram, 22-Dec-2014, trustedCertEntry,
Certificate fingerprint (SHA1): 7B:27:67:B7:DC:12:02:15:0C:90:2F:71:7D:F8:CB:59:5F:3D:34:72server, 14-Dec-2014, keyEntry,
Certificate fingerprint (SHA1): 65:22:8A:B7:B8:EA:53:36:0D:75:E9:74:DF:20:90:DB:BB:C1:AC:4A

What's next ?

Yes, time to change the default password for the Tomcat key store .....

IBM BPM - JavaScript - To Infinity and Beyond

This popped up in my Twitter feed today: -

I suspect that a JavaScript activity within my BPD application runs in an infinite loop. How can I detect such a loop?

developerWorks Answers

Definitely worth a read .....

IBM UrbanCode Deploy - I remember my first time

This is the first of many posts covering a relatively recent acquisition, IBM UrbanCode Deploy (UCD), which is, amongst other things, an automated deployment and delivery solution.

The reason for my interest is that I'm working with a few clients who are considering the use of UCD for automation, in the context of IBM Business Process Manager, IBM Operational Decision Manager, IBM Integration Bus etc.

To start with, I'm going to outline my experiences thus far with the build of a UCD environment, installed on a virtualised Red Hat Enterprise Linux environment, running on my Mac.

Unlike other products, UCD doesn't use IBM Installation Manager or WebSphere Application Server, although it DOES require a relational database. I'm going to use DB2 10.1, which is located on the same VM.

I began by downloading UCD 6.1.1, which is the latest release - as of a few days ago: -

Specifically, I downloaded a 2 GB (!) ISO image: -

IBM UrbanCode Deploy 6.1.1 Multiplatform Multilingual (CN2BAML )

which resulted in this file: -

-rw-r--r--@ 1 hayd  staff   2.0G 12 Dec 12:23 IBM_URBANCODE_DEPLOY_6.1.1_MULTIP.iso

Being an ISO file, I was able to mount this in VMware Fusion as /media: -

ls -alh /media/

total 430M

dr-xr-xr-x.  1 root     root  2.0K Dec  2 13:28 .
dr-xr-xr-x. 25 root     root  4.0K Dec 12 12:59 ..
-rwxr-xr-x.  1 db2inst1 games 430M Dec  1 22:54 ibm-ucd-6110608828.zip
dr-xr-xr-x.  1 root     root  2.0K Dec  2 13:28 java
dr-xr-xr-x.  1 root     root  2.0K Dec  2 13:28 z-os

including: -

ls -alh /media/java/

total 16K
dr-xr-xr-x. 1 root root 2.0K Dec  2 13:28 .
dr-xr-xr-x. 1 root root 2.0K Dec  2 13:28 ..
dr-xr-xr-x. 1 root root 2.0K Aug 21  2013 aix
dr-xr-xr-x. 1 root root 2.0K Aug 21  2013 hpux
dr-xr-xr-x. 1 root root 2.0K Aug 21  2013 linux
dr-xr-xr-x. 1 root root 2.0K Aug 21  2013 solaris
dr-xr-xr-x. 1 root root 2.0K Aug 21  2013 windows
dr-xr-xr-x. 1 root root 2.0K Dec  3 15:10 z-os

ls -alh /media/java/linux/

total 12K
dr-xr-xr-x. 1 root root 2.0K Aug 21  2013 .
dr-xr-xr-x. 1 root root 2.0K Dec  2 13:28 ..
dr-xr-xr-x. 1 root root 2.0K Aug 21  2013 s390
dr-xr-xr-x. 1 root root 2.0K Aug 21  2013 s390_64
dr-xr-xr-x. 1 root root 2.0K Aug 21  2013 x32
dr-xr-xr-x. 1 root root 2.0K Aug 21  2013 x64

ls -alh /media/java/linux/x64/

total 85M
dr-xr-xr-x. 1 root     root  2.0K Aug 21  2013 .
dr-xr-xr-x. 1 root     root  2.0K Aug 21  2013 ..
-rwxr-xr-x. 1 db2inst1 games  85M Aug 21  2013 ibm-java-jre-70-50-linux-x8.gz

I started by extracting the JRE: -

mkdir /opt/IBM/Java
tar xvzf /media/java/linux/x64/ibm-java-jre-70-50-linux-x8.gz -C /tmp/
mv /tmp/ibm-java-x86_64-70/* /opt/IBM/Java/

and testing it: -

/opt/IBM/Java/jre/bin/java -version

java version "1.7.0"
Java(TM) SE Runtime Environment (build pxa6470sr5-20130619_01(SR5))
IBM J9 VM (build 2.6, JRE 1.7.0 Linux amd64-64 Compressed References 20130617_152572 (JIT enabled, AOT enabled)
J9VM - R26_Java726_SR5_20130617_1436_B152572
JIT  - r11.b04_20130528_38954ifx1
GC   - R26_Java726_SR5_20130617_1436_B152572_CMPRSS
J9CL - 20130617_152572)
JCL - 20130616_01 based on Oracle 7u25-b12

Before installing UCD, I created the required DB2 database: -

db2 create database UCD61 automatic storage yes using codeset UTF-8 territory GB pagesize 32768

as db2inst1 and granted permission for the db2user1 ID: -

db2 connect to ucd61
db2 grant dbadm on database to user db2user1

I unpacked UCD: -

unzip /media/ibm-ucd-6110608828.zip -d /tmp

copied the required JDBC drivers into the temporary directory: -

cp /opt/ibm/db2/V10.1/java/db2jcc* /tmp/ibm-ucd-install/lib/ext/

and installed it: -

cd /tmp/ibm-ucd-install/

./install-server.sh 

...

     [echo] Do you accept the license? [y,n] Y
...

     [echo] Installing IBM UrbanCode Deploy version 6.1.1.0.608828
     [echo] Enter the directory where the IBM UrbanCode Deploy should be installed. [Default: /opt/ibm-ucd/server]

...

     [echo] The specified directory does not exist. Do you want to create it? Y,n [Default: Y]
...

     [echo] Installing IBM UrbanCode Deploy to: /opt/ibm-ucd/server
     [echo] Please enter the home directory of the JRE/JDK used to run the server. [Default: /opt/IBM/Java/jre]
...

     [echo] JVM Version detected: 1.7.0
     [echo] JAVA_HOME: /opt/IBM/Java/jre
     [echo] What host name will users access the Web UI at? [Default: ucd61.uk.ibm.com]
...

     [echo] Do you want the Web UI to always use secure connections using SSL? Y,n [Default: Y]
...

     [echo] Enter the port on which the Web UI should listen for secure HTTPS requests. [Default: 8443]
...

     [echo] Enter the port on which the Web UI should redirect unsecured HTTP requests from. [Default: 8080]
...

     [echo] Enter the initial password for the admin user.
...

     [echo] Please type password again.
...

     [echo] Enter the port to use for agent communication. [Default: 7918]
...

     [echo] Do you want the Server and Agent communication to require mutual authentication?  This requires a manual key exchange between the server and each agent. See the documentation for more details. y,N [Default: N]
....

     [echo] Enter the port and hostname of a Rational License Key Server containing product licenses for IBM UrbanCode Deploy, in the form of port@hostname. (e.g. 27000@licenses.example.com) Alternatively, you may leave this blank to begin a 60-day evaluation period. [Default: none]
....

     [echo] Create database schema? Y,n [Default: Y]
...

     [echo] Enter the database type to use. [Default: derby] db2
...

     [echo] Enter the database driver. [Default: com.ibm.db2.jcc.DB2Driver]
...

     [echo] Enter the database connection string. Eg. jdbc:db2://localhost:50000/ibm_ucd jdbc:db2://ucd61.uk.ibm.com:60008/UCD61
...

     [echo] Enter the database username. [Default: ibm_ucd] db2user1
...

     [echo] Enter the database password. [Default: password]
...

     [echo] After starting the server, you may access the web UI by pointing your web-browser at
     [echo] https://ucd61.uk.ibm.com:8443 to complete the Installation.
     [echo] Installer Complete. (press return to exit installer)

...

BUILD SUCCESSFUL
Total time: 2 minutes 24 seconds

I started the server: -

and, ~30 seconds later, checked that it was up-and-running: -

cat /opt/ibm-ucd/server/var/log/deployserver.out 

2014-12-14 11:13:16,619 WARN  main com.urbancode.ds.UDeployServer - Property encryption.keystore.password not set. Using value from property server.keystore.password
2014-12-14 11:13:17,050 INFO  main com.urbancode.ds.UDeployServer - Configuring Agent Network System for single-server setup...
2014-12-14 11:13:19,559 INFO  main com.urbancode.ds.UDeployServer -  done
2014-12-14 11:13:19,703 INFO  main com.urbancode.ds.UDeployServer - IBM UrbanCode Deploy server started.

netstat -aon | grep 8443

tcp        0      0 :::8443                     :::*                        LISTEN      off (0.00/0/0)

db2 list tables for schema db2user1

...

SEC_DB_VERSION                  DB2USER1        T     2014-12-14-11.11.11.872799
SEC_GROUP                       DB2USER1        T     2014-12-14-11.11.12.207976
SEC_GROUP_MAPPER                DB2USER1        T     2014-12-14-11.11.12.271739
SEC_GROUP_MAPPING               DB2USER1        T     2014-12-14-11.11.12.338698
SEC_GROUP_ROLE_ON_TEAM          DB2USER1        T     2014-12-14-11.11.12.399760
...

and, most importantly, tested the server: -

https://ucd61.uk.ibm.com:8443/

and logged in ( note to self, the user ID is, by default, admin although that's NOT obvious ): -

I guessed the user ID, but could have pulled it from the database: -

db2 "select name from db2user1.sec_user"

NAME                                                                                                                                                                                                                                                          
admin                                                                                                                                                                                                                                                         

  1 record(s) selected.

Note this message: -

There is no agent or tag configured to import new component versions, so no new versions will be imported. Please set this on the Settings > System Settings page.

occurs because I haven't yet installed any UCD plugins.

These can be downloaded via the Tools menu: -

Having downloaded the IBM UrbanCode Deploy Agent, I installed it via Resources > Agents > Install New Agent 

Note that I needed to override the Java Home Path ( following on from the Java installation earlier ).

Now the agent is installed, it's ready to be provisioned: -

This is achieved via the Settings > System Settings page: -

We now have a working agent: -

and we're ready to go ......

IBM UrbanCode Deploy - Continuous application delivery to WebSphere Application Server

I'm starting to "play" with UrbanCode Deploy (UCD), in the initial context of continuous application delivery to WebSphere Application Server, moving to the "full fat" model of CAD to IBM BPM and IBM IIB.

This looks to be a useful read: -

Continuous application delivery to WebSphere Application Server

This is even more useful: -

https://developer.ibm.com/urbancode/

PS UCD 6.1.1 was released YESTERDAY - December 11 - guess what I downloaded this morning ?

So I've got UCD installed on a RHEL 6.3 VM, with IBM Java, DB2 10.1 and WAS 8.5.5.

I'll write up the HOW in a few days once I've learned a bit more .....

Journeys in Python - Day 471 - Setting up WebSphere MQ Messaging Providers

So I've previously written about the fun I had setting up WebSphere Application Server to pull messages from a WebSphere MQ Cluster here: -

Lessons Learned - WebSphere MQ, Clustering and Message Driven Beans

This time around, I wanted to streamline my code somewhat, specifically in terms of setting the Native Library Path, as my code wasn't generic enough last time: -

...

Update WAS MQ Provider to support local bindings ( need to add native path )

AdminTask.manageWMQ('"WebSphere MQ Resource Adapter(cells/bpm85Cell1/nodes/AppSrv01Node/servers/foobar|resources.xml#J2CResourceAdapter_1416556034607)"', '[-nativePath /opt/mqm/java/lib64/ -disableWMQ false ]')
AdminConfig.save()
AdminNodeManagement.syncActiveNodes()

....

In other words, my code would only work for THAT particular Resource Adapter.

Borrowing from the nice Mr Steven Robinson Esq; -

How to disable WMQ in WebSphere 8 using manageWMQ command with Jython

I produced a small Jython script that does the job: -

servers = AdminUtilities.convertToList(AdminTask.listServers('[-serverType APPLICATION_SERVER ]'))
ras = AdminUtilities.convertToList(AdminConfig.list('J2CResourceAdapter'))

for serverName in servers :
        name = AdminConfig.showAttribute(serverName, "name")
        if name == "server1" :
print name
                for ra in ras :
                        if ra.find(name) > 0:
                                desc = AdminConfig.showAttribute(ra, "description")
                         if desc.find("WebSphere MQ") > 0:
print "Setting native path for " + desc + " on " + name
AdminTask.manageWMQ(ra, '[-nativePath /opt/ibm/mqm/usr/mqm/java/lib64 -disableWMQ false ]')
AdminConfig.save()

Obviously I could parameterise it, rather than requiring the server name to be hard-coded, but that's a nice-to-have.

:-)

If I run it: -

/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/bin/wsadmin.sh -lang jython -user wasadmin -password passw0rd -f foobar.jy 

WASX7209I: Connected to process "server1" on node localhostNode01 using SOAP connector;  The type of process is: UnManagedProcess
server1
Setting native path for WAS Built In WebSphere MQ Resource Adapter on server1

it does the job perfectly: -

Using IBM HTTP Server and the WebSphere Plugin to load-balance workload across a non-federated WebSphere Application Server environment

This time around, I have a requirement to deploy IBM HTTP Server (IHS) and the WebSphere Plugin to route traffic to WebSphere Application Server (WAS).

So far, so good.

However, the difference is that, this time, I'm NOT leveraging the power of WAS Network Deployment ( WAS ND ). There are no clusters here.

Equally, the two instances of WAS are completely self-contained.

For my proof of concept, I've got a single set of WAS binaries ( I am using WAS 8.5.5.3 ) with a pair of standard profiles, AppSrv01 and AppSrv02. Similarly, I only have a single instance of IHS/Plugin.

However, in the real world, I'd expect the WAS boxes to be separated from one another, perhaps on different boxes, perhaps in different data centres, definitely on different OS hosts ( for resilience ).

So, to recap, I have the following: -

1x installation of IBM HTTP Server Installed into /opt/IBM/HTTPServer

1x installation of WebSphere Plugin Installed into /opt/IBM/WebSphere/Plugins

1x installation of WebSphere Application Server Installed into /opt/IBM/WebSphere/AppServer

So that's one set of binaries for each of the three products.

As I've only got a single instance of IHS / Plugin, I have a single set of configuration artefacts: -

IHS Located in /opt/IBM/HTTPServer/conf/httpd.conf

Plugin Located in /opt/IBM/WebSphere/Plugins/config/plugin-cfg.xml

As I have two instances of WAS, I have two sets of configuration artefacts - in the context of WAS, this is two discrete WAS profiles: -

AppSrv01 Located in /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/

AppSrv02 Located in /opt/IBM/WebSphere/AppServer/profiles/AppSrv02/

These were created as follows: -

/opt/IBM/WebSphere/AppServer/bin/manageprofiles.sh -create -hostName localhost -applyPerfTuningSetting standard -profileName AppSrv01 -adminUserName wasadmin -adminPassword passw0rd -enableAdminSecurity true -nodeName localhostNode01 -cellName localhostNode01Cell -serverName server1 -profilePath /opt/IBM/WebSphere/AppServer/profiles/AppSrv01 -templatePath /opt/IBM/WebSphere/AppServer/profileTemplates/default

/opt/IBM/WebSphere/AppServer/bin/manageprofiles.sh -create -hostName localhost -applyPerfTuningSetting standard -profileName AppSrv02 -adminUserName wasadmin -adminPassword passw0rd -enableAdminSecurity true -nodeName localhostNode01 -cellName localhostNode01Cell -serverName server1 -profilePath /opt/IBM/WebSphere/AppServer/profiles/AppSrv02 -templatePath /opt/IBM/WebSphere/AppServer/profileTemplates/default

This gives me two WAS cells, each containing a single node, and a single server (JVM/instance).

Out-of-the-box, WAS gives me a number of sample applications, one of which, Snoop, is absolutely perfect for testing.

Snoop can be accessed directly from WAS as follows: -

AppSrv01 http://rhel65.uk.ibm.com:9080/servlet/SnoopServlet

AppSrv02 http://rhel65.uk.ibm.com:9081/servlet/SnoopServlet

Note that, in my case, the manageprofiles tool has automatically incremented the port number.

If I had truly located WAS on separate physical/virtual servers, then the port numbers would likely be identical.

Having setup IHS to listen on port 8080: -

Listen 8080
ServerName localhost:8080

and 8443: -

LoadModule ibm_ssl_module modules/mod_ibm_ssl.so
Listen 8443
<VirtualHost *:8443>
SSLEnable
</VirtualHost>
KeyFile /opt/IBM/HTTPServer/ssl/keystore.kdb
SSLDisable

I then set up IHS to use the WAS Plugin: -

LoadModule was_ap22_module "/opt/IBM/WebSphere/Plugins/bin/64bits/mod_was_ap22_http.so"
WebSpherePluginConfig /opt/IBM/WebSphere/Plugins/config/webserver1/plugin-cfg.xml

So far, so good.

However, this assumes that there is only one WAS plugin i.e. IHS is configured to use a single plugin configuration file e.g. plugin-cfg.xml.

But we have TWO disparate WAS cells, each with its own set of ports, applications etc. AND we may well  have a future requirement to load-balance workload in an unique way.

As an example, if we had two off-host WAS servers, each on its own server, but with one having twice as much CPU capacity as the other, we may well want to change the load-balanging algorithm to route 1/3 of the requests to the smaller box and 2/3 to the larger box.

Therefore, we need TWO copies of the WAS Plugin configuration.

Back in the "old" days, it was necessary to manually merge the plugin configuration files together.

Thankfully, WAS 7 introduced us to the pluginMerge.sh tool here: -

/opt/IBM/WebSphere/AppServer/bin/pluginMerge.sh

executed as follows: -

/opt/IBM/WebSphere/AppServer/bin/pluginMerge.sh -l /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/plugin-cfg.xml /opt/IBM/WebSphere/AppServer/profiles/AppSrv02/config/cells/plugin-cfg.xml /opt/IBM/WebSphere/Plugins/config/webserver1/plugin-cfg.xml

This takes the two plugin configuration files from AppSrv01 and AppSrv02, and merges them together in the location within which IHS will then retrieve the single combined file.

For the record, there's another similarly named tool: -

/opt/IBM/WebSphere/AppServer/bin/pluginCfgMerge.sh

which I have yet to try.

This would have worked a treat .....

BUT .....

Having started everything up, when I attempted to access Snoop from IHS: -

http://rhel65.uk.ibm.com:8080/servlet/SnoopServlet

I saw this: -

Not Found

The requested URL /servlet/SnoopServlet was not found on this server.

IBM_HTTP_Server at rhel65.uk.ibm.com Port 8080

Working on the assumption that this was an issue with the Plugin > WAS interaction, rather than IHS > Plugin, I enabled debugging in the plugin configuration file: -

vi /opt/IBM/WebSphere/Plugins/config/webserver1/plugin-cfg.xml

...

    <Log LogLevel="Debug" Name="/opt/IBM/WebSphere/Plugins/logs/http_plugin.log"/>
...

and restarted IHS.

This time around, I saw this: -

[09/Dec/2014:20:04:33.12358] 0000db7e f2593700 - DEBUG: mod_was_ap22_http: as_child_init pid= 0000DB7E
[09/Dec/2014:20:04:36.93650] 0000db7e ece4b700 - DEBUG: lib_util: parseHostHeader: Host: 'rhel65.uk.ibm.com', port 8080
[09/Dec/2014:20:04:36.93659] 0000db7e ece4b700 - DEBUG: ws_common: websphereCheckConfig: Current time is 1418155476, next stat time is 1418155533
[09/Dec/2014:20:04:36.93661] 0000db7e ece4b700 - DETAIL: ws_common: websphereShouldHandleRequest: trying to match a route for: vhost='rhel65.uk.ibm.com'; uri='/servlet/SnoopServlet'
[09/Dec/2014:20:04:36.93663] 0000db7e ece4b700 - DEBUG: ws_common: websphereShouldHandleRequest: NOT config->odrEnabled(reqInfo(d40072a8))
[09/Dec/2014:20:04:36.93666] 0000db7e ece4b700 - DETAIL: ws_common: websphereShouldHandleRequest: No route found

in the log file.

Now I have seen that before.

The problem is that WAS does not "know" about the port 8080 upon which IHS is listening, and therefore will not accept an incoming request from the web server.

This is achieved by the use of the WAS Virtual Host.

Therefore, I needed to "tell" WAS about ports 8080 and 8443 ( the two ports upon which IHS listens ), as follows: -

/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/bin/wsadmin.sh -lang jython 
cellID=AdminControl.getCell() 

AdminConfig.create('HostAlias', AdminConfig.getid('/Cell:'+cellID+'/VirtualHost:default_host/'), '[[hostname "*"] [port "8080"]]') 
AdminConfig.create('HostAlias', AdminConfig.getid('/Cell:'+cellID+'/VirtualHost:default_host/'), '[[hostname "*"] [port "8443"]]') 
AdminConfig.save()

/opt/IBM/WebSphere/AppServer/profiles/AppSrv02/bin/wsadmin.sh -lang jython 
cellID=AdminControl.getCell() 
AdminConfig.create('HostAlias', AdminConfig.getid('/Cell:'+cellID+'/VirtualHost:default_host/'), '[[hostname "*"] [port "8080"]]') 
AdminConfig.create('HostAlias', AdminConfig.getid('/Cell:'+cellID+'/VirtualHost:default_host/'), '[[hostname "*"] [port "8443"]]') 
AdminConfig.save()

and then restart both WAS servers.

I also needed to regenerate the Plugin Configuration: -

/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/bin/GenPluginCfg.sh 
/opt/IBM/WebSphere/AppServer/profiles/AppSrv02/bin/GenPluginCfg.sh 

and then re-merge the two into one: -

/opt/IBM/WebSphere/AppServer/bin/pluginMerge.sh -l /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/plugin-cfg.xml /opt/IBM/WebSphere/AppServer/profiles/AppSrv02/config/cells/plugin-cfg.xml /opt/IBM/WebSphere/Plugins/config/webserver1/plugin-cfg.xml

Finally, I needed to restart IHS, and  ...... 

..... well, it worked happily via HTTP

I could access Snoop via this URL: -

http://rhel65.uk.ibm.com:8080/servlet/SnoopServlet

and see this: -

I could even scroll to the end of the page and see this: -

and this: -

as I reloaded the page.

In other words, I could see that IHS / Plugin were correctly load-balancing between AppSrv01 and AppSrv02.

I did have some additional work to do in the context of SSL however.

In order for the Plugin to be able to correctly connect to WAS via SSL ( port 9443 for AppSrv01 and 9444 for AppSrv02 ) I also needed to import the signer certificates for each of the two WAS cells into the Plugin's trust store.

Looking at the plugin-cfg.xml file, we can see: -

    <Property Name="Keyfile" Value="/opt/IBM/WebSphere/Plugins/etc/plugin-key.kdb"/>
    <Property Name="Stashfile" Value="/opt/IBM/WebSphere/Plugins/etc/plugin-key.sth"/>

The plugin-key.kdb file is the key/trust store for IHS, with the password for that key/trust store being "stashed" in the plugin-key.sth file.

Therefore, I needed to retrieve the SSL certificates for each of the two WAS servers, each into a file: -

openssl s_client -showcerts -connect rhel65.uk.ibm.com:9443 </dev/null  > AppSrv01.cer
openssl s_client -showcerts -connect rhel65.uk.ibm.com:9444 </dev/null  > AppSrv02.cer

and then import each certificate into the .kdb file: -

/opt/IBM/HTTPServer/bin/gskcapicmd -cert -add -db /opt/IBM/WebSphere/Plugins/etc/plugin-key.kdb -pw passw0rd -file AppSrv01.cer 
/opt/IBM/HTTPServer/bin/gskcapicmd -cert -add -db /opt/IBM/WebSphere/Plugins/etc/plugin-key.kdb -pw passw0rd -file AppSrv02.cer 

and then validate: -

/opt/IBM/HTTPServer/bin/gskcapicmd -cert -list -db /opt/IBM/WebSphere/Plugins/etc/plugin-key.kdb -pw passw0rd

which returns: -

....

Certificates found
* default, - personal, ! trusted, # secret key....

! CN=localhost,OU=localhostNode01Cell,OU=localhostNode01,O=IBM,C=US
! "CN=localhost,OU=Root Certificate,OU=localhostNode01Cell,OU=localhostNode01,O=IBM,C=US"
! CN=localhost,OU=localhostNode02Cell,OU=localhostNode02,O=IBM,C=US
! "CN=localhost,OU=Root Certificate,OU=localhostNode02Cell,OU=localhostNode02,O=IBM,C=US"
....

Once I again restarted IHS, I was then able to access Snoop via HTTPS: -

https://rhel65.uk.ibm.com:8443/servlet/SnoopServlet

And that's it, that's all it took.

It was an absolute learning curve, and I thoroughly enjoyed it.

Final point, I did all of this on my own test environment, and I have NOT applied any good practice around security, hardening etc.

For WAS hardening etc. please look at this excellent developerWorks series: -

Advanced security hardening in WebSphere Application Server V7, V8 and V8.5, Part 1: Overview and approach to security hardening

Advanced security hardening in WebSphere Application Server V7, V8 and V8.5, Part 2: Advanced security considerations