Monday, 22 December 2014

IBM UrbanCode Deploy - Working with Apache Tomcat SSL Key Store

One of my friends asked me how one can add SSL certificates to the Apache Tomcat SSL trust store underlying the IBM UrbanCode Deploy automation solution.

In this scenario, he needed to retrieve a certificate from IBM Rational Asset Manager (IRAM) into the UCD key store, in order that a UCD process can access IRAM.

I've done this for IBM HTTP Server and IBM WebSphere Application Server in the past, using the IBM Global Security Toolkit ( GSK ), but Tomcat uses something slightly different.


and this is what I did: -

List Current Certificates in Key Store

/opt/IBM/Java/jre/bin/keytool -list -keystore /opt/ibm-ucd/server/opt/tomcat/conf/tomcat.keystore -storepass changeit 

Keystore type: jks
Keystore provider: IBMJCE

Your keystore contains 2 entries

server, 14-Dec-2014, keyEntry,
Certificate fingerprint (SHA1): 65:22:8A:B7:B8:EA:53:36:0D:75:E9:74:DF:20:90:DB:BB:C1:AC:4A


Get IRAM Certificate

openssl s_client -showcerts -connect ucd61.uk.ibm.com:9443 </dev/null  > ~/iram.cer

depth=1 C = US, O = IBM, OU = ucd61Node01, OU = ucd61Node01Cell, OU = Root Certificate, CN = ucd61.uk.ibm.com
verify error:num=19:self signed certificate in certificate chain
verify return:0
DONE


( In my case, I'm using WAS 8.5.5 on port 9443 in lieu of IRAM )

Note, I needed to manually edit the retrieved certificate to reduce superfluous tags, possible because the WAS certificate is self-signed e.g.: -

-----BEGIN CERTIFICATE-----
MIIDyDCCArCgAwIBAgIGH9jdPEHdMA0GCSqGSIb3DQEBBQUAMIGBMQswCQYDVQQG
EwJVUzEMMAoGA1UEChMDSUJNMRQwEgYDVQQLEwt1Y2Q2MU5vZGUwMTEYMBYGA1UE
CxMPdWNkNjFOb2RlMDFDZWxsMRkwFwYDVQQLExBSb290IENlcnRpZmljYXRlMRkw
FwYDVQQDExB1Y2Q2MS51ay5pYm0uY29tMB4XDTE0MTIxMTIxMjgyMloXDTE1MTIx
MTIxMjgyMlowZjELMAkGA1UEBhMCVVMxDDAKBgNVBAoTA0lCTTEUMBIGA1UECxML
dWNkNjFOb2RlMDExGDAWBgNVBAsTD3VjZDYxTm9kZTAxQ2VsbDEZMBcGA1UEAxMQ
dWNkNjEudWsuaWJtLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AKlFEK54+pN68kp2V1//jU7hiYDjoOvRzECnn6D4CNWAuQ0VOtzaRbOmUmxmPyIb
44kPmxRz34yQj5mjA/0cNGS176EDeGikBlFvMZapJyf2wgEissuliDuaHrBRWrRe
0SWu3rk0vEPZjBAbl7V26vUOMDpFHoHbh3/Um7Qx7Ur9PdV3Qd/9ANhOeyw1G59D
acIhDKuLt3cHWTqLWbozOp0y3Z90sK8dwcOj+z0CH7e4/PHvO1mwlE5I8z9k378e
1FYwtuo/FD+NQMVEpZImTJMrFZVNPGIVp1EOTBC/XMp/cuvVAqcZMoG59bjpjyRd
mWH42L05CH/X1sNw3CI5bWMCAwEAAaNgMF4wSQYDVR0RBEIwQIE+UHJvZmlsZVVV
SUQ6QXBwU3J2MDEtQkFTRS1lMzAzNjNkZi01Y2I1LTQ2MmEtYmM0ZC02Yjg3NTA5
YzRiNTQwEQYDVR0OBAoECEyME/33gIvbMA0GCSqGSIb3DQEBBQUAA4IBAQBeWvNX
g1gOmJLkqIR9agSOxiTvWKAv+nzRIb9sQ9cQX/B0ucFx+Xh58tSO+bfJFDI42QKU
aiUXwtq8pAf+RWvf3kHagpN4GZFKByhp19lBwoceG/3gk/5IGALT+9useJU3N2+T
hNcqJERyAuCH5oBR+goml28yGjWOUAf/6fWf08heGEnfbLMl2S5IiVAoyaRY+byg
KITMlxj9x3YHxME8g7/hL0RyYfRe/Etb0AV4YL2YkAvaPcjFwjOVR35Bv1C1S3zP
q3FJaizXK2M8Albz8UJxnHj+Li3Qnps4wxLci/n40olEbLH6+Lfd1KOc1Nq6817M
6R4QGcCNIClZdIg8
-----END CERTIFICATE-----


otherwise, I end up with: -

keytool error: java.lang.Exception: Input not an X.509 certificate
Add IRAM Certificate to Key Store

/opt/IBM/Java/jre/bin/keytool -importcert -alias iram -file ~/iram.cer -keystore /opt/ibm-ucd/server/opt/tomcat/conf/tomcat.keystore -storepass changeit

Owner: CN=ucd61.uk.ibm.com, OU=ucd61Node01Cell, OU=ucd61Node01, O=IBM, C=US
Issuer: CN=ucd61.uk.ibm.com, OU=Root Certificate, OU=ucd61Node01Cell, OU=ucd61Node01, O=IBM, C=US
Serial number: 1fd8dd3c41dd
Valid from: 11/12/14 21:28 until: 11/12/15 21:28
Certificate fingerprints:
 MD5:  0F:E7:18:C1:69:1B:ED:FC:47:D7:B7:25:7A:5F:E5:8B
 SHA1: 7B:27:67:B7:DC:12:02:15:0C:90:2F:71:7D:F8:CB:59:5F:3D:34:72
 SHA256: 4F:F0:ED:7B:BA:E1:74:2A:20:E2:ED:B6:E8:6B:50:DD:6E:37:3B:0D:19:DB:8B:3C:A4:71:A6:69:44:56:FD:2C
 Signature algorithm name: SHA1withRSA
 Version: 3

Extensions: 

#1: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
[RFC822Name: ProfileUUID:AppSrv01-BASE-e30363df-5cb5-462a-bc4d-6b87509c4b54]]

#2: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 4c 8c 13 fd f7 80 8b db                           L.......
]
]

Trust this certificate? [no]:  
y
Certificate was added to keystore


List Current Certificates in Key Store

/opt/IBM/Java/jre/bin/keytool -list -keystore /opt/ibm-ucd/server/opt/tomcat/conf/tomcat.keystore -storepass changeit 

Keystore type: jks
Keystore provider: IBMJCE

Your keystore contains 2 entries

iram, 22-Dec-2014, trustedCertEntry,
Certificate fingerprint (SHA1): 7B:27:67:B7:DC:12:02:15:0C:90:2F:71:7D:F8:CB:59:5F:3D:34:72
server, 14-Dec-2014, keyEntry,
Certificate fingerprint (SHA1): 65:22:8A:B7:B8:EA:53:36:0D:75:E9:74:DF:20:90:DB:BB:C1:AC:4A


What's next ?

Yes, time to change the default password for the Tomcat key store .....


Thursday, 18 December 2014

IBM BPM - JavaScript - To Infinity and Beyond

This popped up in my Twitter feed today: -

I suspect that a JavaScript activity within my BPD application runs in an infinite loop. How can I detect such a loop?


Definitely worth a read .....

Monday, 15 December 2014

IBM UrbanCode Deploy - I remember my first time


This is the first of many posts covering a relatively recent acquisition, IBM UrbanCode Deploy (UCD), which is, amongst other things, an automated deployment and delivery solution.

The reason for my interest is that I'm working with a few clients who are considering the use of UCD for automation, in the context of IBM Business Process Manager, IBM Operational Decision Manager, IBM Integration Bus etc.

To start with, I'm going to outline my experiences thus far with the build of a UCD environment, installed on a virtualised Red Hat Enterprise Linux environment, running on my Mac.

Unlike other products, UCD doesn't use IBM Installation Manager or WebSphere Application Server, although it DOES require a relational database. I'm going to use DB2 10.1, which is located on the same VM.

I began by downloading UCD 6.1.1, which is the latest release - as of a few days ago: -


Specifically, I downloaded a 2 GB (!) ISO image: -

IBM UrbanCode Deploy 6.1.1 Multiplatform Multilingual (CN2BAML )

which resulted in this file: -

-rw-r--r--@ 1 hayd  staff   2.0G 12 Dec 12:23 IBM_URBANCODE_DEPLOY_6.1.1_MULTIP.iso

Being an ISO file, I was able to mount this in VMware Fusion as /media: -

ls -alh /media/

total 430M
dr-xr-xr-x.  1 root     root  2.0K Dec  2 13:28 .
dr-xr-xr-x. 25 root     root  4.0K Dec 12 12:59 ..
-rwxr-xr-x.  1 db2inst1 games 430M Dec  1 22:54 ibm-ucd-6110608828.zip
dr-xr-xr-x.  1 root     root  2.0K Dec  2 13:28 java
dr-xr-xr-x.  1 root     root  2.0K Dec  2 13:28 z-os

including: -

ls -alh /media/java/

total 16K
dr-xr-xr-x. 1 root root 2.0K Dec  2 13:28 .
dr-xr-xr-x. 1 root root 2.0K Dec  2 13:28 ..
dr-xr-xr-x. 1 root root 2.0K Aug 21  2013 aix
dr-xr-xr-x. 1 root root 2.0K Aug 21  2013 hpux
dr-xr-xr-x. 1 root root 2.0K Aug 21  2013 linux
dr-xr-xr-x. 1 root root 2.0K Aug 21  2013 solaris
dr-xr-xr-x. 1 root root 2.0K Aug 21  2013 windows
dr-xr-xr-x. 1 root root 2.0K Dec  3 15:10 z-os


ls -alh /media/java/linux/

total 12K
dr-xr-xr-x. 1 root root 2.0K Aug 21  2013 .
dr-xr-xr-x. 1 root root 2.0K Dec  2 13:28 ..
dr-xr-xr-x. 1 root root 2.0K Aug 21  2013 s390
dr-xr-xr-x. 1 root root 2.0K Aug 21  2013 s390_64
dr-xr-xr-x. 1 root root 2.0K Aug 21  2013 x32
dr-xr-xr-x. 1 root root 2.0K Aug 21  2013 x64

ls -alh /media/java/linux/x64/

total 85M
dr-xr-xr-x. 1 root     root  2.0K Aug 21  2013 .
dr-xr-xr-x. 1 root     root  2.0K Aug 21  2013 ..
-rwxr-xr-x. 1 db2inst1 games  85M Aug 21  2013 ibm-java-jre-70-50-linux-x8.gz

I started by extracting the JRE: -

mkdir /opt/IBM/Java
tar xvzf /media/java/linux/x64/ibm-java-jre-70-50-linux-x8.gz -C /tmp/

mv /tmp/ibm-java-x86_64-70/* /opt/IBM/Java/

and testing it: -

/opt/IBM/Java/jre/bin/java -version

java version "1.7.0"
Java(TM) SE Runtime Environment (build pxa6470sr5-20130619_01(SR5))
IBM J9 VM (build 2.6, JRE 1.7.0 Linux amd64-64 Compressed References 20130617_152572 (JIT enabled, AOT enabled)
J9VM - R26_Java726_SR5_20130617_1436_B152572
JIT  - r11.b04_20130528_38954ifx1
GC   - R26_Java726_SR5_20130617_1436_B152572_CMPRSS
J9CL - 20130617_152572)
JCL - 20130616_01 based on Oracle 7u25-b12


Before installing UCD, I created the required DB2 database: -

db2 create database UCD61 automatic storage yes using codeset UTF-8 territory GB pagesize 32768

as db2inst1 and granted permission for the db2user1 ID: -

db2 connect to ucd61
db2 grant dbadm on database to user db2user1

I unpacked UCD: -

unzip /media/ibm-ucd-6110608828.zip -d /tmp

copied the required JDBC drivers into the temporary directory: -

cp /opt/ibm/db2/V10.1/java/db2jcc* /tmp/ibm-ucd-install/lib/ext/

and installed it: -

cd /tmp/ibm-ucd-install/
./install-server.sh 

...
     [echo] Do you accept the license? [y,n] Y
...
     [echo] Installing IBM UrbanCode Deploy version 6.1.1.0.608828
     [echo] Enter the directory where the IBM UrbanCode Deploy should be installed. [Default: /opt/ibm-ucd/server]
...
     [echo] The specified directory does not exist. Do you want to create it? Y,n [Default: Y]
...
     [echo] Installing IBM UrbanCode Deploy to: /opt/ibm-ucd/server
     [echo] Please enter the home directory of the JRE/JDK used to run the server. [Default: /opt/IBM/Java/jre]

...
     [echo] JVM Version detected: 1.7.0
     [echo] JAVA_HOME: /opt/IBM/Java/jre
     [echo] What host name will users access the Web UI at? [Default: ucd61.uk.ibm.com]

...
     [echo] Do you want the Web UI to always use secure connections using SSL? Y,n [Default: Y]
...
     [echo] Enter the port on which the Web UI should listen for secure HTTPS requests. [Default: 8443]
...
     [echo] Enter the port on which the Web UI should redirect unsecured HTTP requests from. [Default: 8080]
...
     [echo] Enter the initial password for the admin user.
...
     [echo] Please type password again.
...
     [echo] Enter the port to use for agent communication. [Default: 7918]
...
     [echo] Do you want the Server and Agent communication to require mutual authentication?  This requires a manual key exchange between the server and each agent. See the documentation for more details. y,N [Default: N]
....
     [echo] Enter the port and hostname of a Rational License Key Server containing product licenses for IBM UrbanCode Deploy, in the form of port@hostname. (e.g. 27000@licenses.example.com) Alternatively, you may leave this blank to begin a 60-day evaluation period. [Default: none]
....
     [echo] Create database schema? Y,n [Default: Y]
...
     [echo] Enter the database type to use. [Default: derby] db2
...
     [echo] Enter the database driver. [Default: com.ibm.db2.jcc.DB2Driver]
...
     [echo] Enter the database connection string. Eg. jdbc:db2://localhost:50000/ibm_ucd jdbc:db2://ucd61.uk.ibm.com:60008/UCD61
...
     [echo] Enter the database username. [Default: ibm_ucd] db2user1
...
     [echo] Enter the database password. [Default: password]
...
     [echo] After starting the server, you may access the web UI by pointing your web-browser at
     [echo] https://ucd61.uk.ibm.com:8443 to complete the Installation.
     [echo] Installer Complete. (press return to exit installer)
...
BUILD SUCCESSFUL
Total time: 2 minutes 24 seconds


I started the server: -

and, ~30 seconds later, checked that it was up-and-running: -

cat /opt/ibm-ucd/server/var/log/deployserver.out 

2014-12-14 11:13:16,619 WARN  main com.urbancode.ds.UDeployServer - Property encryption.keystore.password not set. Using value from property server.keystore.password
2014-12-14 11:13:17,050 INFO  main com.urbancode.ds.UDeployServer - Configuring Agent Network System for single-server setup...
2014-12-14 11:13:19,559 INFO  main com.urbancode.ds.UDeployServer -  done
2014-12-14 11:13:19,703 INFO  main com.urbancode.ds.UDeployServer - IBM UrbanCode Deploy server started.


netstat -aon | grep 8443

tcp        0      0 :::8443                     :::*                        LISTEN      off (0.00/0/0)

db2 list tables for schema db2user1

...
SEC_DB_VERSION                  DB2USER1        T     2014-12-14-11.11.11.872799
SEC_GROUP                       DB2USER1        T     2014-12-14-11.11.12.207976
SEC_GROUP_MAPPER                DB2USER1        T     2014-12-14-11.11.12.271739
SEC_GROUP_MAPPING               DB2USER1        T     2014-12-14-11.11.12.338698
SEC_GROUP_ROLE_ON_TEAM          DB2USER1        T     2014-12-14-11.11.12.399760
...


and, most importantly, tested the server: -



and logged in ( note to self, the user ID is, by default, admin although that's NOT obvious ): -


I guessed the user ID, but could have pulled it from the database: -

db2 "select name from db2user1.sec_user"

NAME                                                                                                                                                                                                                                                          
admin                                                                                                                                                                                                                                                         

  1 record(s) selected.

Note this message: -

There is no agent or tag configured to import new component versions, so no new versions will be imported. Please set this on the Settings > System Settings page.

occurs because I haven't yet installed any UCD plugins.

These can be downloaded via the Tools menu: -




Having downloaded the IBM UrbanCode Deploy Agent, I installed it via Resources > Agents > Install New Agent 


Note that I needed to override the Java Home Path ( following on from the Java installation earlier ).

Now the agent is installed, it's ready to be provisioned: -


This is achieved via the Settings > System Settings page: -


We now have a working agent: -


and we're ready to go ......

Friday, 12 December 2014

IBM UrbanCode Deploy - Continuous application delivery to WebSphere Application Server

I'm starting to "play" with UrbanCode Deploy (UCD), in the initial context of continuous application delivery to WebSphere Application Server, moving to the "full fat" model of CAD to IBM BPM and IBM IIB.

This looks to be a useful read: -


This is even more useful: -



PS UCD 6.1.1 was released YESTERDAY - December 11 - guess what I downloaded this morning ?

So I've got UCD installed on a RHEL 6.3 VM, with IBM Java, DB2 10.1 and WAS 8.5.5.

I'll write up the HOW in a few days once I've learned a bit more .....

Journeys in Python - Day 471 - Setting up WebSphere MQ Messaging Providers

So I've previously written about the fun I had setting up WebSphere Application Server to pull messages from a WebSphere MQ Cluster here: -


This time around, I wanted to streamline my code somewhat, specifically in terms of setting the Native Library Path, as my code wasn't generic enough last time: -

...
Update WAS MQ Provider to support local bindings ( need to add native path )

AdminTask.manageWMQ('"WebSphere MQ Resource Adapter(cells/bpm85Cell1/nodes/AppSrv01Node/servers/foobar|resources.xml#J2CResourceAdapter_1416556034607)"', '[-nativePath /opt/mqm/java/lib64/ -disableWMQ false ]')
AdminConfig.save()
AdminNodeManagement.syncActiveNodes()
....

In other words, my code would only work for THAT particular Resource Adapter.

Borrowing from the nice Mr Steven Robinson Esq; -


I produced a small Jython script that does the job: -

servers = AdminUtilities.convertToList(AdminTask.listServers('[-serverType APPLICATION_SERVER ]'))
ras = AdminUtilities.convertToList(AdminConfig.list('J2CResourceAdapter'))

for serverName in servers :
        name = AdminConfig.showAttribute(serverName, "name")
        if name == "server1" :
print name
                for ra in ras :
                        if ra.find(name) > 0:
                                desc = AdminConfig.showAttribute(ra, "description")
                         if desc.find("WebSphere MQ") > 0:
print "Setting native path for " + desc + " on " + name
AdminTask.manageWMQ(ra, '[-nativePath /opt/ibm/mqm/usr/mqm/java/lib64 -disableWMQ false ]')
AdminConfig.save()

Obviously I could parameterise it, rather than requiring the server name to be hard-coded, but that's a nice-to-have.

:-)

If I run it: -

/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/bin/wsadmin.sh -lang jython -user wasadmin -password passw0rd -f foobar.jy 

WASX7209I: Connected to process "server1" on node localhostNode01 using SOAP connector;  The type of process is: UnManagedProcess
server1
Setting native path for WAS Built In WebSphere MQ Resource Adapter on server1

it does the job perfectly: -

Tuesday, 9 December 2014

Using IBM HTTP Server and the WebSphere Plugin to load-balance workload across a non-federated WebSphere Application Server environment

This time around, I have a requirement to deploy IBM HTTP Server (IHS) and the WebSphere Plugin to route traffic to WebSphere Application Server (WAS).

So far, so good.

However, the difference is that, this time, I'm NOT leveraging the power of WAS Network Deployment ( WAS ND ). There are no clusters here.

Equally, the two instances of WAS are completely self-contained.

For my proof of concept, I've got a single set of WAS binaries ( I am using WAS 8.5.5.3 ) with a pair of standard profiles, AppSrv01 and AppSrv02. Similarly, I only have a single instance of IHS/Plugin.

However, in the real world, I'd expect the WAS boxes to be separated from one another, perhaps on different boxes, perhaps in different data centres, definitely on different OS hosts ( for resilience ).

So, to recap, I have the following: -

1x installation of IBM HTTP Server Installed into /opt/IBM/HTTPServer
1x installation of WebSphere Plugin Installed into /opt/IBM/WebSphere/Plugins
1x installation of WebSphere Application Server Installed into /opt/IBM/WebSphere/AppServer

So that's one set of binaries for each of the three products.

As I've only got a single instance of IHS / Plugin, I have a single set of configuration artefacts: -

IHS Located in /opt/IBM/HTTPServer/conf/httpd.conf
Plugin Located in /opt/IBM/WebSphere/Plugins/config/plugin-cfg.xml

As I have two instances of WAS, I have two sets of configuration artefacts - in the context of WAS, this is two discrete WAS profiles: -

AppSrv01 Located in /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/
AppSrv02 Located in /opt/IBM/WebSphere/AppServer/profiles/AppSrv02/

These were created as follows: -

/opt/IBM/WebSphere/AppServer/bin/manageprofiles.sh -create -hostName localhost -applyPerfTuningSetting standard -profileName AppSrv01 -adminUserName wasadmin -adminPassword passw0rd -enableAdminSecurity true -nodeName localhostNode01 -cellName localhostNode01Cell -serverName server1 -profilePath /opt/IBM/WebSphere/AppServer/profiles/AppSrv01 -templatePath /opt/IBM/WebSphere/AppServer/profileTemplates/default
/opt/IBM/WebSphere/AppServer/bin/manageprofiles.sh -create -hostName localhost -applyPerfTuningSetting standard -profileName AppSrv02 -adminUserName wasadmin -adminPassword passw0rd -enableAdminSecurity true -nodeName localhostNode01 -cellName localhostNode01Cell -serverName server1 -profilePath /opt/IBM/WebSphere/AppServer/profiles/AppSrv02 -templatePath /opt/IBM/WebSphere/AppServer/profileTemplates/default
This gives me two WAS cells, each containing a single node, and a single server (JVM/instance).

Out-of-the-box, WAS gives me a number of sample applications, one of which, Snoop, is absolutely perfect for testing.

Snoop can be accessed directly from WAS as follows: -

Note that, in my case, the manageprofiles tool has automatically incremented the port number.

If I had truly located WAS on separate physical/virtual servers, then the port numbers would likely be identical.

Having setup IHS to listen on port 8080: -

Listen 8080
ServerName localhost:8080


and 8443: -

LoadModule ibm_ssl_module modules/mod_ibm_ssl.so
Listen 8443
<VirtualHost *:8443>
SSLEnable
</VirtualHost>
KeyFile /opt/IBM/HTTPServer/ssl/keystore.kdb
SSLDisable

I then set up IHS to use the WAS Plugin: -

LoadModule was_ap22_module "/opt/IBM/WebSphere/Plugins/bin/64bits/mod_was_ap22_http.so"
WebSpherePluginConfig /opt/IBM/WebSphere/Plugins/config/webserver1/plugin-cfg.xml

So far, so good.

However, this assumes that there is only one WAS plugin i.e. IHS is configured to use a single plugin configuration file e.g. plugin-cfg.xml.

But we have TWO disparate WAS cells, each with its own set of ports, applications etc. AND we may well  have a future requirement to load-balance workload in an unique way.

As an example, if we had two off-host WAS servers, each on its own server, but with one having twice as much CPU capacity as the other, we may well want to change the load-balanging algorithm to route 1/3 of the requests to the smaller box and 2/3 to the larger box.

Therefore, we need TWO copies of the WAS Plugin configuration.

Back in the "old" days, it was necessary to manually merge the plugin configuration files together.

Thankfully, WAS 7 introduced us to the pluginMerge.sh tool here: -

/opt/IBM/WebSphere/AppServer/bin/pluginMerge.sh

executed as follows: -

/opt/IBM/WebSphere/AppServer/bin/pluginMerge.sh -l /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/plugin-cfg.xml /opt/IBM/WebSphere/AppServer/profiles/AppSrv02/config/cells/plugin-cfg.xml /opt/IBM/WebSphere/Plugins/config/webserver1/plugin-cfg.xml

This takes the two plugin configuration files from AppSrv01 and AppSrv02, and merges them together in the location within which IHS will then retrieve the single combined file.

For the record, there's another similarly named tool: -

/opt/IBM/WebSphere/AppServer/bin/pluginCfgMerge.sh

which I have yet to try.

This would have worked a treat .....

BUT .....

Having started everything up, when I attempted to access Snoop from IHS: -


I saw this: -

Not Found

The requested URL /servlet/SnoopServlet was not found on this server.

IBM_HTTP_Server at rhel65.uk.ibm.com Port 8080

Working on the assumption that this was an issue with the Plugin > WAS interaction, rather than IHS > Plugin, I enabled debugging in the plugin configuration file: -

vi /opt/IBM/WebSphere/Plugins/config/webserver1/plugin-cfg.xml

...
    <Log LogLevel="Debug" Name="/opt/IBM/WebSphere/Plugins/logs/http_plugin.log"/>
...

and restarted IHS.

This time around, I saw this: -

[09/Dec/2014:20:04:33.12358] 0000db7e f2593700 - DEBUG: mod_was_ap22_http: as_child_init pid= 0000DB7E
[09/Dec/2014:20:04:36.93650] 0000db7e ece4b700 - DEBUG: lib_util: parseHostHeader: Host: 'rhel65.uk.ibm.com', port 8080
[09/Dec/2014:20:04:36.93659] 0000db7e ece4b700 - DEBUG: ws_common: websphereCheckConfig: Current time is 1418155476, next stat time is 1418155533
[09/Dec/2014:20:04:36.93661] 0000db7e ece4b700 - DETAIL: ws_common: websphereShouldHandleRequest: trying to match a route for: vhost='rhel65.uk.ibm.com'; uri='/servlet/SnoopServlet'
[09/Dec/2014:20:04:36.93663] 0000db7e ece4b700 - DEBUG: ws_common: websphereShouldHandleRequest: NOT config->odrEnabled(reqInfo(d40072a8))
[09/Dec/2014:20:04:36.93666] 0000db7e ece4b700 - DETAIL: ws_common: websphereShouldHandleRequest: No route found


in the log file.

Now I have seen that before.

The problem is that WAS does not "know" about the port 8080 upon which IHS is listening, and therefore will not accept an incoming request from the web server.

This is achieved by the use of the WAS Virtual Host.

Therefore, I needed to "tell" WAS about ports 8080 and 8443 ( the two ports upon which IHS listens ), as follows: -

/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/bin/wsadmin.sh -lang jython 
cellID=AdminControl.getCell() 
AdminConfig.create('HostAlias', AdminConfig.getid('/Cell:'+cellID+'/VirtualHost:default_host/'), '[[hostname "*"] [port "8080"]]') 
AdminConfig.create('HostAlias', AdminConfig.getid('/Cell:'+cellID+'/VirtualHost:default_host/'), '[[hostname "*"] [port "8443"]]') 
AdminConfig.save()


/opt/IBM/WebSphere/AppServer/profiles/AppSrv02/bin/wsadmin.sh -lang jython 
cellID=AdminControl.getCell() 
AdminConfig.create('HostAlias', AdminConfig.getid('/Cell:'+cellID+'/VirtualHost:default_host/'), '[[hostname "*"] [port "8080"]]') 
AdminConfig.create('HostAlias', AdminConfig.getid('/Cell:'+cellID+'/VirtualHost:default_host/'), '[[hostname "*"] [port "8443"]]') 
AdminConfig.save()

and then restart both WAS servers.

I also needed to regenerate the Plugin Configuration: -

/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/bin/GenPluginCfg.sh 
/opt/IBM/WebSphere/AppServer/profiles/AppSrv02/bin/GenPluginCfg.sh 


and then re-merge the two into one: -

/opt/IBM/WebSphere/AppServer/bin/pluginMerge.sh -l /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/plugin-cfg.xml /opt/IBM/WebSphere/AppServer/profiles/AppSrv02/config/cells/plugin-cfg.xml /opt/IBM/WebSphere/Plugins/config/webserver1/plugin-cfg.xml

Finally, I needed to restart IHS, and  ...... 

..... well, it worked happily via HTTP

I could access Snoop via this URL: -


and see this: -


I could even scroll to the end of the page and see this: -


and this: -


as I reloaded the page.

In other words, I could see that IHS / Plugin were correctly load-balancing between AppSrv01 and AppSrv02.

I did have some additional work to do in the context of SSL however.

In order for the Plugin to be able to correctly connect to WAS via SSL ( port 9443 for AppSrv01 and 9444 for AppSrv02 ) I also needed to import the signer certificates for each of the two WAS cells into the Plugin's trust store.

Looking at the plugin-cfg.xml file, we can see: -

    <Property Name="Keyfile" Value="/opt/IBM/WebSphere/Plugins/etc/plugin-key.kdb"/>
    <Property Name="Stashfile" Value="/opt/IBM/WebSphere/Plugins/etc/plugin-key.sth"/>

The plugin-key.kdb file is the key/trust store for IHS, with the password for that key/trust store being "stashed" in the plugin-key.sth file.

Therefore, I needed to retrieve the SSL certificates for each of the two WAS servers, each into a file: -

openssl s_client -showcerts -connect rhel65.uk.ibm.com:9443 </dev/null  > AppSrv01.cer
openssl s_client -showcerts -connect rhel65.uk.ibm.com:9444 </dev/null  > AppSrv02.cer

and then import each certificate into the .kdb file: -

/opt/IBM/HTTPServer/bin/gskcapicmd -cert -add -db /opt/IBM/WebSphere/Plugins/etc/plugin-key.kdb -pw passw0rd -file AppSrv01.cer 
/opt/IBM/HTTPServer/bin/gskcapicmd -cert -add -db /opt/IBM/WebSphere/Plugins/etc/plugin-key.kdb -pw passw0rd -file AppSrv02.cer 


and then validate: -

/opt/IBM/HTTPServer/bin/gskcapicmd -cert -list -db /opt/IBM/WebSphere/Plugins/etc/plugin-key.kdb -pw passw0rd

which returns: -

....
Certificates found
* default, - personal, ! trusted, # secret key
....
! CN=localhost,OU=localhostNode01Cell,OU=localhostNode01,O=IBM,C=US
! "CN=localhost,OU=Root Certificate,OU=localhostNode01Cell,OU=localhostNode01,O=IBM,C=US"
! CN=localhost,OU=localhostNode02Cell,OU=localhostNode02,O=IBM,C=US
! "CN=localhost,OU=Root Certificate,OU=localhostNode02Cell,OU=localhostNode02,O=IBM,C=US"

....

Once I again restarted IHS, I was then able to access Snoop via HTTPS: -


And that's it, that's all it took.

It was an absolute learning curve, and I thoroughly enjoyed it.

Final point, I did all of this on my own test environment, and I have NOT applied any good practice around security, hardening etc.

For WAS hardening etc. please look at this excellent developerWorks series: -




Saturday, 6 December 2014

Top ten best practices for WebSphere administrators

This from developerWorks: -

The following list consists of the top ten best practices compiled from numerous common problems that I have seen throughout my technical support years. Most of these issues could easily be prevented by taking these simple precaution steps.  So, if you're a WebSphere administrator, this is your must-read!


PS In my own bashful way, I'm now proud to report that those nice people at developerWorks have allowed me to post to the AIM Support Blog as well : -