A Portal to a Portal: Using IBM HTTP Server and the WebSphere Plugin to load-balance workload across a non-federated WebSphere Application Server environment

This time around, I have a requirement to deploy IBM HTTP Server (IHS) and the WebSphere Plugin to route traffic to WebSphere Application Server (WAS).

So far, so good.

However, the difference is that, this time, I'm NOT leveraging the power of WAS Network Deployment ( WAS ND ). There are no clusters here.

Equally, the two instances of WAS are completely self-contained.

For my proof of concept, I've got a single set of WAS binaries ( I am using WAS 8.5.5.3 ) with a pair of standard profiles, AppSrv01 and AppSrv02. Similarly, I only have a single instance of IHS/Plugin.

However, in the real world, I'd expect the WAS boxes to be separated from one another, perhaps on different boxes, perhaps in different data centres, definitely on different OS hosts ( for resilience ).

So, to recap, I have the following: -

1x installation of IBM HTTP Server Installed into /opt/IBM/HTTPServer

1x installation of WebSphere Plugin Installed into /opt/IBM/WebSphere/Plugins

1x installation of WebSphere Application Server Installed into /opt/IBM/WebSphere/AppServer

So that's one set of binaries for each of the three products.

As I've only got a single instance of IHS / Plugin, I have a single set of configuration artefacts: -

IHS Located in /opt/IBM/HTTPServer/conf/httpd.conf

Plugin Located in /opt/IBM/WebSphere/Plugins/config/plugin-cfg.xml

As I have two instances of WAS, I have two sets of configuration artefacts - in the context of WAS, this is two discrete WAS profiles: -

AppSrv01 Located in /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/

AppSrv02 Located in /opt/IBM/WebSphere/AppServer/profiles/AppSrv02/

These were created as follows: -

/opt/IBM/WebSphere/AppServer/bin/manageprofiles.sh -create -hostName localhost -applyPerfTuningSetting standard -profileName AppSrv01 -adminUserName wasadmin -adminPassword passw0rd -enableAdminSecurity true -nodeName localhostNode01 -cellName localhostNode01Cell -serverName server1 -profilePath /opt/IBM/WebSphere/AppServer/profiles/AppSrv01 -templatePath /opt/IBM/WebSphere/AppServer/profileTemplates/default

/opt/IBM/WebSphere/AppServer/bin/manageprofiles.sh -create -hostName localhost -applyPerfTuningSetting standard -profileName AppSrv02 -adminUserName wasadmin -adminPassword passw0rd -enableAdminSecurity true -nodeName localhostNode01 -cellName localhostNode01Cell -serverName server1 -profilePath /opt/IBM/WebSphere/AppServer/profiles/AppSrv02 -templatePath /opt/IBM/WebSphere/AppServer/profileTemplates/default

This gives me two WAS cells, each containing a single node, and a single server (JVM/instance).

Out-of-the-box, WAS gives me a number of sample applications, one of which, Snoop, is absolutely perfect for testing.

Snoop can be accessed directly from WAS as follows: -

AppSrv01 http://rhel65.uk.ibm.com:9080/servlet/SnoopServlet

AppSrv02 http://rhel65.uk.ibm.com:9081/servlet/SnoopServlet

Note that, in my case, the manageprofiles tool has automatically incremented the port number.

If I had truly located WAS on separate physical/virtual servers, then the port numbers would likely be identical.

Having setup IHS to listen on port 8080: -

Listen 8080
ServerName localhost:8080

and 8443: -

LoadModule ibm_ssl_module modules/mod_ibm_ssl.so
Listen 8443
<VirtualHost *:8443>
SSLEnable
</VirtualHost>
KeyFile /opt/IBM/HTTPServer/ssl/keystore.kdb
SSLDisable

I then set up IHS to use the WAS Plugin: -

LoadModule was_ap22_module "/opt/IBM/WebSphere/Plugins/bin/64bits/mod_was_ap22_http.so"
WebSpherePluginConfig /opt/IBM/WebSphere/Plugins/config/webserver1/plugin-cfg.xml

So far, so good.

However, this assumes that there is only one WAS plugin i.e. IHS is configured to use a single plugin configuration file e.g. plugin-cfg.xml.

But we have TWO disparate WAS cells, each with its own set of ports, applications etc. AND we may well have a future requirement to load-balance workload in an unique way.

As an example, if we had two off-host WAS servers, each on its own server, but with one having twice as much CPU capacity as the other, we may well want to change the load-balanging algorithm to route 1/3 of the requests to the smaller box and 2/3 to the larger box.

Therefore, we need TWO copies of the WAS Plugin configuration.

Back in the "old" days, it was necessary to manually merge the plugin configuration files together.

Thankfully, WAS 7 introduced us to the pluginMerge.sh tool here: -

/opt/IBM/WebSphere/AppServer/bin/pluginMerge.sh

executed as follows: -

/opt/IBM/WebSphere/AppServer/bin/pluginMerge.sh -l /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/plugin-cfg.xml /opt/IBM/WebSphere/AppServer/profiles/AppSrv02/config/cells/plugin-cfg.xml /opt/IBM/WebSphere/Plugins/config/webserver1/plugin-cfg.xml

This takes the two plugin configuration files from AppSrv01 and AppSrv02, and merges them together in the location within which IHS will then retrieve the single combined file.

For the record, there's another similarly named tool: -

/opt/IBM/WebSphere/AppServer/bin/pluginCfgMerge.sh

which I have yet to try.

This would have worked a treat .....

BUT .....

Having started everything up, when I attempted to access Snoop from IHS: -

http://rhel65.uk.ibm.com:8080/servlet/SnoopServlet

I saw this: -

Not Found

The requested URL /servlet/SnoopServlet was not found on this server.

IBM_HTTP_Server at rhel65.uk.ibm.com Port 8080

Working on the assumption that this was an issue with the Plugin > WAS interaction, rather than IHS > Plugin, I enabled debugging in the plugin configuration file: -

vi /opt/IBM/WebSphere/Plugins/config/webserver1/plugin-cfg.xml

...

<Log LogLevel="Debug" Name="/opt/IBM/WebSphere/Plugins/logs/http_plugin.log"/>
...

and restarted IHS.

This time around, I saw this: -

[09/Dec/2014:20:04:33.12358] 0000db7e f2593700 - DEBUG: mod_was_ap22_http: as_child_init pid= 0000DB7E
[09/Dec/2014:20:04:36.93650] 0000db7e ece4b700 - DEBUG: lib_util: parseHostHeader: Host: 'rhel65.uk.ibm.com', port 8080
[09/Dec/2014:20:04:36.93659] 0000db7e ece4b700 - DEBUG: ws_common: websphereCheckConfig: Current time is 1418155476, next stat time is 1418155533
[09/Dec/2014:20:04:36.93661] 0000db7e ece4b700 - DETAIL: ws_common: websphereShouldHandleRequest: trying to match a route for: vhost='rhel65.uk.ibm.com'; uri='/servlet/SnoopServlet'
[09/Dec/2014:20:04:36.93663] 0000db7e ece4b700 - DEBUG: ws_common: websphereShouldHandleRequest: NOT config->odrEnabled(reqInfo(d40072a8))
[09/Dec/2014:20:04:36.93666] 0000db7e ece4b700 - DETAIL: ws_common: websphereShouldHandleRequest: No route found

in the log file.

Now I have seen that before.

The problem is that WAS does not "know" about the port 8080 upon which IHS is listening, and therefore will not accept an incoming request from the web server.

This is achieved by the use of the WAS Virtual Host.

Therefore, I needed to "tell" WAS about ports 8080 and 8443 ( the two ports upon which IHS listens ), as follows: -

/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/bin/wsadmin.sh -lang jython
cellID=AdminControl.getCell()

AdminConfig.create('HostAlias', AdminConfig.getid('/Cell:'+cellID+'/VirtualHost:default_host/'), '[[hostname "*"] [port "8080"]]')
AdminConfig.create('HostAlias', AdminConfig.getid('/Cell:'+cellID+'/VirtualHost:default_host/'), '[[hostname "*"] [port "8443"]]')
AdminConfig.save()

/opt/IBM/WebSphere/AppServer/profiles/AppSrv02/bin/wsadmin.sh -lang jython
cellID=AdminControl.getCell()
AdminConfig.create('HostAlias', AdminConfig.getid('/Cell:'+cellID+'/VirtualHost:default_host/'), '[[hostname "*"] [port "8080"]]')
AdminConfig.create('HostAlias', AdminConfig.getid('/Cell:'+cellID+'/VirtualHost:default_host/'), '[[hostname "*"] [port "8443"]]')
AdminConfig.save()

and then restart both WAS servers.

I also needed to regenerate the Plugin Configuration: -

/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/bin/GenPluginCfg.sh
/opt/IBM/WebSphere/AppServer/profiles/AppSrv02/bin/GenPluginCfg.sh

and then re-merge the two into one: -

Finally, I needed to restart IHS, and ......

..... well, it worked happily via HTTP

I could access Snoop via this URL: -

http://rhel65.uk.ibm.com:8080/servlet/SnoopServlet

and see this: -

I could even scroll to the end of the page and see this: -

and this: -

as I reloaded the page.

In other words, I could see that IHS / Plugin were correctly load-balancing between AppSrv01 and AppSrv02.

I did have some additional work to do in the context of SSL however.

In order for the Plugin to be able to correctly connect to WAS via SSL ( port 9443 for AppSrv01 and 9444 for AppSrv02 ) I also needed to import the signer certificates for each of the two WAS cells into the Plugin's trust store.

Looking at the plugin-cfg.xml file, we can see: -

The plugin-key.kdb file is the key/trust store for IHS, with the password for that key/trust store being "stashed" in the plugin-key.sth file.

Therefore, I needed to retrieve the SSL certificates for each of the two WAS servers, each into a file: -

openssl s_client -showcerts -connect rhel65.uk.ibm.com:9443 </dev/null > AppSrv01.cer
openssl s_client -showcerts -connect rhel65.uk.ibm.com:9444 </dev/null > AppSrv02.cer

and then import each certificate into the .kdb file: -

/opt/IBM/HTTPServer/bin/gskcapicmd -cert -add -db /opt/IBM/WebSphere/Plugins/etc/plugin-key.kdb -pw passw0rd -file AppSrv01.cer
/opt/IBM/HTTPServer/bin/gskcapicmd -cert -add -db /opt/IBM/WebSphere/Plugins/etc/plugin-key.kdb -pw passw0rd -file AppSrv02.cer

and then validate: -

/opt/IBM/HTTPServer/bin/gskcapicmd -cert -list -db /opt/IBM/WebSphere/Plugins/etc/plugin-key.kdb -pw passw0rd

which returns: -

....

Certificates found
* default, - personal, ! trusted, # secret key....

! CN=localhost,OU=localhostNode01Cell,OU=localhostNode01,O=IBM,C=US
! "CN=localhost,OU=Root Certificate,OU=localhostNode01Cell,OU=localhostNode01,O=IBM,C=US"
! CN=localhost,OU=localhostNode02Cell,OU=localhostNode02,O=IBM,C=US
! "CN=localhost,OU=Root Certificate,OU=localhostNode02Cell,OU=localhostNode02,O=IBM,C=US"
....

Once I again restarted IHS, I was then able to access Snoop via HTTPS: -

https://rhel65.uk.ibm.com:8443/servlet/SnoopServlet

And that's it, that's all it took.

It was an absolute learning curve, and I thoroughly enjoyed it.

Final point, I did all of this on my own test environment, and I have NOT applied any good practice around security, hardening etc.

For WAS hardening etc. please look at this excellent developerWorks series: -

Advanced security hardening in WebSphere Application Server V7, V8 and V8.5, Part 1: Overview and approach to security hardening

Advanced security hardening in WebSphere Application Server V7, V8 and V8.5, Part 2: Advanced security considerations