IBM HTTP Server and the Strange Story of Line Numbers

One of my colleagues saw an interesting issue with IBM HTTP Server today.

She's using IHS 8.5.5.0 on Linux.

In essence, whilst she could connect to IHS on port 8080 ( non-SSL port ), she couldn't connect on port 8443 ( SSL port ).

There were no obvious exceptions in the IHS error log, so I asked her to enable SSL tracing ( via the SSLTrace directive in httpd.conf ).

This she did, and this is what we saw in the logs: -

[Thu Nov 12 20:17:34 2015] [debug] ssl_getpwd() entry
[Thu Nov 12 20:17:34 2015] [debug] ssl_getpwd processing :0, ssl flag [0] prompt flag [0]
[Thu Nov 12 20:17:34 2015] [info] mod_unique_id: using ip addr 192.168.1.80
[Thu Nov 12 20:17:35 2015] [debug] SSL initialization for server: webserver.uk.ibm.com, port: 8080
[Thu Nov 12 20:17:35 2015] [notice] Using GSKit version 8.0.14.9
[Thu Nov 12 20:17:35 2015] [debug] SSL initialization for server: webserver.uk.ibm.com, port: 8080
[Thu Nov 12 20:17:35 2015] [info] mod_unique_id: using ip addr 192.168.1.80
[Thu Nov 12 20:17:36 2015] [error] ws_config_parser: handleConfigStart: Unknown property StrictSecurity
[Thu Nov 12 20:17:36 2015] [error] ws_config_parser: handleConfigStart: Unknown property MarkBusyDown
[Thu Nov 12 20:17:36 2015] [error] ws_config_parser: handleConfigStart: Unknown property KillWebServerStartUpOnParseErr
[Thu Nov 12 20:17:36 2015] [error] ws_config_parser: handleConfigStart: Unknown property IISDisableFlushFlag
[Thu Nov 12 20:17:36 2015] [notice] PLUGIN: mod_was_ap22_http: apache_log_header: WebSphere Plugins loaded.
[Thu Nov 12 20:17:36 2015] [notice] PLUGIN: mod_was_ap22_http: apache_log_header: --------------------Plugin Information-----------------------
[Thu Nov 12 20:17:36 2015] [notice] PLUGIN: mod_was_ap22_http: apache_log_header: Bld version: 8.5.0
[Thu Nov 12 20:17:36 2015] [notice] PLUGIN: mod_was_ap22_http: apache_log_header: Bld date: Apr 24 2012, 15:17:46
[Thu Nov 12 20:17:36 2015] [notice] PLUGIN: mod_was_ap22_http: apache_log_header: Webserver: IBM_HTTP_Server
[Thu Nov 12 20:17:36 2015] [notice] PLUGIN: mod_was_ap22_http: apache_log_header: --------------------------------------------------------------
[Thu Nov 12 20:17:36 2015] [notice] Using config file /opt/ibm/HTTPServer/conf/httpd.conf 
[Thu Nov 12 20:17:36 2015] [notice] IBM_HTTP_Server/8.5.0.0 (Unix) configured -- resuming normal operations
[Thu Nov 12 20:17:36 2015] [info] Server built: Mar  7 2012 18:25:14
[Thu Nov 12 20:17:36 2015] [debug] worker.c(1859): AcceptMutex: sysvsem (default: sysvsem)
[Thu Nov 12 20:17:36 2015] [notice] Core file limit is 0; core dumps will be not be written for server crashes
[Thu Nov 12 20:17:36 2015] [debug] mod_mpmstats.c(211): mpmstats daemon started (pid 6494)

which was somewhat strange.

This suggested that IHS was trying to bring up SSL on port 8080 rather than 8443.

Also, I noticed that there was almost no SSL debug appearing in the log, implying that SSL was NOT coming up properly / at all.

This is what she had in httpd.conf: -

Listen 8080
ServerName webserver.uk.ibm.com:8080
LoadModule ibm_ssl_module modules/mod_ibm_ssl.so
Listen 8443
<VirtualHost *:8443>
SSLProtocolDisable SSLv2 SSLv3
SSLEnable
</VirtualHost>
KeyFile /opt/IBM/HttpServer/ssl/keystore.kdb
SSLDisable
SSLTrace

which looks OK although I wouldn't normally expect to see the Listen and ServerName directives immediately next to the SSL configuration block.

Note the inclusion of SSLTrace at my request.

I asked her to enable line number in vi in order to see where precisely the SSL configuration block occurs: -

106 #
107 # Listen: Allows you to bind the web server to specific IP addresses
108 # and/or ports, in addition to the default. See also the <VirtualHost>
109 # directive.
110 #
111 # Change this to Listen on specific IP addresses as shown below to
112 # prevent the web server from accepting connections on all interfaces
113 # (0.0.0.0)
114 #
115 # Change this to "Listen 0.0.0.0:port" to restrict the server to
116 # IPv4.
117 #
118 #Listen 12.34.56.78:80
119 Listen 8080
120 ServerName webserver.uk.ibm.com:8080
121 LoadModule ibm_ssl_module modules/mod_ibm_ssl.so
122 Listen 8443
123 <VirtualHost *:8443>
124 SSLProtocolDisable SSLv2 SSLv3
125 SSLEnable
126 </VirtualHost>
127 KeyFile /opt/IBM/HttpServer/ssl/keystore.kdb
128 SSLDisable
129 SSLTrace

I then compared this to a vanilla IHS configuration: -

106 #
107 # Listen: Allows you to bind the web server to specific IP addresses 
108 # and/or ports, in addition to the default. See also the <VirtualHost>
109 # directive.
110 #
111 # Change this to Listen on specific IP addresses as shown below to 
112 # prevent the web server from accepting connections on all interfaces
113 # (0.0.0.0)
114 #
115 # Change this to "Listen 0.0.0.0:port" to restrict the server to
116 # IPv4.
117 #
118 #Listen 12.34.56.78:80
119 Listen 8080

This made me think "Aha, I wonder if the various blocks need to be on specific lines?".

Again, I looked at the vanilla configuration, in order to locate the SSL block: -

848 # Example SSL configuration which supports SSLv3 and TLSv1
849 # To enable this support:
850 #   1) Create a key database with ikeyman
851 #   2) Update the KeyFile directive below to point to that key database
852 #   3) Uncomment the directives up through the end of the example
853 #
854 #LoadModule ibm_ssl_module modules/mod_ibm_ssl.so
855 #Listen 443
856 #<VirtualHost *:443>
857 #SSLEnable
858 #</VirtualHost>
859 #KeyFile /opt/ibm/HTTPServer/ihsserverkey.kdb
860 #SSLDisable
861 # End of example SSL configuration

I don't know for sure, but I'm guessing that's the root cause.

I had her move the Listen and ServerName directives further up the file, and restart ....

As expected, this did the trick, and IHS came up properly on both port 8080 ( HTTP ) and 8443 ( HTTPS ).

Equally importantly, the debug appeared as expected: -

...

[Thu Nov 12 20:26:19 2015] [info] mod_unique_id: using ip addr 192.168.1.80
[Thu Nov 12 20:26:20 2015] [debug] SSL initialization for server: webserver.uk.ibm.com, port: 8080
[Thu Nov 12 20:26:20 2015] [debug] SSL initialization for server: webserver.uk.ibm.com, port: 8443
[Thu Nov 12 20:26:20 2015] [debug] mod_ibm_ssl.c(1604): Accelerator device has not been enabled (0)
[Thu Nov 12 20:26:20 2015] [debug] SSL support provided for SSLV2 cipher: SSL_RC4_128_WITH_MD5(21)
[Thu Nov 12 20:26:20 2015] [debug] SSL support provided for SSLV2 cipher: SSL_RC4_128_EXPORT40_WITH_MD5(22)
[Thu Nov 12 20:26:20 2015] [debug] SSL support provided for SSLV2 cipher: SSL_RC2_CBC_128_CBC_WITH_MD5(23)
[Thu Nov 12 20:26:20 2015] [debug] SSL support provided for SSLV2 cipher: SSL_RC2_CBC_128_CBC_EXPORT40_WITH_MD5(24)
[Thu Nov 12 20:26:20 2015] [debug] SSL support provided for SSLV2 cipher: SSL_DES_64_CBC_WITH_MD5(26)
[Thu Nov 12 20:26:20 2015] [debug] SSL support provided for SSLV2 cipher: SSL_DES_192_EDE3_CBC_WITH_MD5(27)
[Thu Nov 12 20:26:20 2015] [debug] SSL support NOT provided for SSLV2 cipher: TLS_RSA_WITH_AES_128_CBC_SHA(2F)
[Thu Nov 12 20:26:20 2015] [debug] SSL support NOT provided for SSLV2 cipher: TLS_RSA_WITH_NULL_NULL(30)
[Thu Nov 12 20:26:20 2015] [debug] SSL support NOT provided for SSLV2 cipher: TLS_RSA_WITH_NULL_MD5(31)
[Thu Nov 12 20:26:20 2015] [debug] SSL support NOT provided for SSLV2 cipher: TLS_RSA_WITH_NULL_SHA(32)
[Thu Nov 12 20:26:20 2015] [debug] SSL support NOT provided for SSLV2 cipher: TLS_RSA_EXPORT_WITH_RC4_40_MD5(33)
[Thu Nov 12 20:26:20 2015] [debug] SSL support NOT provided for SSLV2 cipher: TLS_RSA_WITH_RC4_128_MD5(34)
[Thu Nov 12 20:26:20 2015] [debug] SSL support NOT provided for SSLV2 cipher: TLS_RSA_WITH_RC4_128_SHA(35)
[Thu Nov 12 20:26:20 2015] [debug] SSL support NOT provided for SSLV2 cipher: TLS_RSA_WITH_AES_256_CBC_SHA(35b)
[Thu Nov 12 20:26:20 2015] [debug] SSL support NOT provided for SSLV2 cipher: TLS_RSA_EXPORT_WITH_RC2_CBC_40_MD5(36)
[Thu Nov 12 20:26:20 2015] [debug] SSL support NOT provided for SSLV2 cipher: TLS_RSA_WITH_DES_CBC_SHA(39)
...

Having recreated the problem on my own VM, I confirmed that the problem is definitely related to positioning.

Once I moved the SSL block down to line ~848, it just worked.

I need to think about WHY, but the moral of the story is the positioning of Listen, ServerName and the SSL block is obviously more important than I had realised ...