Friday, 24 June 2016

F5 BIG-IP - More learning, more tinkering, more blogging

I'm continuing to learn more about the F5, in my quest to really understand how it all works, in the context of solving a tricksy little SSL handshake problem between an LTM and an IBM HTTP Server box.

Here's a few commands that I've used: -

tmsh show /ltm node

Ltm::Node: (
  Availability   : unknown
  State          : enabled
  Reason         : Node address does not have service checking enabled
  Monitor        : none
  Monitor Status : unchecked
  Session Status : enabled
Traffic                ServerSide  General
  Bits In                       0        -
  Bits Out                      0        -
  Packets In                    0        -
  Packets Out                   0        -
  Current Connections           0        -
  Maximum Connections           0        -
  Total Connections             0        -
  Total Requests                -        0
  Current Sessions              -        0

tmsh show ltm pool all members field-fmt |grep "ltm\ pool\|active-member-cnt\|addr\|monitor-status"

ltm pool {
    active-member-cnt 1
            monitor-status up

Plus this: -

Wednesday, 22 June 2016

WebSphere Plugin and the Case of the GSK_ERROR_BAD_KEYFILE_PASSWORD

We've seen a few instances , where the WebSphere Plugin fails to communicate, via SSL, with WAS.

This manifests itself as Error 500 / HTTP500 when accessing WAS via IHS, using hostname OR service name.

Long story short, it looks like the Plugin SSL configuration files are getting "borked" by something.

The plugin log shows this: -

[22/Jun/2016:13:14:40.16292] 00d3008c 00000001 - ERROR: lib_security: logSSLError: str_security (gsk error 408):  GSK_ERROR_BAD_KEYFILE_PASSWORD
[22/Jun/2016:13:14:40.16294] 00d3008c 00000001 - ERROR: lib_security: initializeSecurity: Failed to initialize GSK environment. Secure transports are not possible.

One symptom is that you cannot query the keystore using the gskcapicmd command, as per this: -

/opt/ibm/HTTPServer/bin/gskcapicmd -cert -list -db plugin-key.kdb -stashed

which returns: -

CTGSK3026W The key file "plugin-key.kdb" does not exist or cannot be read.
CTGSK2016W An invalid database password was encountered.

The same error occurs if you use the correct keystore password e.g. WebAS.

We saw this problem even when we deleted the .KDB and .STH files ( see below ), and propagated them from the WAS cell via the Deployment Manager.

The problem appears to be related to the use of the gskcapicmd command to create certificate requests.

I *think* that, as some point, someone has created certificate requests for the Plugin, which has updated one or more of the related configuration files.

There are a number of files in play here: -

plugin-cfg.xml                                  Configuration file, generated by WAS, and propagated from the Deployment Manager to the IHS box
plugin-key.kdb                                  CMS key database, holding BOTH personal AND signer certificates ( keys and trusts )
plugin-key.sth                                   Encrypted stashed password file
plugin-key.crl                                    Certificate Recovation List
plugin-key.rdb                                  Certificate Request Database

As a test, we moved the .CRL and .RDB files into a different subdirectory, leaving just the .xml, .kdb and .sth files in place.

We were then able to query the .KDB without problems: -

/opt/ibm/HTTPServer/bin/gskcapicmd -cert -list -db plugin-key.kdb –stashed

Certificates found
* default, - personal, ! trusted, # secret key
!       ", OU=RootCertificate, OU=test, OU=dmgr, O=ibm, O=co, C=uk"
-       default

/opt/ibm/HTTPServer/bin/gskcapicmd -cert -list -db plugin-key.kdb -pw WebAS

Certificates found
* default, - personal, ! trusted, # secret key
!       ", OU=RootCertificate, OU=test, OU=dmgr, O=ibm, O=co, C=uk"
-       default

I did some more testing, moving the .CRL and .RDB files back and forth, and have concluded that it IS the .RDB file that "breaks" things.

Once we ended up with JUST the .XML, .KDB and .STH files in place ( in /opt/ibm/WebSphere/Plugins/config/ ), I was able to successfully navigate to WAS via IHS.

Bottom line, there's no need to use the IHS GSK commands ( gskcapicmd ) to request certificates for the WebSphere Plugin in the context of WAS.

If we need a personal WAS certificate, we can generate the Certificate Request using Jython scripts or the ISC, and WAS will take care of updating the KDB etc.

This is different to Plugin -> IBM Integration Bus, where there's no WAS to manage things for us.

The only time we'd ever need to use gskcapicmd against the Plugin KDB was if we wanted to mark the WAS personal certificate as default, in order to ensure that it was used for Plugin -> WAS connectivity, in the context of Mutual Authentication etc ( via the –setdefault  command ).

F5 Load Balancing - My first few forays

I'm currently working on a situation whereby HTTPS load-balancing is inconsistently not working against IBM HTTP Server 8.5.5.

To help me help the client's network team debug this, I've been tinkering with a F5 Local Traffic Manager (LTM) using VMware Fusion on my Mac.

I found a slew of excellent articles on the F5 site including: -

Load balancing got its start in the form of network-based load balancing hardware. It is the essential foundation on which Application Delivery Controllers (ADCs) operate. The second iteration of purpose-built load balancing (following application-based proprietary systems) materialized in the form of network-based appliances. These are the true founding fathers of today's ADCs. Because these devices were application-neutral and resided outside of the application servers themselves, they could load balance using straightforward network techniques. In essence, these devices would present a "virtual server" address to the outside world, and when users attempted to connect, they would forward the connection to the most appropriate real server doing bi-directional network address translation (NAT).

Monitors determine the availability and performance of devices, links, and services on a network. Health monitors check the availability. Performance monitors check the performance and load. If a monitored device, link, or service does not respond within a specified timeout period, or the status indicates that performance is degraded or that the load is excessive, the BIG-IP system can redirect the traffic to another resource.

More importantly, this link: -

was just what I need to get a developer version of F5 VE installed and working.

I ended up with a working F5 Health Monitor probing my IHS server ( on a different Linux VM ), over SSL.

Two things that made a difference: -

(1) Getting the RIGHT cipher

openssl ciphers -v

DHE-RSA-AES256-SHA      SSLv3 Kx=DH       Au=RSA  Enc=AES(256)  Mac=SHA1
DHE-DSS-AES256-SHA      SSLv3 Kx=DH       Au=DSS  Enc=AES(256)  Mac=SHA1
AES256-SHA              SSLv3 Kx=RSA      Au=RSA  Enc=AES(256)  Mac=SHA1
DHE-RSA-CAMELLIA256-SHA SSLv3 Kx=DH       Au=RSA  Enc=Camellia(256) Mac=SHA1
DHE-DSS-CAMELLIA256-SHA SSLv3 Kx=DH       Au=DSS  Enc=Camellia(256) Mac=SHA1
CAMELLIA256-SHA         SSLv3 Kx=RSA      Au=RSA  Enc=Camellia(256) Mac=SHA1
EDH-RSA-DES-CBC3-SHA    SSLv3 Kx=DH       Au=RSA  Enc=3DES(168) Mac=SHA1
EDH-DSS-DES-CBC3-SHA    SSLv3 Kx=DH       Au=DSS  Enc=3DES(168) Mac=SHA1
DES-CBC3-SHA            SSLv3 Kx=RSA      Au=RSA  Enc=3DES(168) Mac=SHA1
DES-CBC3-MD5            SSLv2 Kx=RSA      Au=RSA  Enc=3DES(168) Mac=MD5 
DHE-RSA-AES128-SHA      SSLv3 Kx=DH       Au=RSA  Enc=AES(128)  Mac=SHA1
DHE-DSS-AES128-SHA      SSLv3 Kx=DH       Au=DSS  Enc=AES(128)  Mac=SHA1
AES128-SHA              SSLv3 Kx=RSA      Au=RSA  Enc=AES(128)  Mac=SHA1
DHE-RSA-CAMELLIA128-SHA SSLv3 Kx=DH       Au=RSA  Enc=Camellia(128) Mac=SHA1
DHE-DSS-CAMELLIA128-SHA SSLv3 Kx=DH       Au=DSS  Enc=Camellia(128) Mac=SHA1
CAMELLIA128-SHA         SSLv3 Kx=RSA      Au=RSA  Enc=Camellia(128) Mac=SHA1
RC2-CBC-MD5             SSLv2 Kx=RSA      Au=RSA  Enc=RC2(128)  Mac=MD5 
RC4-SHA                 SSLv3 Kx=RSA      Au=RSA  Enc=RC4(128)  Mac=SHA1
RC4-MD5                 SSLv3 Kx=RSA      Au=RSA  Enc=RC4(128)  Mac=MD5 
RC4-MD5                 SSLv2 Kx=RSA      Au=RSA  Enc=RC4(128)  Mac=MD5 
EDH-RSA-DES-CBC-SHA     SSLv3 Kx=DH       Au=RSA  Enc=DES(56)   Mac=SHA1
EDH-DSS-DES-CBC-SHA     SSLv3 Kx=DH       Au=DSS  Enc=DES(56)   Mac=SHA1
DES-CBC-SHA             SSLv3 Kx=RSA      Au=RSA  Enc=DES(56)   Mac=SHA1
DES-CBC-MD5             SSLv2 Kx=RSA      Au=RSA  Enc=DES(56)   Mac=MD5 
EXP-EDH-RSA-DES-CBC-SHA SSLv3 Kx=DH(512)  Au=RSA  Enc=DES(40)   Mac=SHA1 export
EXP-EDH-DSS-DES-CBC-SHA SSLv3 Kx=DH(512)  Au=DSS  Enc=DES(40)   Mac=SHA1 export
EXP-DES-CBC-SHA         SSLv3 Kx=RSA(512) Au=RSA  Enc=DES(40)   Mac=SHA1 export
EXP-RC2-CBC-MD5         SSLv3 Kx=RSA(512) Au=RSA  Enc=RC2(40)   Mac=MD5  export
EXP-RC2-CBC-MD5         SSLv2 Kx=RSA(512) Au=RSA  Enc=RC2(40)   Mac=MD5  export
EXP-RC4-MD5             SSLv3 Kx=RSA(512) Au=RSA  Enc=RC4(40)   Mac=MD5  export
EXP-RC4-MD5             SSLv2 Kx=RSA(512) Au=RSA  Enc=RC4(40)   Mac=MD5  export

Note that this particular version of the F5 software does NOT support my preferred ECDHE/RSA/GCM ciphers :-(

On my VM, I am running: -

BIG-IP 11.3.0 Build 39.0 VE Trial 11.3.0-HF1 (based on BIGIP 11.3.0HF6)

(2) Getting the Monitor configuration correct

Specifically the send string and the receive response are mega-important

tmsh list ltm monitor https

ltm monitor https davehttps {
    cipherlist DEFAULT:+SHA:+3DES:+kEDH
    compatibility enabled
    defaults-from https
    destination *:pcsync-https
    interval 5
    recv 200
    send "GET /index.html HTTP/1.1\\r\\nHost:\\r\\nConnection: Close\\r\\n\\r\\n"
    time-until-up 0
    timeout 16

I inferred the send string using openssl on the device itself: -

openssl s_client -connect

and pasted this string: -

GET /index.html HTTP/1.1
Connection: Close

into the terminal, and pressed [Enter].

This returned, in part: -

HTTP/1.1 200 OK
Date: Wed, 22 Jun 2016 05:26:41 GMT
Last-Modified: Tue, 06 Jan 2015 17:02:04 GMT
ETag: "da5-50bfec4265b00"
Accept-Ranges: bytes
Content-Length: 3493

which confirms the recv string of 200 ( HTTP 200 OK ).

Now my IHS server is showing regular GET requests from the F5 Monitor: - - - [22/Jun/2016:06:26:54 +0100] "GET /index.html HTTP/1.1" 200 3493 - - [22/Jun/2016:06:26:59 +0100] "GET /index.html HTTP/1.1" 200 3493 - - [22/Jun/2016:06:27:04 +0100] "GET /index.html HTTP/1.1" 200 3493 - - [22/Jun/2016:06:27:09 +0100] "GET /index.html HTTP/1.1" 200 3493 - - [22/Jun/2016:06:27:14 +0100] "GET /index.html HTTP/1.1" 200 3493 - - [22/Jun/2016:06:27:19 +0100] "GET /index.html HTTP/1.1" 200 3493 - - [22/Jun/2016:06:27:24 +0100] "GET /index.html HTTP/1.1" 200 3493 - - [22/Jun/2016:06:27:29 +0100] "GET /index.html HTTP/1.1" 200 3493 - - [22/Jun/2016:06:27:34 +0100] "GET /index.html HTTP/1.1" 200 3493 - - [22/Jun/2016:06:27:39 +0100] "GET /index.html HTTP/1.1" 200 3493 - - [22/Jun/2016:06:27:44 +0100] "GET /index.html HTTP/1.1" 200 3493 - - [22/Jun/2016:06:27:49 +0100] "GET /index.html HTTP/1.1" 200 3493 - - [22/Jun/2016:06:27:54 +0100] "GET /index.html HTTP/1.1" 200 3493


in the access.log.

Now I need to go and configure the F5 "front door" to allow me to actual send traffic to/through the load-balancer to the downstream IHS box.

These links were also of use: -

Monday, 20 June 2016

IBM BPM - Process Center and Unit Test Environment Together

This article from 2014: -

A stand-alone Process Center profile for IBM® Business Process Manager (BPM) is useful for situations where memory and disk space are limited. Rather than install the Process Center as a network deployment environment with at least three profiles running, plus an additional profile to support the unit test environment server, this article describes how a single profile can provide both a Process Center server and a unit test environment server. This content is part of the IBM Business Process Management Journal.

has been updated with a second article: -

A stand-alone Process Center profile for IBM® Business Process Manager (BPM) is useful in situations where memory and disk space are limited. In Part 1, you learned how to create a stand-alone Process Center Profile for Windows environments, and now in Part 2, learn the specific steps for a Linux environment. Rather than install the Process Center as a network deployment environment with at least three profiles running, plus an additional profile to support the unit test environment server, this series describes how a single profile can provide both a Process Center server and a unit test environment server.

Definitely worth a bookmark 

Wednesday, 15 June 2016

Achieve your API strategy with IBM API Connect

This from a former colleague of mine, Carlo Marcoli: -

To thrive in the API economy, you need to strategize your API approach and create, run, manage, and secure your APIs. With this dedicated focus on APIs, your company can share data and services in an easy-to-consume format. It can also create an ecosystem of partners and third parties that is much greater than the ecosystem you reach by using traditional channels. An effective API strategy treats an API as a business product with a well identified lifecycle, target market, and metrics for return on investment (ROI).

Wednesday, 8 June 2016

Webcast - Using IBM UrbanCode with IBM WebSphere to Accelerate Business Transformations

As found on Twitter today: -

Great news for WebSphere Application Server Administrators, IT Managers, Directors and anyone with challenges associated with deploying applications and configurations to WebSphere Application Server in its traditional version, Liberty or as a Service. IBM UrbanCode Deploy has WAS covered! IBM UrbanCode Deploy can reduce overall cycle times and accelerate time-to-test and time-to-market for clients. In this call, attendees will learn how to accelerate WAS deployments to cloud, optimize deployments everywhere, "keep the configuration in Synch", and how to take a manually configured environment (i.e. cell), capture all the configuration and easily reuse it. UrbanCode Deploy can even help you migrate from older to newer versions of WAS in a fraction of the time with UCD automation. Come learn from our experts how this combination is helping customers every day.

Announced - IBM WebSphere Application Server V9.0

Saw this today: -

WebSphere® Application Server V9.0, with its traditional and Liberty run times, continues to offer industry-leading, production-ready, standards-based Java™ EE 7 compliant architecture.

Highlights of Version 9.0 include:

• Certification to the Java EE 7 Web Profile and Java EE 7 Full Platform for WebSphere Application Server traditional, which provides assurance that applications leverage standards-compliant programming models. WebSphere Liberty was certified to Java EE 7 Web Profile and Full Platform in June, 2015.
• Ease of connecting existing on-premises applications with Bluemix® services, which include IBM Watson™ cognitive for optimal business outcomes.
• Enhanced support for creating, documenting, and discovering APIs, and also integrating with API platforms, such as IBM API Connect™.
• Significant improvements in software delivery lifecycle times through seamless integration into DevOps workflows. This enables continuous delivery and removes cross-team dependencies for deployment.
• Accelerated pace of development and deployment by taking advantage of container technology that includes IBM® Container Services and Docker container with support for Docker Data Center. This enables seamless deployment of WebSphere applications to best meet business needs.
• Increased flexibility to deploy WebSphere Application Server workloads with speed and agility on Pivotal Cloud Foundry, Amazon Web Services, Microsoft™ Azure, and Openshift, in addition to IBM Bluemix.
• New WebSphere Application Server on Bluemix, single-tenant offering, which provides an option for deploying WebSphere Application Server applications on an isolated, single-tenant hardware.
• New option to enable VMware customers to quickly provision new or scale existing workloads to the IBM Cloud. This enables clients who start locally and scale globally with cloud capabilities to scale more easily and also comply with data residency and other regulatory mandates.
• Updated WebSphere Extreme Scale that provides ease-of-use enhancements for caching to help improve response times for the most demanding applications and dramatically speed configurations.
• Use of Liberty App Accelerator to provide a quick start development of Java microservices. Spring Boot, Watson™ services, and other technologies can be used with Liberty App Accelerator to easily deploy projects to Bluemix.
• Best practices for creating new Java microservices by using Game On!, an exemplar application, which helps explore microservices concepts.
WebSphere Application Server V9.0 continues to offer the leading, open-standards-based application foundation for traditional workloads and also modern applications that tend to be delivered as services. It enables accelerated delivery of innovative applications with unmatched operational efficiency, reliability, administration, security, and control.

Planned availability date

June 24, 2016