Securing the Database - IBM DB2 10.5 and Transport Layer Security 1.2

This builds upon a series of earlier posts, including: -

WebSphere Application Server 8.5.5.5, TLS 1.2 and DB2

IBM HTTP Server, Transport Layer Security and Google Chrome

Continuing to learn - IBM BPM and IBM Business Monitor to DB2 via SSL/TLS

This week, my colleague, JohnR, and I have been endeavouring to understand more about the way that one can use Transport Layer Security (TLS) 1.2 in the context of IBM DB2, with specific regard to Java client connectivity.

To that end, I've set up my DB2 10.5.0.5 server to only accept incoming connections on a specific port ( 60007 ) via TLS 1.2 using a very specific cipher specification.

This is what I have: -

db2 get dbm config | grep SSL

...
 SSL server keydb file                   (SSL_SVR_KEYDB) = /home/db2inst1/keystore.kdb
 SSL server stash file                   (SSL_SVR_STASH) = /home/db2inst1/keystore.sth
 SSL server certificate label            (SSL_SVR_LABEL) = bam856.uk.ibm.com
 SSL service name                         (SSL_SVCENAME) = db2c_ssl
 SSL cipher specs                      (SSL_CIPHERSPECS) = TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
 SSL versions                             (SSL_VERSIONS) = TLSV12
 SSL client keydb file                  (SSL_CLNT_KEYDB) = 
 SSL client stash file                  (SSL_CLNT_STASH) = 
...

cat /etc/services

...

DB2_db2inst1 60000/tcp
DB2_db2inst1_1 60001/tcp
DB2_db2inst1_2 60002/tcp
DB2_db2inst1_3 60003/tcp
DB2_db2inst1_4 60004/tcp
DB2_db2inst1_END 60005/tcp
db2c_db2inst1 60006/tcp
db2c_ssl 60007/tcp

...

( I've highlighted the most specific aspects above )

This essentially means that DB2 will: -

(a) Listen on port 60007

(b) Only accept incoming connections that use TLS 1.2

(c) Only accept incoming connections that support the TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 cipher specification

(d) Present a signer certificate with the alias bam856.uk.ibm.com

(e) Use a local keystore - /home/db2inst1/keystore.kdb  - to store the signer certificate

(f) Use a local stashed password file - /home/db2inst1/keystore.sth

Having set all of this up, we were somewhat surprised to find that a standard Java class wouldn't connect, instead returning handshake_failure exceptions such: -

com.ibm.db2.jcc.am.DisconnectNonTransientConnectionException: [jcc][t4][2030][11211][3.69.24] A communication error occurred during operations on the connection's underlying socket, socket input stream, 
or socket output stream.  Error location: Reply.fill() - socketInputStream.read (-1).  Message: Received fatal alert: handshake_failure. ERRORCODE=-4499, SQLSTATE=08001

This proved to be for a number of reasons.

These are three of them: -

(1) It's important to use the "right" Java Runtime Environment, as TLS 1.2 support was added relatively recently - we experimented with various versions of Java 7 and Java 8, with varying degrees of success

(2) The AES 256 ciphers require the JRE to be augmented with Unrestricted SDK JCE policy files - this is definitely true for the IBM JRE, and may also be true for Oracle

(3) Not all ciphers work with all JREs - John and I had varying results

In addition, DB2 has a quirk in that the Cipher Specification(s) that are configured in the instance Database Manager Configuration are prefixed with tls_ whereas Java appears to want to prefix them with ssl_ 

As an example, here's an excerpt from the IBM SDK 7.1: -

...

Default enabled cipher suites in order of preference:

• SSL_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384

• SSL_ECDHE_RSA_WITH_AES_256_CBC_SHA384

• SSL_RSA_WITH_AES_256_CBC_SHA256

• SSL_ECDH_ECDSA_WITH_AES_256_CBC_SHA384

...

whereas DB2 wants the suite specified as: -

...

• TLS_RSA_WITH_AES_256_CBC_SHA

• TLS_RSA_WITH_AES_128_CBC_SHA

• TLS_RSA_WITH_3DES_EDE_CBC_SHA

• TLS_RSA_WITH_AES_256_GCM_SHA384

• TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

• TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384

• TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384

• TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384

• TLS_RSA_WITH_AES_256_CBC_SHA256

...

as per this: -

DB2 for Linux UNIX and Windows 10.5.0>Database fundamentals>Security>DB2 security model>Data encryption>Configuring Secure Sockets Layer (SSL) support for a DB2 instance>Supported cipher suites

When I tried to persuade DB2 to use the same consistent naming convention as Java, this is what I saw: -

db2 update dbm config using SSL_CIPHERSPECS SSL_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384

SQL6112N  The configuration parameter was not updated because the resulting 
configuration parameter settings would not be valid.  Reason code "8".

So, in the context of working JREs, I did note that the Java 7 JRE that's included with DB2 10.5.0.5: -

/opt/ibm/db2/V10.5/java/jdk64/jre/bin/java -version

java version "1.7.0"
Java(TM) SE Runtime Environment (build pxa6470sr6-20131015_01(SR6))
IBM J9 VM (build 2.6, JRE 1.7.0 Linux amd64-64 Compressed References 20131013_170512 (JIT enabled, AOT enabled)
J9VM - R26_Java726_SR6_20131013_1510_B170512
JIT  - r11.b05_20131003_47443
GC   - R26_Java726_SR6_20131013_1510_B170512_CMPRSS
J9CL - 20131013_170512)
JCL - 20131011_01 based on Oracle 7u45-b18

returns the handshake_exception whereas that shipped with WebSphere Application Server 8.5.5.5: -

/opt/IBM/WebSphere/AppServer/java/jre/bin/java -version

java version "1.6.0"
Java(TM) SE Runtime Environment (build pxa6460_26sr8fp3-20141218_02(SR8 FP3))
IBM J9 VM (build 2.6, JRE 1.6.0 Linux amd64-64 Compressed References 20141211_226933 (JIT enabled, AOT enabled)
J9VM - R26_Java626_SR8_20141211_2359_B226933
JIT  - r11.b07_20141003_74578.05
GC   - R26_Java626_SR8_20141211_2359_B226933_CMPRSS
J9CL - 20141212_226933)
JCL  - 20141216_01

works perfectly.

Therefore, it's important to ensure that one uses the correct JRE and that one fully tests each required Cipher Suite.

Finally, I mentioned the need to augment the JRE with Unrestricted SDK JCE policy files - this is most clearly documented here: -

...

** Cipher suites that use AES_256 require installation of the   JCE Unlimited Strength Jurisdiction Policy Files.

...

IBM® SDK, Java™ Technology Edition, Version 7 Release 1

<CAVEAT>

As far as I'm aware, the use of the Unrestricted SDK JCE policy files is definitely something that needs to be considered on a case by case basis, as there are license agreement considerations, as described in this example: -

Enabling Strong encryption greater than 128 bit key lengths in Rational Performance Tester

which then links to this: -

Security information - Documentation, example code, and ancillary files relating to IBM's SDKs.

</CAVEAT>

In my own case, having downloaded the policy files, this gave me a ZIP file: -

-rw-r--r-- 1 db2inst1 db2iadm1 4.0K Jun  2 21:06 unrestrictedpolicyfiles.zip

This contains two files: -

-r--r--r--  1 db2inst1 db2iadm1 2253 Oct 12  2012 local_policy.jar
-r--r--r--  1 db2inst1 db2iadm1 2240 Oct 12  2012 US_export_policy.jar

which I placed here: -

/opt/IBM/WebSphere/AppServer/java_1.7_64/jre/lib/security/

having backed up and moved the original versions: -

-rwxr-xr-x.  1 wasadmin wasadmins  3890 Feb 19 17:29 blacklist
-rwxr-xr-x.  1 wasadmin wasadmins 77924 Jun  3 14:49 cacerts
-rwxr-xr-x.  1 wasadmin wasadmins  2532 Apr 16  2012 java.policy
-rwxr-xr-x.  1 wasadmin wasadmins 10560 Apr 16  2012 java.security
-rwxr-xr-x.  1 wasadmin wasadmins    98 Feb 19 17:29 javaws.policy
-r--r--r--   1 root     root       2253 Jun  3 15:18 local_policy.jar
-rwxr-xr-x.  1 wasadmin wasadmins  2640 Feb 19 17:29 local_policy.RAJ
-rwxr-xr-x.  1 wasadmin wasadmins     0 Feb 19 17:29 trusted.libraries
-r--r--r--   1 root     root       2240 Jun  3 15:18 US_export_policy.jar
-rwxr-xr-x.  1 wasadmin wasadmins  2175 Feb 19 17:29 US_export_policy.RAJ

( I have highlighted the original files which I moved to a .RAJ extension and also the new files )

With all of the above in place: -

(a) The right version of Java 7 ( in my case ) to support TLS 1.2

(b) The right TLS cipher specification(s)

(c) The unrestricted policy files - if using AES256 ciphers

This is, of course, over and above the configuration required both server-side ( DB2 ) and client-side ( Java ), in terms of hosting key stores, signer certificates etc.

This is how I extracted the signer certificate from DB2 and stored in a Java KeyStore (JKS) file for use by my Java code: -

openssl s_client -showcerts -connect localhost:60007 </dev/null | openssl x509 -outform DER > ~/db2.cer

depth=0 DC = com, DC = ibm, DC = uk, CN = bam856.uk.ibm.com
verify error:num=18:self signed certificate
verify return:1
depth=0 DC = com, DC = ibm, DC = uk, CN = bam856.uk.ibm.com
verify return:1
DONE

/opt/IBM/WebSphere/AppServer/java/jre/bin/keytool -import -file ~/db2.cer -keystore /tmp/davehay.jks -alias DB22 -storepass davehay

Owner: CN=bam856.uk.ibm.com, DC=uk, DC=ibm, DC=com
Issuer: CN=bam856.uk.ibm.com, DC=uk, DC=ibm, DC=com
Serial number: 686dcce6267d5fb4
Valid from: 28/05/15 13:54 until: 28/05/16 13:54
Certificate fingerprints:
 MD5:  55:22:9D:A3:F8:60:EA:E6:2C:4F:C9:74:59:16:7B:22
 SHA1: B9:07:FB:AC:0C:77:18:4D:B9:52:CD:71:5E:00:DB:93:F4:A9:FA:6A
Trust this certificate? [no]:  y
Certificate was added to keystore

as validated below: -

keytool -list -keystore /tmp/davehay.jks -storepass davehay

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

db2, 03-Jun-2015, trustedCertEntry, 
Certificate fingerprint (SHA1): B9:07:FB:AC:0C:77:18:4D:B9:52:CD:71:5E:00:DB:93:F4:A9:FA:6A

For the record, here's the Java class: -

import java.sql.Connection ;
import java.sql.DriverManager ;
import java.sql.ResultSet ;
import java.sql.Statement ;
import java.sql.SQLException;

import org.omg.CORBA.VersionSpecHelper;

class JdbcTestDB2
{
public static void main (String args[])
{
try
{
Class.forName("com.ibm.db2.jcc.DB2Driver");
}
catch (ClassNotFoundException e)
{
System.err.println (e) ;
System.exit (-1) ;
}
String hostname = "bam856.uk.ibm.com";
int port = 60007;
String dbName = "SAMPLE";
String userName = "db2inst1";
String password = "passw0rd";
String sslConnection = "true";

java.util.Properties properties = new java.util.Properties();
properties.put("user",userName);
properties.put("password", password);
properties.put("sslConnection", sslConnection);
properties.put("sslTrustStoreLocation","/tmp/davehay.jks");
properties.put("sslTrustStorePassword","davehay");

String url = "jdbc:db2://" + hostname + ":" + port + "/" + dbName;
try
{
Connection connection = DriverManager.getConnection(url,properties);

String query = "select EMPNO,FIRSTNME,LASTNAME from DB2INST1.EMPLOYEE" ;

Statement statement = connection.createStatement () ;
ResultSet rs = statement.executeQuery (query) ;

while ( rs.next () )
System.out.println (rs.getString (1) + " " + rs.getString(2) + " " + rs.getString(3)) ;
connection.close () ;
}
catch (java.sql.SQLException e)
{
System.err.println (e) ;
System.exit (-1) ;
}
}
}

Note that I've highlighted the variables that I'm then setting for the JDBC connection using a java.util.Properties object.

Finally, during the problem determination phase, I used the JVM command -Djavax.net.debug=ssl to get more information about the handshake_failure : -

java  -Djavax.net.debug=ssl  JdbcTestDB2

...

*** ClientHello, TLSv1
RandomCookie:  GMT: 1433356707 bytes = { 139, 106, 65, 168, 154, 169, 66, 60, 183, 34, 1, 227, 142, 239, 237, 139, 58, 162, 251, 68, 84, 56, 204, 216, 182, 194, 237, 101 }
Session ID:  {}
Cipher Suites: [TLS_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, SSL_RSA_WITH_RC4_128_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_RC4_128_MD5, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
Compression Methods:  { 0 }
***
main, WRITE: TLSv1 Handshake, length = 67
main, READ: TLSv1.2 Alert, length = 2
main, RECV TLSv1 ALERT:  fatal, handshake_failure
main, called closeSocket()
main, handling exception: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
main, called close()
main, called closeInternal(true)
main, called close()
main, called closeInternal(true)
main, called close()
main, called closeInternal(true)
com.ibm.db2.jcc.am.DisconnectNonTransientConnectionException: [jcc][t4][2030][11211][3.69.24] A communication error occurred during operations on the connection's underlying socket, socket input stream, 
or socket output stream.  Error location: Reply.fill() - socketInputStream.read (-1).  Message: Received fatal alert: handshake_failure. ERRORCODE=-4499, SQLSTATE=08001
...

Note that, whilst the ClientHello phase starts, we never see the resulting ServerHello response.

This is with the wrong JRE: -

ls -al `which java`

lrwxrwxrwx. 1 root root 22 Apr  5 21:52 /usr/bin/java -> /etc/alternatives/java

java -version

java version "1.7.0_65"
OpenJDK Runtime Environment (rhel-2.5.1.2.el6_5-x86_64 u65-b17)
OpenJDK 64-Bit Server VM (build 24.65-b04, mixed mode)

whereas this is what I see with the right JRE: -

...

*** ClientHello, TLSv1.2
RandomCookie:  GMT: 1433356928 bytes = { 209, 47, 22, 227, 221, 42, 210, 36, 159, 234, 33, 130, 46, 110, 132, 83, 32, 121, 46, 38, 107, 8, 238, 212, 19, 125, 148, 178 }
Session ID:  {}
Cipher Suites: [TLS_EMPTY_RENEGOTIATION_INFO_SCSV, SSL_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, SSL_ECDHE_RSA_WITH_AES_256_CBC_SHA384, SSL_RSA_WITH_AES_256_CBC_SHA256, SSL_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, SSL_ECDH_RSA_WITH_AES_256_CBC_SHA384, SSL_DHE_RSA_WITH_AES_256_CBC_SHA256, SSL_DHE_DSS_WITH_AES_256_CBC_SHA256, SSL_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, SSL_ECDHE_RSA_WITH_AES_256_CBC_SHA, SSL_RSA_WITH_AES_256_CBC_SHA, SSL_ECDH_ECDSA_WITH_AES_256_CBC_SHA, SSL_ECDH_RSA_WITH_AES_256_CBC_SHA, SSL_DHE_RSA_WITH_AES_256_CBC_SHA, SSL_DHE_DSS_WITH_AES_256_CBC_SHA, SSL_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, SSL_ECDHE_RSA_WITH_AES_128_CBC_SHA256, SSL_RSA_WITH_AES_128_CBC_SHA256, SSL_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, SSL_ECDH_RSA_WITH_AES_128_CBC_SHA256, SSL_DHE_RSA_WITH_AES_128_CBC_SHA256, SSL_DHE_DSS_WITH_AES_128_CBC_SHA256, SSL_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, SSL_ECDHE_RSA_WITH_AES_128_CBC_SHA, SSL_RSA_WITH_AES_128_CBC_SHA, SSL_ECDH_ECDSA_WITH_AES_128_CBC_SHA, SSL_ECDH_RSA_WITH_AES_128_CBC_SHA, SSL_DHE_RSA_WITH_AES_128_CBC_SHA, SSL_DHE_DSS_WITH_AES_128_CBC_SHA, SSL_ECDHE_ECDSA_WITH_RC4_128_SHA, SSL_ECDHE_RSA_WITH_RC4_128_SHA, SSL_RSA_WITH_RC4_128_SHA, SSL_ECDH_ECDSA_WITH_RC4_128_SHA, SSL_ECDH_RSA_WITH_RC4_128_SHA, SSL_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA, SSL_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, SSL_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA, SSL_ECDH_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_RC4_128_MD5]
Compression Methods:  { 0 }
Extension elliptic_curves, curve names: {secp256r1, secp192r1, secp224r1, secp384r1, secp521r1, secp160k1, secp160r1, secp160r2, secp192k1, secp224k1, secp256k1}
Extension ec_point_formats, formats: [uncompressed]
Extension signature_algorithms, signature_algorithms: SHA512withECDSA, SHA512withRSA, SHA384withECDSA, SHA384withRSA, SHA256withECDSA, SHA256withRSA, SHA224withECDSA, SHA224withRSA, SHA1withECDSA, SHA1withRSA, SHA256withDSA, SHA1withDSA, MD5withRSA
***
main, WRITE: TLSv1.2 Handshake, length = 195
main, READ: TLSv1.2 Handshake, length = 1375
*** ServerHello, TLSv1.2
RandomCookie:  GMT: -128 bytes = { 238, 184, 253, 53, 112, 242, 137, 166, 205, 83, 9, 182, 17, 177, 233, 43, 206, 14, 0, 217, 246, 26, 214, 153, 47, 150, 202, 51 }
Session ID:  {105, 109, 0, 0, 212, 197, 167, 14, 199, 117, 87, 153, 13, 215, 101, 219, 250, 202, 212, 98, 88, 88, 88, 88, 128, 75, 111, 85, 0, 0, 33, 144}
Cipher Suite: SSL_ECDHE_RSA_WITH_AES_256_CBC_SHA384
Compression Method: 0
Extension ec_point_formats, formats: [uncompressed]
Extension renegotiation_info, ri_length: 0, ri_connection_data: { null }
***
JsseJCE:  Using MessageDigest SHA-384 from provider IBMJCE version 1.7
%% Initialized:  [Session-1, SSL_ECDHE_RSA_WITH_AES_256_CBC_SHA384]
** SSL_ECDHE_RSA_WITH_AES_256_CBC_SHA384
...

Again, the ClientHello phase lists a huge number of ciphers being presented from Java to DB2, and the ServerHello shows the single cipher that DB2 is presenting back.

Right, that's it for now :-)