Wednesday, 24 December 2014

IBM UrbanCode Deploy to WebSphere Application Server - "peer not authenticated"

So I have spent quite literally hours over the past two days working ^H^H^H^H^H^H^H playing with this.

I've got UrbanCode Deploy (UCD) 6.1.0 installed on my VM ( running Red Hat Enterprise Linux 6.4 ) and all is working nicely. However, I was trying, and frequently failing, to get UCD to inspect a WebSphere Application Server (WAS) 8.5.5.3 environment using the Configure using WebSphere Topology Discovery workflow: -


I'd already installed the Application Deployment for WebSphere plugin.

I also had a WAS profile created and running happily, and was able to log into WAS using the Integrated Solutions Console (ISC) and wsadmin.sh via SOAP on port 8880.

However, the Topology Discovery workflow kept failing with: -

Caught: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
at com.ibm.jsse2.bc.getPeerCertificates(bc.java:107)
at org.apache.http.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:128)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:572)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:640)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:479)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
at com.urbancode.ud.client.UDRestClient.invokeMethod(UDRestClient.java:134)
at com.urbancode.ud.client.ResourceClient.getResourceByPath(ResourceClient.java:214)
at com.urbancode.ud.client.ResourceClient$getResourceByPath.call(Unknown Source)
at wasConfig$_run_closure5.doCall(wasConfig.groovy:246)
at wasConfig.run(wasConfig.groovy:447)
 
As I say, I spent hours and hours and hours hacking around with this.

My efforts included: -

Enabling Java SSL Debugging

This I achieved in two ways: -

Setting: -

-Djavax.net.debug=ssl

in server.xml: -

/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/ucd61Node01Cell/nodes/ucd61Node01/servers/server1/server.xml 

( Of course, I would/could/should have done this via Jython but not today, today is a day for celebrations )

Setting: -

-Djavax.net.debug=ssl,handshake

in plugin-javaopts.conf: -

/opt/ibm-ucd/agent/conf/plugin-javaopts.conf

This gave me a slew of interesting, but not totally helpful, debug information: -

WAS

...
[24/12/14 18:04:27:327 GMT] 0000005d SystemOut     O *** ServerHello, TLSv1
[24/12/14 18:04:27:327 GMT] 0000005d SystemOut     O RandomCookie:  GMT: 1402667051 bytes = { 125, 232, 230, 27, 197, 137, 249, 
81, 163, 103, 70, 81, 109, 233, 143, 148, 173, 87, 226, 89, 63, 200, 211, 72, 187, 228, 241, 81 }[24/12/14 18:04:27:327 GMT] 0000005d SystemOut     O Session ID:  {84, 155, 0, 36, 45, 184, 111, 190, 179, 72, 26, 92, 2, 182, 6
4, 128, 247, 137, 170, 40, 221, 74, 75, 59, 0, 110, 53, 213, 115, 217, 13, 27}
[24/12/14 18:04:27:327 GMT] 0000005d SystemOut     O Cipher Suite: SSL_RSA_WITH_RC4_128_MD5
[24/12/14 18:04:27:327 GMT] 0000005d SystemOut     O Compression Method: 0
[24/12/14 18:04:27:327 GMT] 0000005d SystemOut     O Extension renegotiation_info, ri_length: 0, ri_connection_data: { null }
[24/12/14 18:04:27:327 GMT] 0000005d SystemOut     O ***
[24/12/14 18:04:27:327 GMT] 0000005d SystemOut     O Cipher suite:  SSL_RSA_WITH_RC4_128_MD5
[24/12/14 18:04:27:328 GMT] 0000005d SystemOut     O JsseJCE:  Using KeyGenerator IbmTlsKeyMaterial from provider TBD via init
[24/12/14 18:04:27:328 GMT] 0000005d SystemOut     O CONNECTION KEYGEN:
[24/12/14 18:04:27:328 GMT] 0000005d SystemOut     O Client Nonce:
[24/12/14 18:04:27:328 GMT] 0000005d SystemOut     O 0000: 54 9b 00 2b 71 e2 2b aa  ca a4 31 d2 f0 53 14 52  T...q.....1..S.R
0010: 15 53 f2 d2 6b 27 55 90  94 58 dd 30 0a 02 56 38  .S..k.U..X.0..V8

[24/12/14 18:04:27:328 GMT] 0000005d SystemOut     O Server Nonce:
[24/12/14 18:04:27:328 GMT] 0000005d SystemOut     O 0000: 54 9b 00 2b 7d e8 e6 1b  c5 89 f9 51 a3 67 46 51  T..........Q.gFQ
0010: 6d e9 8f 94 ad 57 e2 59  3f c8 d3 48 bb e4 f1 51  m....W.Y...H...Q
[24/12/14 18:04:27:328 GMT] 0000005d SystemOut     O Master Secret:
[24/12/14 18:04:27:328 GMT] 0000005d SystemOut     O 0000: 57 d2 31 ca c5 03 13 84  c3 1f 0a 6e ec ce a7 f1  W.1........n....
0010: e7 4c a8 7f 3c 59 52 32  36 4f ce 88 fa 01 18 41  .L...YR26O.....A
0020: da 62 f0 85 55 ac 96 36  b1 f0 d3 87 3f 48 82 65  .b..U..6.....H.e

[24/12/14 18:04:27:328 GMT] 0000005d SystemOut     O Client MAC write Secret:
[24/12/14 18:04:27:328 GMT] 0000005d SystemOut     O 0000: ad 91 42 34 35 4c 2f d7  ad bf 01 0a 24 db 03 d3  ..B45L..........

[24/12/14 18:04:27:328 GMT] 0000005d SystemOut     O Server MAC write Secret:
[24/12/14 18:04:27:328 GMT] 0000005d SystemOut     O 0000: 95 8d 17 be b1 6d 98 89  bc ab 93 e2 d8 55 82 3e  .....m.......U..

[24/12/14 18:04:27:328 GMT] 0000005d SystemOut     O Client write key:
[24/12/14 18:04:27:328 GMT] 0000005d SystemOut     O 0000: 65 8e 3c 03 35 d2 32 29  ca 24 3b 92 ba 19 d7 0b  e...5.2.........

[24/12/14 18:04:27:328 GMT] 0000005d SystemOut     O Server write key:
[24/12/14 18:04:27:328 GMT] 0000005d SystemOut     O 0000: ab 79 bb 21 38 4d dd bd  e5 90 9e 1c 53 34 76 50  .y..8M......S4vP

[24/12/14 18:04:27:328 GMT] 0000005d SystemOut     O ... no IV used for this cipher
[24/12/14 18:04:27:328 GMT] 0000005d SystemOut     O SoapConnectorThreadPool : 1, WRITE: TLSv1 Handshake, length = 81
[24/12/14 18:04:27:329 GMT] 0000005d SystemOut     O JsseJCE:  Using KeyGenerator IbmTlsPrf from provider TBD via init
[24/12/14 18:04:27:329 GMT] 0000005d SystemOut     O HandshakeMessage:  TLS Keygenerator IbmTlsPrf  from provider from init IBMJCE version 1.2
[24/12/14 18:04:27:330 GMT] 0000005d SystemOut     O SoapConnectorThreadPool : 1, WRITE: TLSv1 Change Cipher Spec, length = 1
[24/12/14 18:04:27:331 GMT] 0000005d SystemOut     O JsseJCE:  Using cipher RC4 from provider TBD via init
[24/12/14 18:04:27:331 GMT] 0000005d SystemOut     O CipherBox:  Using cipher RC4 from provider from init IBMJCE version 1.2
[24/12/14 18:04:27:331 GMT] 0000005d SystemOut     O JsseJCE:  Using MAC HmacMD5 from provider TBD via init
[24/12/14 18:04:27:331 GMT] 0000005d SystemOut     O MAC:  Using MessageDigest HmacMD5 from provider IBMJCE version 1.2
[24/12/14 18:04:27:331 GMT] 0000005d SystemOut     O *** Finished
[24/12/14 18:04:27:331 GMT] 0000005d SystemOut     O verify_data:  { 207, 227, 130, 132, 11, 191, 247, 248, 179, 164, 79, 92 }
[24/12/14 18:04:27:331 GMT] 0000005d SystemOut     O ***
[24/12/14 18:04:27:331 GMT] 0000005d SystemOut     O SoapConnectorThreadPool : 1, WRITE: TLSv1 Handshake, length = 32
[24/12/14 18:04:27:332 GMT] 0000005d SystemOut     O SoapConnectorThreadPool : 1, READ: TLSv1 Change Cipher Spec, length = 1
[24/12/14 18:04:27:332 GMT] 0000005d SystemOut     O JsseJCE:  Using cipher RC4 from provider TBD via init
[24/12/14 18:04:27:333 GMT] 0000005d SystemOut     O CipherBox:  Using cipher RC4 from provider from init IBMJCE version 1.2
[24/12/14 18:04:27:333 GMT] 0000005d SystemOut     O JsseJCE:  Using MAC HmacMD5 from provider TBD via init
[24/12/14 18:04:27:333 GMT] 0000005d SystemOut     O MAC:  Using MessageDigest HmacMD5 from provider IBMJCE version 1.2
[24/12/14 18:04:27:333 GMT] 0000005d SystemOut     O SoapConnectorThreadPool : 1, READ: TLSv1 Handshake, length = 32
...
[24/12/14 19:25:24:539 GMT] 0000005b SystemOut     O SoapConnectorThreadPool : 0, received EOFException: error
[24/12/14 19:25:24:539 GMT] 0000005b SystemOut     O SoapConnectorThreadPool : 0, handling exception: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
[24/12/14 19:25:24:539 GMT] 0000005b SystemOut     O SoapConnectorThreadPool : 0, SEND TLSv1 ALERT:  fatal, description = handshake_failure
...


UCD

...
IBMJSSE2 will not enable CBC protection
JsseJCE:  Using SecureRandom IBMSecureRandom from provider IBMJCE version 1.7
JsseJCE:  Using KeyAgreement ECDH from provider IBMJCE version 1.7
JsseJCE:  Using signature SHA1withECDSA from provider TBD via init 
JsseJCE:  Using signature NONEwithECDSA from provider TBD via init 
JsseJCE:  Using KeyFactory EC from provider IBMJCE version 1.7
JsseJCE:  Using KeyPairGenerator EC from provider TBD via init 
JsseJce:  EC is available
main, setSoTimeout(0) called
IBMJSSE2 will allow RFC 5746 renegotiation per com.ibm.jsse2.renegotiate set to none or default
IBMJSSE2 will not require renegotiation indicator during initial handshake per com.ibm.jsse2.renegotiation.indicator set to OPTIONAL or default taken
IBMJSSE2 will not perform identity checking against the peer cert check during renegotiation per com.ibm.jsse2.renegotiation.peer.cert.check set to OFF or default
 
Is initial handshake: true
Ignoring unsupported cipher suite: SSL_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
Ignoring unsupported cipher suite: SSL_ECDHE_RSA_WITH_AES_128_CBC_SHA256
Ignoring unsupported cipher suite: SSL_RSA_WITH_AES_128_CBC_SHA256
Ignoring unsupported cipher suite: SSL_ECDH_ECDSA_WITH_AES_128_CBC_SHA256
Ignoring unsupported cipher suite: SSL_ECDH_RSA_WITH_AES_128_CBC_SHA256
Ignoring unsupported cipher suite: SSL_DHE_RSA_WITH_AES_128_CBC_SHA256
Ignoring unsupported cipher suite: SSL_DHE_DSS_WITH_AES_128_CBC_SHA256
Ignoring unsupported cipher suite: SSL_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
Ignoring unsupported cipher suite: SSL_ECDHE_RSA_WITH_AES_128_CBC_SHA
Ignoring unsupported cipher suite: SSL_RSA_WITH_AES_128_CBC_SHA
Ignoring unsupported cipher suite: SSL_ECDH_ECDSA_WITH_AES_128_CBC_SHA
Ignoring unsupported cipher suite: SSL_ECDH_RSA_WITH_AES_128_CBC_SHA
Ignoring unsupported cipher suite: SSL_DHE_RSA_WITH_AES_128_CBC_SHA
Ignoring unsupported cipher suite: SSL_DHE_DSS_WITH_AES_128_CBC_SHA
Ignoring unsupported cipher suite: SSL_ECDHE_ECDSA_WITH_RC4_128_SHA
Ignoring unsupported cipher suite: SSL_ECDHE_RSA_WITH_RC4_128_SHA
Ignoring unsupported cipher suite: SSL_ECDH_ECDSA_WITH_RC4_128_SHA
Ignoring unsupported cipher suite: SSL_ECDH_RSA_WITH_RC4_128_SHA
Ignoring unsupported cipher suite: SSL_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA
Ignoring unsupported cipher suite: SSL_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA
Ignoring unsupported cipher suite: SSL_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA
Ignoring unsupported cipher suite: SSL_ECDH_RSA_WITH_3DES_EDE_CBC_SHA
%% No cached client session
*** ClientHello, SSLv3
RandomCookie:  GMT: 1402593669 bytes = { 215, 60, 79, 216, 73, 171, 239, 0, 81, 69, 93, 98, 89, 131, 202, 26, 159, 74, 101, 239, 235, 105, 218, 190, 41, 139, 196, 23 }
Session ID:  {}
Cipher Suites: [TLS_EMPTY_RENEGOTIATION_INFO_SCSV, SSL_RSA_WITH_RC4_128_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_RC4_128_MD5]
Compression Methods:  { 0 }
***
main, WRITE: SSLv3 Handshake, length = 55
main, READ: TLSv1 Alert, length = 2
main, RECV TLSv1 ALERT:  fatal, handshake_failure
main, called closeSocket()
main, handling exception: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
main, IOException in getSession():  javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
main, called close()
main, called closeInternal(true)
main, called close()
main, called closeInternal(true)
Caught: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
at com.ibm.jsse2.bc.getPeerCertificates(bc.java:107)
at org.apache.http.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:128)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:572)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:640)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:479)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
at com.urbancode.ud.client.UDRestClient.invokeMethod(UDRestClient.java:134)
at com.urbancode.ud.client.ResourceClient.getResourceByPath(ResourceClient.java:214)
at com.urbancode.ud.client.ResourceClient$getResourceByPath.call(Unknown Source)
at wasConfig$_run_closure5.doCall(wasConfig.groovy:246)
at wasConfig.run(wasConfig.groovy:447)
...

This led me up and down all sorts of lovely lovely garden paths, including: -

  • Fiddling with SSL v3 and TLS v1 protocols, via -Dhttps.protocols=SSLv3 etc.
  • Disabling and enabling session renegotiation via -Dcom.ibm.jsse2.renegotiate
  • Disabling the Server Name Indication (SNI) Extension via -Djsse.enableSNIExtension=false

I also spent an absolute age adding SSL certificates to the JRE keystore: -

export ADDRESS=ucd61.uk.ibm.com
export PORT=8880
echo -n | openssl s_client -connect $ADDRESS:$PORT | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /tmp/$ADDRESS.cert
/opt/IBM/Java/jre/bin/keytool -importcert -trustcacerts -alias was855_3 -keystore /opt/IBM/Java/jre/lib/security/cacerts -storepass changeit -file /tmp/ucd61.uk.ibm.com.cert

but to no avail.

The solution ?

Well, I'd wondered about the Java client version underlying UCD, and checked this: -

/opt/IBM/Java/jre/bin/java -version

java version "1.7.0"
Java(TM) SE Runtime Environment (build pxa6470sr5-20130619_01(SR5))
IBM J9 VM (build 2.6, JRE 1.7.0 Linux amd64-64 Compressed References 20130617_152572 (JIT enabled, AOT enabled)
J9VM - R26_Java726_SR5_20130617_1436_B152572
JIT  - r11.b04_20130528_38954ifx1
GC   - R26_Java726_SR5_20130617_1436_B152572_CMPRSS
J9CL - 20130617_152572)
JCL - 20130616_01 based on Oracle 7u25-b12

and wondered whether, post-POODLE, this was part of the problem.

I downloaded, and installed, the most recent IBM JRE: -


/opt/IBM/Java/jre/bin/java -version

java version "1.7.0"
Java(TM) SE Runtime Environment (build pxa6470_27sr2-20141026_01(SR2))
IBM J9 VM (build 2.7, JRE 1.7.0 Linux amd64-64 Compressed References 20141017_217728 (JIT enabled, AOT enabled)
J9VM - R27_Java727_SR2_20141017_1632_B217728
JIT  - tr.r13.java_20141003_74587.01
GC   - R27_Java727_SR2_20141017_1632_B217728_CMPRSS
J9CL - 20141017_217728)
JCL - 20141004_01 based on Oracle 7u71-b13

and retried the Configure using WebSphere Topology Discovery workflow .... and it worked :-)

The moral of the story ? Check your JRE version.

Now I need to go and undo all the hacks I've made to my environment, but at least I've learned a valuable lesson.

Sources of Inspiration


at

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)

Note to self - use kubectl to query images in a pod or deployment

In both cases, we use JSON ... For a deployment, we can do this: - kubectl get deployment foobar --namespace snafu --output jsonpath="{...