Friday, 18 August 2017

Java to DB2 via TLS 1.2 - A new one on me

So I've been helping a friend debug and resolve a problem whereby his Java 7 code wasn't connecting to a DB2 database via a TLS 1.2 connection.

To validate this, I setup a DB2 instance to support TLS 1.2: -

As root

echo "db2c_ssl 60007/tcp" >> /etc/services

As db2inst1

/home/db2inst1/sqllib/gskit/bin/gsk8capicmd_64 -keydb -create -db /home/db2inst1/keystore.kdb -pw passw0rd -stash

/home/db2inst1/sqllib/gskit/bin/gsk8capicmd_64 -cert -create -db /home/db2inst1/keystore.kdb -pw passw0rd -label "odm.uk.ibm.com" -dn "cn=odm.uk.ibm.com,dc=uk,dc=ibm,dc=com" 

db2 update dbm cfg using SSL_SVR_KEYDB /home/db2inst1/keystore.kdb
db2 update dbm cfg using SSL_SVR_STASH /home/db2inst1/keystore.sth
db2 update dbm cfg using SSL_SVR_LABEL odm.uk.ibm.com
db2 update dbm cfg using SSL_SVCENAME db2c_ssl
db2 update dbm cfg using SSL_VERSIONS TLSV12
db2 update dbm cfg using SSL_CIPHERSPECS TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
db2set DB2COMM=TCPIP,SSL
db2stop
db2start


and then validated what the WAS box was seeing: -

openssl s_client -connect odm.uk.ibm.com:60007 </dev/null

New, TLSv1/SSLv3, Cipher is AES256-GCM-SHA384
Server public key is 1024 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : AES256-GCM-SHA384


Having relieved the DB2 self-signed certificate: -

openssl s_client -showcerts -connect odm.uk.ibm.com:60007 </dev/null | openssl x509 -outform DER > ~/db2.cer

and created a JKS trust store: -

/opt/ibm/WebSphere/AppServer/java/jre/bin/keytool -import -file ~/db2.cer -keystore ~/davehay.jks -alias DB2 -storepass davehay

I then tested my Java code ( see below ) : -

java -cp /opt/ibm/WebSphere/AppServer/ODMjdbcdrivers/DB2/db2jcc.jar:/home/wasadmin DB2connect_ssl2

Instead of working, this generated a trace file ( as per the code ): -

view /tmp/foobar.trc_sds_1 

which contained: -

...
[jcc] BEGIN TRACE_DIAGNOSTICS
[jcc][Thread:main][SQLException@7807b259] java.sql.SQLException
[jcc][Thread:main][SQLException@7807b259] SQL state  = 08001
[jcc][Thread:main][SQLException@7807b259] Error code = -4499
[jcc][Thread:main][SQLException@7807b259] Message    = [jcc][t4][2030][11211][3.61.65] A communication error occurred during operations on the connection's underlying socket, socket input stream,
or socket output stream.  Error location: T4Agent.sendRequest() - flush (-2).  Message: Received fatal alert: handshake_failure. ERRORCODE=-4499, SQLSTATE=08001
[jcc][Thread:main][SQLException@7807b259] Stack trace follows
com.ibm.db2.jcc.am.DisconnectNonTransientConnectionException: [jcc][t4][2030][11211][3.61.65] A communication error occurred during operations on the connection's underlying socket, socket input stream, 
or socket output stream.  Error location: T4Agent.sendRequest() - flush (-2).  Message: Received fatal alert: handshake_failure. ERRORCODE=-4499, SQLSTATE=08001
        at com.ibm.db2.jcc.am.ed.a(ed.java:319)

...

I turned on debugging: -

java -cp /opt/ibm/WebSphere/AppServer/ODMjdbcdrivers/DB2/db2jcc.jar:/home/wasadmin -Djavax.net.debug=ssl DB2connect_ssl2

which told me what I already knew: -

main, WRITE: TLSv1 Handshake, length = 123
main, READ: TLSv1.2 Alert, length = 2
main, RECV TLSv1 ALERT:  fatal, handshake_failure
main, called closeSocket()
main, handling exception: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure

Similarly, DB2 told me that things were awry: -

db2diag -f

which returned: -

2017-08-18-19.27.52.683893+060 I48627542E482         LEVEL: Error
PID     : 128279               TID : 140419925010176 PROC : db2sysc 0
INSTANCE: db2inst1             NODE : 000
APPHDL  : 0-24873
HOSTNAME: odm.uk.ibm.com
EDUID   : 23                   EDUNAME: db2agent () 0
FUNCTION: DB2 UDB, common communication, sqlccMapSSLErrorToDB2Error, probe:30
MESSAGE : DIA3604E The SSL function "gsk_secure_soc_init" failed with the 
          return code "402" in "sqlccSSLSocketSetup".


I've seen this before: -



However, this time around the solution was … different.

I experimented with various tweaks to the Java command-line: -

java -cp /opt/ibm/WebSphere/AppServer/ODMjdbcdrivers/DB2/db2jcc.jar:/home/wasadmin -Dcom.ibm.jsse2.overrideDefaultProtocol=TLSv12 DB2connect_ssl2

and: -

java -cp /opt/ibm/WebSphere/AppServer/ODMjdbcdrivers/DB2/db2jcc.jar:/home/wasadmin -Dcom.ibm.jsse2.overrideDefaultProtocol=TLSv12 -Dhttps.cipherSuites=TLS_RSA_WITH_AES_256_GCM_SHA384 DB2connect_ssl2

etc. but to no avail.

I'd also verified that my JRE was set to support the stronger export-grade ciphers: -

java -cp /home/wasadmin/ CipherTest

PASSED: Max AES key length OK! - >= 256 (2147483647).

At which point, I was scratching my head ….

… and then I read this: -


which said, in part, this: -

...
3. Identify your JCC driver version by issuing the following command:
java -cp db2jcc4.jar com.ibm.db2.jcc.DB2Jcc -version

4. Determine if you need to update your JCC driver.

• If you use Java 8 and an IBM JDK, the minimum recommended level is 4.18.
• Otherwise, update to version 4.22.29 or higher. This is the version that shipped with DB2 v11.1 FP1.
You can download a new JCC driver here. If you deploy your application in a Bluemix environment, either remove the JCC driver ( db2jcc4.jar) in the application's WebContent/WEB-INF/lib/ folder or else ensure that the driver in this folder is up to date before you build and deploy the application. 
...

Given that I'm using DB2 11.1, as per this: -

db2level 

DB21085I  This instance or install (instance name, where applicable: 
"db2inst1") uses "64" bits and DB2 code release "SQL11012" with level 
identifier "0203010F".
Informational tokens are "DB2 v11.1.2.2", "s1706091900", "DYN1701310100AMD64", 
and Fix Pack "2".
Product is installed at "/opt/ibm/db2/V11.1".

I checked the version of the JDBC driver that I was using ( this was shipped with IBM ODM Rules 8.9, which is what I happen to using for my tests, even though this is ONLY Java to DB2 testing ): -

java -cp /opt/ibm/WebSphere/AppServer/ODMjdbcdrivers/DB2/db2jcc.jar com.ibm.db2.jcc.DB2Jcc -version

IBM DB2 JDBC Universal Driver Architecture 3.61.65

java -cp /opt/ibm/WebSphere/AppServer/ODMjdbcdrivers/DB2/db2jcc4.jar com.ibm.db2.jcc.DB2Jcc -version

IBM Data Server Driver for JDBC and SQLJ 4.11.69

which is below the recommended version 4.22.29.

Therefore, I checked the versions that ship with DB2 itself: -

java -cp /opt/ibm/db2/V11.1/java/db2jcc.jar com.ibm.db2.jcc.DB2Jcc -version

IBM DB2 JDBC Universal Driver Architecture 3.72.30

java -cp /opt/ibm/db2/V11.1/java/db2jcc4.jar com.ibm.db2.jcc.DB2Jcc -version

IBM Data Server Driver for JDBC and SQLJ 4.23.42

so I tested my Java class using this updated driver: -

java -cp /opt/ibm/db2/V11.1/java/db2jcc.jar:/home/wasadmin DB2connect_ssl2

and it returned: -

 Connected to database with type 4 url
000010 CHRISTINE
000020 MICHAEL
000030 SALLY
000050 JOHN
000060 IRVING
000070 EVA
000090 EILEEN
000100 THEODORE
000110 VINCENZO
000120 SEAN
000130 DELORES
000140 HEATHER
000150 BRUCE
000160 ELIZABETH
000170 MASATOSHI
000180 MARILYN
000190 JAMES
000200 DAVID
000210 WILLIAM
000220 JENNIFER
000230 JAMES
000240 SALVATORE
000250 DANIEL
000260 SYBIL
000270 MARIA
000280 ETHEL
000290 JOHN
000300 PHILIP
000310 MAUDE
000320 RAMLAL
000330 WING
000340 JASON
200010 DIAN
200120 GREG
200140 KIM
200170 KIYOSHI
200220 REBA
200240 ROBERT
200280 EILEEN
200310 MICHELLE
200330 HELENA
200340 ROY


The moral of the story - the version of the JDBC driver MAY well be more important than first I had realised.

If you need an updated driver, please go here: -


Final point, here's the Java code that I'm using: -

DB2connect_ssl2.java

import java.lang.*;
import java.sql.*;
import java.io.*;

class DB2connect_ssl2
{
public static void main(String argv[])
{
try
{
Connection con = null;
PreparedStatement pstmt = null;

try
{
Class.forName("com.ibm.db2.jcc.DB2Driver").newInstance();
con = DriverManager.getConnection( "jdbc:db2://odm.uk.ibm.com:60007/SAMPLE" +
              ":user=db2inst1;password=passw0rd;" +
              "sslConnection=true;sslTrustStoreLocation=/home/wasadmin/davehay.jks;sslTrustStorePassword=davehay;" +
              "keepAliveTimeout=10;" +
      "traceDirectory=/tmp;" +
      "traceFile=foobar.trc;" +
      "traceFileAppend=false;" +
      "traceLevel=-1;" );
System.out.println(" Connected to database with type 4 url") ;
}
catch (Throwable e)
{
System.out.println("Connect failed" + e);
}

try
{
pstmt = con.prepareStatement("select * from DB2INST1.EMP");
}
catch (Throwable e)
{
System.out.println("Statement Prepare failed" + e);
}

try
{
ResultSet s = pstmt.executeQuery();

        while (s.next())
{
String v_id = s.getString(1);
String v_stream_id = s.getString(2);
System.out.println( v_id + " " + v_stream_id );
}
s.close();
}
catch (Throwable e)
{
System.out.println("fetch processing failed");
}
con.close();
}
catch (Exception e)
{
System.out.println("Connect failed" + e);
}
}
}


and here's another one: -

JdbcTestDB2.java
 
import java.sql.Connection ;
import java.sql.DriverManager ;
import java.sql.ResultSet ;
import java.sql.Statement ;
import java.sql.SQLException;

import org.omg.CORBA.VersionSpecHelper;

class JdbcTestDB2
{
public static void main (String args[])
{
try
{
Class.forName("com.ibm.db2.jcc.DB2Driver");
}
catch (ClassNotFoundException e)
{
System.err.println (e) ;
System.exit (-1) ;
}
String hostname = "odm.uk.ibm.com";
int port = 60007;
String dbName = "SAMPLE";
String userName = "db2inst1";
String password = "passw0rd";
String sslConnection = "true";

java.util.Properties properties = new java.util.Properties();
properties.put("user",userName);
properties.put("password", password);
properties.put("sslConnection", sslConnection);
properties.put("sslTrustStoreLocation","/home/wasadmin/davehay.jks");
properties.put("sslTrustStorePassword","davehay");

String url = "jdbc:db2://" + hostname + ":" + port + "/" + dbName;
try
{
Connection connection = DriverManager.getConnection(url,properties);

String query = "select EMPNO,FIRSTNME,LASTNAME from DB2INST1.EMPLOYEE" ;

Statement statement = connection.createStatement () ;
ResultSet rs = statement.executeQuery (query) ;

while ( rs.next () )
System.out.println (rs.getString (1) + " " + rs.getString(2) + " " + rs.getString(3)) ;
connection.close () ;
}
catch (java.sql.SQLException e)
{
System.err.println (e) ;
System.exit (-1) ;
}
}
}


IBM API Connect - Now a Happy Bunny - Was Notworking, Now Networking

As per the past few posts, I've been learning my way around IBM API Connect, and had it nearly working - until about 15 minutes ago when it ALL started working.

Without getting too deep into the What and How, the TL;DR; is this …..

I have three VMs; the API Manager, the Developer Portal and the DataPower Gateway.


I'd moved DataPower onto a different Mac, as it requires 8 GB RAM.

I'd sensibly configured that VM to use Bridged, rather than NAT, networking, in order that the API Manager ( APIM ) could "see" DataPower.

However, I'd left APIM and the Portal using NAT networking on the main Mac, for no other reason that laziness.

That was where things went wrong.

Whilst I could happily add the DataPower Service and Service into the APIM configuration, using the Cloud Management Console, I was hitting a 404 whenever I tried to test my APIs, even though the Endpoint was correctly configured: -


So here's the thing ….

Whilst it's crucial that APIM can see DataPower, it's also quite important that DataPower can see APIM :-)

One thing that gave me a clue was this UI: -


specifically the Information icon.

This NOW shows: -


but had previously reported the Management Load Balancing Addresses as (a) being the old NAT address of APIM itself and (b) as down.

Once I switched to Bridged, and sorted out my IP addresses, making sure that I was using addresses in the same subnet, to avoid the need for messy routing …. things suddenly became better.

I did sketch out the networking topology a few months back : -


so I really should've known better.

Bottom line, whilst APIM needs to "see" DataPower, DataPower also needs "see" APIM

If it helps, this is also a rather useful introduction to the networking requirements: -

Firewall requirements

IBM API Connect - "Unexpected token ILLEGAL"

So I'm continuing my voyage of discovery with IBM API Connect, and had followed this tutorial to create a simple API using Loopback: -


Annoyingly, when I came to test my API, I was seeing this: -


and, via Debug, this: -


This is the Gateway Script that I have: -

Can you see what I can see ?

Yeah, I'm at home to Cap'n Copy Paste :-(

I fixed my code: -

saved the API and republished the product.

Now we have this: -


which is far far better.

So, for the record, the copy/paste from Firefox to Safari had introduced some glitches with regard to the apostrophes ( single quotes ).

This has bitten me before on Mac, but I'm stupid and forgot :-(

For the record, the DataPower logs also showed similar: -



IBM API Connect - Webhooks not configured

I'm learning my way around IBM API Connect at present, which is giving me a great opportunity to also learn IBM DataPower Gateway, which is nice.

Whilst trying to determine why I'm unable to test my APIs on the DataPower Gateway from within the API Manager, I was looking at some of the underlying Developer Portal commands, including: -

list_sites

which returned: -

5995c388e4b0d25e2a2ee8f2.5995c388e4b0d25e2a2ee8fe => portal.uk.ibm.com/davehay/sb (INSTALLED) <!> Webhooks not configured

and: -

status

System version: 7.x-5.0.7.2-20170629-1629
Distribution version: 7.x-5.0.7.2-20170628-2109

Free disk space: 16G
   Set Hostname: OK
     DNS Server: Reachable (192.168.153.2)
   APIC SSH Key: OK

Configuration:
  APIC Hostname: management.uk.ibm.com
  APIC IP: 192.168.153.130
  Devportal Hostname: portal.uk.ibm.com
  Devportal IP: 192.168.153.132
  APIC Certificate Status (Default): ERROR - The certificate that was presented by the remote was self signed, but with the expectation that it would be CA signed. 
                                             Please ensure that there is a valid CA signed certificate being used by the APIM servers and ensure any intermediate public certificates are presented or are in the trust store.

Node is standalone

Site web check: All sites OK

Site services:
         Webhooks: Some DOWN <!> (portal.uk.ibm.com/davehay/sb)
  Background sync: All sites Up

Services:
  Queue                      is Up
  Database   [Mysql]         is Up (Standalone)
  Web Server [Nginx]         is Up
  PHP Pool   [Php5-fpm]      is Up
  Inetd      [Openbsd-inetd] is Up
  REST       [Restservice]   is Up

FAILURE: Some services are DOWN.


which was less than impressive.

Thankfully the Knowledge Centre saved me: -


which said, in part: -


So I ran the command: -

download_apim_cert

which returned: -

depth=0 C = US, ST = NY, L = Armonk, O = IBM Corporation, CN = API Connect
verify error:num=18:self signed certificate
verify return:1
depth=0 C = US, ST = NY, L = Armonk, O = IBM Corporation, CN = API Connect
verify return:1
DONE
WARNING: The common name provided in the certificate, "Connect", does not match the configured IBM APIC server: "management.uk.ibm.com".  The developer portal will be configured to trust the provided certificate without checking that the server name matches the common name.
Setting user provided https certificate for portal.uk.ibm.com/davehay/sb


I then re-ran the status command: -

status

System version: 7.x-5.0.7.2-20170629-1629
Distribution version: 7.x-5.0.7.2-20170628-2109

Free disk space: 16G
   Set Hostname: OK
     DNS Server: Reachable (192.168.153.2)
   APIC SSH Key: OK

Configuration:
  APIC Hostname: management.uk.ibm.com
  APIC IP: 192.168.153.130
  Devportal Hostname: portal.uk.ibm.com
  Devportal IP: 192.168.153.132
  APIC Certificate Status (Mismatch): WARNING - Only suitable for development and PoC purposes.

Node is standalone

Site web check: All sites OK

Site services:
         Webhooks: All sites Up
  Background sync: All sites Up

Services:
  Queue                      is Up
  Database   [Mysql]         is Up (Standalone)
  Web Server [Nginx]         is Up
  PHP Pool   [Php5-fpm]      is Up
  Inetd      [Openbsd-inetd] is Up
  REST       [Restservice]   is Up

SUCCESS: All services are Up.


and the list_sites command: -

list_sites

5995c388e4b0d25e2a2ee8f2.5995c388e4b0d25e2a2ee8fe => portal.uk.ibm.com/davehay/sb (INSTALLED)

So that's all good then ….

IBM DataPower Playground - on IBM Bluemix

I found this serendipitously whilst looking for something completely different: -

Welcome to the IBM DataPower Playground.

The IBM DataPower Playground is an interactive website that lets you write GatewayScript code and execute it on a cloud hosted DataPower Gateway for learning purposes.

GatewayScript is a new transformation technology for API, Web and Mobile and is available in DataPower as a Processing Policy Action starting from firmware version 7.0. For more information see the documentation.

To get started click on the Samples tab.


For the record, I'm tinkering with DataPower in multiple contexts at present, both DataPower on Docker ( see previous posts ) and DataPower with IBM API Connect ( see previous posts ).

So that's all good then ...

Thursday, 17 August 2017

IBM API Connect - Adding IBM DataPower Gateway into the mix - SOMA import failed

I've been tinkering with an IBM API Connect 5.0.7.2 implementation on my Mac, running all three VMs; the API Manager, the Developer Portal AND the DataPower Gateway on the same box.

However, I saw this: -


when attempting to integrate APIC and IDG.

I dumped out the APIC CMC logs: -

debug postmortem export sftp 10.10.2.3 user fred file /tmp/foobar.tar.gz

and, on the Mac, exploded the resulting tarball, giving me this: -

log/cmc.out:2017-08-17 13:19:30.076 SEVERE [T-83] [com.ibm.datapower.gwapi.impl.GatewaysImpl.deployFilesAndConfig] Deploy: APIMgmt_9A8A3AFACB/gw1: failed: IOException:SOMA import failed
log/cmc.out:Throwable occurred: java.io.IOException: SOMA import failed
log/cmc.out: at com.ibm.datapower.gwapi.soma.Manager.importConfig(Manager.java:657)
log/cmc.out: at com.ibm.datapower.gwapi.soma.Manager.importConfig(Manager.java:625)
log/cmc.out: at com.ibm.datapower.gwapi.soma.Manager.importConfig(Manager.java:621)
log/cmc.out:2017-08-17 13:19:30.084 SEVERE [T-83] [com.ibm.apimgmt.exception.APIGenericException$Serializer.serialize] Error(599597e2e4b0d25e2a2ee700), Http-Code(500), Message(SOMA import failed), User(cmc/admin), Path(post:/gatewayClusters/599426b4e4b04960a3daf1dd/servers/59959798e4b0d25e2a2ee6fa)


As a quick test, I'm moving the IDG VM from the main Mac to another Mac, in order to spread the load, leaving the Manager and Portal VMs on the primary box.

We'll see ……

*UPDATE*

Yes, as I suspected, once I moved the IDG VM to another box, and increased it's RAM from 4 GB to 8 GB, all is now well.


Now to test my API ....

Wednesday, 16 August 2017

What's the opposite of "cat" ? Another in the "I did not know that" series

Thanks to a friend, Paul Smith, for sharing this.

He had a requirement - to take a list of items ( a text file ) and invert the list.

Initially I thought he meant that he wanted to sort the list, which seemed simple.

Here's a list: -

cat list.txt 

Cat
Rabbit
Dog
Gerbil
Hamster
Horse
Rhino
Hippo
Alosaur


so here's the sorted list: -

sort list.txt 

Alosaur
Cat
Dog
Gerbil
Hamster
Hippo
Horse
Rabbit
Rhino


and here's the reverse sort: -

sort -r list.txt 

Rhino
Rabbit
Horse
Hippo
Hamster
Gerbil
Dog
Cat
Alosaur


but NO, I didn't understand his requirement - he wanted the list reversed BUT not sorted.

Introducing …. TAC

Why tac ? 'cos it's the reverse of cat  :-)

Unix and puns, what's not to like

So here's the list, reversed BUT NOT sorted: -

tac list.txt 

Alosaur
Hippo
Rhino
Horse
Hamster
Gerbil
Dog
Rabbit
Cat

Tuesday, 15 August 2017

Getting my head around REST APIs

Almost every product with which I've worked recently, including: -
  • IBM Business Process Manager
  • IBM Operational Decision Manager (Rules)
  • IBM API Connect
  • IBM WebSphere Liberty Profile (Collectives)
  • IBM DataPower Gateway
leverages Representational State Transfer (REST) APIs, so I've got no excuse but to L-E-A-R-N.

Whilst working through the DataPower tutorial here: -


I needed to get a handle on how to drive REST, both in the context of the IDG administration tools, which have a pretty RESTful set of APIs: -


I thus enabled the REST Management service: -

configure; rest-mgmt; admin-state enabled; exit; exit;

Whilst the tutorial recommends the use of the HTTPie tool, I prefer to use curl which is baked into macOS, and doesn't require tools such as brew or port.

So here's my crib of some of the IDG REST APIs, via curl : -

Summary of the Management APIs

curl -k --request GET -url https://localhost:5554/mgmt/ -u admin:admin

Summary of the Status Indicators

curl -k --request GET -url https://localhost:5554/mgmt/status/ -u admin:admin

List available domains

curl -k --request GET -url https://localhost:5554/mgmt/domains/config/ -u admin:admin

Check the CPU usage

curl -k --request GET -url https://localhost:5554/mgmt/status/default/CPUUsage -u admin:admin

As you can imagine, this is a TINY example of the power of the REST APIs.

Then I moved onto the next phase of the tutorial, where we are creating a Multi-Protocol Gateway to "proxy" REST calls through the IDG to IBM Bluemix, using a sample service for Pokemon.

The service is available on Bluemix here: -


and, when poked with a browser, which issues a GET request, simply returns a list of Pokemons in JSON format.

Initially, I tickled the same URL using a REST client plugin called Restlet which sits nicely inside Google Chrome.

Hitting the same URL with a GET request gives me the same list of Pokemon, albeit in a nice UI: -


As part of the tutorial, I needed to add a new Pokemon to the list, using the POST method.

Whilst I could use Restlet for this, I wanted to use curl, so I tested the GET request: -


[{"data":{"moves":"fast"},"height":"34","name":"charmeleon","weight":432,"id":"2"},{"data":{"moves":"very fast"},"height":"90","name":"venusaur","weight":80,"id":"3"},{"data":{"moves":"slow"},"height":"70","name":"ivysaur","weight":200,"id":"1"},{"data":{"moves":"fast"},"height":"50","name":"bulbasuar","weight":100,"id":"4"},{"data":{"moves":"slow"},"height":"70","name":"charmander","weight":122,"id":"9ed78062996515b4db7e1b78d73208b0"}]

I then added a new Pokemon: -

curl --request POST --url https://pokemons.mybluemix.net/api/pokemons --header 'accept: application/json' --header 'content-type: application/json'   --data '{"data":{"moves":"slow"},"height":"70","name":"ivysaur","weight":200,"id":"6"}'

{"data":{"moves":"slow"},"height":"70","name":"ivysaur","weight":200,"id":"6"}

and retrieved the updated list: -

[{"data":{"moves":"fast"},"height":"34","name":"charmeleon","weight":432,"id":"2"},{"data":{"moves":"very fast"},"height":"90","name":"venusaur","weight":80,"id":"3"},{"data":{"moves":"slow"},"height":"70","name":"ivysaur","weight":200,"id":"1"},{"data":{"moves":"fast"},"height":"50","name":"bulbasuar","weight":100,"id":"4"},{"data":{"moves":"slow"},"height":"70","name":"ivysaur","weight":200,"id":"6"},{"data":{"moves":"slow"},"height":"70","name":"charmander","weight":122,"id":"9ed78062996515b4db7e1b78d73208b0"}]

When I (perhaps stupidly) re-ran the POST request with the SAME data, I got this: -

curl --request POST --url https://pokemons.mybluemix.net/api/pokemons --header 'accept: application/json' --header 'content-type: application/json'   --data '{"data":{"moves":"slow"},"height":"70","name":"ivysaur","weight":200,"id":"6"}'

{"error":{"name":"Error","status":409,"message":"Document update conflict. (duplicate?)","error":"conflict","reason":"Document update conflict.","scope":"couch","statusCode":409,"request":{"method":"POST","headers":{"content-type":"application/json","accept":"application/json"},"uri":"https://XXXXXX:XXXXXX@757241f7-e17e-4fc1-b206-91919dd9ced5-bluemix.cloudant.com/pokemon_db","body":"{\"data\":{\"moves\":\"slow\"},\"height\":\"70\",\"name\":\"ivysaur\",\"weight\":200,\"_id\":\"6\",\"loopback__model__name\":\"pokemon\"}"},"headers":{"cache-control":"must-revalidate","content-type":"application/json","date":"Tue, 15 Aug 2017 15:36:22 GMT","x-couch-request-id":"e7ffd29868","x-content-type-options":"nosniff","x-cloudant-backend":"bm-dal-standard3","via":"1.1 lb1.bm-dal-standard3 (Glum/1.37.0)","strict-transport-security":"max-age=31536000","statusCode":409,"uri":"https://XXXXXX:XXXXXX@757241f7-e17e-4fc1-b206-91919dd9ced5-bluemix.cloudant.com/pokemon_db"},"errid":"non_200","description":"couch returned 409"}}

which kinda makes sense.

To prove this, I used the DELETE method: -

{"count":1}

retrieved the now updated list: -


[{"data":{"moves":"fast"},"height":"34","name":"charmeleon","weight":432,"id":"2"},{"data":{"moves":"very fast"},"height":"90","name":"venusaur","weight":80,"id":"3"},{"data":{"moves":"slow"},"height":"70","name":"ivysaur","weight":200,"id":"1"},{"data":{"moves":"fast"},"height":"50","name":"bulbasuar","weight":100,"id":"4"},{"data":{"moves":"slow"},"height":"70","name":"charmander","weight":122,"id":"9ed78062996515b4db7e1b78d73208b0"}]

and then reinserted ID 6: -

curl --request POST --url https://pokemons.mybluemix.net/api/pokemons --header 'accept: application/json' --header 'content-type: application/json'   --data '{"data":{"moves":"slow"},"height":"70","name":"ivysaur","weight":200,"id":"6"}'

{"data":{"moves":"slow"},"height":"70","name":"ivysaur","weight":200,"id":"6"}

and (finally) retrieved the now updated list: -


[{"data":{"moves":"fast"},"height":"34","name":"charmeleon","weight":432,"id":"2"},{"data":{"moves":"very fast"},"height":"90","name":"venusaur","weight":80,"id":"3"},{"data":{"moves":"slow"},"height":"70","name":"ivysaur","weight":200,"id":"1"},{"data":{"moves":"fast"},"height":"50","name":"bulbasuar","weight":100,"id":"4"},{"data":{"moves":"slow"},"height":"70","name":"ivysaur","weight":200,"id":"6"},{"data":{"moves":"slow"},"height":"70","name":"charmander","weight":122,"id":"9ed78062996515b4db7e1b78d73208b0"}]

All of this is of use, because it's enabled me to get a better handle on how I can drive REST from a browser, from a plugin ( Restlet in my case ) and from the command-line, using curl.

Which is all nice.

For the record, here's how I test an API deployed to IBM API Connect: -

curl -k --request POST \
    --url https://apimdev2019.hursley.ibm.com/daves-org/sb/DecisionService/rest/HelloWorldProject/1.0/HelloWorld \
    --header 'accept: application/json' \
    --header 'content-type: application/json' \
    --data '{"request":"David M M Hay"}'


IBM DataPower Gateway - SSH now working

Today has been a day for DataPower: -




So I'm not yet sure why this works, but I was struggling to get SSH to work against an IBM DataPower Gateway Docker image/container.

Whilst the configuration looked OK: -



the service would refuse to become active, instead frustratingly staying as disabled ( 0x0034000d ).

This meant that, whilst I could connect to the DataPower via the web admin interface: -


having previously enabled it: -

configure; web-mgmt 0 9090 9090;

thanks to this: -


I wasn't able to SSH into the box.

Thankfully, I could still attach to the serial terminal using docker attach: -

docker ps -a

CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS                                                                                                      NAMES
14cffe1113a3        ibmcom/datapower    "/bin/drouter"      9 minutes ago       Up 23 seconds       0.0.0.0:5554->5554/tcp, 0.0.0.0:8000-8010->8000-8010/tcp, 0.0.0.0:9022->9022/tcp, 0.0.0.0:9090->9090/tcp   idg

docker attach 14cffe1113a3

login: admin
Password: *****

Welcome to IBM DataPower Gateway console configuration. 
Copyright IBM Corporation 1999-2017 

Version: IDG.7.6.0.0 build 289959 on Jul 20, 2017 1:16:22 PM
Serial number: 0000001

Notice: startup config contains errors.
idg# 


I tried various things, but to no avail.

I then started wondering whether the problem might be related to the fact that I'm running the Docker container as a non-root user, and whether the underlying OS was therefore preventing me from starting a service on a port less than 1024.

For the record, this is kinda Unix 101, one cannot run any service on, say, ports 22 or 80 or 443, unless one is running as root.

It's for that reason that I always configure my web servers ( IBM HTTP Server ) to run on ports 8080 and 8443, because who wants to run a web server as root ? Clue: NOBODY

So I changed the SSH configuration: -

configure terminal
ssh 0.0.0.0 9022
write memory

and then changed my startup script from this: -

docker run -it \
   -v $PWD/config:/drouter/config \
   -v $PWD/local:/drouter/local \
   -e DATAPOWER_ACCEPT_LICENSE=true \
   -e DATAPOWER_INTERACTIVE=true \
   -p 9090:9090 \
   -p 9022:22 \
   -p 5554:5554 \
   -p 8000-8010:8000-8010 \
   --name idg \
   ibmcom/datapower


to this: -

docker run -it \
   -v $PWD/config:/drouter/config \
   -v $PWD/local:/drouter/local \
   -e DATAPOWER_ACCEPT_LICENSE=true \
   -e DATAPOWER_INTERACTIVE=true \
   -p 9090:9090 \
   -p 9022:9022 \
   -p 5554:5554 \
   -p 8000-8010:8000-8010 \
   --name idg \
   ibmcom/datapower


In other words. I'm mapping port 9022 within the container to port 9022 within the host.

Once I did this, I was able to SSH into the box: -

ssh -p 9022 localhost

14cffe1113a3
Unauthorized access prohibited.
login: 
admin
Password: *****

Welcome to IBM DataPower Gateway console configuration. 
Copyright IBM Corporation 1999-2017 

Version: IDG.7.6.0.0 build 289959 on Jul 20, 2017 1:16:22 PM
Serial number: 0000001

Notice: startup config contains errors.
idg# 

and the SSH service looks nice n' happy: -


IBM DataPower Gateway on Docker - So far, now (mostly) so good

So this is how I pulled the image: -

docker pull ibmcom/datapower

resulting in this: -

docker images

REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
ibmcom/datapower    latest              5d04824ab687        2 weeks ago         758MB


and this is how I started the container for the first time: -

docker run -it \
   -v $PWD/config:/drouter/config \
   -v $PWD/local:/drouter/local \
   -e DATAPOWER_ACCEPT_LICENSE=true \
   -e DATAPOWER_INTERACTIVE=true \
   -p 9090:9090 \
   -p 9022:22 \
   -p 5554:5554 \
   -p 8000-8010:8000-8010 \
   --name idg \
   ibmcom/datapower

resulting in this: -

docker ps -a

CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS                                                                                                    NAMES
961a7cb4adf0        ibmcom/datapower    "/bin/drouter"      About an hour ago   Up 2 minutes        0.0.0.0:5554->5554/tcp, 0.0.0.0:8000-8010->8000-8010/tcp, 0.0.0.0:9090->9090/tcp, 0.0.0.0:9022->22/tcp   idg


and this is how I start the image, having previously stopped it ( using the shutdown command ): -

docker start 961a7cb4adf0

and this is how I monitor the logs: -

docker logs 961a7cb4adf0 -f

20170815T115344.287Z [0x8040006b][system][notice] logging target(default-log): Logging started.
20170815T115344.298Z [0x804000fe][system][notice] : Container instance UUID: 69106364-39ba-4935-af9d-4012ef3b7208, Cores: 4, vCPUs: 4, CPU model: Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz, Memory: 3947.4MB, Platform: docker, OS: dpos, Edition: developers-limited, Up time: 0 minutes
20170815T115344.309Z [0x8040001c][system][notice] : DataPower IDG is on-line.
20170815T115344.310Z [0x8100006f][system][notice] : Executing default startup configuration.
20170815T115344.663Z [0x8100006d][system][notice] : Executing system configuration.
20170815T115344.665Z [0x8100006b][mgmt][notice] domain(default): tid(8175): Domain operational state is up.
961a7cb4adf0
Unauthorized access prohibited.
20170815T115346.129Z [0x806000dd][system][notice] cert-monitor(Certificate Monitor): tid(399): Enabling Certificate Monitor to scan once every 1 days for soon to expire certificates
login: 20170815T115352.235Z [0x8100006e][system][notice] : Executing startup configuration.
20170815T115352.242Z [0x8040009f][system][notice] throttle(Throttler): tid(1391): Disabling throttle.
20170815T115352.249Z [0x00350015][mgmt][notice] b2b-persistence(B2BPersistence): tid(111): Operational state down
20170815T115352.295Z [0x00350015][mgmt][notice] smtp-server-connection(default): tid(7071): Operational state down
20170815T115352.295Z [0x00350014][mgmt][notice] smtp-server-connection(default): tid(7071): Operational state up
20170815T115352.304Z [0x81000484][mgmt][warn] : Removed the object 'ProductInsights' with value of 'ProductInsights' from the startup configuration script.
20170815T115352.304Z [0x81000243][cli][error] : required property host is missing
20170815T115352.305Z [][cli][error] : (startup-config:601): exit
20170815T115403.515Z [0x00350015][mgmt][notice] quota-enforcement-server(QuotaEnforcementServer): tid(687): Operational state down
20170815T115403.521Z [0x00350014][mgmt][notice] quota-enforcement-server(QuotaEnforcementServer): tid(687): Operational state up
20170815T115404.567Z [0x00350014][mgmt][notice] web-mgmt(WebGUI-Settings): tid(303): Operational state up
20170815T115404.622Z [0x8100003b][mgmt][notice] domain(default): Domain configured successfully.


I still haven't quite cracked SSH, but that's just a matter of time :-)

I've validated the configuration re SSH: -

cat ~/config/auto-startup.cfg |grep -i ssh

acl "ssh"
ssh "0.0.0.0" "22"
metadata "ssh-password-metadata"
  meta-item "variable" "password" "var://context/INPUT/ssh/password"
  meta-item "variable" "publickey" "var://context/INPUT/ssh/publickey"
  meta-item "variable" "username" "var://context/INPUT/ssh/username"
%if% available "sshdomainclientprofile"
sshdomainclientprofile
  ciphers CHACHA20-POLY1305_AT_OPENSSH.COM
  ciphers AES128-GCM_AT_OPENSSH.COM
  ciphers AES256-GCM_AT_OPENSSH.COM
%if% available "sshserverprofile"
sshserverprofile
  ciphers CHACHA20-POLY1305_AT_OPENSSH.COM
  ciphers AES128-GCM_AT_OPENSSH.COM
  ciphers AES256-GCM_AT_OPENSSH.COM


so it'll be sorted shortly :-)

And more IBM DataPower Gateway - SSH this time around

Following this tutorial: -


I did this: -

configure terminal:
ssh 0.0.0.0 22
write memory

but saw this: -

20170815T105924.939Z [0x81000243][cli][error] : required property host is missing
20170815T105924.939Z [][cli][error] : (startup-config:601): exit


in the startup logs, and this: -

20170815T110105.223Z [0x00b30004][mgmt][error] ssh(SSH Service): tid(111): Invalid local address
20170815T110105.224Z [0x8100003f][mgmt][notice] domain(default): tid(111): Domain configuration has been modified.
20170815T110105.224Z [0x8240001b][audit][error] : tid(111): (admin:default:serial-port:(null)): ssh 'SSH Service' - Invalid local address
20170815T110105.224Z [0x00b30004][cli][error] ssh(SSH Service): Invalid local address


when I tried to start the SSH service: -

ssh start

Can you see what I was doing wrong ?

I was OVERWRITING the IP address/port combination used here: -

ssh 0.0.0.0 22

with this: -

ssh start

somehow assuming that IDG would know what I was talking about.

Can you say "Doh" ??

IBM DataPower Gateway on Docker - Remember the memory

I've been seeing a few interesting exceptions with an IBM DataPower Gateway Docker image, including: -

20170815T101929.233Z [0x81000029][system][error] : tid(8175): Execution of webgui:///clixform.xsl stopped: file:///drouter/webgui/clixform.xsl:3530: Stack overflow
20170815T101929.233Z [0x8100000f][system][critic] : tid(8175): Failed to save configuration to 'config:///auto-startup.cfg'
20170815T101929.234Z [0x8100003c][mgmt][critic] domain(default): tid(8175): Domain configuration not saved.
20170815T101929.235Z [0x810000d2][cli][error] : tid(207): Saving configuration failed - The configuration could not be parsed.

and: -

20170815T102740.943Z [0x81000029][system][error] : tid(8175): Execution of webgui:///clixform.xsl stopped: Out of memory
20170815T102740.943Z [0x8100000f][system][critic] : tid(8175): Failed to save configuration to 'config:///auto-startup.cfg'
20170815T102740.944Z [0x8100003c][mgmt][critic] domain(default): tid(8175): Domain configuration not saved.
20170815T102740.944Z [0x810000d2][cli][error] : tid(207): Saving configuration failed - The configuration could not be parsed..


and, when I started the container, using this script: -

datapower.sh 

docker run -it \
   -v $PWD/config:/drouter/config \
   -v $PWD/local:/drouter/local \
   -e DATAPOWER_ACCEPT_LICENSE=true \
   -e DATAPOWER_INTERACTIVE=true \
   -p 9090:9090 \
   -p 9022:22 \
   -p 5554:5554 \
   -p 8000-8010:8000-8010 \
   --name idg \
   ibmcom/datapower

I saw this: -

20170815T102902.256Z [0x8040006b][system][notice] logging target(default-log): Logging started.
20170815T102902.514Z [0x804000fc][system][error] : Insufficient memory allocation. Required is 4000000 KB, but configured is 2046760 KB.
20170815T102902.514Z [0x804000fe][system][notice] : Container instance UUID: 0e839292-f45e-4ca0-be61-0603482240c4, Cores: 4, vCPUs: 4, CPU model: Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz, Memory: 1998.8MB, Platform: docker, OS: dpos, Edition: developers-limited, Up time: 0 minutes
20170815T102902.531Z [0x8040001c][system][notice] : DataPower IDG is on-line.
20170815T102902.531Z [0x8100006f][system][notice] : Executing default startup configuration.
20170815T102902.918Z [0x8100006d][system][notice] : Executing system configuration.
20170815T102902.922Z [0x8100006b][mgmt][notice] domain(default): tid(8175): Domain operational state is up.
3da718e4a927
Unauthorized access prohibited.
20170815T102906.583Z [0x806000dd][system][notice] cert-monitor(Certificate Monitor): tid(399): Enabling Certificate Monitor to scan once every 1 days for soon to expire certificates
login: 20170815T102912.809Z [0x8100006e][system][notice] : Executing startup configuration.
20170815T102912.819Z [0x8040009f][system][notice] throttle(Throttler): tid(1391): Disabling throttle.
20170815T102912.828Z [0x00350015][mgmt][notice] b2b-persistence(B2BPersistence): tid(111): Operational state down
20170815T102912.850Z [0x0034000d][mgmt][warn] ssh(SSH Service): tid(111): Object is disabled
20170815T102912.879Z [0x00350015][mgmt][notice] smtp-server-connection(default): tid(7071): Operational state down
20170815T102912.879Z [0x00350014][mgmt][notice] smtp-server-connection(default): tid(7071): Operational state up
20170815T102924.506Z [0x00350015][mgmt][notice] quota-enforcement-server(QuotaEnforcementServer): tid(687): Operational state down
20170815T102924.518Z [0x00350014][mgmt][notice] quota-enforcement-server(QuotaEnforcementServer): tid(687): Operational state up
20170815T102924.560Z [0x81000001][cli][error] : *** Unknown command or macro
20170815T102924.560Z [0x81000223][cli][error] : (startup-config:632): save-config overwrite
20170815T102924.566Z [0x00350014][mgmt][notice] web-mgmt(WebGUI-Settings): tid(303): Operational state up
20170815T102924.618Z [0x8100003b][mgmt][notice] domain(default): Domain configured successfully.
20170815T102957.720Z [0x80e0047a][system][error] : tid(49): DataPower QuotaEnforcement task is not responding, restart in progress
20170815T103009.550Z [0x80e0047a][system][error] : tid(65): DataPower QuotaEnforcement task is not responding, restart in progress


This: -


gave me a clue as someone else had the same problem, and was advised to increase the default memory from 2 GB to 4 GB, as follows: -



I did this, and restarted the container: -

20170815T104708.843Z [0x8040006b][system][notice] logging target(default-log): Logging started.
20170815T104709.106Z [0x804000fe][system][notice] : Container instance UUID: a2c12305-548b-4133-98b7-b7323d224fc6, Cores: 4, vCPUs: 4, CPU model: Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz, Memory: 3947.4MB, Platform: docker, OS: dpos, Edition: developers-limited, Up time: 0 minutes
20170815T104709.129Z [0x8040001c][system][notice] : DataPower IDG is on-line.
20170815T104709.130Z [0x8100006f][system][notice] : Executing default startup configuration.
20170815T104709.554Z [0x8100006d][system][notice] : Executing system configuration.
20170815T104709.557Z [0x8100006b][mgmt][notice] domain(default): tid(8175): Domain operational state is up.
1dc74c14aa4f
Unauthorized access prohibited.
20170815T104711.551Z [0x806000dd][system][notice] cert-monitor(Certificate Monitor): tid(399): Enabling Certificate Monitor to scan once every 1 days for soon to expire certificates
login: 20170815T104717.650Z [0x8100006e][system][notice] : Executing startup configuration.
20170815T104717.654Z [0x8040009f][system][notice] throttle(Throttler): tid(1391): Disabling throttle.
20170815T104717.659Z [0x00350015][mgmt][notice] b2b-persistence(B2BPersistence): tid(111): Operational state down
20170815T104717.674Z [0x0034000d][mgmt][warn] ssh(SSH Service): tid(111): Object is disabled
20170815T104717.697Z [0x00350015][mgmt][notice] smtp-server-connection(default): tid(7071): Operational state down
20170815T104717.697Z [0x00350014][mgmt][notice] smtp-server-connection(default): tid(7071): Operational state up
20170815T104729.921Z [0x00350015][mgmt][notice] quota-enforcement-server(QuotaEnforcementServer): tid(687): Operational state down
20170815T104729.927Z [0x00350014][mgmt][notice] quota-enforcement-server(QuotaEnforcementServer): tid(687): Operational state up
20170815T104729.968Z [0x81000001][cli][error] : *** Unknown command or macro
20170815T104729.968Z [0x81000223][cli][error] : (startup-config:632): save-config overwrite
20170815T104729.971Z [0x00350014][mgmt][notice] web-mgmt(WebGUI-Settings): tid(303): Operational state up
20170815T104730.030Z [0x8100003b][mgmt][notice] domain(default): Domain configured successfully.



Even better news … I can now log into the Web Management GUI: -


with the appropriate credentials, whereas I was being blocked before ( even though the same credentials worked on the serial interface ).

Monday, 14 August 2017

IBM API Connect - still learning how to say "Doh!"

I saw this: -


when testing my API, using the local API Connect Toolkit ( apic edit ), even though the same API worked happily on IBM Bluemix.

The suggested URL: -


didn't work, and there was nothing listening on port 4001 on my Mac.

Then I noticed this: -


I clicked on the Play icon, which gave me this: -


further validated thusly: -

netstat -an | grep LISTEN

tcp46      0      0  *.4001                 *.*                    LISTEN     

and now I get this: -


and, more importantly, this: -


so #LifeIsGood 

Thursday, 3 August 2017

IBM BPM - Why won't my Coach Views show up in my Coach ?

As stated previously: -



I'm tinkering with IBM BPM, learning more about the new Responsive Coach Framework in IBM BPM 8.5.7, and the Client-SIde Human Service capabilities.

Having created an External Service and a Service Flow, I then want to create a UI to allow the user to enter their name, and get back a response.

So I've created a Coach View, and marked it as a Template ( for reuse ): -


and then created a pair of Coach Views, one for the input and one for the output.

Having done this, I created a new Client-Side Human Service, and added a pair of coaches: -


However, when I went into each of the two Coaches, via the Coaches tab within the CSHS page, I did NOT see the Coach Views in the palette on the RHS: -



It took me a while to work out why …

When I'd first created the CSHS, I'd ticked the Intended for use on multiple devices checkbox: -

and, as you can guess, when I created the template from which I created the two Coach Views, I did NOT do the same :-(

I went back into the Template, and into the two Coach Views, and suitably checked the appropriate box: -

and now they both appear in the right place: -


which is nice.

IBM BPM and IBM ODM Rules - I continue to learn ...

Following my earlier post: -


I'm continuing to tinker with IBM BPM, specifically using the web-based Process Designer, to integrate with a very simple Decision Service, called, of course, Hello World.

For the record, I'm using IBM BPM on Cloud ( a SaaS offering ) against ODM Rules on Bluemix ( a PaaS offering ).

Having created an External Service, using the YAML that I exported from the Decision Server ( as per the previous post ), I've then created a Service Flow, into which I've dragged the External Service.

I've then bound the External Service to the Service Flow: -


note that I've specified the Operation.

This time around, rather than manually creating Variables within the Service Flow, I chose to have Process Designer do it for me, via the Auto-Map function 

This is with what I ended up: -


However, when I tested the Service Flow by hitting the Play button: -


I saw this: -

CWTBG0019E: Unexpected exception during execution. Exception information: 'com.lombardisoftware.component.common.workflow.WorkflowProcessItemException: CWXSD1028E: The operation callDecisionServiceOperation with HTTP method POST returned an unexpected response status code 500 with response message Internal Server Error'.

So I then tried the Debug button instead: 


and saw this: -


which is where I realised the error of my ways ….

Note that the Request, which has two parts, as per the downstream Decision Service, is EMPTY.

Can you say "Doofus" ? I bet you can ….

Once I added a default ( this is for testing, after all ) value: -


it just worked, returning: -

Service completed

Now to create my Coaches, Templates and Coach Views …...