Build REST APIs using Swagger and IBM Integration Bus – IIB v10

I've been teaching myself more about IBM Integration Bus (IIB), in the context of creating RESTful APIs.

This tutorial was immensely useful: -

Build REST APIs using Swagger and IBM Integration Bus – IIB v10

and I can continue from what it's taught me ….

Munging data - removing duplicates from CSV files

Whilst fiddling with Cloudant yesterday: -

Cloudant - Continuing to tinker

I hit an issue whereby I was trying / failing to upload data that contained duplicates: -

index,name
1,Dave
2,Bob
3,Barney
4,Homer
5,Bart
1,Dave
3,Barney

Note that this is an example file; the real data had 000s of rows :-(

cat duplo.csv | couchimport --transform transform_duplo.js 

  couchimport ****************** +0ms
  couchimport configuration +2ms
  couchimport {
  couchimport  "COUCH_URL": "https://****:****@7fb7794a-fd6f-47a5-a770-a3521ef51df0-bluemix.cloudant.com",
  couchimport  "COUCH_DATABASE": "duplo",
  couchimport  "COUCH_DELIMITER": ",",
  couchimport  "COUCH_FILETYPE": "text",
  couchimport  "COUCH_BUFFER_SIZE": 500,
  couchimport  "COUCH_JSON_PATH": null,
  couchimport  "COUCH_META": null,
  couchimport  "COUCH_PARALLELISM": 1,
  couchimport  "COUCH_PREVIEW": false,
  couchimport  "COUCH_IGNORE_FIELDS": []
  couchimport } +2ms
  couchimport ****************** +0ms
  couchimport { id: '1',
  couchimport   error: 'conflict',
  couchimport   reason: 'Document update conflict.' } +561ms
  couchimport { id: '3',
  couchimport   error: 'conflict',
  couchimport   reason: 'Document update conflict.' } +2ms
  couchimport Written ok:5 - failed: 2 -  (5) +0ms
  couchimport { documents: 5, failed: 2, total: 5, totalfailed: 2 } +0ms
  couchimport writecomplete { total: 5, totalfailed: 2 } +81ms
  couchimport Import complete +0ms

I wanted to strip out the duplicates ( of which there were MANY )

Thankfully the internet showed me how: -

Bash commands/script to remove a line from CSV with duplicate in column

So I did this: -

awk -F, '!seen[$1]++' duplo.csv > duplo_DEDUP.csv

which gave me this: -

index,name
1,Dave
2,Bob
3,Barney
4,Homer
5,Bart

cat duplo_DEDUP.csv | couchimport --transform transform_duplo.js 

  couchimport ****************** +0ms
  couchimport configuration +2ms
  couchimport {
  couchimport  "COUCH_URL": "https://****:****@7fb7794a-fd6f-47a5-a770-a3521ef51df0-bluemix.cloudant.com",
  couchimport  "COUCH_DATABASE": "duplo",
  couchimport  "COUCH_DELIMITER": ",",
  couchimport  "COUCH_FILETYPE": "text",
  couchimport  "COUCH_BUFFER_SIZE": 500,
  couchimport  "COUCH_JSON_PATH": null,
  couchimport  "COUCH_META": null,
  couchimport  "COUCH_PARALLELISM": 1,
  couchimport  "COUCH_PREVIEW": false,
  couchimport  "COUCH_IGNORE_FIELDS": []
  couchimport } +1ms
  couchimport ****************** +1ms
  couchimport Written ok:5 - failed: 0 -  (5) +489ms
  couchimport { documents: 5, failed: 0, total: 5, totalfailed: 0 } +1ms
  couchimport writecomplete { total: 5, totalfailed: 0 } +40ms
  couchimport Import complete +1ms

which is nice :-)

Yay for awk !

macOS - Windows are off the screen ...

I had an issue earlier where my chosen Twitter client, Tweetbot, somehow wandered off the screen, lurking off to the right.

Whilst I could see Tweetbot when I hit the F4 key to open Mission Control, I couldn't actually get to it :-( 

Thankfully the internet ( what, all of it ? ) had the answer: -

…

OS X, particularly recent versions of the operating system, do a good job of corralling application windows by either not allowing a user to resize a window beyond the boundaries of the screen or by automatically snapping a window to a second display for those with multi-monitor setups. But sometimes — due to errors, bugs, or when disconnecting an external monitor — an application window can get "stuck" partially or completely outside of the visible area of the Mac's display, and getting it back can seem impossible. Thankfully, there's a quick and easy step you can take to automatically fix an off screen window in Mac OS X, and it's called Zoom.

…

If you can see the green zoom button, it's the best way to bring the missing portions of your OS X application window back into view. But what if it's the top of the window that's off screen, and you can't see the zoom button at all? In that case, you can achieve the same result via an option in the menu bar.

Simply select your desired application to make it active by click on its icon in the Dock (you should see the application's name in the top-left corner of your OS X Menu Bar, next to the Apple logo). Then, also in the Menu Bar, click the word Window and then Zoom. If you have multiple windows open in the same application, you can also select Zoom All to bring them all to the correct position at once.

…

How to Resize an Off Screen Window in Mac OS X

Bottom line, I hit the Tweetbot icon in the Dock

and then chose Zoom from the Window menu

Now the app is back where it should be

Nice :-)

Cloudant - Continuing to tinker

I'm importing data from a Comma Separated Value (CSV) file into Cloudant, using the most excellent CouchDB tools provided by my IBM colleague, Glynn Bird.

Having created a CSV: -

vi cartoon.csv 

id,givenName,familyName
1,Maggie,Simpson
2,Lisa,Simpson
3,Bart,Simpson
4,Homer,Simpson
5,Fred,Flintstone
6,Wilma,Flintstone
7,Barney,Rubble
8,Betty,Rubble

( with due respect to the creators and owners of The Simpsons and The Flintstones )

I setup my environment: -

export ACCOUNT=0e5c777542c5e2cc2418013429e0824f-bluemix:d088ff753c9e258add92e45128cd161d
acbffedbcec0c8f78b216368ba0503ab

export HOST=d088ff753c9e258add92e45128cd161d-bluemix.cloudant.com

export COUCH_URL=https://$ACCOUNT@$HOST

export COUCH_DATABASE="CARTOON"

export COUCH_DATABASE=`echo $COUCH_DATABASE | tr '[:upper:]' '[:lower:]'`

export COUCH_DELIMITER=","

and created my database: -

curl -X PUT $COUCH_URL/$COUCH_DATABASE

and populated it: -

cat $COUCH_DATABASE.csv | couchimport

This worked well but …. my data had a system-generated _id field whereas I wanted to use my own ID field: -

{
  "_id": "e143bcd25bc620e6aa8f2adc206cf21c",
  "_rev": "1-0152a3e6867ad34da6e882a80f0fbeff",
  "id": "1",
  "givenName": "Maggie",
  "familyName": "Simpson"
}

{
  "_id": "82c1068c830759a904cfdd02ab41b980",
  "_rev": "1-6bbb94301323a3c3f6ff54f1c3c765e5",
  "id": "2",
  "givenName": "Lisa",
  "familyName": "Simpson"
}

Thankfully Glenn kindly advised me how to use a JavaScript function to mitigate this: -

vi ~/transform_cartoon.js

var transform = function(doc) {

  doc._id = doc.id
  delete doc.id
  return doc
}

module.exports = transform

which effectively assigns the _id field to the value of the id field ( as taken from the CSV ) and also drops the original id field.

I dropped the DB: -

curl -X DELETE $COUCH_URL/$COUCH_DATABASE

and recreated it: -

curl -X PUT $COUCH_URL/$COUCH_DATABASE

and then repopulated it: -

cat $COUCH_DATABASE.csv | couchimport --transform ~/transform_cartoon.js

and now we have this: -

{
  "_id": "1",
  "_rev": "1-0e77dbadefba2a95e5cde5bda2ecd695",
  "givenName": "Maggie",
  "familyName": "Simpson"

}

{
  "_id": "2",
  "_rev": "1-fc746edc394ac98b013b7788cc1cca5d",
  "givenName": "Lisa",
  "familyName": "Simpson"
}

If needed, I could modify my transform: -

var transform = function(doc) {
  doc._id = doc.id
  return doc
}

module.exports = transform

to avoid dropping the original id field, to give me this: -

{
  "_id": "1",
  "_rev": "1-0152a3e6867ad34da6e882a80f0fbeff",
  "id": "1",
  "givenName": "Maggie",
  "familyName": "Simpson"
}

{
  "_id": "2",
  "_rev": "1-6bbb94301323a3c3f6ff54f1c3c765e5",
  "id": "2",
  "givenName": "Lisa",
  "familyName": "Simpson"
}

so I have choices :-) 

For more insights, please go here: -

couchimport

Moving data from DocumentDB to Cloudant or CouchDB

Couchimport Revisited

IBM Integration Bus and Cloudant - Baby steps ...

I'm starting to explore the possibilities of integration between IBM Integration Bus ( and thus IBM AppConnect Enterprise ) and IBM Cloudant, both of which are running on the IBM Cloud Platform ( nee Bluemix ).

Having spun up an instance of Cloudant, and worked out how to "feed" it using RESTful APIs, via cURL, I now wanted to find out how one can achieve integration between IIB and Cloudant.

This was of immense help: -

IBM Integration Bus v10 tutorials on Github

https://developer.ibm.com/integration/resources/ibm-integration-bus-v10-tutorials-on-github/

specifically this tutorial: -

Using a LoopBack Request node to insert data into a Cloudant database

and this: -

Using some of the more advanced features of the LoopBackRequest node

https://developer.ibm.com/integration/blog/2016/09/13/using-some-of-the-more-advanced-features-of-the-loopbackrequest-node/

Tutorial: Installing LoopBack connectors

https://www.ibm.com/support/knowledgecenter/en/SSFS6T/com.ibm.apic.toolkit.doc/tapim-connector-install.html

IIB V10.0.0.6 Loopback Request node using MongoDB and Cloudant tutorial

https://developer.ibm.com/integration/blog/2016/10/13/iib-v10-0-0-6-loopback-request-node-using-mongodb-and-cloudant-tutorial/

One of the key requirements is to install a Loopback Connector for IBM Cloudant, as per this exception: -

BIP2087E: Integration node 'TESTNODE_Dave' was unable to process the internal configuration message. 

The entire internal configuration message failed to be processed successfully. 

Use the messages following this message to determine the reasons for the failure. If the problem cannot be resolved after reviewing these messages, contact your IBM Support center. Enabling service trace may help determine the cause of the failure.
BIP4041E: Integration server 'default' received an administration request that encountered an exception. 

While attempting to process an administration request, an exception was encountered. No updates have been made to the configuration of the integration server. 

Review related error messages to determine why the administration request failed.
BIP3879E: The LoopBackRequest node received an error from LoopBack when attempting to connect to the data source name 'CLOUDANT'. Detail: ' WARNING: LoopBack connector "cloudant" is not installed as any of the following modules:   ./connectors/cloudant loopback-connector-cloudant  To fix, run:      npm install loopback-connector-cloudant --save '. 

An error was received when establishing the connection to the configured LoopBack connector data source. 

Check the error information to determine why the error occurred and take the appropriate action to resolve the error. The error detail is a LoopBack connector error message.
BIP2871I: The request made by user 'Dave-PC\Dave' to 'change' the resource '/LoopBack/Loopback_Cloudant' of type 'MessageFlow' on parent 'default' of type 'ExecutionGroup' has the status of 'FAILED'.

Having downloaded/installed NodeJS ( node-v8.11.2-x64.exe ) on the Windows VM, I tried/failed to work out how to resolve the missing dependency.

This gave me the clue: -

Preparing the Integration Node runtime environment to connect to Cloudant

https://developer.ibm.com/integration/blog/2016/09/13/using-some-of-the-more-advanced-features-of-the-loopbackrequest-node/

Namely, I needed to navigate to the appropriate directory under the IIB Toolkit: -

cd C:\ProgramData\IBM\MQSI\node_modules

notepad C:\ProgramData\IBM\MQSI\package.json

{

  "name": "system32",

  "version": "1.0.0",

  "description": "",

  "main": "index.js",

  "dependencies": {

    "loopback-connector-cloudant": "^2.0.5"

  },

  "devDependencies": {},

  "scripts": {

    "test": "echo \"Error: no test specified\" && exit 1"

  },

  "author": "",

  "license": "ISC"

}

npm install loopback-connector-cloudant --save

and, for good belt n' braces reasons, restart the Integration Node.

Once I did this, I was able to run the flow using the Flow Exerciser: -

and insert data into Cloudant using the Create operation: -

or retrieve data from Cloudant using the Retrieve operation: -

which is nice :-)

Final consideration, I needed to create a datasources.json configuration file here: -

C:\ProgramData\IBM\MQSI\connectors\loopback

{

"CLOUDANT":
{
"name": "bluey",
"connector": "cloudant",
"username": "2ha2294a-fd6f-42a5-a220-a4221ef51df0-bluemix",
"password": "5322e20e538422a92f2eaca69db094883125cdfa4db28c20dd82bc3662161108",
"url": "https://2ha2294a-fd6f-42a5-a220-a4221ef51df0-bluemix:5322e20e538422a92f2eaca69db094883125cdfa4db28c20dd82bc3662161108@2ha2294a-fd6f-42a5-a220-a4221ef51df0-bluemix.cloudant.com",
"database": "bluey"
}
}

Thinking about a multitier architecture on IBM Cloud Private?

There's a useful pair of articles pertaining to IBM Cloud Private: -

Thinking about a multitier architecture on IBM Cloud Private?

Decision service lifecycle on IBM Cloud Private

which may be of relevance to a client with whom I'm currently engaged …..