Yet more fun and goodness with Cloudant and couchimport

 One of my friends was wondering why couchimport was apparently working BUT not actually working ....

Running a test such as: -

cat cartoon.csv | couchimport --url https://<<SECRET>>-bluemix.cloudantnosqldb.appdomain.cloud --database cartoon

couchimport
-----------
 url         : "https://<<SECRET>>-bluemix.cloudantnosqldb.appdomain.cloud"
 database    : "cartoon"
 delimiter   : "\t"
 buffer      : 500
 parallelism : 1
 type        : "text"
-----------
  couchimport {"documents":0,"failed":8,"total":0,"totalfailed":8,"statusCodes":{"401":1},"latency":475} +0ms
  couchimport Import complete +0ms

In other words, it does something but reports failed:8

I had to dig back into my memory AND into the docs to work out what was going on....

Specifically this: -

https://www.npmjs.com/package/couchimport#iam-authentication

So, it's a case of "If your name's not down, you're not coming in ....

If Cloudant or CouchDB ( from whence Cloudant came ) was running elsewhere, we could specify user/password credentials but, given that it's running as SaaS on the IBM Cloud, we need a better way ....

Once I realised ( remembered ) that, we were golden....

In essence, the "key" ( do you see what I did there? ) thing is to set an environment variable with an IBM Cloud API key: -

export IAM_API_KEY="<<TOP SECRET>>"

Here's the end-to-end walkthrough : -

Create data to be imported

vi cartoon.csv

id,givenName,familyName

1,Maggie,Simpson

2,Lisa,Simpson

3,Bart,Simpson

4,Homer,Simpson

5,Fred,Flintstone

6,Wilma,Flintstone

7,Barney,Rubble

8,Betty,Rubble

Set environment variables

export COUCH_URL="https://<<TOP SECRET>>-bluemix.cloudantnosqldb.appdomain.cloud"

export IAM_API_KEY="<<TOP SECRET>>"

export COUCH_DATABASE="cartoon"

export COUCH_DELIMITER=","

Generate Access Token

- This is a script that generates an ACCESS_TOKEN variable for my IBM Cloud API key

source ~/genAccessToken.sh

Create database

curl -s -k -X PUT -H 'Authorization: Bearer '"$ACCESS_TOKEN" $COUCH_URL/$COUCH_DATABASE | json_pp

{

   "ok" : true

}

Populate database

cat cartoon.csv | couchimport

couchimport

-----------

 url         : "https://<<SECRET>>-bluemix.cloudantnosqldb.appdomain.cloud"

 database    : "cartoon"

 delimiter   : ","

 buffer      : 500

 parallelism : 1

 type        : "text"

-----------

  couchimport {"documents":8,"failed":0,"total":8,"totalfailed":0,"statusCodes":{"201":1},"latency":381} +0ms

  couchimport Import complete +0ms

Create index

curl -s -k -X POST -H 'Authorization: Bearer '"$ACCESS_TOKEN" -H 'Content-type: application/json' $COUCH_URL/$COUCH_DATABASE/_index -d '{

   "index": {

      "fields": [

         "givenName"

      ]

   },

   "name": "givenName-json-index",

   "type": "json"

}'

Query database

curl -s -k -X POST -H 'Authorization: Bearer '"$ACCESS_TOKEN" -H 'Content-type: application/json' $COUCH_URL/$COUCH_DATABASE/_find -d '{

   "selector": {

      "$or": [

         {

            "givenName": "Maggie"

         },

         {

            "givenName": "Lisa"

         }

      ]

   },

   "fields": [

      "givenName",

      "familyName"

   ],

   "sort": [

      {

         "givenName": "asc"

      }

   ]

}'  | json_pp

{

   "bookmark" : "g2wAAAACaAJkAA5zdGFydGtleV9kb2NpZG0AAAAgNGI5YWZhMzZjNTBiNTg4ZTljMWFmMzUxZjQyNzViMGNoAmQACHN0YXJ0a2V5bAAAAAFtAAAABk1hZ2dpZWpq",

   "docs" : [

      {

         "givenName" : "Lisa",

         "familyName" : "Simpson"

      },

      {

         "givenName" : "Maggie",

         "familyName" : "Simpson"

      }

   ]

}

Can you say "Yay" ? I bet you can .....