Friday, 13 November 2020

Yet more fun and goodness with Cloudant and couchimport

 One of my friends was wondering why couchimport was apparently working BUT not actually working ....

Running a test such as: -

cat cartoon.csv | couchimport --url https://<<SECRET>>-bluemix.cloudantnosqldb.appdomain.cloud --database cartoon

couchimport
-----------
 url         : "https://<<SECRET>>-bluemix.cloudantnosqldb.appdomain.cloud"
 database    : "cartoon"
 delimiter   : "\t"
 buffer      : 500
 parallelism : 1
 type        : "text"
-----------
  couchimport {"documents":0,"failed":8,"total":0,"totalfailed":8,"statusCodes":{"401":1},"latency":475} +0ms
  couchimport Import complete +0ms

In other words, it does something but reports failed:8

I had to dig back into my memory AND into the docs to work out what was going on....

Specifically this: -



So, it's a case of "If your name's not down, you're not coming in ....

If Cloudant or CouchDB ( from whence Cloudant came ) was running elsewhere, we could specify user/password credentials but, given that it's running as SaaS on the IBM Cloud, we need a better way ....

Once I realised ( remembered ) that, we were golden....

In essence, the "key" ( do you see what I did there? ) thing is to set an environment variable with an IBM Cloud API key: -

export IAM_API_KEY="<<TOP SECRET>>"

Here's the end-to-end walkthrough : -

Create data to be imported

vi cartoon.csv

id,givenName,familyName
1,Maggie,Simpson
2,Lisa,Simpson
3,Bart,Simpson
4,Homer,Simpson
5,Fred,Flintstone
6,Wilma,Flintstone
7,Barney,Rubble
8,Betty,Rubble

Set environment variables

export COUCH_URL="https://<<TOP SECRET>>-bluemix.cloudantnosqldb.appdomain.cloud"
export IAM_API_KEY="<<TOP SECRET>>"
export COUCH_DATABASE="cartoon"
export COUCH_DELIMITER=","

Generate Access Token

- This is a script that generates an ACCESS_TOKEN variable for my IBM Cloud API key

source ~/genAccessToken.sh

Create database

curl -s -k -X PUT -H 'Authorization: Bearer '"$ACCESS_TOKEN" $COUCH_URL/$COUCH_DATABASE | json_pp

{
   "ok" : true
}

Populate database

cat cartoon.csv | couchimport

couchimport
-----------
 url         : "https://<<SECRET>>-bluemix.cloudantnosqldb.appdomain.cloud"
 database    : "cartoon"
 delimiter   : ","
 buffer      : 500
 parallelism : 1
 type        : "text"
-----------
  couchimport {"documents":8,"failed":0,"total":8,"totalfailed":0,"statusCodes":{"201":1},"latency":381} +0ms
  couchimport Import complete +0ms

Create index

curl -s -k -X POST -H 'Authorization: Bearer '"$ACCESS_TOKEN" -H 'Content-type: application/json' $COUCH_URL/$COUCH_DATABASE/_index -d '{
   "index": {
      "fields": [
         "givenName"
      ]
   },
   "name": "givenName-json-index",
   "type": "json"
}'

Query database

curl -s -k -X POST -H 'Authorization: Bearer '"$ACCESS_TOKEN" -H 'Content-type: application/json' $COUCH_URL/$COUCH_DATABASE/_find -d '{
   "selector": {
      "$or": [
         {
            "givenName": "Maggie"
         },
         {
            "givenName": "Lisa"
         }
      ]
   },
   "fields": [
      "givenName",
      "familyName"
   ],
   "sort": [
      {
         "givenName": "asc"
      }
   ]
}'  | json_pp

{
   "bookmark" : "g2wAAAACaAJkAA5zdGFydGtleV9kb2NpZG0AAAAgNGI5YWZhMzZjNTBiNTg4ZTljMWFmMzUxZjQyNzViMGNoAmQACHN0YXJ0a2V5bAAAAAFtAAAABk1hZ2dpZWpq",
   "docs" : [
      {
         "givenName" : "Lisa",
         "familyName" : "Simpson"
      },
      {
         "givenName" : "Maggie",
         "familyName" : "Simpson"
      }
   ]
}

Can you say "Yay" ? I bet you can .....

No comments:

Yay, VMware Fusion and macOS Big Sur - no longer "NAT good friends" - forgive the double negative and the terrible pun ...

After macOS 11 Big Sur was released in 2020, VMware updated their Fusion product to v12 and, sadly, managed to break Network Address Trans...