Thursday, 9 September 2021

And there's more - munging base64 in JSON for etcd

Following on from my earlier post: -

etcd - Today I learned ...

I dug into jq more, and found this: -

base64 decoding function #47

specifically this comment: -

As this is showing up on Google a lot, and good documentation on jq is sparse, here is to everybody who lands here:


echo '{"foo": "Ym9iIGxpa2VzIGFsaWNlCg=="}' | jq '.foo | @base64d'

Or even use it when building new objects:

echo '{"foo": "Ym9iIGxpa2VzIGFsaWNlCg=="}' | jq '{encoded: .foo, decoded: .foo | @base64d}'

dating back to 2018.

This led me to a neat-o mechanism to encode the key and value of my JSON document: -

cat dave.json

"value": {
"name":"Dave Hay",

jq -r '.[] |= @base64' dave.json

  "key": "MDEyMzQ1",
  "value": "eyJuYW1lIjoiRGF2ZSBIYXkiLCJpZCI6ImRhdmVoYXkxOTY5In0="

I then used this to generate a new JSON document: -

jq -r '.[] |= @base64' dave.json > dave_encoded.json

which I then fed into etcd: -

curl -X POST --silent --cacert /root/ssl/ca-cert.pem --cert /root/ssl/client-cert.pem --key /root/ssl/client-key.pem https://localhost:2379/v3/kv/put -d @dave_encoded.json | jq

  "header": {
    "cluster_id": "14841639068965178418",
    "member_id": "10276657743932975437",
    "revision": "2",
    "raft_term": "2"

and then confirmed that I could pull the data back out: -

curl -X POST --silent --cacert /root/ssl/ca-cert.pem --cert /root/ssl/client-cert.pem --key /root/ssl/client-key.pem https://localhost:2379/v3/kv/range -d '{"key":"MDEyMzQ1"}' | jq -r .kvs[].value | base64 -d | jq


  "name": "Dave Hay",
  "id": "davehay1969"

which is nice

See, I told you I'd find a way .....

No comments:

Yay, VMware Fusion and macOS Big Sur - no longer "NAT good friends" - forgive the double negative and the terrible pun ...

After macOS 11 Big Sur was released in 2020, VMware updated their Fusion product to v12 and, sadly, managed to break Network Address Trans...