Note to self - using the CRI tool - crictl - to clean up unready pods

 Purely 'cos I know I'll need this again: -

for i in `crictl pods | grep NotReady | awk '{print $1}'`; do crictl rmp $i; done

which is for use when I've got a bunch of NotReady pods waiting to be nuked.

More fun with Docker and Homebrew - authentication this time

Following on from my earlier post: -

Homebrew on macOS - Docker says "No" - well, kinda 

I hit a related problem post the installation of Minikube etc.

When I tried to log into Docker Hub: -

docker login -u davidhay1969

I saw this: -

Password: 

Error saving credentials: error storing credentials - err: exec: "docker-credential-desktop": executable file not found in $PATH, out: ``

This post: -

docker-credential-desktop not installed or not available in PATH

gave me the "solution" - it was another hangover from having Docker Desktop installed, namely this file: -

~/.docker/config.json

For whatever good reason, it was necessary to edit it, and replace credsStore with credStore. 

Once I did this, all was well ...

And, of course, docker login completely rewrote that file, removing any reference to either credsStore or credStore ...

Again... HANGOVER!

Homebrew on macOS - Docker says "No" - well, kinda

Whilst helping out a friend, I was running through a process to get Docker running without Docker Desktop, as per this: -

Run Docker without Docker Desktop on macOS

which makes heavy use of Homebrew.

This has one install stuff such as hyperkit and minikube : -

brew install hyperkit

brew install minikube

which is nice.

However, I was seeing interesting results from Homebrew in general ...

As an example when I ran brew upgrade or brew install hyperkit I was seeing errors such as: -

Error: Permission denied @ apply2files - /usr/local/lib/docker/cli-plugins

and: -

==> Pouring kubernetes-cli--1.22.1.big_sur.bottle.tar.gz

Error: The `brew link` step did not complete successfully

The formula built, but is not symlinked into /usr/local

Could not symlink bin/kubectl

Target /usr/local/bin/kubectl

already exists. You may want to remove it:

  rm '/usr/local/bin/kubectl'

This on macOS 11.6 on my good ole 2014 Mac mini ...

The solution was relatively simple ...

At some point in the past, I'd had Docker Desktop installed, which was obviously running as root via sudo etc.

Therefore, having removed Docker Desktop, I needed to clean up the permissions : -

sudo chown -R davidhay:admin /usr/local/lib

Once I did this, all was well ....

Aide Memoire - Git and SSH rather than HTTPS

 So I'm tinkering with a GitHub project and, having cloned it, was trying to build it using the Makefile: -

cd etcd-operator/

make

which fairly quickly tried/failed to pull in a submodule: -

Cleanup submodule go.mod files
rm -f ./shared-logger/go.mod
Generate submodule go.mod files
git submodule update --init --recursive
Submodule 'shared-logger' (https://github.com/genctl/shared-logger.git) registered for path 'shared-logger'
Cloning into '/root/etcd-operator/shared-logger'...
Username for 'https://github.com': 

Now I have hit this many times before ...

And there is a solution ....

But I couldn't quite remember it ...

Thankfully, Google had the answer, Google is my friend: -

Go: Make error: unable to update repository: fatal: could not read Username for ‘https://github.ibm.com’: terminal prompts disabled

The solution was to update my Git config: -

git config --global url."git@github.com:".insteadOf "https://github.com"

In other words, Git needs to know to use my SSH private key for authentication, rather than the more default HTTPS URL.

Simples!

TIL - building bash using cat and tee and EOF

So I've started to make more use of cat and tee and the EOF symbol when creating documentation describing how one can create various different artefacts from, say, Bash on Linux and macOS.

Typically, I'm focusing upon configuration files e.g. files with .conf and .yaml but, a few days back, I wanted to create a Bash script.

Here's an example of what I'd do for a containerd.conf configuration file: -

cat <<EOF | tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF

Nice and simple, right ?

I then tried the same for a Bash script, here's a trivial example that munges a JSON document - me.json : -

{
    "name": "Dave Hay",
    "eddress": "david_hay@uk.ibm.com",
    "github": "davidhay1969"
}

called ./hello.sh created thusly: -

cat << EOF | tee hello.sh
#!/bin/bash
name=$(cat ~/me.json | jq -r .name)
echo "Hello, "$name
EOF

So, when I paste that into a macOS Terminal session, using /bin/bash this is what I see: -

cat << EOF | tee hello.sh

> #!/bin/bash

> name=$(cat ~/me.json | jq -r .name)

> echo "Hello, "$name

> EOF

#!/bin/bash

name=Dave Hay

echo "Hello, "

with the resulting script looking a bit ... weird ...

cat hello.sh

#!/bin/bash

name=Dave Hay

echo "Hello, "

Having made the script executable: -

chmod +x hello.sh

it doesn't run too well: -

./hello.sh

./hello.sh: line 2: Hay: command not found

Hello,

Thankfully, when I hit the original issue a few days back, something I read ( and, alas, I didn't grab the actual source ), the trick is to wrap the first EOF in single quotes, like this: -

cat << 'EOF' | tee hello.sh

#!/bin/bash

name=$(cat ~/me.json | jq -r .name)

echo "Hello, "$name

EOF

This is what happens when I paste the above into the same Terminal Bash session: -

cat << 'EOF' | tee hello.sh

> #!/bin/bash

> name=$(cat ~/me.json | jq -r .name)

> echo "Hello, "$name

> EOF

#!/bin/bash

name=$(cat ~/me.json | jq -r .name)

echo "Hello, "$name

and the corresponding script looks OK: -

cat hello.sh

#!/bin/bash

name=$(cat ~/me.json | jq -r .name)

echo "Hello, "$name

and, even better, it actually works: -

./hello.sh

Hello, Dave Hay

Can you say "Yay" ?

etcd - base64 isn't the only way

Having written: -

etcd - Today I learned ... 

And there's more - munging base64 in JSON for etcd

I forgot to mention that base64 isn't the only way to write to / read from etcd ...

The encoding is required where we choose to use the gRPC route to etcd via its REST APIs : -

Why gRPC gateway

etcd3 API

v3.5 docs

However, the etcdctl tool does NOT need to use base64 encoding.

Building upon my previous example, where I used the following command: -

jq -r '.[] |= @base64' dave.json > dave_encoded.json

to encode the key and value elements of a JSON document - dave.json - which looks like this: -

{
"key":"012345",
"value": {
"name":"Dave Hay",
"id":"davehay1969"
}
} 

and produce an alternate version - dave_encoded.json - with the base64 encoded elements therein, and then fed etcd using that encoded document: -

curl -X POST --silent --cacert /root/ssl/ca-cert.pem --cert /root/ssl/client-cert.pem --key /root/ssl/client-key.pem https://localhost:2379/v3/kv/put -d @dave_encoded.json | jq

and then use the encoded key - MDEyMzQ1 - to query etcd  and pull the data back out: -

curl -X POST --silent --cacert /root/ssl/ca-cert.pem --cert /root/ssl/client-cert.pem --key /root/ssl/client-key.pem https://localhost:2379/v3/kv/range -d '{"key":"MDEyMzQ1"}' | jq -r .kvs[].value | base64 -d | jq

{
  "name": "Dave Hay",
  "id": "davehay1969"
}

Well, there is an alternate approach, using etcdctl which is further described here: -

Interacting with etcd

So, given that I know the key ID of 012345 I can quite simply ask etcd to provide the value: -

etcdctl --endpoints=localhost:2379 --cacert="/root/ssl/ca-cert.pem" --cert="/root/ssl/client-cert.pem" --key="/root/ssl/client-key.pem" get 012345 | jq

{
  "name": "Dave Hay",
  "id": "davehay1969"
}

which, I think you'll agree, is way simpler.

Bottom line, knowing that the gRPC APIs use base64 is crucial; knowing that etcdctl exists is also rather neat-o.

And there's more - munging base64 in JSON for etcd

Following on from my earlier post: -

etcd - Today I learned ...

I dug into jq more, and found this: -

base64 decoding function #47

specifically this comment: -

As this is showing up on Google a lot, and good documentation on jq is sparse, here is to everybody who lands here:

Example:

echo '{"foo": "Ym9iIGxpa2VzIGFsaWNlCg=="}' | jq '.foo | @base64d'

Or even use it when building new objects:

echo '{"foo": "Ym9iIGxpa2VzIGFsaWNlCg=="}' | jq '{encoded: .foo, decoded: .foo | @base64d}'

dating back to 2018.

This led me to a neat-o mechanism to encode the key and value of my JSON document: -

cat dave.json

{
"key":"012345",
"value": {
"name":"Dave Hay",
"id":"davehay1969"
}
}

jq -r '.[] |= @base64' dave.json

{
  "key": "MDEyMzQ1",
  "value": "eyJuYW1lIjoiRGF2ZSBIYXkiLCJpZCI6ImRhdmVoYXkxOTY5In0="
}

I then used this to generate a new JSON document: -

jq -r '.[] |= @base64' dave.json > dave_encoded.json

which I then fed into etcd: -

curl -X POST --silent --cacert /root/ssl/ca-cert.pem --cert /root/ssl/client-cert.pem --key /root/ssl/client-key.pem https://localhost:2379/v3/kv/put -d @dave_encoded.json | jq

{

  "header": {
    "cluster_id": "14841639068965178418",
    "member_id": "10276657743932975437",
    "revision": "2",
    "raft_term": "2"
  }
}

and then confirmed that I could pull the data back out: -

curl -X POST --silent --cacert /root/ssl/ca-cert.pem --cert /root/ssl/client-cert.pem --key /root/ssl/client-key.pem https://localhost:2379/v3/kv/range -d '{"key":"MDEyMzQ1"}' | jq -r .kvs[].value | base64 -d | jq

{

  "name": "Dave Hay",
  "id": "davehay1969"
}

which is nice

See, I told you I'd find a way .....