Munging Dockerfiles using sed

 So I had a requirement to update a Dockerfile, which I'd pulled from a GitHub repository, without actually adding my changes ( via git add and git commit ) to that repo ...

Specifically, I wanted to add a command to update the to-be-built Alpine image.

Here's how I solved it ...

Having cloned the target repo, which included the Dockerfile ( example below ): -

FROM alpine:3.7
RUN apk add --no-cache mysql-client
ENTRYPOINT ["mysql"]

At present, this is what happens when I build the image: -

docker build -f Dockerfile .

Sending build context to Docker daemon  2.048kB
Step 1/3 : FROM alpine:3.7
3.7: Pulling from library/alpine
5d20c808ce19: Pull complete 
Digest: sha256:8421d9a84432575381bfabd248f1eb56f3aa21d9d7cd2511583c68c9b7511d10
Status: Downloaded newer image for alpine:3.7
 ---> 6d1ef012b567
Step 2/3 : RUN apk add --no-cache mysql-client
 ---> Running in 07bc9ca0e14a
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/community/x86_64/APKINDEX.tar.gz
(1/6) Installing mariadb-common (10.1.41-r0)
(2/6) Installing ncurses-terminfo-base (6.0_p20171125-r1)
(3/6) Installing ncurses-terminfo (6.0_p20171125-r1)
(4/6) Installing ncurses-libs (6.0_p20171125-r1)
(5/6) Installing mariadb-client (10.1.41-r0)
(6/6) Installing mysql-client (10.1.41-r0)
Executing busybox-1.27.2-r11.trigger
OK: 41 MiB in 19 packages
Removing intermediate container 07bc9ca0e14a
 ---> 43862371f8a4
Step 3/3 : ENTRYPOINT ["mysql"]
 ---> Running in d8b08c967cc1
Removing intermediate container d8b08c967cc1
 ---> 1ee30800ffbd
Successfully built 1ee30800ffbd

I wanted to add a command to run apk upgrade before the ENTRYPOINT entry ( so, in essence, inserting it between lines 2 and 3 )

This is what I did: -

sed -i '' -e "2s/^//p; 2s/^.*/RUN apk --no-cache upgrade/" Dockerfile

which results in: -

FROM alpine:3.7

RUN apk add --no-cache mysql-client

RUN apk --no-cache upgrade

ENTRYPOINT ["mysql"]

Now the build looks like this: -

docker build -f Dockerfile .

Sending build context to Docker daemon  2.048kB

Step 1/4 : FROM alpine:3.7

3.7: Pulling from library/alpine

5d20c808ce19: Pull complete 

Digest: sha256:8421d9a84432575381bfabd248f1eb56f3aa21d9d7cd2511583c68c9b7511d10

Status: Downloaded newer image for alpine:3.7

 ---> 6d1ef012b567

Step 2/4 : RUN apk add --no-cache mysql-client

 ---> Running in b29668e70377

fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/main/x86_64/APKINDEX.tar.gz

fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/community/x86_64/APKINDEX.tar.gz

(1/6) Installing mariadb-common (10.1.41-r0)

(2/6) Installing ncurses-terminfo-base (6.0_p20171125-r1)

(3/6) Installing ncurses-terminfo (6.0_p20171125-r1)

(4/6) Installing ncurses-libs (6.0_p20171125-r1)

(5/6) Installing mariadb-client (10.1.41-r0)

(6/6) Installing mysql-client (10.1.41-r0)

Executing busybox-1.27.2-r11.trigger

OK: 41 MiB in 19 packages

Removing intermediate container b29668e70377

 ---> 971a3d538edf

Step 3/4 : RUN apk --no-cache upgrade

 ---> Running in 8dfa10b481ad

fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/main/x86_64/APKINDEX.tar.gz

fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/community/x86_64/APKINDEX.tar.gz

(1/2) Upgrading musl (1.1.18-r3 -> 1.1.18-r4)

(2/2) Upgrading musl-utils (1.1.18-r3 -> 1.1.18-r4)

Executing busybox-1.27.2-r11.trigger

OK: 41 MiB in 19 packages

Removing intermediate container 8dfa10b481ad

 ---> 35d7cbec77c0

Step 4/4 : ENTRYPOINT ["mysql"]

 ---> Running in c0d7d310d396

Removing intermediate container c0d7d310d396

 ---> 4ad02b88f4d4

Successfully built 4ad02b88f4d4

In essence, the sed command duplicates line 2: -

RUN apk add --no-cache mysql-client

as line 3, and then replaces the newly duplicated text with the required replacement: -

RUN apk --no-cache upgrade

Neat, eh ?

I've got this in a script, and it works a treat .....

Tinkering with YQ, it's like JQ but for YAML rather than JSON

 A colleague was tinkering with Mike Farah's excellent yq tool, and had asked about updating it on his Mac.

I've got yq installed via Homebrew: -

brew update

which threw up: -

Error: 

  homebrew-core is a shallow clone.

To `brew update`, first run:

  git -C /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core fetch --unshallow

so I did this: -

git -C /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core fetch --unshallow

and then: -

brew upgrade yq

which resulted in the latest version: -

yq --version

yq version 4.5.1

Subsequently, my friend was then looking to use the evaluate feature of yq to run queries against YAML documents, with the query string being defined in an environment variable.

Therefore, I had to try this ...

Firstly, I created sample.yaml 

name:

 - Ben

 - Dave

 - Tom

 - Jerry

which yq correctly evaluates: -

yq eval sample.yaml

name:

  - Ben

  - Dave

  - Tom

  - Jerry

and: -

yq eval .name sample.yaml

- Ben

- Dave

- Tom

- Jerry

I then set the environment variable to define my search query: -

export FRIEND="Ben"

and updated my yq query: -

yq eval ".name = \"$FRIEND\"" sample.yaml

name: Ben

and then changed my variable: -

export FRIEND="Tom"

and re-ran my query: -

yq eval ".name = \"$FRIEND\"" sample.yaml

name: Tom

Thanks to this: -

how to pass environment variable as value #468

yq: evaluate

https://mikefarah.gitbook.io/yq/commands/evaluate

JQ - more filtering, more fun

 Building upon earlier posts, I wanted to streamline the output from a REST API that returns a list of running containers: -

curl -s -k -X GET https://my.endpoint.com/containers -H 'Accept: application/json' -H 'Content-Type: application/json' -H 'Authorization: Bearer '"$ACCESS_TOKEN" | jq '.data[] | select (.Names[] | contains("dave"))' | jq .Names

[
  "/davehay_k8s_worker_1"
]
[
  "/davehay_k8s_master"
]
[
  "/davehay_k8s_worker_2"
]

Specifically, I wanted to remove the extraneous square brackets and double quotes ...

Here we go ...

curl -s -k -X GET https://my.endpoint.com/containers -H 'Accept: application/json' -H 'Content-Type: application/json' -H 'Authorization: Bearer '"$ACCESS_TOKEN"  | jq '.data[] | select (.Names[] | contains("dave"))' | jq -r .Names[]

/davehay_k8s_worker_1

/davehay_k8s_master

/davehay_k8s_worker_2

As with everything, there's a better way ....

But this works for me :-)

I learn something new each and every day - finding big files on macOS

 This came up in the context of a colleague trying to work out what's eating his Mac's disk.

Whilst I'm familiar with using the built-in Storage Management app, which includes a Recommendations tab: -

I'd not realised that there's an easy way to look for files, based upon size, using the mdfind command: -

mdfind "kMDItemFSSize >$[1024*1024*1024]"

/System/Library/dyld/dyld_shared_cache_x86_64
/System/Library/dyld/dyld_shared_cache_x86_64h
/System/Library/dyld/dyld_shared_cache_arm64e
/System/Library/dyld/aot_shared_cache
/Applications/HCL Notes.app
/Applications/Docker.app
/Users/hayd/Virtual Machines.localized/Ubuntu.vmwarevm
/Users/hayd/Library/Containers/com.docker.docker/Data/vms/0/data/Docker.raw
/Applications/Microsoft Excel.app
/Applications/iMovie.app
/Applications/Microsoft Word.app
/Applications/Microsoft PowerPoint.app
/Applications/VMware Fusion.app
/Users/hayd/Virtual Machines.localized/Windows10.vmwarevm

Now my example is looking for really large files - 1024^3 - or, to be more specific, files exceeding 1 GB in size, but it's good to know .....

Gah, ImagePullBackOff with Calico CNI running on Kubernetes

Whilst deploying Calico Node etc. to my cluster, via: -

kubectl apply -f calico.yaml

and whilst checking the running Pods, via: -

kubectl get pods -A

I was seeing: -

...

kube-system   calico-node-9srv6                         0/1     Init:ErrImagePull   0          8s

...

I dug into that failing Pod with: -

kubectl describe pod calico-node-9srv6 --namespace kube-system

which showed: -

...

  Type     Reason   Age                    From                   Message

  ----     ------   ----                   ----                   -------

  Normal   BackOff  17m (x303 over 88m)    kubelet, 50c933ad26be  Back-off pulling image "us.icr.io/davehay/calico/cni:latest-s390x"

  Warning  Failed   2m56s (x368 over 88m)  kubelet, 50c933ad26be  Error: ImagePullBackOff

...

Now I knew that it wasn't an authentication issue, as my YAML was also defining a Secret, as per my previous post: -

Argh, Kubernetes and YAML hell

and had defined that Secret within the YAML, using: -

...

imagePullSecrets:

- name: my_secret

...

So why wasn't it working .... ?

And then it struck me .... DOH!

My Pod is running inside the kube-system Namespace ....

You know where I'm going with this, am I right ?

Yes, my Secret was NOT inside the same kube-system Namespace, but was in default.

Once I updated my YAML to redefine my Secret: -

...

apiVersion: v1

kind: Secret

data:

  .dockerconfigjson:

    <HERE'S THE BASE64 ENCODED STUFF>

metadata:

  name: my_secret

  namespace: kube-system

type: kubernetes.io/dockerconfigjson

...

re-applied the YAML, and deleted the failing Pod, all was well

Argh, Kubernetes and YAML hell

I was trying to create a Kubernetes (K8s) Secret, containing existing Docker credentials, as per this: -

Create a Secret based on existing Docker credentials

and kept hitting syntax errors with the YAML.

For reference, in this scenario, we've already logged into a container registry, such as IBM Container Registry or Docker Hub, and want to grab the credentials that Docker itself "caches" in ~/.docker/config.json

Wait, what ? You didn't know that Docker helpfully does that ? Another good reason to NOT leave yourself logged into a container registry when you step away from your box ....

Anyhow, as per the above linked documentation, the trick is to encapsulate the content of that file, encoded using Base64, into a YAML file that looks something like this: -

---
apiVersion: v1
kind: Secret
data:
  .dockerconfigjson:
    <HERE'S THE BASE64 ENCODED STUFF>
metadata:
  name: my_secret
type: kubernetes.io/dockerconfigjson

The trick is to get the Base64 encoded stuff just right ....

I was doing this: -

cat ~/.docker/config.json | base64 

which resulted in: -

ewoJImF1dGhzIjoge30sCgkiSHR0cEhlYWRlcnMiOiB7CgkJIlVzZXItQWdlbnQiOiAiRG9ja2Vy

LUNsaWVudC8xOS4wMy42IChsaW51eCkiCgl9Cn0=

I kept seeing exceptions such as: -

error: error parsing secret.yaml: error converting YAML to JSON: yaml: line 7: could not find expected ':'

and: -

Error from server (BadRequest): error when creating "secret.yaml": Secret in version "v1" cannot be handled as a Secret: v1.Secret.ObjectMeta: v1.ObjectMeta.TypeMeta: Kind: Data: decode base64: illegal base64 data at input byte 76, error found in #10 byte of ...|BLAHBLAH=="},"kind":"|..., bigger context ...|BLAHBLAHBLAHBLAHBLAHBLAHBLAHBLAHBLAHBLAHBLAHBLAHBLAHBLAH=="},"kind":"Secret","metadata":{"annotations":{"kube|...

when I tried to apply the YAML: -

kubectl apply -f secret.yaml

And then I re-read the documentation, for the 11th time, and saw: -

base64 encode the docker file and paste that string, unbroken as the value for field data[".dockerconfigjson"]

Can you see what I was doing wrong ?

Yep, I wasn't "telling" the Base64 encoded to produce an unbroken ( and, more importantly, unwrapped ) string.

This time I did it right: -

cat ~/.docker/config.json | base64 --wrap=0

resulting in this: -

ewoJImF1dGhzIjoge30sCgkiSHR0cEhlYWRlcnMiOiB7CgkJIlVzZXItQWdlbnQiOiAiRG9ja2VyLUNsaWVudC8xOS4wMy42IChsaW51eCkiCgl9Cn0=root@379cd9170839:~# 

Having discarded the user@hostname stuff, I was left with this: -

ewoJImF1dGhzIjoge30sCgkiSHR0cEhlYWRlcnMiOiB7CgkJIlVzZXItQWdlbnQiOiAiRG9ja2VyLUNsaWVudC8xOS4wMy42IChsaW51eCkiCgl9Cn0=

I updated my YAML: -

---

apiVersion: v1

kind: Secret

data:

  .dockerconfigjson: ewoJImF1dGhzIjoge30sCgkiSHR0cEhlYWRlcnMiOiB7CgkJIlVzZXItQWdlbnQiOiAiRG9ja2VyLUNsaWVudC8xOS4wMy42IChsaW51eCkiCgl9Cn0=

metadata:

  name: my_secret

type: kubernetes.io/dockerconfigjson

and applied it: -

kubectl apply -f secret.yaml 

secret/armadamultiarch created

and we're off to the races!