Not in Kansas anymore - apparently I don't exist

Whilst trying to upgrade the Tekton CLI tool tkn using Homebrew: -

brew upgrade tektoncd/tools/tektoncd-cli

==> Upgrading 1 outdated package:

tektoncd/tools/tektoncd-cli 0.8.0 -> 0.15.0

==> Upgrading tektoncd/tools/tektoncd-cli 0.8.0 -> 0.15.0 

==> Downloading https://github.com/tektoncd/cli/releases/download/v0.15.0/tkn_0.15.0_Darwin_x86_64.tar.gz

==> Downloading from https://github-releases.githubusercontent.com/181939372/9d001b00-4060-11eb-9efa-57717f8e92f7?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJY

######################################################################## 100.0%

Error: An exception occurred within a child process:

  ArgumentError: user david_hay@uk.ibm.com doesn't exist

To narrow things down, I ran: -

brew upgrade

but that failed with much the same: -

==> Upgrading tektoncd/tools/tektoncd-cli 0.8.0 -> 0.15.0 

==> Downloading https://github.com/tektoncd/cli/releases/download/v0.15.0/tkn_0.15.0_Darwin_x86_64.tar.gz

Already downloaded: /Users/hayd/Library/Caches/Homebrew/downloads/d911addd12ba79ea06e5d8d0002a0523ec0b53d6aa5407a94f6e9836f3ff6fa5--tkn_0.15.0_Darwin_x86_64.tar.gz

Error: An exception occurred within a child process:

  ArgumentError: user david_hay@uk.ibm.com doesn't exist

Following this: -

Heroku cli (brew install heroku): ArgumentError: user emixam23 doesn't exist

I checked my environment: -

printenv | grep david_hay@uk.ibm.com

which returned: -

USER=david_hay@uk.ibm.com

Given that my macOS user is actually hayd, I suspect that this is the issue.

I reset that environment variable: -

export USER=hayd

and tried again: -

brew upgrade tektoncd/tools/tektoncd-cli

Updating Homebrew...

==> Auto-updated Homebrew!

Updated 1 tap (homebrew/core).

==> Updated Formulae

Updated 1 formula.

==> Upgrading 1 outdated package:

tektoncd/tools/tektoncd-cli 0.8.0 -> 0.15.0

==> Upgrading tektoncd/tools/tektoncd-cli 0.8.0 -> 0.15.0 

==> Downloading https://github.com/tektoncd/cli/releases/download/v0.15.0/tkn_0.15.0_Darwin_x86_64.tar.gz

Already downloaded: /Users/hayd/Library/Caches/Homebrew/downloads/d911addd12ba79ea06e5d8d0002a0523ec0b53d6aa5407a94f6e9836f3ff6fa5--tkn_0.15.0_Darwin_x86_64.tar.gz

==> Caveats

Bash completion has been installed to:

  /usr/local/etc/bash_completion.d

==> Summary

🍺  /usr/local/Cellar/tektoncd-cli/0.15.0: 8 files, 43.7MB, built in 7 seconds

Removing: /usr/local/Cellar/tektoncd-cli/0.8.0... (8 files, 32.4MB)

Job done!

For the record, tkn was back-level: -

tkn version

Client version: 0.8.0

Pipeline version: unknown

and is now up-to-date: -

tkn version

Client version: 0.15.0

Pipeline version: v0.22.0

Triggers version: v0.12.1

( which, usefully, also reports the versions of Tekton Pipelines and Triggers deployed to my cluster )

Two of my favourite things - Kubernetes and jq

 As per recent posts, I've been falling in love with jq and using it for more and more and more ....

Today, it's Kubernetes meets jq

Specifically, working with secrets ... and following on from an earlier post: -

Gah, again with the ImagePullBackOff

So I'm creating a dummy secret, wrapped around some credentials for an IBM Container Registry (ICR) instance: -

kubectl create secret docker-registry foobar --docker-server='https://us.icr.io' --docker-username='iamapikey' --docker-password='THIS_IS_NOT_A_VALID_APIKEY'

secret/foobar created

Now the secret of type docker-registry essentially wraps the credentials ( whether it be for Docker Hub or an ICR instance or similar ) up in a Base64-encoded "blob", as evidenced by the query: -

kubectl get secret foobar --output json

{

    "apiVersion": "v1",

    "data": {

        ".dockerconfigjson": "eyJhdXRocyI6eyJodHRwczovL3VzLmljci5pbyI6eyJ1c2VybmFtZSI6ImlhbWFwaWtleSIsInBhc3N3b3JkIjoiVEhJU19JU19OT1RfQV9WQUxJRF9BUElLRVkiLCJhdXRoIjoiYVdGdFlYQnBhMlY1T2xSSVNWTmZTVk5mVGs5VVgwRmZWa0ZNU1VSZlFWQkpTMFZaIn19fQ=="

    },

    "kind": "Secret",

    "metadata": {

        "creationTimestamp": "2021-03-16T16:57:52Z",

        "managedFields": [

            {

                "apiVersion": "v1",

                "fieldsType": "FieldsV1",

                "fieldsV1": {

                    "f:data": {

                        ".": {},

                        "f:.dockerconfigjson": {}

                    },

                    "f:type": {}

                },

                "manager": "kubectl-create",

                "operation": "Update",

                "time": "2021-03-16T16:57:52Z"

            }

        ],

        "name": "foobar",

        "namespace": "default",

        "resourceVersion": "19983",

        "selfLink": "/api/v1/namespaces/default/secrets/foobar",

        "uid": "26bdd49b-49c6-4133-a331-3e9cb6150a26"

    },

    "type": "kubernetes.io/dockerconfigjson"

}

so we can quickly inspect ( decode ) the secret, using jq to parse the output: -

kubectl get secret foobar --output json | jq -r .data[] | base64 -d

{"auths":{"https://us.icr.io":{"username":"iamapikey","password":"THIS_IS_NOT_A_VALID_APIKEY","auth":"aWFtYXBpa2V5OlRISVNfSVNfTk9UX0FfVkFMSURfQVBJS0VZ"}}}

This is a useful way to check input against output, and thus avoid GIGO.

The other jq related tip is in the context of Calico Node, where I was looking to inspect a daemonset to grab one specific data element - known as IP_AUTODETECTION_METHOD - in short, this can be used to ensure that the Calico Node pods use a specific network adapter inside each of the K8s nodes.

So I'd take a generic kubectl command such as: -

kubectl get daemonset/calico-node -n kube-system --output json

and then parse the output to only retrieve the value of IP_AUTODETECTION_METHOD : -

{

  "name": "IP_AUTODETECTION_METHOD",

  "value": "interface=eth.*"

}

I could, if I so chose, override that by editing the daemonset : -

kubectl set env daemonset/calico-node -n kube-system IP_AUTODETECTION_METHOD=interface=eth0

and then re-run the kubectl get command to check that the value had changed ....

Nice !

Gah, again with the ImagePullBackOff

 So, following on from this: -

Gah, ImagePullBackOff with Calico CNI running on Kubernetes

I was again seeing this: -

kube-system   calico-node-lxmk4                          0/1     Init:ImagePullBackOff   0          5m26s

and, upon further digging: -

kubectl describe pod calico-node-lxmk4 --namespace kube-system

Type     Reason     Age                    From                   Message

----     ------     ----                   ----                   -------

Normal   Scheduled  5m47s                  default-scheduler      Successfully assigned kube-system/calico-node-lxmk4 to 667ceb40fc75

Normal   Pulling    4m24s (x4 over 5m46s)  kubelet, 667ceb40fc75  Pulling image "us.icr.io/mynamespace/calico/cni:v3.16.5"

Warning  Failed     4m23s (x4 over 5m45s)  kubelet, 667ceb40fc75  Failed to pull image "us.icr.io/mynamespace/calico/cni:v3.16.5": rpc error: code = Unknown desc = Error response from daemon: Get https://us.icr.io/v2/mynamespace/calico/cni/manifests/v3.16.5: unauthorized: The login credentials are not valid, or your IBM Cloud account is not active.

Warning  Failed     4m23s (x4 over 5m45s)  kubelet, 667ceb40fc75  Error: ErrImagePull

Warning  Failed     3m57s (x7 over 5m45s)  kubelet, 667ceb40fc75  Error: ImagePullBackOff

Normal   BackOff    46s (x21 over 5m45s)   kubelet, 667ceb40fc75  Back-off pulling image "us.icr.io/mynamespace/calico/cni:v3.16.5"

Note that my images are coming from IBM Container Registry, rather than Docker Hub, and that's the key .....

I was following this: -

Pull an Image from a Private Registry

which describes how one can generate a K8s secret from an existing docker login by grabbing the content of ~/.docker/config.json

Therefore, I was doing this: -

kubectl create secret generic regcred --from-file=.dockerconfigjson=/root/.docker/config.json --type=kubernetes.io/dockerconfigjson

having previously logged in: -

echo "<MY API KEY>" | docker login -u iamapikey --password-stdin us.icr.io

which creates/updates /root/.docker/config.json

And that's where I was failing .....

Finally, after a few hours of head-banging, I looked back through my notes and realised that, for previous activities, including Tekton Pipelines / Triggers, I used a different approach to generate the secret: -

kubectl create secret docker-registry regcred --namespace kube-system --docker-server='https://us.icr.io' --docker-username='iamapikey' --docker-password='<MY API KEY>'

And, of course, it worked .....

Every day is ......

Fun with IBM Container Registry, Vulnerability Advisor and Nginx

 So I'm tinkering with IBM Container Registry (ICR) at present, and am testing the Vulnerability Advisor (VA) feature, by building/tagging/pushing a basic Nginx image.

Having configured my Nginx server for HTTPS ( HTTP over TLS ) - or so I thought - I was baffled that VA kept throwing up configuration errors: -

The scan results show that 5 ISSUES were found for the image.
Configuration Issues Found
==========================
Configuration Issue ID                                Policy Status   Security Practice                                  How to Resolve   
application_configuration:nginx.ssl_certificate_key   Active          Specifies the private key file for server cert.    ssl_certificate_key is not present in   
                                                                                                                         /etc/nginx/nginx.conf or   
                                                                                                                         /etc/nginx/sites-enabled/default.   
application_configuration:nginx.ssl_ciphers           Active          Specifies ciphers used in TLS.                     ssl_ciphers is not present in   
                                                                                                                         /etc/nginx/nginx.conf or   
                                                                                                                         /etc/nginx/sites-enabled/default. Defaults may not   
                                                                                                                         be secure.   
application_configuration:nginx.server_tokens         Active          Enables or disables emitting nginx version in      server_tokens is present but value is off. nginx   
                                                                      error messages and in the Server response header   will sends its version in HTTP responses which can   
                                                                      field.                                             be used by attackers for version-specific attacks   
                                                                                                                         against this nginx server.   
                                                                                                                         File: /etc/nginx/nginx.conf   
application_configuration:nginx.ssl_protocols         Active          Enables the specified protocols.                   ssl_protocols is not present in   
                                                                                                                         /etc/nginx/nginx.conf or   
                                                                                                                         /etc/nginx/sites-enabled/default.   
application_configuration:nginx.ssl_certificate       Active          Specifies a file with the certificate in the PEM   ssl_certificate is not present in   
                                                                      format for the given virtual server.               /etc/nginx/nginx.conf or   
                                                                                                                         /etc/nginx/sites-enabled/default.   
OK

even though I thought I'd configured Nginx to support the required configuration items e.g. server_tokens and ssl_protocols etc.

Well, I kinda had ....

I'd added these items: -

ssl_certificate     /etc/nginx/nginx.crt;

ssl_certificate_key /etc/nginx/nginx.key;

ssl_ciphers         EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH;

ssl_protocols       TLSv1.2;

ssl_prefer_server_ciphers   on;

server_tokens       on;

 

into nginx.conf BUT in the wrong place.

I had them in the http{} section rather than in the server{} section.

After some further digging, I realised that all but server_tokens should go in the server{} block, so we end up with this: -

user  nginx;

worker_processes  1;

error_log  /var/log/nginx/error.log warn;

pid        /var/run/nginx.pid;

events {

    worker_connections  1024;

}

http {

    include       /etc/nginx/mime.types;

    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '

                      '$status $body_bytes_sent "$http_referer" '

                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;

    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;

    server_tokens   off;

 

    server {

        listen                      443 ssl default_server;

        listen                      [::]:443 ssl default_server ;

        server_name                 example.com www.example.com;

        root                        /usr/share/nginx/html;

        ssl_certificate             /etc/nginx/nginx.crt;

        ssl_certificate_key         /etc/nginx/nginx.key;

        ssl_ciphers                 EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH;

        ssl_protocols               TLSv1.2;

        ssl_prefer_server_ciphers   on;

    }

}

and, more importantly, this: -

The scan results show that NO ISSUES were found for the image.

OK

For further reading, there's a useful tutorial covering ICR and VA here: -

IBM Cloud Container Registry and Vulnerability Advisor workflow tutorial