Apple Watch - stop talking to me

For some peculiar reason, the Workout app on my Apple Watch started talking to me this morning …

Whilst that's not necessarily a problem, hearing a disembodied voice coming through my AirPods at 0600ish, whilst out on my morning walk was somewhat disconcerting

Thankfully a quick Google came to the rescue: -

watchOS 8: Siri on Apple Watch can announce your progress during a workout

which led me to the Apple Watch app on my iPhone : -

Why this started today, given that I updated to watchOS 8 a week or two back …. 🤷‍♀️

Podman and IBM Container Registry - there's more ...

Following on from my most recent Podman-related posts I'm creating an image to test my RedHat OpenShift Kubernetes Service (ROKS) deployment.

Given that I'm living in the IBM Cloud CLI, I thought I'd try the ic cr command-line: -

ic cr build --no-cache de.icr.io/roks_oct2021/hello_world:latest --file Dockerfile .

FAILED

The 'build' command is deprecated, you must specify the --accept-deprecation option to use this command. For more information see: https://www.ibm.com/cloud/blog/announcements/ibm-cloud-container-registry-deprecating-container-builds

Aw shucks, of course ...

Thankfully, I have Podman ...

podman build --no-cache -t de.icr.io/roks_oct2021/hello_world:latest -f Dockerfile .

and then, having created myself a namespace in ICR: -

ic cr namespace-add roks_oct2021

I can push the image: -

podman push de.icr.io/roks_oct2021/hello_world:latest

and we're off to the races ...

ic cr images

Listing images...

Repository                           Tag      Digest         Namespace      Created          Size    Security status   

de.icr.io/roks_oct2021/hello_world   latest   029263beb4d4   roks_oct2021   43 minutes ago   16 MB   No Issues   

OK

ic cr va de.icr.io/roks_oct2021/hello_world:latest

Checking security issues for 'de.icr.io/roks_oct2021/hello_world:latest'...

Image 'de.icr.io/roks_oct2021/hello_world:latest' was last scanned on Tue Oct 26 12:36:34 UTC 2021

The scan results show that NO ISSUES were found for the image.

OK

ic cr va --extended de.icr.io/roks_oct2021/hello_world:latest

Checking security issues for 'de.icr.io/roks_oct2021/hello_world:latest'...

Image 'de.icr.io/roks_oct2021/hello_world:latest' was last scanned on Tue Oct 26 12:36:34 UTC 2021

The scan results show that NO ISSUES were found for the image.

OK

which is nice !

Ooops, Podman broke my IBM Container Registry - well, kinda

I was digging into an IBM Container Registry (ICR), specifically to look at an image that I'd just built/pushed.

This is on a Mac upon which I've installed Podman, to replace Docker Desktop, as per previous posts.

Having logged into IBM Cloud ( I have a script for that ), I logged into the ICR instance: -

ic cr login

which responded: -

Logging in to 'us.icr.io'...

FAILED

Failed to 'docker login' to 'us.icr.io' with error: Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM

Error: unable to connect to Podman. failed to create sshClient: Connection to bastion host (ssh://core@localhost:53095/run/user/1000/podman/podman.sock) failed.: dial tcp [::1]:53095: connect: connection refused

At which point, I realised where I'd gone wrong - I'd rebooted my Mac since last I did this, and, at a guess, the Podman Machine doesn't autostart.

I manually started it: -

podman machine start

INFO[0000] waiting for clients...                       

INFO[0000] listening tcp://0.0.0.0:7777                 

INFO[0000] new connection from  to /var/folders/b5/8vqr9tt54v94jxzs0_k2qq2m0000gn/T/podman/qemu_podman-machine-default.sock 

Waiting for VM ...

Machine "podman-machine-default" started successfully

and then attempted to log into ICR: -

ic cr login

Logging in to 'us.icr.io'...

Logged in to 'us.icr.io'.

OK

Next step is to find out how to autostart the Podman Machine "service" .... 

IBM Cloud - OCP clusters, Ingress and Certificate Manager

So this is definitely a work-in-progress but I may have resolved an issue that I was seeing with a newly created OpenShift Container Platform (OCP) cluster.

TL;DR; the command ic cs cluster ls showed my cluster state as warning and never as ready.

When I inspected the cluster using ic cs cluster get --cluster $cluster_name I saw: -

Ingress Subdomain:              - †   

Ingress Secret:                 - †   

Ingress Status:                 -   

Ingress Message:                -   

and: -

† Your Ingress subdomain and secret might not be ready yet. For more info by cluster type, see 'https://ibm.biz/ingress-sub' for Kubernetes or 'https://ibm.biz/ingress-sub-ocp' for OpenShift.

and, after a while, this: -

Ingress Message:                Could not upload certificates to Certificate Manager instance. Ensure you have the correct IAM permissions. For more info, see http://ibm.biz/ingress-secret   

I followed the suggested link: -

Why does no Ingress secret exist after cluster creation?

and ended up with: -

Why does no Ingress subdomain exist after I create clusters of the same or similar name?

which said, in part: -     

What’s happening

You create and delete a cluster multiple times, such as for automation purposes.

Every time that you create the cluster, you use either the same name or a name that is very similar to previous names that you used. When you run ibmcloud ks cluster get --cluster <cluster>, your cluster is in a normal state but no Ingress Subdomain or Ingress Secret are available.

Why it’s happening

When you create and delete a cluster that uses the same name multiple times, the Ingress subdomain for that cluster in the format <cluster_name>.<globally_unique_account_HASH>-0000.<region>.containers.appdomain.cloud is registered and unregistered each time.

The certificate for the subdomain is also generated and deleted each time. If you create and delete a cluster with the same name 5 times or more within 7 days, you might reach the Let's Encrypt Duplicate Certificate rate limit, because the same Ingress subdomain and certificate are registered every time that you create the cluster. Because very long cluster names are truncated to 24 characters in the Ingress subdomain for the cluster, you can also reach the rate limit if you use multiple cluster names that have the same first 24 characters.

Given that I'm writing a document guiding one through the process of deploying OCP on IBM Cloud, I have been re-using the same cluster name e.g. roks-oct2021 over and over the past few days.

Working on the hypothesis that that's the root cause, I've changed the way that I generate the cluster name for my document: -

export cluster_name="roks_`date +%s`"

which uses the date in epoch format e.g. run the command three times in sequence: -

date +%s

1635176609

date +%s

1635176610

date +%s

1635176611

and note the difference.

I've just deleted and recreated my cluster, and it's looking good thus far: -

ic cs cluster ls

OK

Name              ID                     State       Created          Workers   Location    Version                 Resource Group Name   Provider   

roks_1635176017   c5rcsngf0kf7u096q2e0   deploying   10 minutes ago   2         Frankfurt   4.8.11_1526_openshift   default               vpc-gen2   

The state shows as deploying rather than warning and, even more promisingly, the number of Worker ( Computer Nodes ) shows as 2 rather than 0.

We'll see ....

Today I learned - one reason why one may not be able to authenticate to a RedHat OpenShift cluster running on IBM Cloud ...

Today I mainly be tinkering with RedHat OpenShift Container Platform (OCP) on IBM Cloud, and am running through a set of steps that I've written to document the end-to-end setup.

Having created a cluster on Friday, I tried to authenticate to it today: -

export apikey="my_api_key_goes_here"

oc login https://control-plane-endpoint-url:31452 -u apikey -p $apikey

but this threw up: -

Error from server (InternalError): Internal error occurred: unexpected response: 500

After a spot of digging, I realised why this was the case.

This was a newly created cluster BUT I'd not yet retrieved the cluster configuration from IBM Cloud.

The cluster name is roks-oct2021 so I needed to run: -

ic cs cluster config --cluster roks-oct2021

OK

The configuration for roks-oct2021 was downloaded successfully.

Added context for roks-oct2021 to the current kubeconfig file.

You can now execute 'kubectl' commands against your cluster. For example, run 'kubectl get nodes'.

If you are accessing the cluster for the first time, 'kubectl' commands might fail for a few seconds while RBAC synchronizes.

Once I'd done this, all was well - I was able to run: -

oc login https://control-plane-endpoint-url:31452 -u apikey -p $apikey

which responded: -

Login successful.

You have access to 66 projects, the list has been suppressed. You can list all projects with 'oc projects'

Using project "default".

Yay, VMware Fusion and macOS Big Sur - no longer "NAT good friends" - forgive the double negative and the terrible pun ...

After macOS 11 Big Sur was released in 2020, VMware updated their Fusion product to v12 and, sadly, managed to break Network Address Translation (NAT), as per their release notes: -

VMware Fusion 12 Release Notes

Unable to access port forwarding on a NAT virtual machine, if the traffic is routed through the host IP stack on Big Sur hosts

On Big Sur hosts, if user configures NAT port forwarding in Fusion, the service running in the virtual machine is unavailable on the macOS host using localhost:exposedPort, hostIP:exposedPort, or 127.0.0.1:exposedPort; Port forwarding is also not accessible inside a NAT virtual machine using hostIP:exposedPort.

Thankfully, as of now, with Fusion 12.2.0 this is now resolved: -

VMware Fusion 12.2.0 Release Notes

Resolved Issues

Unable to access port forwarding on a NAT virtual machine if the traffic is routed through the host IP stack on Big Sur hosts

On Big Sur hosts, if a user configures NAT port forwarding in Fusion, the service running in the virtual machine is unavailable on the macOS host using localhost:exposedPort, hostIP:exposedPort, or 127.0.0.1:exposedPort; Port forwarding is also not accessible inside a NAT virtual machine using hostIP:exposedPort.

This issue is now resolved.

I updated this AM, and am now running Fusion 12.2.0 on macOS 11.6 and all is well - my Windows 10 VM happily shares my Mac's connection using NAT, and can tunnel via my Cisco AnyConnect VPN, which is nice ....

For my future self - don't try and use crictl to deploy pods into an existing Kubernetes cluster

I'm doing some work with the Kata Containers v2 runtime, and was trying to test it using crictl 

First I created a pair of YAML documents: -

tee podsandbox-config.yaml <<EOF
metadata:
  attempt: 1
  name: busybox-sandbox
  namespace: default
  uid: hdishd83djaidwnduwk28bcsb
log_directory: /tmp
linux:
  namespaces:
    options: {}
EOF

tee container-config.json <<EOF
{
  "metadata": {
      "name": "busybox"
  },
  "image":{
      "image": "busybox"
  },
  "command": [
      "top"
  ],
  "log_path":"busybox.log",
  "linux": {
  }
}
EOF

and then I used the first of those to create a Pod Sandbox: -

sandbox=$(crictl runp -r kata podsandbox-config.yaml)

However, the resulting pod soon disappeared - I managed to check its state before it disappeared: -

crictl pods | grep kata

9fafade8c3216       23 seconds ago      NotReady            busybox-sandbox                                  default             1                   kata

65fc059b8129d       40 minutes ago      Ready               nginx-kata                                       default             0                   kata

and inspected the NotReady pod: -

crictl inspectp 9fafade8c3216 | jq .status.state

"SANDBOX_NOTREADY"

Thankfully someone else had hit this issue over in the cri-o project: -

Pod sandbox becomes SANDBOX_NOTREADY after four seconds #1432 

specifically this comment: -

I guess you might be using crictl to create pod/container on a running kubernetes node. Kubelet deletes unwanted containers/pods, please don't do that on a running kubernetes node.

See https://kubernetes.io/docs/tasks/debug-application-cluster/crictl/#example-crictl-commands

Ah, yes, that'd be it !

Back to K8s ..........