For my future self - don't try and use crictl to deploy pods into an existing Kubernetes cluster

I'm doing some work with the Kata Containers v2 runtime, and was trying to test it using crictl 

First I created a pair of YAML documents: -

tee podsandbox-config.yaml <<EOF
metadata:
  attempt: 1
  name: busybox-sandbox
  namespace: default
  uid: hdishd83djaidwnduwk28bcsb
log_directory: /tmp
linux:
  namespaces:
    options: {}
EOF

tee container-config.json <<EOF
{
  "metadata": {
      "name": "busybox"
  },
  "image":{
      "image": "busybox"
  },
  "command": [
      "top"
  ],
  "log_path":"busybox.log",
  "linux": {
  }
}
EOF

and then I used the first of those to create a Pod Sandbox: -

sandbox=$(crictl runp -r kata podsandbox-config.yaml)

However, the resulting pod soon disappeared - I managed to check its state before it disappeared: -

crictl pods | grep kata

9fafade8c3216       23 seconds ago      NotReady            busybox-sandbox                                  default             1                   kata

65fc059b8129d       40 minutes ago      Ready               nginx-kata                                       default             0                   kata

and inspected the NotReady pod: -

crictl inspectp 9fafade8c3216 | jq .status.state

"SANDBOX_NOTREADY"

Thankfully someone else had hit this issue over in the cri-o project: -

Pod sandbox becomes SANDBOX_NOTREADY after four seconds #1432 

specifically this comment: -

I guess you might be using crictl to create pod/container on a running kubernetes node. Kubelet deletes unwanted containers/pods, please don't do that on a running kubernetes node.

See https://kubernetes.io/docs/tasks/debug-application-cluster/crictl/#example-crictl-commands

Ah, yes, that'd be it !

Back to K8s ..........