Tuesday, 28 September 2021

Nuking a pesky K8s namespace from orbit ...

I was having problems with a recalcitrant Kubernetes namespace that just did not want to be deleted ...

kubectl get namespaces | grep -v Active

NAME              STATUS        AGE
foobar            Terminating   7d22h

kubectl get namespace foobar --output json

    "apiVersion": "v1",
    "kind": "Namespace",
    "metadata": {
        "creationTimestamp": "2021-09-20T09:30:17Z",
        "deletionTimestamp": "2021-09-27T17:18:57Z",
        "name": "foobar",
        "resourceVersion": "772406",
        "uid": "22863d2f-f956-4da8-bfd5-dd70a4d1685a"
    "spec": {
        "finalizers": [
    "status": {
        "conditions": [
                "lastTransitionTime": "2021-09-27T17:19:04Z",
                "message": "All resources successfully discovered",
                "reason": "ResourcesDiscovered",
                "status": "False",
                "type": "NamespaceDeletionDiscoveryFailure"
                "lastTransitionTime": "2021-09-27T17:19:04Z",
                "message": "All legacy kube types successfully parsed",
                "reason": "ParsedGroupVersions",
                "status": "False",
                "type": "NamespaceDeletionGroupVersionParsingFailure"
                "lastTransitionTime": "2021-09-27T17:19:04Z",
                "message": "All content successfully deleted, may be waiting on finalization",
                "reason": "ContentDeleted",
                "status": "False",
                "type": "NamespaceDeletionContentFailure"
                "lastTransitionTime": "2021-09-27T17:19:04Z",
                "message": "Some resources are remaining: etcdbackups.etcd.database.coreos.com has 1 resource instances",
                "reason": "SomeResourcesRemain",
                "status": "True",
                "type": "NamespaceContentRemaining"
                "lastTransitionTime": "2021-09-27T17:19:04Z",
                "message": "Some content in the namespace has finalizers remaining: backup-operator-periodic in 1 resource instances",
                "reason": "SomeFinalizersRemain",
                "status": "True",
                "type": "NamespaceFinalizersRemaining"
        "phase": "Terminating"

This: -

Namespace "stuck" as Terminating, How do I remove it?

gave me some inspiration: -

kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get -n foobar

NAME                  AGE
foobar-etcd-cluster   7d22h

kubectl get etcdbackups -A

NAMESPACE   NAME                  AGE

foobar      foobar-etcd-cluster   7d22h

kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -n foobar

NAME                                                      AGE
etcdbackup.etcd.database.coreos.com/foobar-etcd-cluster   7d22h
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress

This bit was the most useful: -

kubectl get namespace foobar --output json > /tmp/foobar.json

I edited the resulting JSON document: -

vi /tmp/foobar.json

removing the kubernetes finaliser, from this: -

    }, "spec": { "finalizers": [ "kubernetes" ] },
to this: -

    }, "spec": { "finalizers": [ ] },

and then started the K8s proxy: -

kubectl proxy

in one terminal and, from another terminal, ran cURL: -

cd /tmp

curl -k -H "Content-Type: application/json" -X PUT --data-binary @foobar.json https://localhost:8001/api/v1/namespaces/foobar/finalize

Sadly this responded with: -

curl: (35) error:1400410B:SSL routines:CONNECT_CR_SRVR_HELLO:wrong version number


Well, that wasn't ever gonna work; I was trying to hit the API server using HTTPS rather than HTTP - I guess that kubectl proxy only listens using HTTP.

Therefore, the solution was to hit the endpoint using HTTP rather than HTTPS.


However, once I changed from localhost to all was well: -

curl -k -H "Content-Type: application/json" -X PUT --data-binary @foobar.json

and now all is well: -

kubectl get namespaces

NAME              STATUS   AGE
default           Active   9d
ibm-cert-store    Active   9d
ibm-operators     Active   9d
ibm-system        Active   9d
kube-node-lease   Active   9d
kube-public       Active   9d
kube-system       Active   9d

No comments:

Yay, VMware Fusion and macOS Big Sur - no longer "NAT good friends" - forgive the double negative and the terrible pun ...

After macOS 11 Big Sur was released in 2020, VMware updated their Fusion product to v12 and, sadly, managed to break Network Address Trans...