Nuking a pesky K8s namespace from orbit ...

I was having problems with a recalcitrant Kubernetes namespace that just did not want to be deleted ...

kubectl get namespaces | grep -v Active

NAME              STATUS        AGE
foobar            Terminating   7d22h

kubectl get namespace foobar --output json

{
    "apiVersion": "v1",
    "kind": "Namespace",
    "metadata": {
        "creationTimestamp": "2021-09-20T09:30:17Z",
        "deletionTimestamp": "2021-09-27T17:18:57Z",
        "name": "foobar",
        "resourceVersion": "772406",
        "uid": "22863d2f-f956-4da8-bfd5-dd70a4d1685a"
    },
    "spec": {
        "finalizers": [
            "kubernetes"
        ]
    },
    "status": {
        "conditions": [
            {
                "lastTransitionTime": "2021-09-27T17:19:04Z",
                "message": "All resources successfully discovered",
                "reason": "ResourcesDiscovered",
                "status": "False",
                "type": "NamespaceDeletionDiscoveryFailure"
            },
            {
                "lastTransitionTime": "2021-09-27T17:19:04Z",
                "message": "All legacy kube types successfully parsed",
                "reason": "ParsedGroupVersions",
                "status": "False",
                "type": "NamespaceDeletionGroupVersionParsingFailure"
            },
            {
                "lastTransitionTime": "2021-09-27T17:19:04Z",
                "message": "All content successfully deleted, may be waiting on finalization",
                "reason": "ContentDeleted",
                "status": "False",
                "type": "NamespaceDeletionContentFailure"
            },
            {
                "lastTransitionTime": "2021-09-27T17:19:04Z",
                "message": "Some resources are remaining: etcdbackups.etcd.database.coreos.com has 1 resource instances",
                "reason": "SomeResourcesRemain",
                "status": "True",
                "type": "NamespaceContentRemaining"
            },
            {
                "lastTransitionTime": "2021-09-27T17:19:04Z",
                "message": "Some content in the namespace has finalizers remaining: backup-operator-periodic in 1 resource instances",
                "reason": "SomeFinalizersRemain",
                "status": "True",
                "type": "NamespaceFinalizersRemaining"
            }
        ],
        "phase": "Terminating"
    }
}

This: -

Namespace "stuck" as Terminating, How do I remove it?

gave me some inspiration: -

kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get -n foobar

...
NAME                  AGE
foobar-etcd-cluster   7d22h
...

kubectl get etcdbackups -A

NAMESPACE   NAME                  AGE

foobar      foobar-etcd-cluster   7d22h

kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -n foobar

NAME                                                      AGE
etcdbackup.etcd.database.coreos.com/foobar-etcd-cluster   7d22h
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress

This bit was the most useful: -

kubectl get namespace foobar --output json > /tmp/foobar.json

I edited the resulting JSON document: -

vi /tmp/foobar.json

removing the kubernetes finaliser, from this: -

    }, "spec": { "finalizers": [ "kubernetes" ] },

to this: -

    }, "spec": { "finalizers": [ ] },

and then started the K8s proxy: -

kubectl proxy

in one terminal and, from another terminal, ran cURL: -

cd /tmp

curl -k -H "Content-Type: application/json" -X PUT --data-binary @foobar.json https://localhost:8001/api/v1/namespaces/foobar/finalize

Sadly this responded with: -

curl: (35) error:1400410B:SSL routines:CONNECT_CR_SRVR_HELLO:wrong version number

**UPDATE**

Well, that wasn't ever gonna work; I was trying to hit the API server using HTTPS rather than HTTP - I guess that kubectl proxy only listens using HTTP.

Therefore, the solution was to hit the endpoint using HTTP rather than HTTPS.

**UPDATE**

However, once I changed from localhost to 127.0.0.1 all was well: -

curl -k -H "Content-Type: application/json" -X PUT --data-binary @foobar.json http://127.0.0.1:8001/api/v1/namespaces/foobar/finalize

and now all is well: -

kubectl get namespaces

NAME              STATUS   AGE
default           Active   9d
ibm-cert-store    Active   9d
ibm-operators     Active   9d
ibm-system        Active   9d
kube-node-lease   Active   9d
kube-public       Active   9d
kube-system       Active   9d