Debugging kubectl - and finding the REST API

I've been digging into some networking-related issues with a Worker node in my IBM Kubernetes Service (IKS) cluster.

Anyway......

Whilst trying to work out what was going wrong, when running the command: -

kubectl describe node

I wanted to get as much detail as possible.

Thankfully, kubectl has a nice debug option: -

kubectl --v=X

where X can range from 1 to 8.....

However, apparently only 6,7 and 8 actually do anything.

From this: -

Exploring kubectl HTTP request verbose option and custom kube-config based on kubernetes RBAC

the three levels of verbosity are: -

...

Verbosity level 6

This level will display all requested resources with HTTP response code and time take to complete each request

...

Verbosity level 7

This level will display HTTP request with headers

...

Verbosity level 8

This level will display HTTP request with contents.

...

Here's a simple example, showing all three levels of verbosity: -

kubectl --v=6 version

I0205 10:13:49.337013   70861 loader.go:357] Config loaded from file /Users/hayd/.bluemix/plugins/container-service/clusters/dmhIKSCluster/kube-config-dal10-dmhIKSCluster.yml
I0205 10:13:49.915443   70861 round_trippers.go:406] GET https://c2.us-south.containers.cloud.ibm.com:26705/version 200 OK in 576 milliseconds
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.11", GitCommit:"637c7e288581ee40ab4ca210618a89a555b6e7e9", GitTreeState:"clean", BuildDate:"2018-11-26T14:38:32Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.6+IKS", GitCommit:"002d263ed027db260968616b951fb46f2bab9bb1", GitTreeState:"clean", BuildDate:"2019-01-09T08:07:22Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}

kubectl --v=7 version

I0205 10:13:55.886496   70862 loader.go:357] Config loaded from file /Users/hayd/.bluemix/plugins/container-service/clusters/dmhIKSCluster/kube-config-dal10-dmhIKSCluster.yml
I0205 10:13:55.887120   70862 round_trippers.go:384] GET https://c2.us-south.containers.cloud.ibm.com:26705/version
I0205 10:13:55.887132   70862 round_trippers.go:391] Request Headers:
I0205 10:13:55.887137   70862 round_trippers.go:394]     Accept: application/json, */*
I0205 10:13:55.887142   70862 round_trippers.go:394]     User-Agent: kubectl/v1.10.11 (darwin/amd64) kubernetes/637c7e2
I0205 10:13:56.503585   70862 round_trippers.go:409] Response Status: 200 OK in 616 milliseconds
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.11", GitCommit:"637c7e288581ee40ab4ca210618a89a555b6e7e9", GitTreeState:"clean", BuildDate:"2018-11-26T14:38:32Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.6+IKS", GitCommit:"002d263ed027db260968616b951fb46f2bab9bb1", GitTreeState:"clean", BuildDate:"2019-01-09T08:07:22Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}

kubectl --v=8 version

I0205 10:13:59.300776   70863 loader.go:357] Config loaded from file /Users/hayd/.bluemix/plugins/container-service/clusters/dmhIKSCluster/kube-config-dal10-dmhIKSCluster.yml
I0205 10:13:59.301266   70863 round_trippers.go:384] GET https://c2.us-south.containers.cloud.ibm.com:26705/version
I0205 10:13:59.301276   70863 round_trippers.go:391] Request Headers:
I0205 10:13:59.301281   70863 round_trippers.go:394]     Accept: application/json, */*
I0205 10:13:59.301286   70863 round_trippers.go:394]     User-Agent: kubectl/v1.10.11 (darwin/amd64) kubernetes/637c7e2
I0205 10:13:59.883868   70863 round_trippers.go:409] Response Status: 200 OK in 582 milliseconds
I0205 10:13:59.883907   70863 round_trippers.go:412] Response Headers:
I0205 10:13:59.883923   70863 round_trippers.go:415]     Date: Tue, 05 Feb 2019 10:13:59 GMT
I0205 10:13:59.883938   70863 round_trippers.go:415]     Content-Type: application/json
I0205 10:13:59.883951   70863 round_trippers.go:415]     Content-Length: 267
I0205 10:13:59.885396   70863 request.go:874] Response Body: {
  "major": "1",
  "minor": "11",
  "gitVersion": "v1.11.6+IKS",
  "gitCommit": "002d263ed027db260968616b951fb46f2bab9bb1",
  "gitTreeState": "clean",
  "buildDate": "2019-01-09T08:07:22Z",
  "goVersion": "go1.10.3",
  "compiler": "gc",
  "platform": "linux/amd64"
}
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.11", GitCommit:"637c7e288581ee40ab4ca210618a89a555b6e7e9", GitTreeState:"clean", BuildDate:"2018-11-26T14:38:32Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.6+IKS", GitCommit:"002d263ed027db260968616b951fb46f2bab9bb1", GitTreeState:"clean", BuildDate:"2019-01-09T08:07:22Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}

Now the other thing to notice is that the HTTP request is shown in all three cases: -

GET https://c2.us-south.containers.cloud.ibm.com:26705/version

Therefore, we can also use a REST client such as cURL to access the same information - obviously, in the example of kubectl version this will only show us the server-side version; the client-side is local and requires no network access.

Here's an example using cURL: -

curl -k -X GET -H "Authorization: Bearer $TOKEN" https://c2.us-south.containers.cloud.ibm.com:26705/version

{

  "major": "1",

  "minor": "11",

  "gitVersion": "v1.11.6+IKS",

  "gitCommit": "002d263ed027db260968616b951fb46f2bab9bb1",

  "gitTreeState": "clean",

  "buildDate": "2019-01-09T08:07:22Z",

  "goVersion": "go1.10.3",

  "compiler": "gc",

  "platform": "linux/amd64"

}

There's a couple of things to note: -

• We're using the -k switch to tell cURL to ignore the fact that the endpoint ( the K8S cluster ) is using a self-signed, rather than CA-signed certificate. Whilst self-signed is perfectly secure, the signer certificate won't be in the local ( to the cURL client ) trust store
• We're passing the content of an environment variable - $TOKEN - which contains the Bearer authentication token; this is stored in the local KUBECONFIG configuration file, and is generated when one logs into the underlying IBM Cloud environment, using the command: -
• ibmcloud login -a https://api.ng.bluemix.net --sso

I've written a basic script to grab the authorisation token from the KUBECONFIG file and stuff it into the TOKEN environment variable: -

export TOKEN=`cat $KUBECONFIG|grep -i id-token|awk '{print $2}'`

So, having established that kubectl is really using REST over HTTPS under the covers, here's a more  useful example: -

kubectl get nodes

NAME      STATUS     ROLES     AGE       VERSION

x.x.x.x   NotReady      17h       v1.11.5

x.x.x.x   Ready          6d        v1.11.6+IKS

x.x.x.x   Ready          5d        v1.11.5

and here's the cURL equivalent: -

curl -k -X GET -H "Authorization: Bearer $TOKEN" https://c2.us-south.containers.cloud.ibm.com:26705/api/v1/nodes

which provides a LOT more output :-)