Wednesday, 23 January 2019

W00t, IBM Microclimate running on IBM Cloud Private ...

So another "Voyage of Discovery" post .....

I'm tinkering with IBM Microclimate : -

Microclimate provides an end-to-end, cloud-native solution for creating, building, testing and deploying applications. The solution offers services and tools to help you create and modernize applications in one seamless experience. It covers each step of the process from writing and testing code to building and deployment. The solution enables containerized development, rapid iteration with real-time performance insights, intelligent feedback, diagnostic services, an integrated DevOps pipeline and deployment to the cloud.

also well documented here: -

Microclimate is an end to end development environment that lets you rapidly create, edit, and deploy applications. Applications are run in containers from day one and can be delivered into production on Kubernetes through an automated DevOps pipeline using Jenkins. Microclimate can be installed locally or on IBM Cloud Private, and currently supports Java, Node.js, and Swift.

I've played with this before: -

Playing with Microclimate on IBM Cloud Private

Microclimate on IBM Cloud Private - Permission to write

and will be posting my own build notes, but I'm still following the official documentation here: -

Having followed all of the pre-requisite steps ( which mainly involve creating lots of artefacts using kubectl ), and having installed the Helm chart, I was following this tutorial: -

Create and deploy Lagom Reactive applications with Microclimate

but found that the resulting Docker container would never start.

I dug around within IBM Cloud Private (ICP) or, to be more accurate, within Kubernetes, upon which ICP is built.

Microclimate comprises a number of Pods, sitting within a dedicated namespace - I'm using micro-climate : -

kubectl get pods -n micro-climate

NAME                                                              READY     STATUS    RESTARTS   AGE
mc-adamjava-381f2a40-1ef8-11e9-bb42-adamjava-5c9697c464-dbqdk     1/1       Running   0          27m
mc-yoda-22f5f600-1efb-11e9-964a-yoda-bbb88b5d4-lqzwh              1/1       Running   0          52m
microclimate-ibm-microclimate-67cfd99c7b-bj7p2                    1/1       Running   0          55m
microclimate-ibm-microclimate-admin-editor-77ddbdd86-xzlbj        2/2       Running   0          53m
microclimate-ibm-microclimate-admin-filewatcher-6cc6c785cf6lsjx   1/1       Running   0          53m
microclimate-ibm-microclimate-admin-loadrunner-856b4b48b6-jqlmc   1/1       Running   0          53m
microclimate-ibm-microclimate-atrium-7f75d754fd-fp244             1/1       Running   0          1h
microclimate-ibm-microclimate-devops-568c4c5989-kjcqs             1/1       Running   0          1h
microclimate-jenkins-678584959-64jlm                              1/1       Running   0          1h

Given that all were running happily, I chose to dive into the logs of the File Watcher pod: -

kubectl logs -f microclimate-ibm-microclimate-admin-filewatcher-6cc6c785cf6lsjx -n micro-climate

and spotted this: -

[ERROR Tue Jan 22 13:58:43 UTC 2019 | Project: foobar | File Name: null | Function Name: null | Line Number: null] _tickCallback : 189 | unauthorized: authentication required

This took me down the rabbit hole of testing that I could push Docker images to the local registry that's part of the ICP cluster: -

Pushing Docker images to IBM Cloud Private

but Microclimate still refused to play ball.

It did, however, confirm my suspicion that the problem was with the credentials between Microclimate and the Docker registry.

I looked back at my build notes, and saw that I'd wrongly read this: -

Create the Microclimate registry secret

This secret is used by both Microclimate and Microclimate's pipelines. It allows images to be pushed and pulled from the private registry on your Kubernetes cluster.

Use the following code to create a Docker registry secret:

kubectl create secret docker-registry microclimate-registry-secret \
  --docker-server=:8500 \
  --docker-username= \
  --docker-password= \

Verify that the secret was created successfully and exists in the target namespace for Microclimate before you continue. This secret does not need to be patched to a service account as the Microclimate installation will manage this step.

as meaning that the secret needed to point at DockerHub i.e. the official Docker registry rather than the ICP registry.

After much faffing around, including a full nuke of the Helm chart, I was able to resolve this ... mainly thanks to some awesome support from the IBM Microclimate developer team in Hursley :-)

At one point, I stupidly retraced the very same steps ( using creds for DockerHub ) because my notes were out-of-sync with reality.

However, once I deleted the microclimate-registry-secret  : -

kubectl delete secret microclimate-registry-secret -n micro-climate

and recreated it: -

kubectl create secret docker-registry microclimate-registry-secret   --docker-server=mycluster.icp:8500   --docker-username=admin   --docker-password=admin

pointing at the ICP server ( my cluster.icp:8500 ) and using the ICP credentials, things started to behave.

Just in case, I nuked the main Microclimate pod: -

kubectl delete pod microclimate-ibm-microclimate-67cfd99c7b-zqs8m 

which forced the Replica Set to spawn a new pod ( remember, cattle, not pets ).

At that point, I was able to create a new Java / Lagom project, and watch it happily start and work :-)

I also noticed that my other failing-to-start project magically started working.

So, the long story very short ( TL;DR; ), get the darn credentials right.

For the record, I also nuked/recreated the other secret - microclimate-pipeline-secret - as follows: -

kubectl create secret docker-registry microclimate-pipeline-secret \
  --docker-server=mycluster.icp:8500 \
  --docker-username=admin \
  --docker-password=admin \

So things are now working - now I'm going to rip it all down and REDO FROM START, to see what I've learned :-)

PS For what it's worth, a quick cheat to get the logs of the File Watcher pod, regardless of it's name is: -

kubectl logs -f `kubectl get pods -n micro-climate | grep -i watcher | awk '{print $1}'` -n micro-climate

No comments:

Grokking grep

A colleague was tinkering with grep  and, thanks to him, I discovered a bit more about the trusty little utility. I had not really explored ...