TIL Docker - more command-line switches

I have a list of Docker containers running on one of my IBM Z boxes: -

docker ps -a

CONTAINER ID        IMAGE               COMMAND             CREATED              STATUS              PORTS               NAMES
6d1e6c4dde9c        fc23c99c5777        "entry.sh init"     45 seconds ago       Up 44 seconds                           Dave-0000863x5o8rs
16ef58de3f35        fc23c99c5777        "entry.sh init"     About a minute ago   Up About a minute                       Dave-000085nktqvy0
6c92bd119be7        fc23c99c5777        "entry.sh init"     About a minute ago   Up About a minute                       Dave-000085brfoc7a
4da853f8f67c        fc23c99c5777        "entry.sh init"     2 minutes ago        Up 2 minutes                            Dave-000084ud02iu8
7fa273f30533        fc23c99c5777        "entry.sh init"     3 minutes ago        Up 3 minutes                            Dave-000084be142eo
5c517f926f9f        fc23c99c5777        "entry.sh init"     3 minutes ago        Up 3 minutes                            Dave-000083swh54zl
ea7e9f9843b6        fc23c99c5777        "entry.sh init"     4 minutes ago        Up 4 minutes                            Dave-000083b1da2s0

I want to grab just the container name, in order to do some scripting.

Ordinarily, I'd just use awk as per this: -

docker ps -a | awk '{print $11}'

a
a
Dave-000085brfoc7a
Dave-000084ud02iu8
Dave-000084be142eo
Dave-000083swh54zl
Dave-000083b1da2s0

Guess what ?

Docker has a better way ..... here's --filter

docker ps --filter name='Dave'

CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
6d1e6c4dde9c        fc23c99c5777        "entry.sh init"     4 minutes ago       Up 4 minutes                            Dave-0000863x5o8rs
16ef58de3f35        fc23c99c5777        "entry.sh init"     5 minutes ago       Up 5 minutes                            Dave-000085nktqvy0
6c92bd119be7        fc23c99c5777        "entry.sh init"     5 minutes ago       Up 5 minutes                            Dave-000085brfoc7a
4da853f8f67c        fc23c99c5777        "entry.sh init"     6 minutes ago       Up 6 minutes                            Dave-000084ud02iu8
7fa273f30533        fc23c99c5777        "entry.sh init"     7 minutes ago       Up 7 minutes                            Dave-000084be142eo
5c517f926f9f        fc23c99c5777        "entry.sh init"     7 minutes ago       Up 7 minutes                            Dave-000083swh54zl
ea7e9f9843b6        fc23c99c5777        "entry.sh init"     8 minutes ago       Up 8 minutes                            Dave-000083b1da2s0

But, hang on, all that's done is filter by name .... we want JUST the name column to appear - here's --format 

docker ps --filter name='Dave*' --format {{.Names}}

Dave-0000863x5o8rs
Dave-000085nktqvy0
Dave-000085brfoc7a
Dave-000084ud02iu8
Dave-000084be142eo
Dave-000083swh54zl
Dave-000083b1da2s0

That's the ticket ....

Now we can get really purty: -

docker ps --filter name='Dave*' --format "My name is {{.Names}} and I have been {{.Status}}"

My name is Dave-000083b1da2s0 and I have been Up About an hour

With thanks to this: -

docker ps

Updating my iPad - watching what's going on

I'm restoring my iPad, reverting to a backup, and am watching the lack of progress within Finder: -

I found a better way: -

while true;  do clear;  ls -lhrt ~/Library/iTunes/iPad\ Software\ Updates;  sleep 20;  done

which is WAY better.

Now I can keep an eye on the download, ready for the update/restore

Thanks to this: -

Question: Q: iOS Software Download Progress in Finder

for the insight 

And an update - doing the same whilst I update an iPhone to iOS 14 ....

while true; do clear; ls -lhrt ~/Library/iTunes/iPhone\ Software\ Updates/; sleep 20; done

total 7536776

-rw-r--r--  1 hayd  staff   3.6G 21 Sep 14:50 iPhone_5.5_14.0_18A373_Restore.ipsw.download

And this - IPSW Downloads gives me an idea how big the download should be i.e. 4.34 GB

A follow-up - Java and Maven

Following my earlier post: -

Another cup of Java, Joe

I forgot to mention that I'm running my Jenkins Plugin using Apache Maven 

Given that I'm using Java 8: -

java -version

java version "1.8.0_251"
Java(TM) SE Runtime Environment (build 1.8.0_251-b08)
Java HotSpot(TM) 64-Bit Server VM (build 25.251-b08, mixed mode)

I wanted to ensure that Maven was using the same version: -

mvn -version

Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
Maven home: /usr/local/Cellar/maven/3.6.3_1/libexec
Java version: 1.8.0_251, vendor: Oracle Corporation, runtime: /Library/Java/JavaVirtualMachines/jdk1.8.0_251.jdk/Contents/Home/jre
Default locale: en_GB, platform encoding: UTF-8
OS name: "mac os x", version: "10.15.6", arch: "x86_64", family: "Mac"

So that's a "Yep" !

Another cup of Java, Joe

For my most recent project, I'm co-developing a plugin for Jenkins which required me to dust off my Java skills - it's been a while since I was coding Enterprise Java Beans, servlets, portlets and JavaServer Pages etc.

For the record, my first job in Hursley in 1997 required me to have some knowledge of Java - I went to a book store ( remember them ? ) and bought a book on .... JavaScript.

Surely they're the same, I thought ....... Can you say "Doh" ?

Anyhow, this most recent challenge requires me to code against the Jenkins and Docker APIs using Java 8.

On my Mac, I have multiple versions of Java installed, including Java 8 itself, as per System Preferences -> Java : -

However, I'm also using Microsoft Visual Studio as my Integrated Development Environment (IDE), which, as per this, requires Java 11.

Therefore, I have three Java versions installed.

One nice thing - macOS has a neat way to manage the installed Java Runtime Environments (JRE) and Java Development Kits (JDK) : -

/usr/libexec/java_home -V

Matching Java Virtual Machines (3):
    11.0.8, x86_64: "AdoptOpenJDK 11" /Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home
    1.8.0_262, x86_64: "AdoptOpenJDK 8" /Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home
    1.8.0_251, x86_64: "Java SE 8" /Library/Java/JavaVirtualMachines/jdk1.8.0_251.jdk/Contents/Home

/Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home

And one can choose the version thusly: -

export JAVA_HOME=`/usr/libexec/java_home -v 1.8.0_251`; java -version

java version "1.8.0_251"
Java(TM) SE Runtime Environment (build 1.8.0_251-b08)
Java HotSpot(TM) 64-Bit Server VM (build 25.251-b08, mixed mode)

export JAVA_HOME=`/usr/libexec/java_home -v 1.8.0_262`; java -version

openjdk version "1.8.0_262"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_262-b10)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.262-b10, mixed mode)

export JAVA_HOME=`/usr/libexec/java_home -v 11.0.8`; java -version

openjdk version "11.0.8" 2020-07-14
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.8+10)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.8+10, mixed mode)

which allows me to set the requisite version in ~/.bash_profile.

vi ~/.bash_profile

export JAVA_HOME=`/usr/libexec/java_home -v 1.8.0_251`; java -version

source ~/.bash_profile

java version "1.8.0_251"
Java(TM) SE Runtime Environment (build 1.8.0_251-b08)
Java HotSpot(TM) 64-Bit Server VM (build 25.251-b08, mixed mode)

The upside of having java -version added to the profile is that the Java version is reported whenever I open Terminal: -

kubectl and the failing versions

I was seeing the following: -

18:00:24  error: error fetching provider config: parsing time "0" as "Mon, 02 Jan 2006 15:04:05 MST": cannot parse "0" as "Mon"

from a Smoke Test running in Jenkins against a Kubernetes cluster.

In part, the Smoke Test uses kubectl to retrieve nodes, deployments and services.

Long story short, this was an issue of K8s versions.

There was a clue: -

17:59:57  Client Version: v1.18.6

17:59:57  Server Version: v1.16.2+IKS

My cluster is running K8s v1.16.2 whereas the version of kubectl is v1.18.6.

Therefore, there was a 2+ version mismatch between the two.

The problem was relatively simple ...

The Jenkins Pipeline was invoking a Bash script which was running kubectl from the path - which was picking up the default, latest version - v1.18.6 - whereas it should've been running it from the current directory ( via ./kubectl ).

Once I realised this, and remediated it, all was well .......

Pruning Docker - And there's more ....

Now I've written about this before, in the context of cleaning down Docker environments, but I found a useful tip today, to help remove old/unwanted Docker images from a box.

These are images that are effectively unnamed / untagged, as per this example: -

docker images

<none>                                                           <none>                         4a492ff5f010        11 days ago         2.66GB

My Ubuntu build server had LOADS of these.

Put it this way, the command: -

df -kmh /var

returned this: -

/dev/mapper/basevg-var  433G  318G   94G  78% /var

most of which was taken up by /var/lib/docker

These unbranded images are known as "Dangling Images" and can be queried thusly: -

docker images -f "dangling=true" 

REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
<none>              <none>              4a492ff5f010        11 days ago         2.66GB

and pruned accordingly: -

docker rmi `docker images -f "dangling=true" -q`

As mentioned before, I also use this command: -

docker rm `docker ps -a|grep Exited|awk '{print $1}'`

to remove stopped containers.

SSH and Fun with Qualities of Service

An IBM colleague was having fun and games SSHing into certain internal VMs, and was seeing: -

client_loop: send disconnect: Broken pipe

Through trial and error, including judicious use of the ssh -v flag, we reached a conclusion that it was somehow related to his connectivity via his home WiFi network, combined with the corporate VPN.

At one stage, I had validate that he could SSH into GitHub: -

ssh -v git@github.com

just to confirm that the problem appeared to only occur with the combination of SSH, home internet connection and corporate VPN.

A quick Google took me here: -

OpenSSH broken-"client_loop: send disconnect: Broken pipe"

which took me here: -

Fix "Broken Pipe" error on SSH connection in Fedora/VMWare

which in part, said this: -

Searching for solutions, I finally found one that worked at https://superuser.com/questions/1371506/from-vmware-and-fedora-29-ssh-to-remote-machine-cause-broken-pipe.

The solution is to set the IPQoS option to throughput. You can do it from the command line or in the SSH config file.

Set option via CLI

You can set the IPQoS option via the command-line whenever you connect like this:

ssh example.com -o IPQoS=throughput

Set option in SSH config

You can set the IPQoS option in the config file (e.g. $HOME/.ssh/config) like this:

# This will apply to all hosts.
Host *

    

IPQoS=throughput

I had my colleague try adding -o IPQoS=throughput to his existing SSH command and ... it bloomin' well worked .....

There was a similar thread on the VMware forum a few years back: -

SSH connections dropped due to QoS flags

Nice!

Modernize a monolithic Node.js application into a microservices architecture using IBM Cloud Pak for Applications

From one of my IBM colleagues, we have this: -

This tutorial shows how to transform a traditional monolithic core banking application, which is implemented in Node.js, into a modern microservices architecture by using IBM Cloud Pak for Applications.

Cloud Pak for Applications speeds the development of applications that are built for Kubernetes by using agile DevOps processes. Running on Red Hat OpenShift, the Cloud Pak provides a hybrid, multicloud foundation that is built on open standards, enabling workloads and data to run anywhere. It integrates two main open source projects: Kabanero and Appsody.

Modernize a monolithic Node.js application into a microservices architecture using IBM Cloud Pak for Applications

TIL - More about Developer Advocacy on IBM Z

During this evening's most excellent MainframerZ Meetup event, one of our awesome Developer Advocates, Jenn Francis, mentioned a couple of really really useful links: -

https://developer.ibm.com/components/ibmz/

https://www.ibm.com/community/z/

Encrypt Kubernetes secrets with IBM Cloud Hyper Protect Crypto Services

This was recently authored by a couple of my IBM colleagues: -

Encrypt Kubernetes secrets with IBM Cloud Hyper Protect Crypto Services

Create a secret in Kubernetes, create a root key in Hyper Protect Crypto Services, and enable KMS encryption in Kubernetes

This tutorial shows you how to encrypt your Kubernetes secrets using IBM Cloud Hyper Protect Crypto Services as the KMS provider. You'll learn how to create a secret in IBM Cloud Kubernetes, create a root key in Hyper Protect Crypto Services, and encrypt the secrets and etcd component of your Kubernetes master with the root key in your Hyper Protect Crypto Services instance.

https://developer.ibm.com/tutorials/encrypt-kubernetes-secrets-with-hyper-protect-crypto-services/