Wednesday, 4 August 2021

Tinkering with Rust - an underscore

Whilst tinkering with the Kata Containers kata-agent component, I was trying to work out what the underscore meant in this line of code: -

let _ = spec.save(config_path.to_str().unwrap());

Thankfully, this GitHub issue: -

Documet let _ = ... behavior saliently, or even warn about it #40096

led me to this section of the Rust language book: -

Ignoring Values in a Pattern

You’ve seen that it’s sometimes useful to ignore values in a pattern, such as in the last arm of a match, to get a catchall that doesn’t actually do anything but does account for all remaining possible values. There are a few ways to ignore entire values or parts of values in a pattern: using the _ pattern (which you’ve seen), using the _ pattern within another pattern, using a name that starts with an underscore, or using .. to ignore remaining parts of a value. Let’s explore how and why to use each of these patterns.

and: -

We’ve used the underscore (_) as a wildcard pattern that will match any value but not bind to the value. Although the underscore _ pattern is especially useful as the last arm in a match expression, we can use it in any pattern, including function parameters, as shown in Listing 18-17.

Ignoring an Entire Value with _



Wednesday, 28 July 2021

Fun and games with sed and unterminated commands in Jenkins

So it took me ~3 hours to fix a Bug that should've taken ~10 minutes ...

I was trying to mitigate an issue with one of our Alpine Linux-based images, where our IBM Container Registry (ICR) Vulnerability Advisor (VA) tool was (rightly) complaining about our exposure to CVE-2021-36159 with apk-tools.

I knew that the mitigation was to update the Dockerfile to ensure that this package was updated.

However, given that I'm building from someone else's GH project, where I don't control the Dockerfile per se, I wanted to have my Jenkins Pipeline job amend the Dockerfile "in flight".

So the Dockerfile had the line: -

FROM alpine:3.13 AS run

so I used a bit of sed sweetness to add: -

RUN apk --no-cache upgrade apk-tools

How hard can it be ?

I even tested it using Bash: -

echo "RUN apk --no-cache add procps" > /tmp/foo.txt

cat /tmp/foo.txt

RUN apk --no-cache add procps

sed -i 's/RUN apk --no-cache add procps/RUN apk --no-cache add procps\nRUN apk --no-cache upgrade apk-tools/g' /tmp/foo.txt

cat /tmp/foo.txt

RUN apk --no-cache add procps
RUN apk --no-cache upgrade apk-tools

Easy right ?

Nah, not with Bash embedded in Groovy via a Jenkinsfile ...

Each and every time I ran my Pipeline, the sed command threw up: -

10:45:38  sed: -e expression #1, char 61: unterminated `s' command

etc.

I tried various different incarnations .... with different separators, including / and ; but to no avail.

The internet had the answer, as per usual ....


specifically this: -

2. Insert lines using Regular expression

which provided the following example: -

sed '/PATTERN/ i <LINE-TO-BE-ADDED>' FILE.txt

I tested this manually: -

echo "RUN apk --no-cache add procps" > /tmp/foo.txt

cat /tmp/foo.txt

RUN apk --no-cache add procps

sed -i "/RUN apk --no-cache add procps/a RUN apk --no-cache upgrade apk-tools" /tmp/foo.txt

cat /tmp/foo.txt

RUN apk --no-cache add procps
RUN apk --no-cache upgrade apk-tools

And, better still, it worked within Jenkins: -

11:42:04  Step 14/29 : RUN apk --no-cache add procps
11:42:04   ---> Running in 9505a71400bb
11:42:04  fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/main/s390x/APKINDEX.tar.gz
11:42:04  fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/community/s390x/APKINDEX.tar.gz
11:42:04  (1/5) Installing libintl (0.20.2-r2)
11:42:04  (2/5) Installing ncurses-terminfo-base (6.2_p20210109-r0)
11:42:04  (3/5) Installing ncurses-libs (6.2_p20210109-r0)
11:42:04  (4/5) Installing libproc (3.3.16-r0)
11:42:04  (5/5) Installing procps (3.3.16-r0)
11:42:04  Executing busybox-1.32.1-r6.trigger
11:42:04  OK: 7 MiB in 19 packages

11:42:05  Removing intermediate container 9505a71400bb
11:42:05   ---> a4947b0b1d8d
11:42:05  Step 15/29 : RUN apk --no-cache upgrade apk-tools

which is nice :-)

I've raised an issue with the original project's GH repo, as it'd be better to get apk-tools upgraded at "source" so to speak, but I'm rather happy with my experience - every day is, indeed, a school day


Tuesday, 27 July 2021

Fun and games with sudo and Go in Ubuntu 20.04

Whilst reviewing a colleague's documentation, where he was describing how to code in Go, I noticed a discrepancy in the way that things work when one runs a command as a non-root user vs. running as root, via Super User Do ( sudo )

Having downloaded / unpacked Go thusly: -

sudo wget -c https://golang.org/dl/go1.16.6.linux-amd64.tar.gz -O - | sudo tar -xz -C /usr/local

resulting in Go being installed in /usr/local/go with the actual go binary being in /usr/local/go/bin

ls -al /usr/local/go/bin

total 17128
drwxr-xr-x  2 root root     4096 Jul 12 20:04 .
drwxr-xr-x 10 root root     4096 Jul 12 20:01 ..
-rwxr-xr-x  1 root root 14072999 Jul 12 20:04 go
-rwxr-xr-x  1 root root  3453176 Jul 12 20:04 gofmt

we needed to run a build ( of containerd ) as root, via sudo make and sudo make install

However, this failed: -

cd ~/containerd

sudo make

+ bin/ctr

/bin/sh: 1: go: not found

make: *** [Makefile:219: bin/ctr] Error 127

even though I'd confirmed that go was installed: -

which go

/usr/local/go/bin/go

ls -al `which go`

-rwxr-xr-x 1 root root 14072999 Jul 12 20:04 /usr/local/go/bin/go

go version

go version go1.16.6 linux/amd64

as I'd previously added it to my PATH via this line in ~/.profile : -

export PATH=$PATH:/usr/local/go/bin

Of course, the wrinkle was that I'm running go as root, via sudo ....

This totally helped: -

bash profile works for user but not sudo

specifically the answer that had me validate the PATH in both cases: -

echo 'echo $PATH' | sh

/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/go/bin

echo 'echo $PATH' | sudo sh 

/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin

Note the difference .... ( apart from /usr/games of course ) ?

The answer then had me alias sudo in ~/.bashrc as follows: -

alias sudo='sudo env PATH=$PATH'

validated with: -

alias

alias alert='notify-send --urgency=low -i "$([ $? = 0 ] && echo terminal || echo error)" "$(history|tail -n1|sed -e '\''s/^\s*[0-9]\+\s*//;s/[;&|]\s*alert$//'\'')"'
alias egrep='egrep --color=auto'
alias fgrep='fgrep --color=auto'
alias grep='grep --color=auto'
alias hist='history | cut -c 8-'
alias l='ls -CF'
alias la='ls -A'
alias ll='ls -alF'
alias ls='ls --color=auto'
alias sudo='sudo env PATH=$PATH'

and now the validation works: -

echo 'echo $PATH' | sh

/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/go/bin

echo 'echo $PATH' | sudo sh 

/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/go/bin

and, better still, the build works: -

cd ~/containerd/

sudo make
+ bin/ctr
+ bin/containerd
+ bin/containerd-stress
+ bin/containerd-shim
+ bin/containerd-shim-runc-v1
+ bin/containerd-shim-runc-v2
+ binaries

sudo make install

+ install bin/ctr bin/containerd bin/containerd-stress bin/containerd-shim bin/containerd-shim-runc-v1 bin/containerd-shim-runc-v2

which is nice

Also, make me a sandwich .....

Sunday, 25 July 2021

Nesting VMs - not quite as cosy as it sounds....

I wrote about this a few months back: -

Kata Containers and Ubuntu Linux - lessons learned - 3/many - a WIP

in the context of VM nesting being a pain ....

For context, I'm trying ( and failing ) to get Kata Containers fully running inside an Ubuntu VM running on VMware Fusion on my Mac.

This is what I currently have: -

Host: macOS 11.5 Big Sur

Virtualisation: VMware Fusion 12.1.2

Guest: Ubuntu 20.04.2 LTS

Kernel: 5.4.0-80-generic #90-Ubuntu SMP

kata-runtime  : 2.1.0

QEMU: 5.2.0

I've been experimenting with various container runtimes here, including containerd and CRI-O 

However, each and every time I'm hitting the same nested virtualisation issue

Most recently, when I try and use crictl and runp to start a container using the Kata 2.0 runtime: -

sudo crictl runp test/testdata/sandbox_config.json

FATA[0002] run pod sandbox: rpc error: code = Unknown desc = CreateContainer failed: failed to launch qemu: exit status 1, error messages from qemu log: qemu-system-x86_64: error: failed to set MSR 0x48d to 0x5600000016
qemu-system-x86_64: ../target/i386/kvm.c:2701: kvm_buf_set_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.
: unknown 

I've tried various hacks to mitigate this but I just cannot get past it ...

More digging is required, or this combination is a bust - thankfully I have many other options, including IBM Cloud Virtual Servers.......

More to come ....

For a future me - remembering to stop VMware Tools service when running Kata Containers on Ubuntu on VMware on macOS

I keep writing and re-writing this script, because I forgot to store it somewhere memorable ....

The problem to be solved is that the Kata Containers runtime, aka kata-runtime, will not start unless I remember to stop the VMware Tools service, aka open-vm-tools.service and unload a few dependent modules.

At present, I'm running Kata via a snap installation - I'm not terribly keen on that approach, as there seem to be too many fiddly configuration things to-do, but I'll look at that another day, and perhaps raise an issue in the Kata repo.

Meantime, this is what I have: -

/snap/kata-containers/current/usr/bin/kata-runtime -version

kata-runtime  : 2.1.0

   commit   : 0f822919268e4095dd9bdbbb2351248b53746501

   OCI specs: 1.0.1-dev

and, when I run the check tool: -

/snap/kata-containers/current/usr/bin/kata-runtime check

I get this: -

WARN[0000] Not running network checks as super user      arch=amd64 name=kata-runtime pid=1431 source=runtime
WARN[0000] modprobe insert module failed                 arch=amd64 error="exit status 1" module=vhost_vsock name=kata-runtime output="modprobe: ERROR: could not insert 'vhost_vsock': Device or resource busy\n" pid=1431 source=runtime
ERRO[0000] kernel property not found                     arch=amd64 description="Host Support for Linux VM Sockets" name=vhost_vsock pid=1431 source=runtime type=module
System is capable of running Kata Containers
System can currently create Kata Containers

The vhost_vsock module cannot be loaded, as evidenced by this: -

modprobe vhost_vsock

modprobe: ERROR: could not insert 'vhost_vsock': Device or resource busy

lsmod | grep vhost

vhost_net              32768  0
tap                    24576  1 vhost_net
vhost                  49152  1 vhost_net

Somewhere I realised / discovered / found out that VMware Tools, more specifically open-vm-tools, was responsible :-)

Given that, for my use cases at the moment, I don't need VMware Tools for e.g. GUI interactions between guest and host, shared folders etc. I chose to simply disable it.

I've got a script for that: -

~/stop_tools.sh 

#!/bin/bash
systemctl stop open-vm-tools.service
rmmod vmw_vsock_virtio_transport_common
rmmod vmw_vsock_vmci_transport
rmmod vsock
rmmod vhost_net
rmmod vhost

So I can run my script: -

~/stop_tools.sh 

and check that neither vhost* or vsock* modules are now loaded: -

lsmod | grep vs

<NOTHING RETURNED>

lsmod | grep vh

<NOTHING RETURNED>

and then load the vhost_vsock module: -

modprobe vhost_vsock

and check the loaded modules: -

lsmod | grep vs

vhost_vsock            24576  0
vmw_vsock_virtio_transport_common    32768  1 vhost_vsock
vhost                  49152  1 vhost_vsock
vsock                  36864  2 vmw_vsock_virtio_transport_common,vhost_vsock

lsmod | grep vh

vhost_vsock            24576  0
vmw_vsock_virtio_transport_common    32768  1 vhost_vsock
vhost                  49152  1 vhost_vsock
vsock                  36864  2 vmw_vsock_virtio_transport_common,vhost_vsock

and, even more importantly, use Kata: -

/snap/kata-containers/current/usr/bin/kata-runtime check

WARN[0000] Not running network checks as super user      arch=amd64 name=kata-runtime pid=1911 source=runtime
System is capable of running Kata Containers
System can currently create Kata Containers

/snap/kata-containers/current/usr/bin/kata-runtime kata-check

WARN[0000] Not running network checks as super user      arch=amd64 name=kata-runtime pid=1926 source=runtime
System is capable of running Kata Containers
System can currently create Kata Containers

I could probably hack a better solution and/or uninstall VMware Tools but .....

Friday, 23 July 2021

A reprise - growing disks in Ubuntu

 I've written this many times over the years, with many different Linux distributions, but had a need to REDO FROM START yesterday, growing the disk within an Ubuntu VM running on VMware Fusion.

So, for my future self, this is what worked for me ( using Ubuntu 20.04.02 LTS )

Look at the current disk

df -kmh /

Filesystem                         Size  Used Avail Use% Mounted on

/dev/mapper/ubuntu--vg-ubuntu--lv   19G  8.5G  9.2G  49% /

Increase VM from 20 to 50 GB

This is easily done via VMware Fusion: -


I grew the disk from 20 GB to 50 GB whilst the VM was shutdown, and then booted it up.

Inspect the current partition layout

fdisk /dev/sda -l

GPT PMBR size mismatch (41943039 != 104857599) will be corrected by write.

Disk /dev/sda: 50 GiB, 53687091200 bytes, 104857600 sectors

Disk model: VMware Virtual S

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disklabel type: gpt

Disk identifier: DA7D21AD-C6E5-4549-B753-391F2EB88C8E


Device       Start      End  Sectors Size Type

/dev/sda1     2048     4095     2048   1M BIOS boot

/dev/sda2     4096  2101247  2097152   1G Linux filesystem

/dev/sda3  2101248 41940991 39839744  19G Linux filesystem

Create a new partition

I did this via CLI, but could've quite easily used fdisk interactively. Also, I took the defaults on start, end, sectors, partition type etc.

(

echo "n"

echo -e "\n"

echo -e "\n"

echo -e "\n"

echo "w"

) | fdisk /dev/sda

Inspect the new partition layout

fdisk /dev/sda -l

Disk /dev/sda: 50 GiB, 53687091200 bytes, 104857600 sectors

Disk model: VMware Virtual S

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disklabel type: gpt

Disk identifier: DA7D21AD-C6E5-4549-B753-391F2EB88C8E


Device        Start       End  Sectors Size Type

/dev/sda1      2048      4095     2048   1M BIOS boot

/dev/sda2      4096   2101247  2097152   1G Linux filesystem

/dev/sda3   2101248  41940991 39839744  19G Linux filesystem

/dev/sda4  41940992 104857566 62916575  30G Linux filesystem

Create a Physical Volume using the new sda4 partition

pvcreate /dev/sda4

  Physical volume "/dev/sda4" successfully created.

Extend the existing Volume Group to use the new sda4 partition  

vgextend /dev/ubuntu-vg/ /dev/sda4

  Volume group "ubuntu-vg" successfully extended

Extend the Logical Volume to fit

- Note that I deliberately chose the value of 29G to fit inside the newly added 30GB of disk

lvextend -L +29G /dev/ubuntu-vg/ubuntu-lv

  Size of logical volume ubuntu-vg/ubuntu-lv changed from <19.00 GiB (4863 extents) to <48.00 GiB (12287 extents).

  Logical volume ubuntu-vg/ubuntu-lv successfully resized.

Resize the file-system to fit  

resize2fs /dev/mapper/ubuntu--vg-ubuntu--lv

resize2fs 1.45.5 (07-Jan-2020)

Filesystem at /dev/mapper/ubuntu--vg-ubuntu--lv is mounted on /; on-line resizing required

old_desc_blocks = 3, new_desc_blocks = 6

The filesystem on /dev/mapper/ubuntu--vg-ubuntu--lv is now 12581888 (4k) blocks long.

Look at the current disk

df -kmh /

Filesystem                         Size  Used Avail Use% Mounted on

/dev/mapper/ubuntu--vg-ubuntu--lv   48G  8.5G   37G  19% /

Celebrate !


Thursday, 15 July 2021

Gah, problems building Kata Containers but totally self-inflicted

 Whilst trying to build the kernel using Kata Containers, I kept hitting a problem which appeared to be with the make command.

As an example: -

./build-kernel.sh setup

/root/go/github.com/src/tools/packaging/kernel/../scripts/lib.sh: line 30: pushd: /root/go/src/github.com/kata-containers/tests: No such file or directory
/root/go/github.com/src/tools/packaging/kernel/../scripts/lib.sh: line 31: .ci/install_yq.sh: No such file or directory
/root/go/github.com/src/tools/packaging/kernel/../scripts/lib.sh: line 32: popd: directory stack empty
INFO: Config version: 85
INFO: Kernel version: 5.10.25
INFO: /root/go/github.com/src/tools/packaging/kernel/kata-linux-5.10.25-85 already exist
Kernel source ready: /root/go/github.com/src/tools/packaging/kernel/kata-linux-5.10.25-85

I assumed that this was a missing pre-requisite on my part, harking back to a previous post: -

Kata Containers and Ubuntu Linux - lessons learned - 4/many

Well, kinda but not quite ...

I'm not 100% sure what led me to the realisation but I did finally notice this in the above message: -

/root/go/src/github.com/kata-containers/tests

'cos I'd (inadvertently) cloned the Kata Containers repo into ... 

/root/go/github.com/src

Not sure why but I did ...

The developer guide does make it clear: -

$ go get -d -u github.com/kata-containers/kata-containers
$ cd $GOPATH/src/github.com/kata-containers/kata-containers/src/runtime
$ make && sudo -E PATH=$PATH make install

Build and install the Kata Containers runtime

Once I did it properly: -

git clone git@github.com:kata-containers/kata-containers.git $GOPATH/src/github.com

cd $GOPATH/src/github.com/tools/packaging/kernel

./build-kernel.sh setup

./build-kernel.sh build

all was well, funnily enough

๐Ÿ˜ฎ‍๐Ÿ’จ๐Ÿ˜ฎ‍๐Ÿ’จ๐Ÿ˜ฎ‍๐Ÿ’จ๐Ÿ˜ฎ‍๐Ÿ’จ๐Ÿ˜ฎ‍๐Ÿ’จ๐Ÿ˜ฎ‍๐Ÿ’จ

Tinkering with Rust - an underscore

Whilst tinkering with the Kata Containers kata-agent component, I was trying to work out what the underscore meant in this line of code: - ...