Monday, 25 November 2019

"WARNING:root:could not open file '/etc/apt/sources.list'"

We saw this: -

WARNING:root:could not open file '/etc/apt/sources.list'

during a complex series of Docker builds.

After a LOT of investigation, we concluded that it was a combination of a slightly malformed /etc/apt/sources.list file AND the use of the add-apt-repository command.

In essence, sources.list was being slightly truncated by a previous build process, meaning that the very last character of the file was NOT a CR/LF, but was some "special character".

The subsequent use of the add-apt-repository in a Dockerfile: -

add-apt-repository "deb [arch=s390x] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

was completely over-writing the sources.list file with the following TWO lines: -

deb [arch=s390x] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable
# deb-src [arch=s390x] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable

I've not quite reproduced the problem manually, despite my best efforts with a hex editor ...

However, the TL;DR; of the situation is that malformed lines in sources.list may be the root cause of all sorts of problems .....

Thursday, 14 November 2019

IBM Cloud Hyper Protect Virtual Servers

This is, in part, upon what I've been working this past six months or so ....

Glad to see a useful rundown of the services and it's capabilities : -

IBM Cloud Hyper Protect Virtual Servers 

Gain complete authority over workloads with sensitive data or your business IP 

What are IBM Cloud Hyper Protect Virtual Servers?

IBM Cloud™ Hyper Protect Virtual Servers provides complete authority over workloads with sensitive data within a secure virtual server environment. No one, not even cloud admins, can access your workloads.

Built on the security and reliability of the LinuxONE platform, your business IP is protected through data-at-rest and runtime encryption. You can extend and consume on-prem LinuxONE resources in the cloud for faster development, testing and backup, without sacrificing security.

Monday, 11 November 2019

Fun with OpenSSL Certificate Requests and space characters in Subject Names

I've got a command within a Dockerfile that generates a Certificate Service Request, via the openssl req command.

This references an environment variable that contains the required Subject Name: -

export CRT_SUBJ="/C=US/ST=New York/L=Armonk/O=International Business Machines Corporation/CN=securebuildserver"

When I reference this variable in my Dockerfile: -

RUN openssl req -new -x509 -nodes -out server.crt -keyout server.key -subj $CRT_SUBJ

this fails with: -

unknown option York/L=Armonk/O=International
req [options] outfile

Note that it's failing to digest the Subject Name, specifically at the first space character, between New and York.

The solution ?

Wrap the environment variable in double-quotes: -

RUN openssl req -new -x509 -nodes -out server.crt -keyout server.key -subj "$CRT_SUBJ"

which works a treat ......

Friday, 1 November 2019

Using awk to munge SSH private keys

One of those "because I needed to" answers ....

Having generated a SSH public/private key pair: -

ssh-keygen -t rsa -b 4096 -f /tmp/this_is_a_test -N ""

I wanted to get the private key: -

-----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAACFwAAAAdzc2gtcn
NhAAAAAwEAAQAAAgEAnonsGDeVoFfKyCTzsbof7SL5TWE2a5vWfEHq9fHFNH1WhRYKYm9z
E0SLTnS/uYUT4rs8LfxT4vUYGgoNrtY9BH3V8U624CAsSuKdVc4PQdDt5r7CIZmhqIhEh1
ooMB/Ih0C83j6mU653n5bOBDTf1C8wSBBsZnZoivf0OX/MsHutJZCm1xgMT3NStBPFYhpO
sqqWmMXrOJEBgNR880ECB2SBB29XmmhI2SYlzp3ZvTnGS1FCo4LctGkN7uaMFTMZx4QJgz
v7DCeJuhzOcOSw54F4vJX3JGI9c49CBL7g2bxkzh83UpFZiOzVnwzabzkFIRgfC5eFfcxk
NUxTpCPMpfZX0MAKRMJb59wPSXHPA0pisBEIu9gZOUrDeFmZlwUPuVH4i5wCb/rW2Ynkyf
TKhz8bdC2dCj9/BAUv3axsWNjfgA4J/PvUgZ8yr3+zceRUcG+rzi8AFimR5H4QHMzTP9js
PQ3cQ73OUpK0yl/ER+ibqhfkuz2HlEml8WOEEx5ChzSa+jvxzBBsYS3y4ebGbDbn/hBY2w
L/ZSNeXbfx9iq7docW6U2tVcmuMDN1Voadd1V9cKIWsbJ+/UuPWyLLTw2PjpmyNPQJXPrr
63vXaYskKZOZJ9pFqygZJxe6wepc+XluPkbpf1EmXZKy2wJyJPC/DImCmPDgoT6aslQVCg
sAAAdIY7RrvWO0a70AAAAHc3NoLXJzYQAAAgEAnonsGDeVoFfKyCTzsbof7SL5TWE2a5vW
fEHq9fHFNH1WhRYKYm9zE0SLTnS/uYUT4rs8LfxT4vUYGgoNrtY9BH3V8U624CAsSuKdVc
4PQdDt5r7CIZmhqIhEh1ooMB/Ih0C83j6mU653n5bOBDTf1C8wSBBsZnZoivf0OX/MsHut
JZCm1xgMT3NStBPFYhpOsqqWmMXrOJEBgNR880ECB2SBB29XmmhI2SYlzp3ZvTnGS1FCo4
LctGkN7uaMFTMZx4QJgzv7DCeJuhzOcOSw54F4vJX3JGI9c49CBL7g2bxkzh83UpFZiOzV
nwzabzkFIRgfC5eFfcxkNUxTpCPMpfZX0MAKRMJb59wPSXHPA0pisBEIu9gZOUrDeFmZlw
UPuVH4i5wCb/rW2YnkyfTKhz8bdC2dCj9/BAUv3axsWNjfgA4J/PvUgZ8yr3+zceRUcG+r
zi8AFimR5H4QHMzTP9jsPQ3cQ73OUpK0yl/ER+ibqhfkuz2HlEml8WOEEx5ChzSa+jvxzB
BsYS3y4ebGbDbn/hBY2wL/ZSNeXbfx9iq7docW6U2tVcmuMDN1Voadd1V9cKIWsbJ+/UuP
WyLLTw2PjpmyNPQJXPrr63vXaYskKZOZJ9pFqygZJxe6wepc+XluPkbpf1EmXZKy2wJyJP
C/DImCmPDgoT6aslQVCgsAAAADAQABAAACAES9iKD3eOkNJ9+gJgnpqe4oO/BRNkdySo2W
B0qcieU4c0tBQz9rE3pHPh/Gf1rxWK32gSFvRvAhVVH1CfnE6rAH37vBZYnIaWwO1ileLz
9aV8y06XeoJW/tpckKTmKPZWM/rqSuhW/DNhr8SFeKlYJq84RLmrTSooGfAUXtI9NhzLF9
Bto8kxnf5vwtlmfzACx3ytRy/IqKVRyHBaZziW2rJkcS/+jzTqLY54uu5bomUtRVZ49+Lj
QLWP8894A5p/PaTEWavD+I5S7NTrWkcw/uE2uxmncsIXj5pKZqHRRLm2nEHzitzSSRjA5q
hEu1bYPr8YoVKMxi0xBJB/sjipe8XHPUGskw0XBhklwvGAq6HFMWWNScgNl5wDksb4TPwU
REjcEBsd4CrW1mUNZcaj0lib55TEGcaU9tghmMSmMmjHMF58ZvE3DqaXL+VfjQS1YXiFYX
UBJP4QQl3gDG5ypRCrr69FLahigGCnd/FIWev8BhoFolHft2gb/IJu36JeJ+24V4bHT7oG
2G2LLIoNFe0XL0GvlbyNEKRgtLcAijBhDUZs9UEztH4gmFwAPGisbT+hhlhdC3kpxNARTT
HskggYe//W1ZRX2/pIrw5oRn9g1UZ5A0Syla7Cw0JI0Jgcz5C5YQnMTD58jHOZZeb3zBEZ
XfNrkXPkXEnBkO6dgBAAABAAIePVO0JuF16FYKjdFhKwvGKmgc0+kulT16o9kjFlcj94lx
3MnAMldO8uPEm7IBdQtcF5k6MN3lLjSQF0AN/byJ7tK9+N6o/UQ6VFSR2TkcPc41/J6xH2
oERUXt7pJyygkRjDSotWFnZBzA6LcMiRi5NDjq9wXhNPPbl7Bi1LokZ0Bo6szVfwFSL5F2
cyr20yXqyIX3hIduHfhRrNsMt+DgPHsg45Az3xu1H8eyT2K8YkyMjCDeNmm/PXLv2bQa/f
dszZ+KOa4ORiNj/GnPyqM2vZJwY5wk0RCgCujr9pg7zgzjGxw2BsRxE1/rAASiqnn1iGT3
EPKBFPKViKJ2TtsAAAEBANImJ7PZLFiPwXui9TK/QdhiY9a9+ZE0HOAhVTMvJEMKAtwqak
RpjUVerlnX6KB/F7AX9TV472XTQqJSoHvwhImkWhe5EtHzKjkr/Al8pyQo44yy1Nw0vEfM
ryI7oZddx+tNZDguncyn12H3mxd3hI/qt9Y8rgmcGLsjq3T73OvVDcXueofFj2JEzzcdOj
9ebtfJIrd93xsJXlGr0KdHG0Ozz8uL6nBUpvWgQjp+cyNBkmw9NbsX7zP8h2xmVuY4ofr+
1PvyoKHzY0TdidUKEk6qgskd9wSoxsb7iWEWGpfKeMzIrGRqBfdZI14nCRLQI+vvYX+Z8a
0gFDNNWcpwL8cAAAEBAMEhFQGuhm9SxQ1fFoofYP5Vgx9ftIkCB7VbVhAjqYoKJRxGAhFf
hNSAYlx2bHQAQCMhuufmdQGlBKPcBbVZYkh1uo2224sWrMBPqU0yQlV3UAgtCAd+3jTpWj
KV3OT0cs8pHfj/+Wlt9CldfsOUN1pfXlong1oLejgMKZceyYCpwb/9LjoQupbmC9g4To+X
COgINdHPnpV9sePwoILfJIhWwHCpzw4LSIeuozxWg54J5l0ibMeWfMB/ZgTzFxTmmg1Htr
g5Z7hCKhjrHWwfT5w8oYHuwLpEaKYcZhIq/D2ujWfQGbPp8rbORCj+Ssz8SBOsSk+PpEGG
DivcpWlUW50AAAAOaGF5ZEBEYXZlcy1NQlABAgMEBQ==
-----END OPENSSH PRIVATE KEY-----

into a format suitable to be pasted into a JSON document: -

  "GITHUB_KEY":"-----BEGIN OPENSSH PRIVATE KEY-----
-----END OPENSSH PRIVATE KEY-----\n",

with the \n characters replacing the CR/LF at the end of each line.

Initially I munged the file using TextEdit and search / replace, but that's so "Like a cave person".

Thankfully, there's (always) a better way ...

awk '{printf "%s\\n", $0}' /tmp/this_is_a_test

which returns: -

-----BEGIN OPENSSH PRIVATE KEY-----\nb3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAACFwAAAAdzc2gtcn\nNhAAAAAwEAAQAAAgEAnonsGDeVoFfKyCTzsbof7SL5TWE2a5vWfEHq9fHFNH1WhRYKYm9z\nE0SLTnS/uYUT4rs8LfxT4vUYGgoNrtY9BH3V8U624CAsSuKdVc4PQdDt5r7CIZmhqIhEh1\nooMB/Ih0C83j6mU653n5bOBDTf1C8wSBBsZnZoivf0OX/MsHutJZCm1xgMT3NStBPFYhpO\nsqqWmMXrOJEBgNR880ECB2SBB29XmmhI2SYlzp3ZvTnGS1FCo4LctGkN7uaMFTMZx4QJgz\nv7DCeJuhzOcOSw54F4vJX3JGI9c49CBL7g2bxkzh83UpFZiOzVnwzabzkFIRgfC5eFfcxk\nNUxTpCPMpfZX0MAKRMJb59wPSXHPA0pisBEIu9gZOUrDeFmZlwUPuVH4i5wCb/rW2Ynkyf\nTKhz8bdC2dCj9/BAUv3axsWNjfgA4J/PvUgZ8yr3+zceRUcG+rzi8AFimR5H4QHMzTP9js\nPQ3cQ73OUpK0yl/ER+ibqhfkuz2HlEml8WOEEx5ChzSa+jvxzBBsYS3y4ebGbDbn/hBY2w\nL/ZSNeXbfx9iq7docW6U2tVcmuMDN1Voadd1V9cKIWsbJ+/UuPWyLLTw2PjpmyNPQJXPrr\n63vXaYskKZOZJ9pFqygZJxe6wepc+XluPkbpf1EmXZKy2wJyJPC/DImCmPDgoT6aslQVCg\nsAAAdIY7RrvWO0a70AAAAHc3NoLXJzYQAAAgEAnonsGDeVoFfKyCTzsbof7SL5TWE2a5vW\nfEHq9fHFNH1WhRYKYm9zE0SLTnS/uYUT4rs8LfxT4vUYGgoNrtY9BH3V8U624CAsSuKdVc\n4PQdDt5r7CIZmhqIhEh1ooMB/Ih0C83j6mU653n5bOBDTf1C8wSBBsZnZoivf0OX/MsHut\nJZCm1xgMT3NStBPFYhpOsqqWmMXrOJEBgNR880ECB2SBB29XmmhI2SYlzp3ZvTnGS1FCo4\nLctGkN7uaMFTMZx4QJgzv7DCeJuhzOcOSw54F4vJX3JGI9c49CBL7g2bxkzh83UpFZiOzV\nnwzabzkFIRgfC5eFfcxkNUxTpCPMpfZX0MAKRMJb59wPSXHPA0pisBEIu9gZOUrDeFmZlw\nUPuVH4i5wCb/rW2YnkyfTKhz8bdC2dCj9/BAUv3axsWNjfgA4J/PvUgZ8yr3+zceRUcG+r\nzi8AFimR5H4QHMzTP9jsPQ3cQ73OUpK0yl/ER+ibqhfkuz2HlEml8WOEEx5ChzSa+jvxzB\nBsYS3y4ebGbDbn/hBY2wL/ZSNeXbfx9iq7docW6U2tVcmuMDN1Voadd1V9cKIWsbJ+/UuP\nWyLLTw2PjpmyNPQJXPrr63vXaYskKZOZJ9pFqygZJxe6wepc+XluPkbpf1EmXZKy2wJyJP\nC/DImCmPDgoT6aslQVCgsAAAADAQABAAACAES9iKD3eOkNJ9+gJgnpqe4oO/BRNkdySo2W\nB0qcieU4c0tBQz9rE3pHPh/Gf1rxWK32gSFvRvAhVVH1CfnE6rAH37vBZYnIaWwO1ileLz\n9aV8y06XeoJW/tpckKTmKPZWM/rqSuhW/DNhr8SFeKlYJq84RLmrTSooGfAUXtI9NhzLF9\nBto8kxnf5vwtlmfzACx3ytRy/IqKVRyHBaZziW2rJkcS/+jzTqLY54uu5bomUtRVZ49+Lj\nQLWP8894A5p/PaTEWavD+I5S7NTrWkcw/uE2uxmncsIXj5pKZqHRRLm2nEHzitzSSRjA5q\nhEu1bYPr8YoVKMxi0xBJB/sjipe8XHPUGskw0XBhklwvGAq6HFMWWNScgNl5wDksb4TPwU\nREjcEBsd4CrW1mUNZcaj0lib55TEGcaU9tghmMSmMmjHMF58ZvE3DqaXL+VfjQS1YXiFYX\nUBJP4QQl3gDG5ypRCrr69FLahigGCnd/FIWev8BhoFolHft2gb/IJu36JeJ+24V4bHT7oG\n2G2LLIoNFe0XL0GvlbyNEKRgtLcAijBhDUZs9UEztH4gmFwAPGisbT+hhlhdC3kpxNARTT\nHskggYe//W1ZRX2/pIrw5oRn9g1UZ5A0Syla7Cw0JI0Jgcz5C5YQnMTD58jHOZZeb3zBEZ\nXfNrkXPkXEnBkO6dgBAAABAAIePVO0JuF16FYKjdFhKwvGKmgc0+kulT16o9kjFlcj94lx\n3MnAMldO8uPEm7IBdQtcF5k6MN3lLjSQF0AN/byJ7tK9+N6o/UQ6VFSR2TkcPc41/J6xH2\noERUXt7pJyygkRjDSotWFnZBzA6LcMiRi5NDjq9wXhNPPbl7Bi1LokZ0Bo6szVfwFSL5F2\ncyr20yXqyIX3hIduHfhRrNsMt+DgPHsg45Az3xu1H8eyT2K8YkyMjCDeNmm/PXLv2bQa/f\ndszZ+KOa4ORiNj/GnPyqM2vZJwY5wk0RCgCujr9pg7zgzjGxw2BsRxE1/rAASiqnn1iGT3\nEPKBFPKViKJ2TtsAAAEBANImJ7PZLFiPwXui9TK/QdhiY9a9+ZE0HOAhVTMvJEMKAtwqak\nRpjUVerlnX6KB/F7AX9TV472XTQqJSoHvwhImkWhe5EtHzKjkr/Al8pyQo44yy1Nw0vEfM\nryI7oZddx+tNZDguncyn12H3mxd3hI/qt9Y8rgmcGLsjq3T73OvVDcXueofFj2JEzzcdOj\n9ebtfJIrd93xsJXlGr0KdHG0Ozz8uL6nBUpvWgQjp+cyNBkmw9NbsX7zP8h2xmVuY4ofr+\n1PvyoKHzY0TdidUKEk6qgskd9wSoxsb7iWEWGpfKeMzIrGRqBfdZI14nCRLQI+vvYX+Z8a\n0gFDNNWcpwL8cAAAEBAMEhFQGuhm9SxQ1fFoofYP5Vgx9ftIkCB7VbVhAjqYoKJRxGAhFf\nhNSAYlx2bHQAQCMhuufmdQGlBKPcBbVZYkh1uo2224sWrMBPqU0yQlV3UAgtCAd+3jTpWj\nKV3OT0cs8pHfj/+Wlt9CldfsOUN1pfXlong1oLejgMKZceyYCpwb/9LjoQupbmC9g4To+X\nCOgINdHPnpV9sePwoILfJIhWwHCpzw4LSIeuozxWg54J5l0ibMeWfMB/ZgTzFxTmmg1Htr\ng5Z7hCKhjrHWwfT5w8oYHuwLpEaKYcZhIq/D2ujWfQGbPp8rbORCj+Ssz8SBOsSk+PpEGG\nDivcpWlUW50AAAAOaGF5ZEBEYXZlcy1NQlABAgMEBQ==\n-----END OPENSSH PRIVATE KEY-----\n

Of course, an alternate mechanism ( or many ) exists : -

sed 's/$/\\n/' /tmp/this_is_a_test | tr -d '\n'

Thanks Internet, you rock: -

Yay, VMware Fusion and macOS Big Sur - no longer "NAT good friends" - forgive the double negative and the terrible pun ...

After macOS 11 Big Sur was released in 2020, VMware updated their Fusion product to v12 and, sadly, managed to break Network Address Trans...