"WARNING:root:could not open file '/etc/apt/sources.list'"

We saw this: -

WARNING:root:could not open file '/etc/apt/sources.list'

during a complex series of Docker builds.

After a LOT of investigation, we concluded that it was a combination of a slightly malformed /etc/apt/sources.list file AND the use of the add-apt-repository command.

In essence, sources.list was being slightly truncated by a previous build process, meaning that the very last character of the file was NOT a CR/LF, but was some "special character".

The subsequent use of the add-apt-repository in a Dockerfile: -

add-apt-repository "deb [arch=s390x] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

was completely over-writing the sources.list file with the following TWO lines: -

deb [arch=s390x] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable

# deb-src [arch=s390x] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable

I've not quite reproduced the problem manually, despite my best efforts with a hex editor ...

However, the TL;DR; of the situation is that malformed lines in sources.list may be the root cause of all sorts of problems .....

IBM Cloud Hyper Protect Virtual Servers

This is, in part, upon what I've been working this past six months or so ....

Glad to see a useful rundown of the services and it's capabilities : -

IBM Cloud Hyper Protect Virtual Servers 

Gain complete authority over workloads with sensitive data or your business IP 

What are IBM Cloud Hyper Protect Virtual Servers?

IBM Cloud™ Hyper Protect Virtual Servers provides complete authority over workloads with sensitive data within a secure virtual server environment. No one, not even cloud admins, can access your workloads.

Built on the security and reliability of the LinuxONE platform, your business IP is protected through data-at-rest and runtime encryption. You can extend and consume on-prem LinuxONE resources in the cloud for faster development, testing and backup, without sacrificing security.

Fun with OpenSSL Certificate Requests and space characters in Subject Names

I've got a command within a Dockerfile that generates a Certificate Service Request, via the openssl req command.

This references an environment variable that contains the required Subject Name: -

export CRT_SUBJ="/C=US/ST=New York/L=Armonk/O=International Business Machines Corporation/CN=securebuildserver"

When I reference this variable in my Dockerfile: -

RUN openssl req -new -x509 -nodes -out server.crt -keyout server.key -subj $CRT_SUBJ

this fails with: -

unknown option York/L=Armonk/O=International
req [options] outfile

Note that it's failing to digest the Subject Name, specifically at the first space character, between New and York.

The solution ?

Wrap the environment variable in double-quotes: -

RUN openssl req -new -x509 -nodes -out server.crt -keyout server.key -subj "$CRT_SUBJ"

which works a treat ......

Using awk to munge SSH private keys

One of those "because I needed to" answers ....

Having generated a SSH public/private key pair: -

ssh-keygen -t rsa -b 4096 -f /tmp/this_is_a_test -N ""

I wanted to get the private key: -

-----BEGIN OPENSSH PRIVATE KEY-----

b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAACFwAAAAdzc2gtcn

NhAAAAAwEAAQAAAgEAnonsGDeVoFfKyCTzsbof7SL5TWE2a5vWfEHq9fHFNH1WhRYKYm9z

E0SLTnS/uYUT4rs8LfxT4vUYGgoNrtY9BH3V8U624CAsSuKdVc4PQdDt5r7CIZmhqIhEh1

ooMB/Ih0C83j6mU653n5bOBDTf1C8wSBBsZnZoivf0OX/MsHutJZCm1xgMT3NStBPFYhpO

sqqWmMXrOJEBgNR880ECB2SBB29XmmhI2SYlzp3ZvTnGS1FCo4LctGkN7uaMFTMZx4QJgz

v7DCeJuhzOcOSw54F4vJX3JGI9c49CBL7g2bxkzh83UpFZiOzVnwzabzkFIRgfC5eFfcxk

NUxTpCPMpfZX0MAKRMJb59wPSXHPA0pisBEIu9gZOUrDeFmZlwUPuVH4i5wCb/rW2Ynkyf

TKhz8bdC2dCj9/BAUv3axsWNjfgA4J/PvUgZ8yr3+zceRUcG+rzi8AFimR5H4QHMzTP9js

PQ3cQ73OUpK0yl/ER+ibqhfkuz2HlEml8WOEEx5ChzSa+jvxzBBsYS3y4ebGbDbn/hBY2w

L/ZSNeXbfx9iq7docW6U2tVcmuMDN1Voadd1V9cKIWsbJ+/UuPWyLLTw2PjpmyNPQJXPrr

63vXaYskKZOZJ9pFqygZJxe6wepc+XluPkbpf1EmXZKy2wJyJPC/DImCmPDgoT6aslQVCg

sAAAdIY7RrvWO0a70AAAAHc3NoLXJzYQAAAgEAnonsGDeVoFfKyCTzsbof7SL5TWE2a5vW

fEHq9fHFNH1WhRYKYm9zE0SLTnS/uYUT4rs8LfxT4vUYGgoNrtY9BH3V8U624CAsSuKdVc

4PQdDt5r7CIZmhqIhEh1ooMB/Ih0C83j6mU653n5bOBDTf1C8wSBBsZnZoivf0OX/MsHut

JZCm1xgMT3NStBPFYhpOsqqWmMXrOJEBgNR880ECB2SBB29XmmhI2SYlzp3ZvTnGS1FCo4

LctGkN7uaMFTMZx4QJgzv7DCeJuhzOcOSw54F4vJX3JGI9c49CBL7g2bxkzh83UpFZiOzV

nwzabzkFIRgfC5eFfcxkNUxTpCPMpfZX0MAKRMJb59wPSXHPA0pisBEIu9gZOUrDeFmZlw

UPuVH4i5wCb/rW2YnkyfTKhz8bdC2dCj9/BAUv3axsWNjfgA4J/PvUgZ8yr3+zceRUcG+r

zi8AFimR5H4QHMzTP9jsPQ3cQ73OUpK0yl/ER+ibqhfkuz2HlEml8WOEEx5ChzSa+jvxzB

BsYS3y4ebGbDbn/hBY2wL/ZSNeXbfx9iq7docW6U2tVcmuMDN1Voadd1V9cKIWsbJ+/UuP

WyLLTw2PjpmyNPQJXPrr63vXaYskKZOZJ9pFqygZJxe6wepc+XluPkbpf1EmXZKy2wJyJP

C/DImCmPDgoT6aslQVCgsAAAADAQABAAACAES9iKD3eOkNJ9+gJgnpqe4oO/BRNkdySo2W

B0qcieU4c0tBQz9rE3pHPh/Gf1rxWK32gSFvRvAhVVH1CfnE6rAH37vBZYnIaWwO1ileLz

9aV8y06XeoJW/tpckKTmKPZWM/rqSuhW/DNhr8SFeKlYJq84RLmrTSooGfAUXtI9NhzLF9

Bto8kxnf5vwtlmfzACx3ytRy/IqKVRyHBaZziW2rJkcS/+jzTqLY54uu5bomUtRVZ49+Lj

QLWP8894A5p/PaTEWavD+I5S7NTrWkcw/uE2uxmncsIXj5pKZqHRRLm2nEHzitzSSRjA5q

hEu1bYPr8YoVKMxi0xBJB/sjipe8XHPUGskw0XBhklwvGAq6HFMWWNScgNl5wDksb4TPwU

REjcEBsd4CrW1mUNZcaj0lib55TEGcaU9tghmMSmMmjHMF58ZvE3DqaXL+VfjQS1YXiFYX

UBJP4QQl3gDG5ypRCrr69FLahigGCnd/FIWev8BhoFolHft2gb/IJu36JeJ+24V4bHT7oG

2G2LLIoNFe0XL0GvlbyNEKRgtLcAijBhDUZs9UEztH4gmFwAPGisbT+hhlhdC3kpxNARTT

HskggYe//W1ZRX2/pIrw5oRn9g1UZ5A0Syla7Cw0JI0Jgcz5C5YQnMTD58jHOZZeb3zBEZ

XfNrkXPkXEnBkO6dgBAAABAAIePVO0JuF16FYKjdFhKwvGKmgc0+kulT16o9kjFlcj94lx

3MnAMldO8uPEm7IBdQtcF5k6MN3lLjSQF0AN/byJ7tK9+N6o/UQ6VFSR2TkcPc41/J6xH2

oERUXt7pJyygkRjDSotWFnZBzA6LcMiRi5NDjq9wXhNPPbl7Bi1LokZ0Bo6szVfwFSL5F2

cyr20yXqyIX3hIduHfhRrNsMt+DgPHsg45Az3xu1H8eyT2K8YkyMjCDeNmm/PXLv2bQa/f

dszZ+KOa4ORiNj/GnPyqM2vZJwY5wk0RCgCujr9pg7zgzjGxw2BsRxE1/rAASiqnn1iGT3

EPKBFPKViKJ2TtsAAAEBANImJ7PZLFiPwXui9TK/QdhiY9a9+ZE0HOAhVTMvJEMKAtwqak

RpjUVerlnX6KB/F7AX9TV472XTQqJSoHvwhImkWhe5EtHzKjkr/Al8pyQo44yy1Nw0vEfM

ryI7oZddx+tNZDguncyn12H3mxd3hI/qt9Y8rgmcGLsjq3T73OvVDcXueofFj2JEzzcdOj

9ebtfJIrd93xsJXlGr0KdHG0Ozz8uL6nBUpvWgQjp+cyNBkmw9NbsX7zP8h2xmVuY4ofr+

1PvyoKHzY0TdidUKEk6qgskd9wSoxsb7iWEWGpfKeMzIrGRqBfdZI14nCRLQI+vvYX+Z8a

0gFDNNWcpwL8cAAAEBAMEhFQGuhm9SxQ1fFoofYP5Vgx9ftIkCB7VbVhAjqYoKJRxGAhFf

hNSAYlx2bHQAQCMhuufmdQGlBKPcBbVZYkh1uo2224sWrMBPqU0yQlV3UAgtCAd+3jTpWj

KV3OT0cs8pHfj/+Wlt9CldfsOUN1pfXlong1oLejgMKZceyYCpwb/9LjoQupbmC9g4To+X

COgINdHPnpV9sePwoILfJIhWwHCpzw4LSIeuozxWg54J5l0ibMeWfMB/ZgTzFxTmmg1Htr

g5Z7hCKhjrHWwfT5w8oYHuwLpEaKYcZhIq/D2ujWfQGbPp8rbORCj+Ssz8SBOsSk+PpEGG

DivcpWlUW50AAAAOaGF5ZEBEYXZlcy1NQlABAgMEBQ==

-----END OPENSSH PRIVATE KEY-----

into a format suitable to be pasted into a JSON document: -

  "GITHUB_KEY":"-----BEGIN OPENSSH PRIVATE KEY-----

-----END OPENSSH PRIVATE KEY-----\n",

with the \n characters replacing the CR/LF at the end of each line.

Initially I munged the file using TextEdit and search / replace, but that's so "Like a cave person".

Thankfully, there's (always) a better way ...

awk '{printf "%s\\n", $0}' /tmp/this_is_a_test

which returns: -

-----BEGIN OPENSSH PRIVATE KEY-----\nb3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAACFwAAAAdzc2gtcn\nNhAAAAAwEAAQAAAgEAnonsGDeVoFfKyCTzsbof7SL5TWE2a5vWfEHq9fHFNH1WhRYKYm9z\nE0SLTnS/uYUT4rs8LfxT4vUYGgoNrtY9BH3V8U624CAsSuKdVc4PQdDt5r7CIZmhqIhEh1\nooMB/Ih0C83j6mU653n5bOBDTf1C8wSBBsZnZoivf0OX/MsHutJZCm1xgMT3NStBPFYhpO\nsqqWmMXrOJEBgNR880ECB2SBB29XmmhI2SYlzp3ZvTnGS1FCo4LctGkN7uaMFTMZx4QJgz\nv7DCeJuhzOcOSw54F4vJX3JGI9c49CBL7g2bxkzh83UpFZiOzVnwzabzkFIRgfC5eFfcxk\nNUxTpCPMpfZX0MAKRMJb59wPSXHPA0pisBEIu9gZOUrDeFmZlwUPuVH4i5wCb/rW2Ynkyf\nTKhz8bdC2dCj9/BAUv3axsWNjfgA4J/PvUgZ8yr3+zceRUcG+rzi8AFimR5H4QHMzTP9js\nPQ3cQ73OUpK0yl/ER+ibqhfkuz2HlEml8WOEEx5ChzSa+jvxzBBsYS3y4ebGbDbn/hBY2w\nL/ZSNeXbfx9iq7docW6U2tVcmuMDN1Voadd1V9cKIWsbJ+/UuPWyLLTw2PjpmyNPQJXPrr\n63vXaYskKZOZJ9pFqygZJxe6wepc+XluPkbpf1EmXZKy2wJyJPC/DImCmPDgoT6aslQVCg\nsAAAdIY7RrvWO0a70AAAAHc3NoLXJzYQAAAgEAnonsGDeVoFfKyCTzsbof7SL5TWE2a5vW\nfEHq9fHFNH1WhRYKYm9zE0SLTnS/uYUT4rs8LfxT4vUYGgoNrtY9BH3V8U624CAsSuKdVc\n4PQdDt5r7CIZmhqIhEh1ooMB/Ih0C83j6mU653n5bOBDTf1C8wSBBsZnZoivf0OX/MsHut\nJZCm1xgMT3NStBPFYhpOsqqWmMXrOJEBgNR880ECB2SBB29XmmhI2SYlzp3ZvTnGS1FCo4\nLctGkN7uaMFTMZx4QJgzv7DCeJuhzOcOSw54F4vJX3JGI9c49CBL7g2bxkzh83UpFZiOzV\nnwzabzkFIRgfC5eFfcxkNUxTpCPMpfZX0MAKRMJb59wPSXHPA0pisBEIu9gZOUrDeFmZlw\nUPuVH4i5wCb/rW2YnkyfTKhz8bdC2dCj9/BAUv3axsWNjfgA4J/PvUgZ8yr3+zceRUcG+r\nzi8AFimR5H4QHMzTP9jsPQ3cQ73OUpK0yl/ER+ibqhfkuz2HlEml8WOEEx5ChzSa+jvxzB\nBsYS3y4ebGbDbn/hBY2wL/ZSNeXbfx9iq7docW6U2tVcmuMDN1Voadd1V9cKIWsbJ+/UuP\nWyLLTw2PjpmyNPQJXPrr63vXaYskKZOZJ9pFqygZJxe6wepc+XluPkbpf1EmXZKy2wJyJP\nC/DImCmPDgoT6aslQVCgsAAAADAQABAAACAES9iKD3eOkNJ9+gJgnpqe4oO/BRNkdySo2W\nB0qcieU4c0tBQz9rE3pHPh/Gf1rxWK32gSFvRvAhVVH1CfnE6rAH37vBZYnIaWwO1ileLz\n9aV8y06XeoJW/tpckKTmKPZWM/rqSuhW/DNhr8SFeKlYJq84RLmrTSooGfAUXtI9NhzLF9\nBto8kxnf5vwtlmfzACx3ytRy/IqKVRyHBaZziW2rJkcS/+jzTqLY54uu5bomUtRVZ49+Lj\nQLWP8894A5p/PaTEWavD+I5S7NTrWkcw/uE2uxmncsIXj5pKZqHRRLm2nEHzitzSSRjA5q\nhEu1bYPr8YoVKMxi0xBJB/sjipe8XHPUGskw0XBhklwvGAq6HFMWWNScgNl5wDksb4TPwU\nREjcEBsd4CrW1mUNZcaj0lib55TEGcaU9tghmMSmMmjHMF58ZvE3DqaXL+VfjQS1YXiFYX\nUBJP4QQl3gDG5ypRCrr69FLahigGCnd/FIWev8BhoFolHft2gb/IJu36JeJ+24V4bHT7oG\n2G2LLIoNFe0XL0GvlbyNEKRgtLcAijBhDUZs9UEztH4gmFwAPGisbT+hhlhdC3kpxNARTT\nHskggYe//W1ZRX2/pIrw5oRn9g1UZ5A0Syla7Cw0JI0Jgcz5C5YQnMTD58jHOZZeb3zBEZ\nXfNrkXPkXEnBkO6dgBAAABAAIePVO0JuF16FYKjdFhKwvGKmgc0+kulT16o9kjFlcj94lx\n3MnAMldO8uPEm7IBdQtcF5k6MN3lLjSQF0AN/byJ7tK9+N6o/UQ6VFSR2TkcPc41/J6xH2\noERUXt7pJyygkRjDSotWFnZBzA6LcMiRi5NDjq9wXhNPPbl7Bi1LokZ0Bo6szVfwFSL5F2\ncyr20yXqyIX3hIduHfhRrNsMt+DgPHsg45Az3xu1H8eyT2K8YkyMjCDeNmm/PXLv2bQa/f\ndszZ+KOa4ORiNj/GnPyqM2vZJwY5wk0RCgCujr9pg7zgzjGxw2BsRxE1/rAASiqnn1iGT3\nEPKBFPKViKJ2TtsAAAEBANImJ7PZLFiPwXui9TK/QdhiY9a9+ZE0HOAhVTMvJEMKAtwqak\nRpjUVerlnX6KB/F7AX9TV472XTQqJSoHvwhImkWhe5EtHzKjkr/Al8pyQo44yy1Nw0vEfM\nryI7oZddx+tNZDguncyn12H3mxd3hI/qt9Y8rgmcGLsjq3T73OvVDcXueofFj2JEzzcdOj\n9ebtfJIrd93xsJXlGr0KdHG0Ozz8uL6nBUpvWgQjp+cyNBkmw9NbsX7zP8h2xmVuY4ofr+\n1PvyoKHzY0TdidUKEk6qgskd9wSoxsb7iWEWGpfKeMzIrGRqBfdZI14nCRLQI+vvYX+Z8a\n0gFDNNWcpwL8cAAAEBAMEhFQGuhm9SxQ1fFoofYP5Vgx9ftIkCB7VbVhAjqYoKJRxGAhFf\nhNSAYlx2bHQAQCMhuufmdQGlBKPcBbVZYkh1uo2224sWrMBPqU0yQlV3UAgtCAd+3jTpWj\nKV3OT0cs8pHfj/+Wlt9CldfsOUN1pfXlong1oLejgMKZceyYCpwb/9LjoQupbmC9g4To+X\nCOgINdHPnpV9sePwoILfJIhWwHCpzw4LSIeuozxWg54J5l0ibMeWfMB/ZgTzFxTmmg1Htr\ng5Z7hCKhjrHWwfT5w8oYHuwLpEaKYcZhIq/D2ujWfQGbPp8rbORCj+Ssz8SBOsSk+PpEGG\nDivcpWlUW50AAAAOaGF5ZEBEYXZlcy1NQlABAgMEBQ==\n-----END OPENSSH PRIVATE KEY-----\n

Of course, an alternate mechanism ( or many ) exists : -

sed 's/$/\\n/' /tmp/this_is_a_test | tr -d '\n'

Thanks Internet, you rock: -

Replace newlines with literal \n