Right Sed Fred!

I've been using the Stream Editor (sed) for the past few years, and especially love it's ability to do in-place editing of a file as per this example: -

sed -i'' "s/PidFile\ logs/PidFile\ ${Product}\/logs/g" /opt/ibm/HTTPServer/${Product}/conf/httpd.conf

so I was somewhat surprised to find it that it seemed to bork on macOS Catalina this morning: -

as per this terrible ( REALLY ) example: -

sed -i'' 's/PRIVATE/PUBLIC/g' dave.key 

sed: 1: "dave.key": extra characters at the end of d command

even though the same command worked on Ubuntu 18.04.02.

Assuming it to be a version thing, I checked the version of sed shipped with Ubuntu: -

sed --version

sed (GNU sed) 4.4

Copyright (C) 2017 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later .

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law.

Written by Jay Fenlason, Tom Lord, Ken Pizzini,

and Paolo Bonzini.

GNU sed home page: .

General help using GNU software: .

E-mail bug reports to: .

and, guess what, the --version or -v or --v switches don't even work on the macOS version :-( 

It transpires that the macOS version is based upon BSD Unix, which kinda makes sense given its roots from Mach etc.

This helped: -

BSD/macOS Sed vs. GNU Sed vs. the POSIX Sed specification

The key thing ....

On macOS, the cheat for in-place editing is simple ....

Rather than this: -

sed -i'' 's/PRIVATE/PUBLIC/g' dave.key 

I used this: -

sed -i '' 's/PRIVATE/PUBLIC/g' dave.key 

Yep, I added a space between -i and the single quote symbols !

PS Do NOT NOT NOT hand-edit your private OR public keys!!!

"unable to set private key file" - more fun with openSSL and certificates

Another long story cut short, but I saw this: -

curl: (58) unable to set private key file: 'dave.pem' type PEM

from my Ansible/Python code, whilst attempting to use a PEM certificate that I'd generated myself: -

Generate Private Key

openssl genrsa -out key.pem 2048

Generate Certificate Service Request

openssl req -subj '/C=GB/O=IBM/CN=davehay' -new -key key.pem -out csr.pem

Generate Personal Certificate

openssl x509 -req -days 9999 -in csr.pem -signkey key.pem -out cert.pem

Having munged the key and certificate into a single PEM file: -

cat key.pem cert.pem > dave.pem

I found that my Python code was then validating the private key within dave.pem : - 

cert_pkey.split('-----BEGIN PRIVATE KEY-----')

which meant that it was failing ...

Simple solution, right ?

Yeah, I edited dave.pem to remove the characters RSA from the PEM file: -

sed -i '' 's/RSA //g' dave.pem 

Problem solved, right ?

NAH!!

My code, which uses cURL under the covers, then failed with: -

curl: (58) unable to set private key file: 'dave.pem' type PEM

This blog post: -

curl: (58) unable to set private key file: ‘server.key’ type PEM

described how one can validate the private key and its certificate: -

openssl x509 -noout -modulus -in dave.pem | openssl md5

which returns a MD5 checksum: -

0d6b9d546ff1b65284ec32096bea2904

and: -

openssl rsa -noout -modulus -in dave.pem | openssl md5

which SHOULD return a MD5 checksum, but instead returned: -

unable to load Private Key

4686818796:error:0DFFF0A8:asn1 encoding routines:CRYPTO_internal:wrong tag:/BuildRoot/Library/Caches/com.apple.xbs/Sources/libressl/libressl-47.11.1/libressl-2.8/crypto/asn1/tasn_dec.c:1144:

4686818796:error:0DFFF03A:asn1 encoding routines:CRYPTO_internal:nested asn1 error:/BuildRoot/Library/Caches/com.apple.xbs/Sources/libressl/libressl-47.11.1/libressl-2.8/crypto/asn1/tasn_dec.c:317:Type=X509_ALGOR

4686818796:error:0DFFF03A:asn1 encoding routines:CRYPTO_internal:nested asn1 error:/BuildRoot/Library/Caches/com.apple.xbs/Sources/libressl/libressl-47.11.1/libressl-2.8/crypto/asn1/tasn_dec.c:646:Field=pkeyalg, Type=PKCS8_PRIV_KEY_INFO

4686818796:error:09FFF00D:PEM routines:CRYPTO_internal:ASN1 lib:/BuildRoot/Library/Caches/com.apple.xbs/Sources/libressl/libressl-47.11.1/libressl-2.8/crypto/pem/pem_pkey.c:143:

d41d8cd98f00b204e9800998ecf8427e

Yeah, you guessed it, I broke my private key by removing RSA :-)

It was relatively easy to fix, it was all down to the way that I was generating my key and certificate. I switched to this: -

openssl req -subj '/C=GB/O=IBM/CN=davehay' -new -newkey rsa:2048 -days 365 -nodes -x509 -sha256 -keyout dave.key -out dave.crt

and, after munging the key and certificate: -

cat dave.key dave.crt > dave.pem

I ended up with a PEM file that I did NOT need to edit i.e. it contained the key (!) string: -

-----BEGIN PRIVATE KEY-----

and, more importantly, it validated without problems: -

openssl x509 -noout -modulus -in dave.pem | openssl md5

1c03038c6be240c22d759bfef58e9db2

openssl rsa -noout -modulus -in dave.pem | openssl md5

1c03038c6be240c22d759bfef58e9db2

and, even more importantly, my code works!!!

Moral of the story ? Don't manually hack your keys, instead check the way that you're generating them in the first place :-)

More about OpenSSL and PKCS12 certificates .... some INFO

So, having written this: -

Client Authentication and tinkering with various certificate formats 

a few days back, I realised that I'd neglected to describe how one can validate a PKCS12 certificate.

So here we go ...

Having exported my PEM file into PKCS12 format: -

openssl pkcs12 -export -out dave.p12 -in dave.pem

or: -

openssl pkcs12 -export -out dave.pfx -in dave.pem

I can validate the resulting certificate / private key as follows: -

openssl pkcs12 -info -in dave.p12

 

Enter Import Password:

MAC Iteration 2048

MAC verified OK

PKCS7 Encrypted data: pbeWithSHA1And40BitRC2-CBC, Iteration 2048

Certificate bag

Bag Attributes

    localKeyID: C4 40 FE 9E 1A 30 48 24 B1 59 94 1C 9A 6A EB 65 04 DA A9 AF 

subject=/CN=51bceeff97c5

issuer=/CN=51bceeff97c5

-----BEGIN CERTIFICATE-----

MIICojCCAYoCAQEwDQYJKoZIhvcNAQEEBQAwFzEVMBMGA1UEAwwMNTFiY2VlZmY5

N2M1MB4XDTE5MTAxMzA3NDIzNFoXDTIwMTAxMjA3NDIzNFowFzEVMBMGA1UEAwwM

NTFiY2VlZmY5N2M1MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA380u

JElYYA8TqhQdflnQW/A73OTpsc67meB9Adi4vTvwc9VP51BI/IkupRfC71IFJg/c

wmPZWQC8Mx/TF2s5XOPWrAOQRGu4cmh2fOG9E4ZJCSq9kzqGZ3BBUTyqC6ZtGad/

VBloOjw7e9D2sWuVDZqwWqo6nGcF2i/ZVDeUYyBa9ul8+opes7ufRZdbDI9s5iOc

LH+6cxL3efmLZIUVMg8/jvC9nzIRTYk0mmYsyPAJcvIvy9RoxLHG62UjmS2lLJN3

RE/jyDj29Xi5Aokplz5pnYoAxzPo9YmUAwlLDD6vYJCa6B2q5votu7NFJ6ttF4Io

/bolg+PyYfLE2nsHOwIDAQABMA0GCSqGSIb3DQEBBAUAA4IBAQCdxsDo+8E7VqMo

I3bke3j2vSnB/KLE+5UpeRsXhO4zRVAcJbVEuppJFkCwxL0ZLOfoQH+yGcdANa7v

9Fh+JTlDInokrwOIQDsh1X09s/Ca3LlWx9pnuOC39gA6XeqO3b75FXq9FLBhNFGO

LSiNtruqXcIgEK3IK2T9DRfR/D1B9TlAONGfAFpIT+szog2LFG3NwyNiaxRRNcR0

ZLpSqWOkkBrKKSaDTn4KRynh5Hs/ZbcsApybXz56rr+NsR9o6T7IhFkAUtzLhW+U

+YqT07fCJ8MHy6FMIz80HpVlktA1Tj790Ynk4YAL9IB2AObxXdSRcoSHBkl4frQP

Jm+0QNEr

-----END CERTIFICATE-----

PKCS7 Data

Shrouded Keybag: pbeWithSHA1And3-KeyTripleDES-CBC, Iteration 2048

Bag Attributes

    localKeyID: C4 40 FE 9E 1A 30 48 24 B1 59 94 1C 9A 6A EB 65 04 DA A9 AF 

Key Attributes:

Enter PEM pass phrase:

Note that this works for a PKCS12 file, whether it's got a .P12 or .PFX extension or otherwise ...

openssl pkcs12 -info -in dave.foobarsnafu

Enter Import Password:

MAC Iteration 2048

MAC verified OK

PKCS7 Encrypted data: pbeWithSHA1And40BitRC2-CBC, Iteration 2048

Certificate bag

Bag Attributes

    localKeyID: C4 40 FE 9E 1A 30 48 24 B1 59 94 1C 9A 6A EB 65 04 DA A9 AF 

subject=/CN=51bceeff97c5

issuer=/CN=51bceeff97c5

-----BEGIN CERTIFICATE-----

MIICojCCAYoCAQEwDQYJKoZIhvcNAQEEBQAwFzEVMBMGA1UEAwwMNTFiY2VlZmY5

N2M1MB4XDTE5MTAxMzA3NDIzNFoXDTIwMTAxMjA3NDIzNFowFzEVMBMGA1UEAwwM

NTFiY2VlZmY5N2M1MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA380u

JElYYA8TqhQdflnQW/A73OTpsc67meB9Adi4vTvwc9VP51BI/IkupRfC71IFJg/c

wmPZWQC8Mx/TF2s5XOPWrAOQRGu4cmh2fOG9E4ZJCSq9kzqGZ3BBUTyqC6ZtGad/

VBloOjw7e9D2sWuVDZqwWqo6nGcF2i/ZVDeUYyBa9ul8+opes7ufRZdbDI9s5iOc

LH+6cxL3efmLZIUVMg8/jvC9nzIRTYk0mmYsyPAJcvIvy9RoxLHG62UjmS2lLJN3

RE/jyDj29Xi5Aokplz5pnYoAxzPo9YmUAwlLDD6vYJCa6B2q5votu7NFJ6ttF4Io

/bolg+PyYfLE2nsHOwIDAQABMA0GCSqGSIb3DQEBBAUAA4IBAQCdxsDo+8E7VqMo

I3bke3j2vSnB/KLE+5UpeRsXhO4zRVAcJbVEuppJFkCwxL0ZLOfoQH+yGcdANa7v

9Fh+JTlDInokrwOIQDsh1X09s/Ca3LlWx9pnuOC39gA6XeqO3b75FXq9FLBhNFGO

LSiNtruqXcIgEK3IK2T9DRfR/D1B9TlAONGfAFpIT+szog2LFG3NwyNiaxRRNcR0

ZLpSqWOkkBrKKSaDTn4KRynh5Hs/ZbcsApybXz56rr+NsR9o6T7IhFkAUtzLhW+U

+YqT07fCJ8MHy6FMIz80HpVlktA1Tj790Ynk4YAL9IB2AObxXdSRcoSHBkl4frQP

Jm+0QNEr

-----END CERTIFICATE-----

PKCS7 Data

Shrouded Keybag: pbeWithSHA1And3-KeyTripleDES-CBC, Iteration 2048

Bag Attributes

    localKeyID: C4 40 FE 9E 1A 30 48 24 B1 59 94 1C 9A 6A EB 65 04 DA A9 AF 

Key Attributes:

Enter PEM pass phrase:

WebSphere Liberty Profile and DB2 in Docker - There's more ...

I wrote about this a few months back: -

 IBM WebSphere Liberty Profile and IBM DB2 and Docker - An approach 

and had a query from a colleague last week.

So I did some of again ....

The context is that my colleague was looking to run a Java class, using the DB2 JDBC JCC4 drivers, where DB2 was running in a Docker container.

So I replicated this, using Liberty as my Java runtime ....

Here's the details: -

Create a directory on the host for DB2 to use for its datastore

mkdir /tmp/db2data

Start a DB2 container


docker run -itd --name mydb2 --privileged=true -p 50000:50000 -e LICENSE=accept -e DB2INST1_PASSWORD=p455w0rd -e DBNAME=testdb -v /tmp/db2data:/database ibmcom/db2

Check the container logs for the DB2 startup

docker logs mydb2 -f

Open a shell into the DB2 container

docker exec -it mydb2 /bin/bash

Switch to db2inst1 instance account

su - db2inst1

Create the DB2 sample database

db2sampl

Quit back to the host OS

exit

exit

Copy the DB2 JDBC JCC4 driver AND license file out of the DB2 container

docker cp mydb2:/opt/ibm/db2/V11.5/java/db2jcc4.jar .

docker cp mydb2:/opt/ibm/db2/V11.5/java/db2jcc_license_cu.jar .

Create a Dockerfile to spin up a Liberty container using the DB2 JAR files

vi Dockerfile

FROM websphere-liberty:latest

ENV LICENSE accept

COPY db2jcc4.jar /

COPY db2jcc_license_cu.jar /

COPY JdbcTestDB2.class /

CMD ["java","-cp","/:/db2jcc4.jar","JdbcTestDB2","carded1.fyre.ibm.com","50000","sample","db2inst1 ","p455w0rd"]

Start the Liberty container

docker run -dt --name mywlp wlp:latest

Check the Liberty logs

000010 CHRISTINE HAAS

000020 MICHAEL THOMPSON

000030 SALLY KWAN

000050 JOHN GEYER

000060 IRVING STERN

000070 EVA PULASKI

000090 EILEEN HENDERSON

000100 THEODORE SPENSER

000110 VINCENZO LUCCHESSI

000120 SEAN O'CONNELL

000130 DELORES QUINTANA

000140 HEATHER NICHOLLS

000150 BRUCE ADAMSON

000160 ELIZABETH PIANKA

000170 MASATOSHI YOSHIMURA

000180 MARILYN SCOUTTEN

000190 JAMES WALKER

000200 DAVID BROWN

000210 WILLIAM JONES

000220 JENNIFER LUTZ

000230 JAMES JEFFERSON

000240 SALVATORE MARINO

000250 DANIEL SMITH

000260 SYBIL JOHNSON

000270 MARIA PEREZ

000280 ETHEL SCHNEIDER

000290 JOHN PARKER

000300 PHILIP SMITH

000310 MAUDE SETRIGHT

000320 RAMLAL MEHTA

000330 WING LEE

000340 JASON GOUNOT

200010 DIAN HEMMINGER

200120 GREG ORLANDO

200140 KIM NATZ

200170 KIYOSHI YAMAMOTO

200220 REBA JOHN

200240 ROBERT MONTEVERDE

200280 EILEEN SCHWARTZ

200310 MICHELLE SPRINGER

200330 HELENA WONG

200340 ROY ALONZO

For the record, the Java class that I'm running is here: -

vi JdbcTestDB2.java

import java.sql.Connection ;

import java.sql.DriverManager ;

import java.sql.ResultSet ;

import java.sql.Statement ;

import java.sql.SQLException;

class JdbcTestDB2

{

 public static void main (String args[])

 {

  try

  {

   Class.forName("com.ibm.db2.jcc.DB2Driver");

  }

  catch (ClassNotFoundException e)

  {

   System.err.println (e) ;

   System.exit (-1) ;

   }

   String hostname      = args[0];

   String port          = args[1];

   String dbName        = args[2];

   String userName      = args[3];

   String password      = args[4];

   String sslConnection = "false";

   java.util.Properties properties = new java.util.Properties();

   properties.put("user",userName);

   properties.put("password", password);

  String url = "jdbc:db2://" + hostname + ":" + port + "/" + dbName;

  try

  {

      Connection connection = DriverManager.getConnection(url,properties);

      String query = "select EMPNO,FIRSTNME,LASTNAME from DB2INST1.EMPLOYEE" ;

      Statement statement = connection.createStatement () ;

   ResultSet rs = statement.executeQuery (query) ;

   while ( rs.next () )

    System.out.println (rs.getString (1) + " " + rs.getString(2) + " " + rs.getString(3)) ;

    connection.close () ;

  }

    catch (java.sql.SQLException e)

  {

   System.err.println (e) ;

   System.exit (-1) ;

  }

 }

}

and I can easily run the same class from the host OS: -

/opt/ibm/java/jre/bin/java -cp $(pwd):$(pwd)/db2jcc4.jar JdbcTestDB2 carded1.fyre.ibm.com 50000 sample db2inst1 p455w0rd

For what it's worth, I have the IBM Java 8 SDK: -

ibm-java-sdk-8.0-5.41-x86_64-archive.bin

installed on the host OS ( Ubuntu 18.04.3 LTS )

Darknet Diaries and the Python Cheatsheet

I've recently been mainlining on back-episodes of the Darknet Diaries podcast, and, in one episode, the host, Jack Rhysider mentioned a Python cheat sheet that he'd written, and was sharing here: -

https://darknetdiaries.com/python/

Definitely worth listening to the podcast and, if you're coding in Python, consider getting the tutorial - it'll cost you one email address, which is cheap at thrice the price !

Client Authentication and tinkering with various certificate formats

So, long story short, we have a service that builds Docker images, INSIDE a Docker container, and pushes the tagged and built images to Docker Hub ...

But that's not important right now ...

The key ( apologies for the pun ) thing is that the service exposes a series of actions via a REST API that is protected by TLS 1.2, to which we authenticate via a personal certificate.

This particular certificate is actually a PEM ( Privacy Enhanced Mail ) Base64-encoded DER file: -

dave.pem

which contains the personal certificate AND its private key: -

-----BEGIN CERTIFICATE-----
MIICojCCAYoCAQEwDQYJKoZIhvcNAQEEBQAwFzEVMBMGA1UEAwwMNTFiY2VlZmY5
N2M1MB4XDTE5MTAxMzA3NDIzNFoXDTIwMTAxMjA3NDIzNFowFzEVMBMGA1UEAwwM
...
-----END CERTIFICATE-----
-----BEGIN PRIVATE KEY-----
MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDfzS4kSVhgDxOq
FB1+WdBb8Dvc5OmxzruZ4H0B2Li9O/Bz1U/nUEj8iS6lF8LvUgUmD9zCY9lZALwz
H9MXazlc49asA5BEa7hyaHZ84b0ThkkJKr2TOoZncEFRPKoLpm0Zp39UGWg6PDt7
...
-----END PRIVATE KEY-----

Using that certificate, we can authenticate to the REST endpoint, such as this example: -

curl -k https://192.168.1.24:443/image --cert dave.pem

{"root_ssh_enabled":false,"status":"initialized"}

So far, so good.

We then wanted to perform a series of security tests against the same endpoint, using a product called AppScan Standard ( this used to be an IBM Rational offering, and has been recently transitioned to HCL ).

This DOES support client authentication BUT doesn't support a PEM file.

Therefore, we needed to convert the PEM file into a different format, Public-Key Cryptography Standards (PKCS), as either a .p12 or .pfx file.

This is nice n' easy using the Swiss Army knife of security - openssl - as per this: -

openssl pkcs12 -export -out dave.p12 -in dave.pem

and then validate it via cURL: -

curl -k https://192.168.1.24:443/image --cert-type p12 --cert dave.p12

{"root_ssh_enabled":false,"status":"initialized"}

So now we're good to go .....

Retreading my steps - deleting directories with hyphens ...

I had an issue whilst trying to sort out some directories on my Mac, containing hyphens ( - ) as part of their names: -

Untracked files:
  (use "git add ..." to include in what will be committed)

--header
--url

which were causing fgrep to choke: -

fgrep -R

usage: fgrep [-abcDEFGHhIiJLlmnOoqRSsUVvwxZ] [-A num] [-B num] [-C[num]]

[-e pattern] [-f file] [--binary-files=value] [--color=when]

[--context[=num]] [--directories=action] [--label] [--line-buffered]

[--null] [pattern] [file ...]

I couldn't easily delete them: -

rm -Rf "--header"

rm: illegal option -- -

usage: rm [-f | -i] [-dPRrvW] file ...

       unlink file

ls *header*

ls: illegal option -- -

usage: ls [-ABCFGHLOPRSTUWabcdefghiklmnopqrstuwx1] [file ...]

rm -Rf '--url'

rm: illegal option -- -
usage: rm [-f | -i] [-dPRrvW] file ...
       unlink file

but I remembered hitting this a while back ....

which takes me back to a post that I wrote back in 2010: -

 Removing files with hyphenated filenames in Linux 


I'd had the same problem a few days ago - the problem is that the hyphen ( - ) character is seen by the rm command as an option.

The solution ? A quick rummage around Google threw this up: -

http://serverfault.com/questions/37823/how-to-delete-file-with-option-character-in-name-in-linux

Use the "--" option to tell rm that there are no more options coming, hence the command becomes: -

rm -Rf -- -6895560/

Job done :-)



So I tried that: -

rm -Rf -- --url

rm -Rf -- --header

which did the trick.

And now fgrep is happy ....