Wednesday, 16 October 2019

Right Sed Fred!

I've been using the Stream Editor (sed) for the past few years, and especially love it's ability to do in-place editing of a file as per this example: -

sed -i'' "s/PidFile\ logs/PidFile\ ${Product}\/logs/g" /opt/ibm/HTTPServer/${Product}/conf/httpd.conf

so I was somewhat surprised to find it that it seemed to bork on macOS Catalina this morning: -

as per this terrible ( REALLY ) example: -

sed -i'' 's/PRIVATE/PUBLIC/g' dave.key 

sed: 1: "dave.key": extra characters at the end of d command

even though the same command worked on Ubuntu 18.04.02.

Assuming it to be a version thing, I checked the version of sed shipped with Ubuntu: -

sed --version

sed (GNU sed) 4.4
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later .
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Jay Fenlason, Tom Lord, Ken Pizzini,
and Paolo Bonzini.
GNU sed home page: .
General help using GNU software: .
E-mail bug reports to: .

and, guess what, the --version or -v or --v switches don't even work on the macOS version :-( 

It transpires that the macOS version is based upon BSD Unix, which kinda makes sense given its roots from Mach etc.

This helped: -


The key thing ....

On macOS, the cheat for in-place editing is simple ....

Rather than this: -

sed -i'' 's/PRIVATE/PUBLIC/g' dave.key 

I used this: -

sed -i '' 's/PRIVATE/PUBLIC/g' dave.key 

Yep, I added a space between -i and the single quote symbols !

PS Do NOT NOT NOT hand-edit your private OR public keys!!!

"unable to set private key file" - more fun with openSSL and certificates

Another long story cut short, but I saw this: -

curl: (58) unable to set private key file: 'dave.pem' type PEM

from my Ansible/Python code, whilst attempting to use a PEM certificate that I'd generated myself: -

Generate Private Key

openssl genrsa -out key.pem 2048

Generate Certificate Service Request

openssl req -subj '/C=GB/O=IBM/CN=davehay' -new -key key.pem -out csr.pem

Generate Personal Certificate

openssl x509 -req -days 9999 -in csr.pem -signkey key.pem -out cert.pem

Having munged the key and certificate into a single PEM file: -

cat key.pem cert.pem > dave.pem

I found that my Python code was then validating the private key within dave.pem : - 

cert_pkey.split('-----BEGIN PRIVATE KEY-----')

which meant that it was failing ...

Simple solution, right ?

Yeah, I edited dave.pem to remove the characters RSA from the PEM file: -

sed -i '' 's/RSA //g' dave.pem 

Problem solved, right ?

NAH!!

My code, which uses cURL under the covers, then failed with: -

curl: (58) unable to set private key file: 'dave.pem' type PEM

This blog post: -


described how one can validate the private key and its certificate: -

openssl x509 -noout -modulus -in dave.pem | openssl md5

which returns a MD5 checksum: -

0d6b9d546ff1b65284ec32096bea2904

and: -

openssl rsa -noout -modulus -in dave.pem | openssl md5

which SHOULD return a MD5 checksum, but instead returned: -

unable to load Private Key
4686818796:error:0DFFF0A8:asn1 encoding routines:CRYPTO_internal:wrong tag:/BuildRoot/Library/Caches/com.apple.xbs/Sources/libressl/libressl-47.11.1/libressl-2.8/crypto/asn1/tasn_dec.c:1144:
4686818796:error:0DFFF03A:asn1 encoding routines:CRYPTO_internal:nested asn1 error:/BuildRoot/Library/Caches/com.apple.xbs/Sources/libressl/libressl-47.11.1/libressl-2.8/crypto/asn1/tasn_dec.c:317:Type=X509_ALGOR
4686818796:error:0DFFF03A:asn1 encoding routines:CRYPTO_internal:nested asn1 error:/BuildRoot/Library/Caches/com.apple.xbs/Sources/libressl/libressl-47.11.1/libressl-2.8/crypto/asn1/tasn_dec.c:646:Field=pkeyalg, Type=PKCS8_PRIV_KEY_INFO
4686818796:error:09FFF00D:PEM routines:CRYPTO_internal:ASN1 lib:/BuildRoot/Library/Caches/com.apple.xbs/Sources/libressl/libressl-47.11.1/libressl-2.8/crypto/pem/pem_pkey.c:143:
d41d8cd98f00b204e9800998ecf8427e

Yeah, you guessed it, I broke my private key by removing RSA :-)

It was relatively easy to fix, it was all down to the way that I was generating my key and certificate. I switched to this: -

openssl req -subj '/C=GB/O=IBM/CN=davehay' -new -newkey rsa:2048 -days 365 -nodes -x509 -sha256 -keyout dave.key -out dave.crt

and, after munging the key and certificate: -

cat dave.key dave.crt > dave.pem

I ended up with a PEM file that I did NOT need to edit i.e. it contained the key (!) string: -

-----BEGIN PRIVATE KEY-----

and, more importantly, it validated without problems: -

openssl x509 -noout -modulus -in dave.pem | openssl md5

1c03038c6be240c22d759bfef58e9db2

openssl rsa -noout -modulus -in dave.pem | openssl md5

1c03038c6be240c22d759bfef58e9db2

and, even more importantly, my code works!!!

Moral of the story ? Don't manually hack your keys, instead check the way that you're generating them in the first place :-)

Monday, 14 October 2019

More about OpenSSL and PKCS12 certificates .... some INFO

So, having written this: -

Client Authentication and tinkering with various certificate formats 

a few days back, I realised that I'd neglected to describe how one can validate a PKCS12 certificate.

So here we go ...

Having exported my PEM file into PKCS12 format: -

openssl pkcs12 -export -out dave.p12 -in dave.pem

or: -

openssl pkcs12 -export -out dave.pfx -in dave.pem

I can validate the resulting certificate / private key as follows: -

openssl pkcs12 -info -in dave.p12
 
Enter Import Password:
MAC Iteration 2048
MAC verified OK
PKCS7 Encrypted data: pbeWithSHA1And40BitRC2-CBC, Iteration 2048
Certificate bag
Bag Attributes
    localKeyID: C4 40 FE 9E 1A 30 48 24 B1 59 94 1C 9A 6A EB 65 04 DA A9 AF 
subject=/CN=51bceeff97c5
issuer=/CN=51bceeff97c5
-----BEGIN CERTIFICATE-----
MIICojCCAYoCAQEwDQYJKoZIhvcNAQEEBQAwFzEVMBMGA1UEAwwMNTFiY2VlZmY5
N2M1MB4XDTE5MTAxMzA3NDIzNFoXDTIwMTAxMjA3NDIzNFowFzEVMBMGA1UEAwwM
NTFiY2VlZmY5N2M1MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA380u
JElYYA8TqhQdflnQW/A73OTpsc67meB9Adi4vTvwc9VP51BI/IkupRfC71IFJg/c
wmPZWQC8Mx/TF2s5XOPWrAOQRGu4cmh2fOG9E4ZJCSq9kzqGZ3BBUTyqC6ZtGad/
VBloOjw7e9D2sWuVDZqwWqo6nGcF2i/ZVDeUYyBa9ul8+opes7ufRZdbDI9s5iOc
LH+6cxL3efmLZIUVMg8/jvC9nzIRTYk0mmYsyPAJcvIvy9RoxLHG62UjmS2lLJN3
RE/jyDj29Xi5Aokplz5pnYoAxzPo9YmUAwlLDD6vYJCa6B2q5votu7NFJ6ttF4Io
/bolg+PyYfLE2nsHOwIDAQABMA0GCSqGSIb3DQEBBAUAA4IBAQCdxsDo+8E7VqMo
I3bke3j2vSnB/KLE+5UpeRsXhO4zRVAcJbVEuppJFkCwxL0ZLOfoQH+yGcdANa7v
9Fh+JTlDInokrwOIQDsh1X09s/Ca3LlWx9pnuOC39gA6XeqO3b75FXq9FLBhNFGO
LSiNtruqXcIgEK3IK2T9DRfR/D1B9TlAONGfAFpIT+szog2LFG3NwyNiaxRRNcR0
ZLpSqWOkkBrKKSaDTn4KRynh5Hs/ZbcsApybXz56rr+NsR9o6T7IhFkAUtzLhW+U
+YqT07fCJ8MHy6FMIz80HpVlktA1Tj790Ynk4YAL9IB2AObxXdSRcoSHBkl4frQP
Jm+0QNEr
-----END CERTIFICATE-----
PKCS7 Data
Shrouded Keybag: pbeWithSHA1And3-KeyTripleDES-CBC, Iteration 2048
Bag Attributes
    localKeyID: C4 40 FE 9E 1A 30 48 24 B1 59 94 1C 9A 6A EB 65 04 DA A9 AF 
Key Attributes:
Enter PEM pass phrase:

Note that this works for a PKCS12 file, whether it's got a .P12 or .PFX extension or otherwise ...

openssl pkcs12 -info -in dave.foobarsnafu

Enter Import Password:
MAC Iteration 2048
MAC verified OK
PKCS7 Encrypted data: pbeWithSHA1And40BitRC2-CBC, Iteration 2048
Certificate bag
Bag Attributes
    localKeyID: C4 40 FE 9E 1A 30 48 24 B1 59 94 1C 9A 6A EB 65 04 DA A9 AF 
subject=/CN=51bceeff97c5
issuer=/CN=51bceeff97c5
-----BEGIN CERTIFICATE-----
MIICojCCAYoCAQEwDQYJKoZIhvcNAQEEBQAwFzEVMBMGA1UEAwwMNTFiY2VlZmY5
N2M1MB4XDTE5MTAxMzA3NDIzNFoXDTIwMTAxMjA3NDIzNFowFzEVMBMGA1UEAwwM
NTFiY2VlZmY5N2M1MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA380u
JElYYA8TqhQdflnQW/A73OTpsc67meB9Adi4vTvwc9VP51BI/IkupRfC71IFJg/c
wmPZWQC8Mx/TF2s5XOPWrAOQRGu4cmh2fOG9E4ZJCSq9kzqGZ3BBUTyqC6ZtGad/
VBloOjw7e9D2sWuVDZqwWqo6nGcF2i/ZVDeUYyBa9ul8+opes7ufRZdbDI9s5iOc
LH+6cxL3efmLZIUVMg8/jvC9nzIRTYk0mmYsyPAJcvIvy9RoxLHG62UjmS2lLJN3
RE/jyDj29Xi5Aokplz5pnYoAxzPo9YmUAwlLDD6vYJCa6B2q5votu7NFJ6ttF4Io
/bolg+PyYfLE2nsHOwIDAQABMA0GCSqGSIb3DQEBBAUAA4IBAQCdxsDo+8E7VqMo
I3bke3j2vSnB/KLE+5UpeRsXhO4zRVAcJbVEuppJFkCwxL0ZLOfoQH+yGcdANa7v
9Fh+JTlDInokrwOIQDsh1X09s/Ca3LlWx9pnuOC39gA6XeqO3b75FXq9FLBhNFGO
LSiNtruqXcIgEK3IK2T9DRfR/D1B9TlAONGfAFpIT+szog2LFG3NwyNiaxRRNcR0
ZLpSqWOkkBrKKSaDTn4KRynh5Hs/ZbcsApybXz56rr+NsR9o6T7IhFkAUtzLhW+U
+YqT07fCJ8MHy6FMIz80HpVlktA1Tj790Ynk4YAL9IB2AObxXdSRcoSHBkl4frQP
Jm+0QNEr
-----END CERTIFICATE-----
PKCS7 Data
Shrouded Keybag: pbeWithSHA1And3-KeyTripleDES-CBC, Iteration 2048
Bag Attributes
    localKeyID: C4 40 FE 9E 1A 30 48 24 B1 59 94 1C 9A 6A EB 65 04 DA A9 AF 
Key Attributes:
Enter PEM pass phrase:

WebSphere Liberty Profile and DB2 in Docker - There's more ...

I wrote about this a few months back: -

 IBM WebSphere Liberty Profile and IBM DB2 and Docker - An approach 

and had a query from a colleague last week.

So I did some of again ....

The context is that my colleague was looking to run a Java class, using the DB2 JDBC JCC4 drivers, where DB2 was running in a Docker container.

So I replicated this, using Liberty as my Java runtime ....

Here's the details: -

Create a directory on the host for DB2 to use for its datastore

mkdir /tmp/db2data

Start a DB2 container


docker run -itd --name mydb2 --privileged=true -p 50000:50000 -e LICENSE=accept -e DB2INST1_PASSWORD=p455w0rd -e DBNAME=testdb -v /tmp/db2data:/database ibmcom/db2

Check the container logs for the DB2 startup

docker logs mydb2 -f

Open a shell into the DB2 container

docker exec -it mydb2 /bin/bash

Switch to db2inst1 instance account

su - db2inst1

Create the DB2 sample database

db2sampl

Quit back to the host OS

exit

exit

Copy the DB2 JDBC JCC4 driver AND license file out of the DB2 container

docker cp mydb2:/opt/ibm/db2/V11.5/java/db2jcc4.jar .
docker cp mydb2:/opt/ibm/db2/V11.5/java/db2jcc_license_cu.jar .

Create a Dockerfile to spin up a Liberty container using the DB2 JAR files

vi Dockerfile

FROM websphere-liberty:latest
ENV LICENSE accept
COPY db2jcc4.jar /
COPY db2jcc_license_cu.jar /
COPY JdbcTestDB2.class /
CMD ["java","-cp","/:/db2jcc4.jar","JdbcTestDB2","carded1.fyre.ibm.com","50000","sample","db2inst1 ","p455w0rd"]

Start the Liberty container

docker run -dt --name mywlp wlp:latest

Check the Liberty logs


000010 CHRISTINE HAAS
000020 MICHAEL THOMPSON
000030 SALLY KWAN
000050 JOHN GEYER
000060 IRVING STERN
000070 EVA PULASKI
000090 EILEEN HENDERSON
000100 THEODORE SPENSER
000110 VINCENZO LUCCHESSI
000120 SEAN O'CONNELL
000130 DELORES QUINTANA
000140 HEATHER NICHOLLS
000150 BRUCE ADAMSON
000160 ELIZABETH PIANKA
000170 MASATOSHI YOSHIMURA
000180 MARILYN SCOUTTEN
000190 JAMES WALKER
000200 DAVID BROWN
000210 WILLIAM JONES
000220 JENNIFER LUTZ
000230 JAMES JEFFERSON
000240 SALVATORE MARINO
000250 DANIEL SMITH
000260 SYBIL JOHNSON
000270 MARIA PEREZ
000280 ETHEL SCHNEIDER
000290 JOHN PARKER
000300 PHILIP SMITH
000310 MAUDE SETRIGHT
000320 RAMLAL MEHTA
000330 WING LEE
000340 JASON GOUNOT
200010 DIAN HEMMINGER
200120 GREG ORLANDO
200140 KIM NATZ
200170 KIYOSHI YAMAMOTO
200220 REBA JOHN
200240 ROBERT MONTEVERDE
200280 EILEEN SCHWARTZ
200310 MICHELLE SPRINGER
200330 HELENA WONG
200340 ROY ALONZO

For the record, the Java class that I'm running is here: -

vi JdbcTestDB2.java

import java.sql.Connection ;
import java.sql.DriverManager ;
import java.sql.ResultSet ;
import java.sql.Statement ;
import java.sql.SQLException;

class JdbcTestDB2
{
 public static void main (String args[])
 {
  try
  {
   Class.forName("com.ibm.db2.jcc.DB2Driver");
  }
  catch (ClassNotFoundException e)
  {
   System.err.println (e) ;
   System.exit (-1) ;
   }
   String hostname      = args[0];
   String port          = args[1];
   String dbName        = args[2];
   String userName      = args[3];
   String password      = args[4];
   String sslConnection = "false";

   java.util.Properties properties = new java.util.Properties();
   properties.put("user",userName);
   properties.put("password", password);

  String url = "jdbc:db2://" + hostname + ":" + port + "/" + dbName;
  try
  {
      Connection connection = DriverManager.getConnection(url,properties);

      String query = "select EMPNO,FIRSTNME,LASTNAME from DB2INST1.EMPLOYEE" ;

      Statement statement = connection.createStatement () ;
   ResultSet rs = statement.executeQuery (query) ;

   while ( rs.next () )
    System.out.println (rs.getString (1) + " " + rs.getString(2) + " " + rs.getString(3)) ;
    connection.close () ;
  }
    catch (java.sql.SQLException e)
  {
   System.err.println (e) ;
   System.exit (-1) ;
  }
 }
}

and I can easily run the same class from the host OS: -

/opt/ibm/java/jre/bin/java -cp $(pwd):$(pwd)/db2jcc4.jar JdbcTestDB2 carded1.fyre.ibm.com 50000 sample db2inst1 p455w0rd

For what it's worth, I have the IBM Java 8 SDK: -

ibm-java-sdk-8.0-5.41-x86_64-archive.bin

installed on the host OS ( Ubuntu 18.04.3 LTS )

Darknet Diaries and the Python Cheatsheet

I've recently been mainlining on back-episodes of the Darknet Diaries podcast, and, in one episode, the host, Jack Rhysider mentioned a Python cheat sheet that he'd written, and was sharing here: -

https://darknetdiaries.com/python/

Definitely worth listening to the podcast and, if you're coding in Python, consider getting the tutorial - it'll cost you one email address, which is cheap at thrice the price !

Sunday, 13 October 2019

Client Authentication and tinkering with various certificate formats

So, long story short, we have a service that builds Docker images, INSIDE a Docker container, and pushes the tagged and built images to Docker Hub ...

But that's not important right now ...

The key ( apologies for the pun ) thing is that the service exposes a series of actions via a REST API that is protected by TLS 1.2, to which we authenticate via a personal certificate.

This particular certificate is actually a PEM ( Privacy Enhanced Mail ) Base64-encoded DER file: -

dave.pem

which contains the personal certificate AND its private key: -

-----BEGIN CERTIFICATE-----
MIICojCCAYoCAQEwDQYJKoZIhvcNAQEEBQAwFzEVMBMGA1UEAwwMNTFiY2VlZmY5
N2M1MB4XDTE5MTAxMzA3NDIzNFoXDTIwMTAxMjA3NDIzNFowFzEVMBMGA1UEAwwM
...
-----END CERTIFICATE-----
-----BEGIN PRIVATE KEY-----
MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDfzS4kSVhgDxOq
FB1+WdBb8Dvc5OmxzruZ4H0B2Li9O/Bz1U/nUEj8iS6lF8LvUgUmD9zCY9lZALwz
H9MXazlc49asA5BEa7hyaHZ84b0ThkkJKr2TOoZncEFRPKoLpm0Zp39UGWg6PDt7
...
-----END PRIVATE KEY-----

Using that certificate, we can authenticate to the REST endpoint, such as this example: -

curl -k https://192.168.1.24:443/image --cert dave.pem

{"root_ssh_enabled":false,"status":"initialized"}

So far, so good.

We then wanted to perform a series of security tests against the same endpoint, using a product called AppScan Standard ( this used to be an IBM Rational offering, and has been recently transitioned to HCL ).

This DOES support client authentication BUT doesn't support a PEM file.

Therefore, we needed to convert the PEM file into a different format, Public-Key Cryptography Standards (PKCS), as either a .p12 or .pfx file.

This is nice n' easy using the Swiss Army knife of security - openssl - as per this: -

openssl pkcs12 -export -out dave.p12 -in dave.pem

and then validate it via cURL: -

curl -k https://192.168.1.24:443/image --cert-type p12 --cert dave.p12

{"root_ssh_enabled":false,"status":"initialized"}

So now we're good to go .....

Thursday, 3 October 2019

Retreading my steps - deleting directories with hyphens ...

I had an issue whilst trying to sort out some directories on my Mac, containing hyphens ( - ) as part of their names: -

Untracked files:
  (use "git add ..." to include in what will be committed)

--header
--url

which were causing fgrep to choke: -

fgrep -R

usage: fgrep [-abcDEFGHhIiJLlmnOoqRSsUVvwxZ] [-A num] [-B num] [-C[num]]
[-e pattern] [-f file] [--binary-files=value] [--color=when]
[--context[=num]] [--directories=action] [--label] [--line-buffered]
[--null] [pattern] [file ...]

I couldn't easily delete them: -

rm -Rf "--header"

rm: illegal option -- -
usage: rm [-f | -i] [-dPRrvW] file ...
       unlink file

ls *header*

ls: illegal option -- -
usage: ls [-ABCFGHLOPRSTUWabcdefghiklmnopqrstuwx1] [file ...]

rm -Rf '--url'

rm: illegal option -- -
usage: rm [-f | -i] [-dPRrvW] file ...
       unlink file

but I remembered hitting this a while back ....

which takes me back to a post that I wrote back in 2010: -

 Removing files with hyphenated filenames in Linux 


I'd had the same problem a few days ago - the problem is that the hyphen ( - ) character is seen by the rm command as an option.

The solution ? A quick rummage around Google threw this up: -

http://serverfault.com/questions/37823/how-to-delete-file-with-option-character-in-name-in-linux

Use the "--" option to tell rm that there are no more options coming, hence the command becomes: -

rm -Rf -- -6895560/

Job done :-)



So I tried that: -

rm -Rf -- --url

rm -Rf -- --header

which did the trick.

And now fgrep is happy ....


Fun with OpenSSL Certificate Requests and space characters in Subject Names

I've got a command within a Dockerfile that generates a Certificate Service Request, via the openssl req  command. This references an ...