I've been tinkering with the most recent version of etcd namely 3.5.0 recently, having built it from their GitHub project.
My initial and main requirement was to test etcd with SSL/TLS, specifically both server-side X509 certificate-based encryption ( to validate endpoint integrity and on-the-wire data encryption ) AND client-side certificate-based authentication ( to ensure that only authenticated clients could access the etcd service ).
Having got this done and dusted, albeit with a self-signed certificate process, with a locally-created Certificate Authority (CA) being used to create the keys and certificates being presented: -
(a) by the etcd server to its clients
(b) the client consuming etcd, both via its REST APIs, being consumed by curl and also the native etcdctl utility.
Now I'm looking at the REST API in more depth, having previously only tested the /version endpoint: -
curl --silent --cacert /root/ssl/ca-cert.pem --cert /root/ssl/client-cert.pem --key /root/ssl/client-key.pem https://localhost:2379/version | jq
{
"etcdserver": "3.5.0",
"etcdcluster": "3.5.0"
}
I then wanted to try sending data to, and receiving data from, etcd again using its REST APIs, recognising that the purpose of etcd is: -
etcd is a strongly consistent, distributed key-value store that provides a reliable way to store data that needs to be accessed by a distributed system or cluster of machines.
Source: https://etcd.io/
Thankfully, the etcd documentation does cover this nicely: -
Why gRPC gateway
Interestingly, this uses the POST HTTP method for almost all of the operations, both PUT and GET, which was new to me ...
The documentation does cover this: -
Put and get keys
Watch keys
etc.
Interestingly, we have this: -
The gateway accepts a JSON mapping for etcd’s protocol buffer message definitions. Note that key and value fields are defined as byte arrays and therefore must be base64 encoded in JSON. The following examples use curl, but any HTTP/JSON client should work all the same.
Source: Using gRPC gateway
Using a test JSON document: -
cat dave.json
{
"key":"012345",
"value": {
"name":"Dave Hay",
"id":"davidhay1969"
}
}
I need to encode the key and the value into base64; the Q&D way to do this is using jq and base64
cat dave.json | jq -r .key | base64
MDEyMzQ1Cg==
cat dave.json | jq -r .value | base64
ewogICJuYW1lIjogIkRhdmUgSGF5IiwKICAiaWQiOiAiZGF2ZWhheTE5NjkiCn0K
and then insert those into a new JSON document: -
cat dave_b64.json
{
"key":"MDEyMzQ1Cg==",
"value": "ewogICJuYW1lIjogIkRhdmUgSGF5IiwKICAiaWQiOiAiZGF2ZWhheTE5NjkiCn0K"
}
We can then put that into etcd again using the REST API: -
curl -X POST --silent --cacert /root/ssl/ca-cert.pem --cert /root/ssl/client-cert.pem --key /root/ssl/client-key.pem https://localhost:2379/v3/kv/put -d @dave_b64.json | jq
{
"header": {
"cluster_id": "14841639068965178418",
"member_id": "10276657743932975437",
"revision": "8",
"raft_term": "2"
}
}
and then query etcd to get it back out again ...
Now, for some reason, etcd uses the range endpoint rather than, say, get but that's fine ....
curl -X POST --silent --cacert /root/ssl/ca-cert.pem --cert /root/ssl/client-cert.pem --key /root/ssl/client-key.pem https://localhost:2379/v3/kv/range -d '{"key":"MDEyMzQ1Cg=="}' | jq
{
"header": {
"cluster_id": "14841639068965178418",
"member_id": "10276657743932975437",
"revision": "8",
"raft_term": "2"
},
"kvs": [
{
"key": "MDEyMzQ1Cg==",
"create_revision": "7",
"mod_revision": "8",
"version": "2",
"value": "ewogICJuYW1lIjogIkRhdmUgSGF5IiwKICAiaWQiOiAiZGF2ZWhheTE5NjkiCn0K"
}
],
"count": "1"
}
Note that we get back the base64 encrypted value field: -
"value": "ewogICJuYW1lIjogIkRhdmUgSGF5IiwKICAiaWQiOiAiZGF2ZWhheTE5NjkiCn0K"
We can quite easily decode this manually: -
echo "ewogICJuYW1lIjogIkRhdmUgSGF5IiwKICAiaWQiOiAiZGF2ZWhheTE5NjkiCn0K" | base64 -d
{
"name": "Dave Hay",
"id": "davehay1969"
}
or decode it on the fly: -
curl -X POST --silent --cacert /root/ssl/ca-cert.pem --cert /root/ssl/client-cert.pem --key /root/ssl/client-key.pem https://localhost:2379/v3/kv/range -d '{"key":"MDEyMzQ1Cg=="}' | jq -r .kvs[].value | base64 -d
{
"name": "Dave Hay",
"id": "davehay1969"
}
I'm sure I can script up a process to create the JSON document containing Base64-coded data without too much problem ....
TL;DR; writing to / reading from etcd feels very achievable, recognising that there's a bit of a learning curve to (a) understand the REST API and (b) handle the Base64 encoding / decoding process ....
