How to run Containers as a Service – Secure Docker communications

Last March, Dimitri did an excellent series on ‘How to run Containers as a Service’. In part 1, Dimitri describes containers, the
problems of running containers and possible solutions. In part 2, he published a very elaborate guide on how to configure Containers as a Service using vRealize Automation. But when upgrading vRealize Automation to the new 7.3 release this breaks. Investigating this revealed that vRealize Automation 7.3 requires secure Docker communications and the unsecured http option used in part 2 no longer works.

Why?

By default, Docker runs via a non-networked Unix socket. If you need Docker to be reachable via the network in a safe manner, you can enable TLS by specifying the tlsverify flag and pointing Docker’s tlscacert flag to a trusted CA certificate. In the daemon mode, it will only allow connections from clients authenticated by a certificate signed by that CA. In the client mode, it will only connect to servers with a certificate signed by that CA.

So how do we secure Docker communications?

Generate Certificates on the Photon OS Template

1. Connect to the Photon OS template via SSH
2. Start and enable docker using the following commands

$ systemctl start docker
$ systemctl enable docker

3. Generate self-signed Certificates using the following commands:

$ openssl genrsa -aes256 -out ca-key.pem 4096
$ openssl req -new -x509 -days 365 -key ca-key.pem -sha256 -out ca.pem

Provide the certificate details and Photon template FQDN, when prompted for Common Name input.

4. Now that we have a CA, you can create a server key and certificate signing request (CSR). Make sure that “Common Name” (i.e., server FQDN or YOUR name) matches the hostname you will use to connect to Docker.

$ openssl genrsa -out server-key.pem 4096
$ openssl req -subj "/CN=[FQDN Docker host]" -sha256 -new -key server-key.pem -out server.csr
$ echo subjectAltName = DNS:[FQDN Docker host],IP:[IP ranges] > extfile.cnf
$ openssl x509 -req -days 365 -sha256 -in server.csr -CA ca.pem -CAkey ca-key.pem \ -CAcreateserial -out server-cert.pem -extfile extfile.cnf

5. For client authentication, create a client key, a certificate signing request, an extensions config file and sign the private key.

$ openssl genrsa -out key.pem 4096
$ openssl req -subj '/CN=client' -new -key key.pem -out client.csr
$ echo extendedKeyUsage = clientAuth > extfile.cnf
$ openssl x509 -req -days 365 -sha256 -in client.csr -CA ca.pem -CAkey ca-key.pem \ -CAcreateserial -out cert.pem -extfile extfile.cnf

6. Next, make the Docker daemon only accept connections from clients providing a certificate trusted by the CA and enable certificate validation. Copy your CA certificate, your server certificate, and your client certificate to your Docker client machine.

$ openssl genrsa -out key.pem 4096
$ dockerd --tlsverify --tlscacert=ca.pem --tlscert=server-cert.pem --tlskey=server-key.pem \ -H=0.0.0.0:2376
$ docker --tlsverify --tlscacert=ca.pem --tlscert=cert.pem --tlskey=key.pem \ -H=[FQDN Docker host]:2376 version

7. Open docker file using vi and add the following line to this file.

$ vi /etc/default/docker

DOCKER_OPTS="--tlsverify --tlscacert=/root/ca.pem --tlscert=/root/server-cert.pem --tlskey=/root/server-key.pem -H=0.0.0.0:2376"

8. Incoming traffic on port 2376 is blocked by default so it must be allowed through the firewall. To do this, add the firewall rule and restart the service to make the changes effective.

$ echo "iptables -I INPUT 1 -p tcp --dport 2376 -j ACCEPT" >>  /etc/systemd/scripts/iptables
$ systemctl restart iptables
$ systemctl restart docker

9. We’re almost done. The final step is to display and save the contents of the public and private certificate keys.

$ cat /root/server-cert.pem
$ cat /root/server-key.pem

Copy and paste the certificates into a text file which you will need to use in a later stage.

Verify secure Docker communications

Now the secure Docker communications is setup we need to check if we performed the steps above in a correct manner, we need to verify if the certificate validation if done correctly. We can do this by adding the template to the Containers Tab within vRealize Automation. To do this do the following:

1. 1. Open your vRA console and go to the Containers tab.
2. 2. Select [Add a Host].
3. 3. Provide the Photon OS template URL in the address field (syntax = https://[IP address]:2376).
4. 4. Enter new credentials using the [New Credentials] option.
5. 5. Enter the correct credentials and select the [Certificate] option.
6. 6. Paste the Docker certificates we saved to notepad earlier in the appropriate fields.
7. 7. Confirm and save the credentials by using the [Verify] button and select [Yes] to confirm the certificate.

If you performed the actions correctly you should see a green message ribbon saying: “Verified successfully!”

8. Now that secure Docker communications is verified, shutdown the Photon VM and convert to a vSphere template

You can now continue with ‘Create a container host Blueprint and configure Container Placement‘ in part 2 of Dimitri’s series on Containers as a Service.