In this blog post we are going to setup a kubernetes cluster with automated certificate generation using certbot. Will cover up some interesting concepts of kubernetes along the way, like:
So let's get started.
When using a traditional VM's/instances
, one has access to ssh into a fixed instance with an assigned and fixed public IP that the DNS can resolve to. Once the DNS is set to resolve your hostname to your instance, you can install certbot on it and generate the certs on that instance.
With kubernetes, things get a bit tricky. You will still have a set of instances in your cluster, but they aren't directly accessible from outside the cluster. Plus you cannot preempt on which node your nginx
or other ingress pod will be scheduled to run. Hence the most straight forward way to setup is doing everything through kubernetes
and docker
. This will also provide us with a few advantages:
I'll be using GCP for some parts of this blog post but replicating those parts on other cloud platforms is pretty straight forward.
We'll start by reserving a static IP for our cluster and then forward a DNS record to that IP so that certbot can easily resolve to our cluster.
Make sure to keep the region of the static IP and your cluster same.
Now that we have the static IP, we can move on to the kubernetes part - the fun part.
The first thing that we setup is the load balancer service, we will then use this service to resolve to the pods running our certbot client and later on our own application pods. Below is the YAML for our LoadBalancer service. It utilizes the static IP we created in the previous step.
svc.yml
apiVersion: v1 kind: Service metadata: name: certbot-lb labels: app: certbot-lb spec: type: LoadBalancer loadBalancerIP: X.X.X.X ports: - port: 80 name: "http" protocol: TCP - port: 443 name: "tls" protocol: TCP selector: app: certbot-generator
Most of it is self explanatory, few fields of interest are:
spec.type
: LoadBalancer
- This spins up a Google Cloud LoadBalancer on GCP and AWS Elastic LoadBalancer on AWSspec.loadBalancerIP
- This assign the previously generated static IP to our loadBalancer
, now all traffic coming to our IP address is funneled into this load balancer.ports.port
- We opened 2 TCP ports, port 80
and port 443
for accepting HTTP and HTTPS traffic respectively.spec.selector
- These are the set of labels that allow us to govern which pods can our loadBalancer resolve to. We'll later use the same set of labels in our Job
and Pod
templatesLet's deploy this service to our cluster.
$ kubectl apply -f svc.yml
service "certbot-lb" created
If we see the status of our service now, we should see this
$ kubectl get svc certbot-lb NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE certbot-lb LoadBalancer X.X.X.X X.X.X.X 80:30271/TCP,443:32224/TCP 1m
If you're trying this on minikube, LoadBalancer service is not available. Besides, it makes no sense trying to generate a SSL cert on your local environment.
Next we have to think about where to run our certbot container.
The Job controllers of kubernetes allows us to schedule pods which run to completion. That is, these are jobs that if finished without error, need not be run again in future. This is exactly what we want when generating SSL certs. Once the certbot process is done and has given us the certificates, we no longer need that container to be running. Also we don't want kubernetes to restart this pod when it exits. However we can ask kubernetes to automatically reschedule the pod in case is case the pod exists with a failure/error.
So lets write a Job spec file that will generate the SSL cert for us using certbot. Certbot provides an official docker container that we can just reuse in our case.
jobs.yml
apiVersion: batch/v1 kind: Job metadata: name: certbot spec: template: metadata: labels: app: certbot-generator spec: containers: - name: certbot image: certbot/certbot command: ["certbot"] args: ["certonly", "--noninteractive", "--agree-tos", "--staging", "--standalone", "-d", "staging.ishankhare.com", "-m", "me@ishankhare.com"] ports: - containerPort: 80 - containerPort: 443 restartPolicy: "Never"
Few important fields here are:
spec.template.metadata.labels
- This matches with the spec.selector
specified in our LoadBalancer
service. This brings our pod under our loadBalancer. Now everything coming on port 80 and 443 will be funneled to our pod.spec.template.spec.containers[0].image
- The certbot/certbot
docker image. Kubernetes will do a docker pull certbot/certbot
on the server for us when scheduling this pod.spec.template.spec.containers[0].command
- The command to run in the pod.spec.template.spec.containers[0].args
- The arguments to the above command. We're using the standalone mode of certbot to generate the certs here as it makes things straightforward. You can read more about this command in certbot docsspec.template.spec.containers[0].ports
- We've opened port 80 and 443 for our container.Before deploying this to our cluster, make sure that the domain you specify is actually pointing to the static IP we setup in the previous steps.
Let's deploy this to our cluster:
$ kubectl apply -f jobs.yml
job.batch "certbot" created
This Job
will spin-up a single pod for us, we can get that:
$ kubectl get pods NAME READY STATUS RESTARTS AGE certbot-sgd4w 1/1 Running 0 5s
We can now see the STDOUT logs on this pod
$ kubectl logs -f certbot-sgd4w Saving debug log to /var/log/letsencrypt/letsencrypt.log Plugins selected: Authenticator standalone, Installer None Obtaining a new certificate Performing the following challenges: http-01 challenge for staging.ishankhare.com Waiting for verification... Cleaning up challenges IMPORTANT NOTES: - Congratulations! Your certificate and chain have been saved at: /etc/letsencrypt/live/staging.ishankhare.com/fullchain.pem Your key file has been saved at: /etc/letsencrypt/live/staging.ishankhare.com/privkey.pem Your cert will expire on 2019-03-04. To obtain a new or tweaked version of this certificate in the future, simply run certbot again. To non-interactively renew *all* of your certificates, run "certbot renew" - Your account credentials have been saved in your Certbot configuration directory at /etc/letsencrypt. You should make a secure backup of this folder now. This configuration directory will also contain certificates and private keys obtained by Certbot so making regular backups of this folder is ideal.
After this, the certbot process exits without error and so does our pod. If we now list our pods
$ kubectl get pods NAME READY STATUS RESTARTS AGE certbot-sgd4w 0/1 Completed 0 45m
We see only a single pod whose status is completed.
We can add
.spec.ttlSecondsAfterFinish
to ourjobs.yml
like so:spec: ttlSecondsAfterFinish: 10This will automatically garbage collect our certbot pod after its finished. Only supported in kubernetes v1.12 alpha. Hence not recommended right now.
We currently have 2 options when it comes to saving the certs:
PersistentVolume
and a PersistentVolumeClaim
1Gi
. Hence this is highly inefficient.kubernetes Secrets
. (Recommended)in cluster configuration
, which is straightforward to use and is usually the recommended way in such cases.ClusterRole
and ClusterRoleBinding
.rbac
(RoleBasedAccessControl)To understand this, we need to go a little deeper into our current architecture. Our current setup looks roughly like this:
As we can see here, the certs generated by our certbot
Job Controller Pod are already inside the cluster. We want these cert credentials to be stored on the secrets.
When we fetch/create/modify secrets in normal flow, its usually done through the kubectl
client like:
$ kubectl get secrets NAME TYPE DATA AGE default-token-hks8k kubernetes.io/service-account-token 3 1m $ kubectl create secret Create a secret using specified subcommand. Available Commands: docker-registry Create a secret for use with a Docker registry generic Create a secret from a local file, directory or literal value tls Create a TLS secret Usage: kubectl create secret [flags] [options]
But, in our case, we want to store these credentials as secrets from inside a Pod
which is itself inside the cluster. This is exactly what I meant by in-cluster access above.
We'll need the first 2 mentioned above in the same Pod that is trying to access the secrets. Hence its best to extend the
certbot/certbot
docker image with the above dependencies added in the container image itself.
This will change our cluster architecture a bit. The image below shows the rough cluster arch with our modified setup.
Let's first create our Dockerfile
for our extended container:
FROM certbot/certbot COPY ./script.sh /script.sh RUN wget https://storage.googleapis.com/kubernetes-release/release/v1.6.3/bin/linux/amd64/kubectl RUN chmod +x kubectl RUN mv kubectl /usr/local/bin ENTRYPOINT /script.sh
We have also used an extra
script.sh
in our container. We'll use this script as our entrypoint instead of the default entrypoint ofcertbot/certbot
image. This allows us to start our auxiliarykube-proxy
and use kubectl as we desire. We present thescript.sh
below:
#!/bin/sh # start kube proxy kubectl proxy & certbot certonly --noninteractive --force-renewal --agree-tos --staging --standalone -d staging.ishankhare.com -m me@ishankhare.com kubectl create secret tls cert --cert=/etc/letsencrypt/live/staging.ishankhare.com/fullchain.pem \ --key=/etc/letsencrypt/live/staging.ishankhare.com/privkey.pem # kill the kubectl process running in background kill %1
/etc/letsencrypt/live/staging.ishankhare.com/
, we now use these paths to create the tls
type Secret using kubectl create secret tls
. This command has the following syntax: $ kubectl create secret tls -h Create a TLS secret from the given public/private key pair. The public/private key pair must exist before hand. The public key certificate must be .PEM encoded and match the given private key. Examples: # Create a new TLS secret named tls-secret with the given key pair: kubectl create secret tls tls-secret --cert=path/to/tls.cert --key=path/to/tls.key
Hence our command above will create a secret named cert
Since we are now using an updated Dockerfile with own own script as its entrypoint, we can modify the Job manifest file like below:
apiVersion: batch/v1 kind: Job metadata: #labels: # app: certbot-generator name: certbot spec: template: metadata: labels: app: certbot-generator spec: containers: - name: certbot image: ishankhare07/certbot:0.0.6 - containerPort: 80 - containerPort: 443 restartPolicy: "Never"
With this we are ready to run our job now.
First verify that our LoadBalancer service is up:
$ kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE certbot-lb LoadBalancer 10.11.240.100 X.X.X.X 80:31855/TCP,443:32457/TCP 32m # delete our older job $ kubectl delete job certbot job.batch "certbot" deleted
Now we deploy our new Job file
$ kubectl apply -f jobs.yml job.batch "certbot" created # list the pod created for the job $ kubectl get pods NAME READY STATUS RESTARTS AGE certbot-5nn5h 1/1 Running 0 3s # get STDOUT logs for this pod $ kubectl logs -f certbot-5nn5h Saving debug log to /var/log/letsencrypt/letsencrypt.log Plugins selected: Authenticator standalone, Installer None Obtaining a new certificate Performing the following challenges: http-01 challenge for staging.ishankhare.com Waiting for verification... Cleaning up challenges Starting to serve on 127.0.0.1:8001IMPORTANT NOTES: - Congratulations! Your certificate and chain have been saved at: /etc/letsencrypt/live/staging.ishankhare.com/fullchain.pem Your key file has been saved at: /etc/letsencrypt/live/staging.ishankhare.com/privkey.pem Your cert will expire on 2019-03-13. To obtain a new or tweaked version of this certificate in the future, simply run certbot again. To non-interactively renew *all* of your certificates, run "certbot renew" - Your account credentials have been saved in your Certbot configuration directory at /etc/letsencrypt. You should make a secure backup of this folder now. This configuration directory will also contain certificates and private keys obtained by Certbot so making regular backups of this folder is ideal. Error from server (Forbidden): secrets is forbidden: User "system:serviceaccount:default:default" cannot create secrets in the namespace "default"
The certificate generation was successful. But we cannot yet write to Secrets. The last line in the above output shows us that.
Error from server (Forbidden): secrets is forbidden: User "system:serviceaccount:default:default" cannot create secrets in the namespace "default"
Why is that? Because RBAC....
RBAC or Role Based Access Control in kubernetes consists of 2 parts:
Role
/ClusterRole
RoleBinding
/ClusterRoleBinding
We will be using ClusterRole
and ClusterRoleBinding
approach to provide cluster wide access to our Pod. More fine-grained, role-based access can be provided with the Role
and RoleBinding
approach which can be referred from the docs - https://kubernetes.io/docs/reference/access-authn-authz/rbac/
Let's create 2 files called rbac-cr.yml
and rbac-crb.yml
with the following contents:
rbac-cr.yml
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: namespace: default name: secret-reader rules: - apiGroups: [""] resources: ["secrets"] verbs: ["get", "list", "create"]
This allows access to Secrets
, in-particular to get
, list
and create
Secrets. The last verb create
is what concerns us.
rbac-crb.yml
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: secret-reader subjects: - kind: User name: <your account here> namespace: default - kind: ServiceAccount name: default namespace: default roleRef: kind: ClusterRole name: cluster-admin apiGroup: rbac.authorization.k8s.io
Fields of interest here are .subjects
. It is an array and we have defined two kind
s in this.
kind: User
: This refers to the current user, who is executing these commands using kubectl. This is required so as to grant the current user enough permissions to grant the .rules.resources
and .rules.verbs
related access that we have defined in the rbac-cr.yml
i.e. our ClusterRole
definition.kind: ServiceAccount
: This refers to the account use inside the cluster when our pod will be creating the secrets using the kube-proxy
.We push this to our kubernetes cluster in this particular order only:
$ kubectl apply -f rbac-crb.yml clusterrolebinding.rbac.authorization.k8s.io "secret-reader" created $ kubectl apply -f rbac-cr.yml clusterrole.rbac.authorization.k8s.io "secret-reader" created
The order is important, because we first need to create ClusterRoleBinding
(defined in rbac-crb.yml) and then using that binding on our account, we are going to apply a ClusterRole
(defined in rbac-cr.yml).
I say re-run because since previous job still exists:
$ kubectl get jobs NAME DESIRED SUCCESSFUL AGE certbot 1 1 1h
and because of this job, a stagnant Pod exists as well:
$ kubectl get pods NAME READY STATUS RESTARTS AGE certbot-5nn5h 0/1 Completed 0 1h
If we just try to apply our jobs.yml file now, kubernetes sees that nothing as changed in our yml manifest and chooses to take no action:
$ kubectl apply -f jobs.yml
job.batch "certbot" unchanged
Hence we delete and apply our job from scratch:
$ kubectl delete job certbot job.batch "certbot" deleted $ kubectl apply -f jobs.yml job.batch "certbot" created # get the pod created by the above job $ kubectl get pods NAME READY STATUS RESTARTS AGE certbot-c9h6m 1/1 Running 0 2s
We try to see the STDOUT logs of this container
$ kubectl logs -f certbot-c9h6m Saving debug log to /var/log/letsencrypt/letsencrypt.log Plugins selected: Authenticator standalone, Installer None Obtaining a new certificate Performing the following challenges: http-01 challenge for staging.ishankhare.com Waiting for verification... Cleaning up challenges Starting to serve on 127.0.0.1:8001IMPORTANT NOTES: - Congratulations! Your certificate and chain have been saved at: /etc/letsencrypt/live/staging.ishankhare.com/fullchain.pem Your key file has been saved at: /etc/letsencrypt/live/staging.ishankhare.com/privkey.pem Your cert will expire on 2019-03-13. To obtain a new or tweaked version of this certificate in the future, simply run certbot again. To non-interactively renew *all* of your certificates, run "certbot renew" - Your account credentials have been saved in your Certbot configuration directory at /etc/letsencrypt. You should make a secure backup of this folder now. This configuration directory will also contain certificates and private keys obtained by Certbot so making regular backups of this folder is ideal. secret "cert" created
The last line says secret "cert" created
We can now see this recently created secret:
$ kubectl get secret cert
NAME TYPE DATA AGE
cert kubernetes.io/tls 2 9m
If you actually want to get the contents of the cert you can:
$ kubectl get secret cert -o yaml > cert.yml
# or
$ kubectl get secret cert -o json > cert.json
It is always desirable to redirect the above streams to a file rather than printing them directly to the console.
With this goal achieved, I'll wrap up this post here now. I'll be back with more posts soon detailing on:
Init Containers
along with the Job Controllers
used in this post and our good old Pods
and Deployments
to see how we can make
these interdependent services wait for the depending services to complete before spawning themselves.Do let me know what you think of this post and if you have any questions in the comments section below.
This post was originally published on my blog ishankhare.dev