etcd

Trident’s use of etcd

Trident uses etcd to maintain state for the objects that it manages.

By default, Trident deploys an etcd container as part of the Trident pod. This is a single node etcd cluster managed by Trident that’s backed by a highly reliable volume from a NetApp storage system. This is perfectly acceptable for production.

Using an external etcd cluster

In some cases there may already be a production etcd cluster available that you would like Trident to use instead, or you would like to build one.

Note

Kubernetes itself uses an etcd cluster for its objects. While it’s technically possible to use that cluster to store Trident’s state as well, it is highly discouraged by the Kubernetes community.

Beginning with Trident 18.01, this is a supported configuration provided that the etcd cluster is using v3. It is also highly recommend that you encrypt communication to the remote cluster with TLS.

The instructions in this section cover both the case where you are deploying Trident for the first time with an external etcd cluster and the case where you already have a Trident deployment that uses the local etcd container and you want to move to an external etcd cluster without losing any state maintained by Trident.

Step 1: Bring down Trident

First, make sure that you have started Trident successfully at least once with version 18.01 or above. Previous versions of Trident used etcdv2, and Trident needs to start once with a more recent version to automatically upgrade its state to the etcdv3 format.

Now we need to bring down Trident so that no new state is written to etcd. This is most easily done by running the uninstall script, which retains all state by default.

The uninstall script is located in the Trident installer bundle that you downloaded to install Trident.

trident-installer$ ./uninstall_trident.sh -n <namespace>

Step 2: Copy scripts and the deployment file

As part of this step, we copy three files to the root of the Trident installer bundle directory:

trident-installer$ ls extras/external-etcd/trident/
etcdcopy-job.yaml  install_trident_external_etcd.sh  trident-deployment-external-etcd.yaml
trident-installer$ cp extras/external-etcd/trident/* .

etcdcopy-job.yaml contains the definition for an application that copies etcd data from one endpoint to another. If you are setting up a new Trident instance with an external cluster for the first time, you can ignore this file and Step 3.

trident-deployment-external-etcd.yaml contains the Deployment definition for the Trident instance that is configured to work with an external cluster. This file is used by the install_trident_external_etcd.sh script.

As the contents of trident-deployment-external-etcd.yaml and install_trident_external_etcd.sh suggest, the install process is much simpler with an external etcd cluster as there is no need to run Trident launcher to provision a volume, PVC, and PV for the Trident deployment.

Step 3: Copy etcd data from the local endpoint to the remote endpoint

Once you make sure that Trident is not running as instructed in Step 1, configure etcdcopy-job.yaml with the information about the destination cluster. In this example, we are copying data from the local etcd instance used by the terminated Trident deployment to the remote cluster.

etcdcopy-job.yaml makes reference to the Kubernetes Secret etcd-client-tls, which was created automatically if you installed the sample etcd cluster. If you already have a production etcd cluster set up, you need to generate the Secret yourself and adjust the parameters taken by the etcd-copy container in etcdcopy-job.yaml.

For example, etcdcopy-job.yaml is based on a Secret that was created using the following command:

trident-installer/extras/external-etcd$ kubectl --namespace=trident create secret generic etcd-client-tls --from-file=etcd-client-ca.crt=./certs/ca.pem --from-file=etcd-client.crt=./certs/client.pem --from-file=etcd-client.key=./certs/client-key.pem

Based on how you set up your external cluster and how you name the files that make up the Secret, you may have to modify etcdv3_dest arguments in etcdcopy-job.yaml as well:

- -etcdv3_dest
- https://trident-etcd-client:2379
- -etcdv3_dest_cacert
- /root/certs/etcd-client-ca.crt
- -etcdv3_dest_cert
- /root/certs/etcd-client.crt
- -etcdv3_dest_key
- /root/certs/etcd-client.key

Once etcdcopy-job.yaml is configured properly, you can start migrating data between the two etcd endpoints:

trident-installer$ kubectl create -f etcdcopy-job.yaml
job "etcd-copy" created
trident-installer$ kubectl get pod -aw
NAME                             READY     STATUS      RESTARTS   AGE
etcd-copy-fzhqm                  1/2       Completed   0          14s
etcd-operator-3986959281-782hx   1/1       Running     0          1d
etcdctl                          1/1       Running     0          1d
trident-etcd-0000                1/1       Running     0          1d
trident-installer$ kubectl logs etcd-copy-fzhqm -c etcd-copy
time="2017-11-03T14:36:35Z" level=debug msg="Read key from the source." key="/trident/v1/backend/solidfire_10.250.118.144"
time="2017-11-03T14:36:35Z" level=debug msg="Wrote key to the destination." key="/trident/v1/backend/solidfire_10.250.118.144"
time="2017-11-03T14:36:35Z" level=debug msg="Read key from the source." key="/trident/v1/storageclass/solidfire"
time="2017-11-03T14:36:35Z" level=debug msg="Wrote key to the destination." key="/trident/v1/storageclass/solidfire"
trident-installer$ kubectl delete -f etcdcopy-job.yaml
job "etcd-copy" deleted

The logs for etcd-copy should indicate that Job has successfully copied Trident’s state to the remote etcd cluster.

Step 4: Install Trident with an external etcd cluster

Prior to running the install script, please adjust trident-deployment-external-etcd.yaml to reflect your setup. More specifically, you may need to change the etcdv3 endpoint and Secret if you did not rely on the instructions on this page to set up your etcd cluster.

trident-installer$ ./install_trident_external_etcd.sh -n trident

That’s it! Trident is now up and running against an external etcd cluster. You should now be able to run tridentctl and see all of the same configuration you had before.

Building your own etcd cluster

We needed to be able to easily create etcd clusters with RBAC and TLS enabled for testing purposes. We think that the tools we built to do that are also a useful way to help others understand how to do that.

This provides a reference to show how Trident operates with an external etcd cluster, and it should be generic enough to use for applications other than Trident.

These instructions use the etcd operator and are based on the information found in Cluster Spec, Cluster TLS Guide, etcd Client Service, Operator RBAC Setup, and Generating Self-signed Certificates.

Installing

The Trident installer bundle includes a set of scripts and configuration files to set up an external cluster. These files can be found under trident-installer/extras/external-etcd/.

To install the etcd cluster in namespace trident, run the following command:

trident-installer$ cd extras/external-etcd/
trident-installer/extras/external-etcd$ ./install_etcd.sh -n trident
Installer assumes you have deployed Kubernetes. If this is an OpenShift deployment, make sure 'oc' is in the $PATH.
cfssl and cfssljson have already been downloaded.
serviceaccount "etcd-operator" created
clusterrole "etcd-operator" created
clusterrolebinding "etcd-operator" created
deployment "etcd-operator" created
secret "etcd-client-tls" created
secret "etcd-server-tls" created
secret "etcd-peer-tls" created
etcdcluster "trident-etcd" created

The above script creates a few Kubernetes objects, including the following:

trident-installer/extras/external-etcd$ kubectl get pod
NAME                             READY     STATUS    RESTARTS   AGE
etcd-operator-3986959281-m048l   1/1       Running   0          1m
trident-etcd-0000                1/1       Running   0          20s
trident-installer/extras/external-etcd$ kubectl get service
NAME                  CLUSTER-IP    EXTERNAL-IP   PORT(S)             AGE
trident-etcd          None          <none>        2379/TCP,2380/TCP   1m
trident-etcd-client   10.99.21.44   <none>        2379/TCP            1m
trident-installer/extras/external-etcd$ kubectl get secret
NAME                        TYPE                                  DATA      AGE
default-token-ql7s3         kubernetes.io/service-account-token   3         72d
etcd-client-tls             Opaque                                3         1m
etcd-operator-token-nsh2n   kubernetes.io/service-account-token   3         1m
etcd-peer-tls               Opaque                                3         1m
etcd-server-tls             Opaque                                3         1m

The Kubernetes Secrets shown above are constructed using the CA, certificates, and private keys generated by the installer script:

trident-installer/extras/external-etcd$ ls certs/
ca-config.json  ca-key.pem  client-csr.json  gen-ca.sh      gen-server.sh  peer-key.pem  server-csr.json
ca.csr          ca.pem      client-key.pem   gen-client.sh  peer.csr       peer.pem      server-key.pem
ca-csr.json     client.csr  client.pem       gen-peer.sh    peer-csr.json  server.csr    server.pem

For more information about the Secrets used by the operator, please see Cluster TLS Guide and Generating Self-signed Certificates.

Testing

To verify the cluster we brought up in the previous step is working properly, we can run the following commands:

trident-installer/extras/external-etcd$ kubectl create -f kubernetes-yaml/etcdctl-pod.yaml
trident-installer/extras/external-etcd$ kubectl exec etcdctl -- etcdctl --endpoints=https://trident-etcd-client:2379 --cert=/root/certs/etcd-client.crt --key=/root/certs/etcd-client.key --cacert=/root/certs/etcd-client-ca.crt member list -w table
trident-installer/extras/external-etcd$ kubectl exec etcdctl -- etcdctl --endpoints=https://trident-etcd-client:2379 --cert=/root/certs/etcd-client.crt --key=/root/certs/etcd-client.key --cacert=/root/certs/etcd-client-ca.crt put foo bar
trident-installer/extras/external-etcd$ kubectl exec etcdctl -- etcdctl --endpoints=https://trident-etcd-client:2379 --cert=/root/certs/etcd-client.crt --key=/root/certs/etcd-client.key --cacert=/root/certs/etcd-client-ca.crt get foo
trident-installer/extras/external-etcd$ kubectl exec etcdctl -- etcdctl --endpoints=https://trident-etcd-client:2379 --cert=/root/certs/etcd-client.crt --key=/root/certs/etcd-client.key --cacert=/root/certs/etcd-client-ca.crt del foo
trident-installer/extras/external-etcd$ kubectl delete -f kubernetes-yaml/etcdctl-pod.yaml

The above commands invoke the etcdctl binary inside the etcdctl pod to interact with the etcd cluster. Please see kubernetes-yaml/etcdctl-pod.yaml to understand how client credentials are supplied using the etcd-client-tls Secret to the etcdctl pod. It is important to note that etcd operator requires a working kube-dns pod as it relies on a Kubernetes Service to communicate with the etcd cluster.

Uninstalling

To uninstall the etcd cluster in namespace trident, run the following:

trident-installer/extras/external-etcd$ ./uninstall_etcd.sh -n trident