Automating GKE creation on CircleCI builds

Last time we launched an IRIS application in the Google Cloud using its GKE service.

And, although creating a cluster manually (or through gcloud) is easy, the modern Infrastructure-as-Code (IaC) approach advises that the description of the Kubernetes cluster should be stored in the repository as code as well. How to write this code is determined by the tool that’s used for IaC.

In the case of Google Cloud, there are several options, among them Deployment Manager and Terraform. Opinions are divided as to which is better: if you want to learn more, read this Reddit thread Opinions on Terraform vs. Deployment Manager? and the Medium article Comparing GCP Deployment Manager and Terraform. 

For this article we’ll choose Terraform, since it’s less tied to a specific vendor and you can use your IaC with different cloud providers.

We’ll assume you’ve read the earlier article and already have a Google account, and that you’ve created a project named “Development,” as in the previous article. In this article, its ID is shown as <PROJECT_ID>. In the examples below, change it to the ID of your own project. 

Keep in mind that Google isn’t free, although it has a free tier. Be sure to control your expenses.

We’ll also assume that you’ve already forked the original repository. We’ll call this fork “my-objectscript-rest-docker-template” and refer to its root directory as “<root_repo_dir>” throughout this article.

All code samples are stored in this repo to simplify copying and pasting.

The following diagram depicts the whole deployment process in one picture:

So, let's install the latest version of Terraform at the time of writing:

 

The version is important here, because many examples on the Internet use earlier versions, and 0.12 brought many changes.

We want Terraform to perform certain actions (use certain APIs) in our GCP account. To enable this, create a Service Account with the name 'terraform', and enable the Kubernetes Engine API. Don’t worry about how we’re going to achieve this — just read further and your questions will be addressed.

Let's try an example with the gcloud utility, although we could also use the Web Console.

We're going to use a couple different commands in the following examples. See the following documentation topics for more details on these commands and features.

• gcloud iam service-accounts create
• Granting roles to a service account for specific resources
• gcloud iam service-accounts keys create
• Enabling an API in your Google Cloud project

Now let's walk through the example.

 

Because we worked with gcloud in the previous article, we won’t discuss all of the setup details here. For this example, run the following commands:

 

Now let's add a few roles to the terraform service account besides “Kubernetes Engine Admin” (container.admin). These roles will be useful to us in the future.

 

Note that the last entry creates your account.json file. Be sure to keep this file secret.

 

Next, let’s describe the GKE cluster in Terraform’s HCL language. Note that we use several placeholders here; replace them with your values:

 

Here’s the HCL configuration for the cluster in practice:

 

To make sure the HCL code is in the proper format, Terraform provides a handy formatting command you can use:

 

The code snippet shown above indicates that the created resources will be provided by Google, and the resources themselves are google_container_cluster and google_container_node_pool, which we designate preemptible for costs savings. We also choose to create our own pool instead of using the default.

Let’s focus briefly on the following setting:

 

Terraform writes everything it's done into the status file and then uses this file for other work. For convenient sharing, it’s better to store this file somewhere in a remote place. A typical place is a Google Bucket.

Let's create this bucket. Use the name of your bucket instead of the placeholder <BUCKET_NAME>. Before bucket creation let’s check if <BUCKET_NAME> is available as it has to be unique across all GCP:

 

Good answer:

 

"Busy" answer means you have to choose another name:

 

Let’s also enable versioning, as Terraform recommends.

 

Terraform is modular and needs to add a Google provider plugin to create something in GCP. We use the following command to do this:

 

Let's look at what Terraform is going to do to create a GKE cluster:

 

The command output includes details of the plan. If you have no objections, let's implement this plan:

 

By the way, to delete the resources created by Terraform, run this command from the <root_repo_dir>/terraform/ directory:

 

Let’s leave the cluster as is for a while and move on. But first note that we don’t want to push everything into the repository, so we’ll add several files to the exceptions:

 

Using Helm

In the previous article, we stored Kubernetes manifests as yaml files in the <root_repo_dir>/k8s/ directory, which we then sent to the cluster using the "kubectl apply" command. 

This time we'll try a different approach: using the Kubernetes package manager Helm, which has recently been updated to version 3. Please, use version 3 or later because version 2 had Kubernetes-side security issues (see Running Helm in production: Security best practices for details). First, we’ll pack the Kubernetes manifests from our k8s/ directory into a Helm package, which is known as a chart. A Helm chart installed in Kubernetes is called a release. In a minimal configuration, a chart will consist of several files:

 

Their purpose is well-described on the official site. The best practices for creating your own charts are described in the The Chart Best Practices Guide in the Helm documentation. 

Here’s what the contents of our files look like:

 

 

 

 

 

To create the Helm charts, install the Helm client and the kubectl command-line utility.

 

Create a namespace called iris. It would be nice if this were created during the deployment, but so far this is not the case.

First, add credentials for the cluster created by Terraform to kube-config:

 

Confirm (without kicking off a real deploy) that Helm is going to create the following in Kubernetes:

 

The output—the Kubernetes manifests—has been omitted for space here. If everything looks good, let’s deploy:

 

We see that Helm has deployed our application, but since we haven’t created the Docker image eu.gcr.io/iris-rest:v1 yet, Kubernetes can’t pull it (ImagePullBackOff):

 

Let’s finish with it for now:

 

The CircleCI Side

Now that we’ve tried out Terraform and the Helm client, let’s put them to use during the deployment process on the CircleCI side.

 

You’ll need to add several environment variables to your project on CircleCI side:

The GCLOUD_SERVICE_KEY is the CircleCI service account key, and TF_SERVICE_ACCOUNT_KEY is the Terraform service account key. Recall that the service account key is the whole content of account.json file.

Next, let’s push our changes to a repository:

 

The CircleCI UI dashboard should show that everything is ok:

 

 

Terraform is an idempotent tool and if the GKE cluster is present, the "terraform" job won’t do anything. If the cluster doesn’t exist, it will be created before Kubernetes deployment.
Finally, let’s check IRIS availability:

 

Conclusion

Terraform and Helm are standard DevOps tools and should be fine integrated with IRIS deployment.

They do require some learning, but after some practice, they can really save you time and effort.