Use CircleCI to Drive CI/CD of Microservices into GKE

7 min readMar 6, 2021

As an early stage startup with a small engineering team, we build and deploy code into GCP/GKE. CircleCI was first introduced through the recommendation of a contracting firm. Later our founding engineering team quickly picked up the expertise on CircleCI and started expanding its usage in various components in our software stack. This article will present our journey of using CircleCI to drive CI/CD of microservices into GKE, the managed Kubernetes environment in GCP. Hope this article will be useful for engineering organizations in the similar journey.

What is CircleCI?

In the high level, CircleCI is a continuous integration and delivery platform helps software teams rapidly release code by automating the build, test, and deploy process. Below is a list of specific attributes about CircleCI:

CircleCI is a SaaS service: It’s much easier to get started compared with setting up a on-premise Jenkins cluster.
Very affordable: with a free tier of one concurrent job executor, and $30/month for 80 concurrent job executors which is more than enough for a small engineering team.
Developer friendly: build configuration sits together with the source code and is defined/maintained by developers.
Good Docker support.
Good integrations with Kubernetes and GCP through ORBs (reusable snippets of code)
Workflow with with manual approval steps.

We will explain some of the points above in the later sections with examples.

Our Development Pipeline

We have four environments in our development pipeline:

Dev: code merged in master branches will be deployed into this environment first.
Staging: builds passed sanity check in Dev environment will be promoted into this environment on demand.
Prod: builds passed QA testing in Staging environment will be promoted into this production environment following a weekly schedule. This environment is customer facing.
Perf: specific builds is deployed in this environment for performance testing only. This environment can be scaled down to zero nodes for cost saving when no performance testing is ongoing.

Below is the high-level architecture of our backend software stack that are deployed and running in GKE clusters. Each colored component is built and deployed by one separate CircleCI pipelines. The same software stack is deployed into all of the 4 environments mentioned above through the CircleCI.

For simplicity, we will take the green-coloredWeb UI microservice as the example to explain how it is being built and deployed into GKE through CircleCI. The Web component is NodeJS/ReactJS based, which is very typical for startup companies to build their initial product upon.

Use CircleCI Pipeline to Build/Deploy Web Microservice

Create GitHub Checks

All our source code repositories reside in GitHub. We need to configure GitHub checks in CircleCI administration console, so that it will provide workflow status messages and rerunning a workflow from the GitHub Checks page.

Create CircleCI Context per Environment

For each of the four development environments mentioned above, a CircleCI context with a list of Environment Variables need to be defined for the CircleCI build scripts to pick up while performing the build/deployment operations into the specific environment. The Environment Variables could include:

Database URLs/passwords
Dependent Service URLs/API Keys
API Keys for SaaS services
GCP service accounts
Google Project Id
GCP cluster
GCP compute zone

Each of the environments could be running in a different GCP project/compute zone/cluster. So these environment variable values should be different for different environments you might deploy your service into. Below is a screen shot of all the CircleCI contexts we have defined. It include those defined for dev/staging/perf/prod environments.

Autopsy of the `.circleci/config.yml` Pipeline Definition

The CircleCI workflows are defined in the .circleci/config.yml file in the root of the source code repository. It will be picked up by the CircleCI SaaS service when the workflow is being triggered per the triggering conditions.

First part of the config.yml: configuring the workflow version and ORBs to be imported.

version: 2.1
orbs:
  gcr: circleci/gcp-gcr@0.8.0
  k8s: circleci/kubernetes@0.11.1
  gke: circleci/gcp-gke@1.1.0

The three ORBs imported in the above snippet are needed for the build jobs to interact with GCR (google container registry), the target Kubernetes cluster in GKE.

Second part of config.yml: defining the executors.

executors:
  build-executor:
    docker:
      - image: circleci/node:12-buster-browsers
      - image: cypress/browsers:chrome67
      - image: redis
      - image: circleci/postgres:11-alpine
        environment:
          POSTGRES_USER: testuser
          POSTGRES_DB: testdb
  deploy-executor:
    docker:
      - image: circleci/python:3.7.9

There are two executors defined in the following snippet. One is build-executor, it’s used for building and testing the NodeJS/ReactJS based web component in a docker environment with multiple docker images applied. Another is deploy-executor, it’s for creating a python based docker image for running GCP/kubectl commands for pushing docker images into GCR and deploy microservice into GKE.

Third part of the config.yml: defining jobs.

jobs:
  test:
    executor: build-executor
    steps:
      - checkout
      - run: npm run subprojects:install
      - run: npm run lint
      - run: npm run test:server
      - run: npm run test:front-end
      - run: ./scripts/test-integration-ci

The snippet above defines a test job that utilizes the build-executorenvironment. It checks out the source code first, then use the npm command to drive the NodeJS module installation, linting and testing of theserver and front-end sub-modules in a typical NodeJS application.

build-docker-publish-and-rollout:
    executor: gke-executor
    steps:
      - checkout
      - gcr/gcr-auth
      - gke/install
      - k8s/install-kubectl
      - setup_remote_docker:
          docker_layer_caching: true
          version: 19.03.13
      - gcr/build-image:
          image: web
          registry-url: gcr.io
          tag: $CIRCLE_BUILD_NUM
      - gcr/push-image:
          image: web
          registry-url: gcr.io
          tag: $CIRCLE_BUILD_NUM
      - run:
          name: "Replace parameters in configs.yml/deployment.yml with values from the CircleCI context."
          command: |
            sed -i 's#GOOGLE_PROJECT_ID#'"$GOOGLE_PROJECT_ID"'#g' k8s-deploy/gcp/deployment.yml
            sed -i 's#DOCKER_IMAGE_TAG#'"$CIRCLE_BUILD_NUM"'#g' k8s-deploy/gcp/deployment.yml
            # sed -i 's#CIRCLE_QUERY_SERVICE_URL#'"$QUERY_SERVICE_URL"'#g' k8s-deploy/gcp/configs.yml
      - run:
          name: "Deploy web configs.yml/deployment.yml to the target K8S cluster"
          command: |
            gcloud container clusters get-credentials $GCP_CLUSTER --region $GOOGLE_COMPUTE_ZONE --project $GOOGLE_PROJECT_ID
            kubectl apply -f k8s-deploy/gcp/configs.yml
            kubectl apply -f k8s-deploy/gcp/deployment.yml

The above snippet defines a build-docker-publish-and-rollout job that utilizes the deploy-executorenvironment to perform the following sequence of actions:

Check out the source code
Initialize GCR/GKE/K8S related modules
Setup remote docker layer caching
Build the docker image of the web microservice
Push the docker image into GCR (google container registry)
Replace parameters in the local configs.yml/deployment.yml (K8S config map and deployment manifest template) with proper values from the CircleCI environment context. Please refer to the previous section on setting up CircleCI contexts.
Initialize gcloud client to point to the target GKE cluster and deploy the web microservice using kubectl commands.

Fourth part of the config.yml: defining workflows.

workflows:
  version: 2
  test-before-merge:
    jobs:
      - test:
          filters:
            branches:
              ignore: master

Above snippet defined one test-before-merge workflow that runs the test job defined in the jobs section. It is triggered for non-master feature branch commits to catch issues before the pull request can be merged into the master branch.

The following snippet defined a test-and-deploy-web workflow that is triggered only after the code is merged into the master branch:

test-and-deploy-web:
    jobs:
      - test:
          filters:
            branches:
              only: master
      - build-docker-publish-and-rollout:
          name: build-publish-rollout-to-dev
          requires:
            - test
          context: dev
      - deploy-to-perf-approval: 
          filters:
            branches:
              only: master
          type: approval 
      - build-docker-publish-and-rollout:
          name: build-publish-rollout-to-perf
          requires:
            - deploy-to-perf-approval
          context: perf
      - deploy-to-staging-approval: 
          filters:
            branches:
              only: master
          type: approval 
      - build-docker-publish-and-rollout:
          name: build-publish-rollout-to-staging
          requires:
            - deploy-to-staging-approval
          context: staging
      - deploy-to-prod-approval: 
          filters:
            branches:
              only: master
          type: approval 
      - build-docker-publish-and-rollout:
          name: build-publish-rollout-to-prod
          requires:
            - test
            - deploy-to-prod-approval
          context: prod

This workflow will build/publish docker images into GCR first and then deploy the new web build into Dev environment. Then it can deploy the same build to Staging/Prod/Perf environments upon approval.

Above are the screen shots of the web pipeline in CircleCI console. It shows that the build has been deployed into Dev environment. And it’s also deployed into the Staging environment after the user approval in thedeploy-to-staging-approval step.

Conclusion

CircleCI is a SaaS based CI/CD platform. It’s easy to get started and very affordable. It’s very developer friendly as the CI/CD workflow is defined as code in YAML format and sit side by side with the source code. We have use it extensively in our company to drive the CI/CD of microservices into GKE in the google cloud. We are very happy with what CircleCI has enabled us to achieve in terms of engineering velocity.