Building a good Terraform CI/CD pipeline

In my job, I work a lot with Terraform, which makes a good CI/CD pipeline crucial. I’ve worked with a few implementations and there are four things that I consider to be baseline requirements of a Terraform CI/CD pipeline:

• the pipeline is secure
• the production infrastructure is kept in sync with the main branch of the infrastructure as code repository
• no one needs to run Terraform CLI commands manually in the normal workflow (although I believe developers should have the ability to do so, as in my experience they will need)
• code reviewers can review the Terraform plan a proposed code change would create as well as the code change itself

One of the simplest ways I’ve come across of setting this up is using CircleCI with the Terraform orb created by Ovotech. It requires you to integrate with Github, provide a couple of keys and then create a config.yml file…..and that’s it! A config.yml file like the one in their docs:

version: '2.1'
orbs:
  terraform: 'circleci/terraform@dev:alpha'
workflows:
  deploy_infrastructure:
    jobs:
      - terraform/fmt:
          checkout: true
          context: terraform
      - terraform/validate:
          checkout: true
          context: terraform
          requires:
            - terraform/fmt
      - terraform/plan:
          checkout: true
          context: terraform
          persist-workspace: true
          requires:
            - terraform/validate
      - terraform/apply:
          attach-workspace: true
          context: terraform
          filters:
            branches:
              only: master
          requires:
            - terraform/plan

will check that your terraform is valid and formatted, add the output of the Terraform plan onto your PR for easy reviewing and then apply it on merge. It has a few really nice additional features as well, such as when you merge a PR, it checks the plan on the PR against the plan it’s about to apply and fails if there are any differences. You can never apply changes that haven’t been code reviewed!

However, using CircleCI in this way has some security risks. If a bad actor got permissions to open pull requests in your Github repository, then they would be able to execute arbitrary commands against your infrastructure with the permissions that the CircleCI Terraform orb is using, by changing the contents of the config.yml file in their pull request. Given running Terraform normally requires high levels of permissions, this could be very damaging.

Another security concern with using CircleCI like this is that the CircleCI virtual machine running the code needs to be able to access your resources. If you’re taking a defence in depth approach to networking then resources like your kubernetes cluster and your databases are likely to be buried in private subnets within your VPC, in order to restrict the access the public internet has to them. Ideally, such resources have no access to the public internet, which is not possible if you need them to be managed by Terraform run by CircleCI, particularly as CircleCI isn’t able to offer a small number of IPs to allowlist.

A more secure option for a Terraform CI/CD platform for cloud infrastructure is to run the build within your cloud VPC and to manage the build pipeline configuration within Terraform as well, so that changes can’t be made to the build pipeline without going through the review and apply process. I don’t know of a good open source Terraform module for doing this, for any cloud provider, so if you’ll probably need to build your own.

I’ve worked on a couple of different implementations of this, both in AWS, one using CodeBuild alone and the other using a combination of CodeBuild and CodePipeline. The main things I’ve learnt are that it requires quite a bit of effort to lock down permissions so that the plan process doesn’t allow a vulnerability at the PR stage (making sure the plan role didn’t have unfettered access to SecretsManager was key), bootstrapping has to be done manually and that posting plans back to the originating PR is a really excellent feature and well worth the effort. Don’t force your reviewers to log into AWS just to review your code properly!

To summarise, if you want a quick, effective Terraform CI/CD pipeline, use CircleCI with the ovotech Terraform orb. That does come with a security risk, if a user gains access to your Github repository, however, so I’d recommend building your own pipeline. If you’re aware of a good open source Terraform module that provides a within-VPC Terraform CI/CD pipeline, do let me know!