Keyless authentication from CircleCI to Google Cloud

Imagine an organisation that runs its applications, code and processing on Google Cloud (GCP). Sometimes it has to trust other systems to change the infrastructure. When an engineer makes a change on GCP they use their own identity so one can track what they’ve done and when it happened. When a trusted third party, like CircleCI needs to deploy the latest code changes it needs its own identity so it can update Cloud Functions, or create more load balancers.

CircleCI cannot log in to GCP as engineers do, so we give it a service account and private key to access the GCP APIs. This is analogous to a keycard for getting through a locked door but creates problems when we have to think about keeping the service account key secure. If anyone was to find your secure key pass they can get into any rooms or buildings that you can, using your identity.

https://twitter.com/sethvargo/status/1384160529145466888

The service account key is a long-lived JSON token that is typically stored in CircleCI and used in a build and deployment workflows as an environment variable. This introduces security risks; by default, the tokens do not expire until the year 9999, it’s difficult to trace where the JSON tokens have been saved once they are downloaded from GCP and if an engineer was to use one it conceals their own identity.

We’ve got a problem with long-lived security tokens.

Github rolled out keyless authentication to GCP at the end of 2021. At the start of April CircleCI released OpenID Connect tokens for use with cloud providers. This would allow us to trust CircleCI workflows in GCP and stop using long-lived service account tokens.

Each workflow in CircleCI is a sequence of steps that builds, tests and deploys our application code to Google Cloud. We can configure Google Cloud to trust our CircleCI workflows without using static keys. At the start of each workflow run a signed identity token (JWT) is created that describes which workflow is running, which user triggered it and where the token was issued from. In this case the issuer is CircleCI. The token is cryptographically signed and we can verify the integrity of the signature with a publicly available key. We can trust that this token was issued by CircleCI and no one else.

Next, we configure our Google Cloud projects to permit these CircleCI workflow identities access to the GCP APIs to make code and infrastructure changes. We set up a Workload Identity federated identity pool that sets out some rules for validating the identity tokens and exchanging them for a short-lived API access token.

You will recall from earlier that our static tokens expire in the year 9999, sometime in the distant future. Now we can create tokens that expire in 30 minutes, dramatically reducing the impact if the token were to leak. We also don’t need to securely store and track those troublesome JSON tokens any longer.

What does this look like in practice?

We should create a Workload Identity Pool that allows us to use federated identity to authenticate with Google Cloud.

1​​resource "google_iam_workload_identity_pool" "circleci" {
2  provider                  = google-beta
3  project                   = var.project_id
4  description               = "CircleCI Workload Identity Pool"
5  workload_identity_pool_id = "circleci-oidc-pool"
6}

Then we associate a pool provider for CircleCI. We can extract the claims and metadata from the CircleCI token which allow us to map to a service account specific to that CircleCI project. We could go further and map Github users to specific service accounts, but this is likely sufficient for us right now.

 1resource "google_iam_workload_identity_pool_provider" "circleci" {
 2  provider                           = google-beta
 3  workload_identity_pool_id          = google_iam_workload_identity_pool.circleci.workload_identity_pool_id
 4  workload_identity_pool_provider_id = "circleci-provider"
 5  display_name                       = "CircleCI"
 6  description                        = "CircleCI Identity Pool Provider"
 7  disabled                           = false
 8  attribute_mapping = {
 9    "google.subject"          = "assertion.sub"
10    "attribute.project_id"    = "assertion.sub.extract('/project/{project_id}/')"
11    "attribute.organisation"  = "assertion.aud"
12    "attribute.circleci_user" = "assertion.sub.extract('/user/{user_id}')"
13  }
14  oidc {
15    issuer_uri = "https://oidc.circleci.com/org/<org-id>"
16    allowed_audiences = [
17      "<org-id>"
18    ]
19  }
20}

Almost done, we need to allow our federated CircleCI identity to use our GCP service account. We can map the attributes we extract from the token.

1resource "google_service_account_iam_member" "sa" {
2  service_account_id = var.service_account_id
3  role               = "roles/iam.workloadIdentityUser"
4  member             = format("principalSet://iam.googleapis.com/projects/%s/locations/global/workloadIdentityPools/circleci-oidc-pool/attribute.project_id/%s", var.project_id, var.circleci_project_id)
5}

As long as our CircleCI workflow has a context attached, an environment variable CIRCLE_OIDC_TOKEN will be populated with the token. We can write this value to a file for a later step.

1echo "$CIRCLE_OIDC_TOKEN" > circleci-token.txt

In our workflow we can now exchange our CircleCI JWT for a GCP service account access token. We specify the service account email, Workload Identity pool provider ID and an output file for our credentials.

1gcloud iam workload-identity-pools create-cred-config projects/1068313249650/locations/global/workloadIdentityPools/circleci-oidc/providers/circleci \
2    --service-account=[email protected] \
3    --output-file=credentials.json \
4    --credential-source-file=circleci-token.txt \
5    --credential-source-type=text

We now set two environment variables for applications that use the Google SDKs to detect the credentials and perform the token exchange.

1export GOOGLE_APPLICATION_CREDENTIALS=credentials.json
2export CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE=credentials.json

We can now use terraform or gcloud CLI to operate against our GCP infrastructure without needing a long-lived service account token. Even if the tokens are accidentally printed to CircleCI output or logged somewhere they expire in 30 minutes so the window for an attack is greatly reduced.

Check out the CircleCI docs on their OIDC token exchange for more details.