How to decrypt Terraform states stored in GitLab backend

cover

Assuming you are using the GitLab Terraform state feature in your self managed instance and you are using the embded backup utility provided by GitLab.

The Terraform state files are encrypted before they are stored. This means that you cannot retreiv the content at rest. For this purpose, GitLab use application secrets (and derive new secrets from thoses keys when needed) to encrypt sensitive content.

You want to retreiv the content of a state file from a GitLab backup. Like explained in this issue, it's not possible to easily retreiv a decrypted content is the instance is offline.

Recover procedure

Retreiving GitLab instance secrets

Using the GitLab documentation about application secrets you need to retreiv the value of the db_key_base.

Retreiving project informations

Terraform state storage is linked to a GitLab project. If the project still exists in your instance, save the project_id for later.

If you have deleted the project we are going to need the database backup (also stored in the GitLab backup).

First search in the public.routes table for your project. You can filter by the source_type=Project and path=path/of/your/project. Write down the value in the namespace_id column.

Then look at the public.projects table filtering the namespace_id retreived just before. Write down the id of the project.

Calculating the state storage path

States are stored with hashed path.

The first part is the hash256 sum of the project id. You can use the following command to calculate it :

echo -n "<project_id>" | openssl dgst -sha256

The second part can be retreived from the database in the table public.terraform_states. Filter by the project_id and state name save the uuid.

The full folder path of the state file is now :

<part_1[0:1]>/<part_1[2:3]>/<part_1>/<part_2>

If you have enabled versionning state are stored with the filename <serial_number>.tfstate

Retreiving the content

This is the hardest part and you need to be a little familiar with Ruby.

GitLab use a tool named Lockbox to encrypt Terraform state content before storing them. We are going to use the same tool to decrypt our files.

Fill and use the following script :

#################
# Configuration #
#################

# Retreived from GitLab rails secrets
# https://docs.gitlab.com/ee/development/application_secrets.html
# This is a dummy key base. Don't bother using it
db_key_base = "<FILL_DB_KEY_BASE>"

# The project ID in GitLab
project_id = "<FILL_PROJECT_ID>"

# The file to decrypt
input_file = "<FILL_ENCRYPTED_STATE_FILE_PATH>"

# The file where to write the terraform state content
output_file = "<FILL_DECRYPTED_STATE_FILE_PATH>"

#############
# ALGORITHM #
#############

# Compute encryption key
key = OpenSSL::HMAC.digest('SHA256', db_key_base, project_id)

# Generate LockBox tool
# https://github.com/ankane/lockbox
lockbox = Lockbox.new(key: key)

encrypted_state_content = File.binread(input_file)
state_content = lockbox.decrypt_str(encrypted_state_content)
File.write(output_file, state_content)

I'm not an expert in Ruby but here is a simple bash script to setup the correct environment :

#!/bin/bash

rails new myapp
cd myapp/

echo 'gem "lockbox"' >> Gemfile
bundle install

bundle exec rails runner decode.rb