---
title: Decrypt Terraform States in GitLab Backend
date: 2022-10-05
slug: decrypt-terraform-state-gitlab-backend
authors:
- lunik
description: This article explain how you can retreive and decrypt Terraform states stored in GitLab backend from a GitLab backup
tags:
- terraform
- gitlab
- terraform-state
- state
- backend
---

<!--
# CHANGELOG

-->

![cover](/blog/img/posts/2022-10-05-decrypt-terraform-state-gitlab-backend/cover.jpeg)

Assuming you are using the [GitLab Terraform state feature][gitlab-terraform-doc] in your self managed instance and you are using the embded [backup utility provided by GitLab][gitlab-backup-doc].

The Terraform state files are encrypted before they are stored. This means that you cannot retreiv the content at rest.
For this purpose, GitLab use [application secrets][gitlab-application-secrets-doc] (and derive new secrets from thoses keys when needed) to encrypt sensitive content.

You want to retreiv the content of a state file from a GitLab backup. Like explained in [this issue][gitlab-terraform-state-offline-issue], it's not possible to easily retreiv a decrypted content is the instance is offline.

<!-- truncate -->

## Recover procedure

### Retreiving GitLab instance secrets

Using the [GitLab documentation about application secrets][gitlab-application-secrets-doc] you need to retreiv the value of the `db_key_base`.

### Retreiving project informations

Terraform state storage is linked to a GitLab project. If the project still exists in your instance, save the `project_id` for later.


If you have deleted the project we are going to need the database backup (also stored in the GitLab backup).

First search in the `public.routes` table for your project. You can filter by the `source_type=Project` and `path=path/of/your/project`. Write down the value in the `namespace_id` column.

Then look at the `public.projects` table filtering the `namespace_id` retreived just before. Write down the `id` of the project.

### Calculating the state storage path

States are stored with hashed path.

The first part is the hash256 sum of the project `id`. You can use the following command to calculate it :
```bash
echo -n "<project_id>" | openssl dgst -sha256
```

The second part can be retreived from the database in the table `public.terraform_states`. Filter by the `project_id` and state `name` save the `uuid`.


The full folder path of the state file is now :
```
<part_1[0:1]>/<part_1[2:3]>/<part_1>/<part_2>
```

If you have enabled versionning state are stored with the filename `<serial_number>.tfstate`

### Retreiving the content

This is the hardest part and you need to be a little familiar with [Ruby][ruby-lang-website].

GitLab use a tool named [Lockbox][lockbox-github] to encrypt Terraform state content before storing them. We are going to use the same tool to decrypt our files.

Fill and use the following script :
```ruby
#################
# Configuration #
#################

# Retreived from GitLab rails secrets
# https://docs.gitlab.com/ee/development/application_secrets.html
# This is a dummy key base. Don't bother using it
db_key_base = "<FILL_DB_KEY_BASE>"

# The project ID in GitLab
project_id = "<FILL_PROJECT_ID>"

# The file to decrypt
input_file = "<FILL_ENCRYPTED_STATE_FILE_PATH>"

# The file where to write the terraform state content
output_file = "<FILL_DECRYPTED_STATE_FILE_PATH>"

#############
# ALGORITHM #
#############

# Compute encryption key
key = OpenSSL::HMAC.digest('SHA256', db_key_base, project_id)

# Generate LockBox tool
# https://github.com/ankane/lockbox
lockbox = Lockbox.new(key: key)

encrypted_state_content = File.binread(input_file)
state_content = lockbox.decrypt_str(encrypted_state_content)
File.write(output_file, state_content)
```

I'm not an expert in [Ruby][ruby-lang-website] but here is a simple bash script to setup the correct environment : 
```bash
#!/bin/bash

rails new myapp
cd myapp/

echo 'gem "lockbox"' >> Gemfile
bundle install

bundle exec rails runner decode.rb
```

## Resources

- [GitLab source code used to encrypt state files][gitlab-state-uploaded-gitlab]
- [GitLab issue for documenting the recover procedure][gitlab-recover-procedure-issue]

<!-- links -->

[gitlab-terraform-doc]: https://docs.gitlab.com/ee/user/infrastructure/iac/terraform_state.html
[gitlab-backup-doc]: https://docs.gitlab.com/charts/backup-restore/
[gitlab-terraform-state-offline-issue]: https://gitlab.com/gitlab-org/gitlab/-/issues/335739
[gitlab-application-secrets-doc]: https://docs.gitlab.com/ee/development/application_secrets.html
[ruby-lang-website]: https://www.ruby-lang.org/
[lockbox-github]: https://github.com/ankane/lockbox
[gitlab-state-uploaded-gitlab]: https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/uploaders/terraform/state_uploader.rb
[gitlab-recover-procedure-issue]: https://gitlab.com/gitlab-org/gitlab/-/issues/342225