Creating an S3 backend for Terraform

In the beginning, the backend was local. And it was good. But then lo, the project needed to be portable and used by others. And so, the developer said let their be S3. And the backend was ported to a remote state on an S3 bucket. And it was good. And the people rejoiced!

Moving Terraform state to a remote backend is a common occurrence in a Terraform project. Some projects simply start off with a remote backend right from the start, but others need to migrate from the local state to a remote state. This is a quick little article on how to create a Terraform S3 backend without needing to touch your AWS console at all. 😃

1 Create a separate directory for your S3 config files and state

This section is entirely up to the developer but my preference is to keep a separate section of my project to store configurations for common resources used by my backend and modules in a /common directory. But its not necessary, just my preference.

2 Setup your configuration initially with the necessary AWS providers.

terraform {
    required_providers {
        aws = {
          source  = "hashicorp/aws"
          version = "~> 5"
        }
    }
}

provider "aws" {
  region  = "us-east-1"
  profile = "terraform"
}

3 If you haven't already done so, run terraform init to initialize your working directory.

4 Next, add configurations for the S3 and DynamoDB resources that you'll need to store your state file and state lock. Its recommended to setup your S3 bucket with encryption, so you'll need to add the necessary resources to setup encryption as well.

##
## AWS resources to support remote_backend_config
##


# S3 bucket with its super unique name
resource "aws_s3_bucket" "terraform_state" {
  bucket        = "my-super-unique-terraform-state-name"
  force_destroy = true
}

# Good idea to enable versioning
resource "aws_s3_bucket_versioning" "tf_bucket_version" {
  bucket = aws_s3_bucket.terraform_state.id
  versioning_configuration {
    status = "Enabled"
  }
}

# AWS KMS Key to support bucket encryption
resource "aws_kms_key" "tf_kms_key" {
  description             = "This is used to encrypt bucket objects"
  deletion_window_in_days = 10
}

# Encryption policy
resource "aws_s3_bucket_server_side_encryption_configuration" "tf_state_sse" {
  bucket = aws_s3_bucket.terraform_state.id

  rule {
    apply_server_side_encryption_by_default {
      kms_master_key_id = aws_kms_key.tf_kms_key.id
      sse_algorithm     = "aws:kms"
    }
  }
}

# Dynamo DB table used for state locking
resource "aws_dynamodb_table" "terraform_locks" {
  name         = "terraform-state-locking-table"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "LockID"
  attribute {
    name = "LockID"
    type = "S"
  }

}

5 Run terraform apply. This will create all of the resources that we setup in our config file in AWS for our remote backend. We will be running with local state for a moment, but it will be migrated in the next couple of steps.

6 Now you need to change your terraform backend to S3. Modify your terraform block so it looks like this:

terraform {
  backend "s3" {
    bucket         = "my-super-unique-terraform-state-name"
    key            = "terraform/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "terraform-state-locking-table"
    encrypt        = true
  }

  required_providers {
        aws = {
          source  = "hashicorp/aws"
          version = "~> 5"
        }
    }
}

7 Run terraform init again. It will discover you have changed state and ask to migrate. Say yes.

<$> terraform init

Initializing the backend...
Do you want to copy existing state to the new backend?
  Pre-existing state was found while migrating the previous "local" backend to the
  newly configured "s3" backend. No existing state was found in the newly
  configured "s3" backend. Do you want to copy this state to the new "s3"
  backend? Enter "yes" to copy and "no" to start with an empty state.

  Enter a value: yes


Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Congrats! Your all done setting up your remote backend and backend resources. All without having to touch an AWS Console!

Now you can use this remote backend on your project. If you have multiple directories making up your project, you can store the terraform.tfstate files for each at different keys (directories) within S3. Just change the key value to a unique value. For instance, I like to follow the directory structure of my terraform project and mirror that within my S3 key (directory) structure. For example:

Common Directory:

key = "terraform/common/terraform.tfstate"

of for Prod:

key = "terraform/prod/terraform.tfstate"

Hope this was helpful. Enjoy!