annexe 2 - tfstates

1. Goals

Terraform uses states to remember what it has done. It will then compare those states wih the new resources you can eventually add.
Just check the doc here.

By default, the states are stored locally, but they can be stored remotely in Consul. This is a great feature:

We will use this to be able to destroy and recreate our bastion host at will: this can help to save a lot of money

2. Activation of the Consul backend

We need to activate the Consul backend in all Terraform projects.
The configuration file is already present but not activated.

[bastion] (ansible_virtualenv) ~/
$ cd ~/ansible_playbooks

[bastion] (ansible_virtualenv) ~/ansible_playbooks
$ find . -name "backend.tf.disabled" -exec sh -c 'cp "$1" "${1%.disabled}"' _ {} \;

You will also need a Consul token to be able to write in the Consul KV store.
For the demo, just take the global management token or generate one from the Consul UI.

[bastion] (ansible_virtualenv) ~/ansible_playbooks
$ export CONSUL_HTTP_TOKEN=XXXX-XXXX-XXXX-XXXX

3. Push of the states in Consul

We must go in all projects and “init” them one more time. Terraform wil take care of pushing the existing states in Consul.

[bastion] (ansible_virtualenv) ~/ansible_playbooks
$ cd echo/terraform/demo/echo

[bastion] (ansible_virtualenv) ~/ansible_playbooks/echo/terraform/demo/echo
$ cd system_socat_green
$ terraform init
$ cd ..

[bastion] (ansible_virtualenv) ~/ansible_playbooks/echo/terraform/demo/echo
$ cd system_socat_blue
$ terraform init
$ cd ..

[bastion] (ansible_virtualenv) ~/ansible_playbooks/echo/terraform/demo/echo
$ cd system_haproxy
$ terraform init
$ cd ..

[bastion] (ansible_virtualenv) ~/ansible_playbooks/echo/terraform/demo/echo
$ cd network
$ terraform init
$ cd ..

[bastion] (ansible_virtualenv) ~/ansible_playbooks/echo/terraform/demo/echo
$ cd ssl
$ terraform init
$ cd ..

[bastion] (ansible_virtualenv) ~/ansible_playbooks/echo/terraform/demo/echo
$ cd ~/ansible_playbooks/infrasecrets/terraform/demo

[bastion] (ansible_virtualenv) ~/ansible_playbooks/infrasecrets/terraform/demo
$ cd infrasecrets/system
$ terraform init
$ cd ../..

[bastion] (ansible_virtualenv) ~/ansible_playbooks/infrasecrets/terraform/demo
$ cd infrasecrets/network
$ terraform init
$ cd ../..

[bastion] (ansible_virtualenv) ~/ansible_playbooks/infrasecrets/terraform/demo
$ cd core-network
$ terraform init
$ cd ..

You can check in the Consul UI that the states are now stored:

Consul KV

Consul KV Code

4. Destroy the bastion host

I hope we didn’t miss anything because we are going to destroy the bastion host.
Go back your workstation.

[workstation] (ansible_virtualenv) ~/demo_big_infra
$ cd ~/demo_big_infra/infrasecrets/terraform/demo/bastion/system
$ terraform destroy -var-file '../../vars_network.tf'

5. Well, now, recreate it!

And now, we recreate it.

[workstation] (ansible_virtualenv) ~/demo_big_infra/infrasecrets/terraform/demo/bastion/system
$ terraform apply -var-file '../../vars_network.tf'

The SSH fingerprint of the new host is of course different from the previous one.
But thanks to SSHFP, there is no need to check everything is OK!

Of course, we need Ansible to provision it, just like the first time:

[workstation] (ansible_virtualenv) ~/demo_big_infra/infrasecrets/terraform/demo/bastion/system
$ cd ~/demo_big_infra/infrasecrets
$ ansible-playbook INIT_bastion.yml -i inventories/demo/hosts_bastion.lst -D -e @inventories/demo/extra_vars_terraform.yml

6. Connect to the bastion

Let’s try to connect to the bastion.

[workstation] (ansible_virtualenv) ~/demo_big_infra
$ ssh -i ~/.ssh/demo_big_infra -A persecutor@bastion-eu-west-3a-0.terror.ninja

Activate the virtualenv and export the API Keys and Tokens:

[bastion] ~/
$ source ansible_virtualenv/bin/activate
$ export AWS_ACCESS_KEY_ID=CHANGE_WITH_KEY_ID
$ export AWS_SECRET_ACCESS_KEY=CHANGE_WITH_SECRET_KEY
$ export CLOUDFLARE_EMAILCLOUDFLARE_EMAIL=CHANGE_WITH_EMAIL
$ export CLOUDFLARE_TOKEN=CHANGE_WITH_TOKEN
$ export BOTO_USE_ENDPOINT_HEURISTICS=True

We can now finish the installation by joining the existing Consul cluster.

To execute Ansible, you will need to replace the following Ansible extra-vars parameters:

  • my_vault_ops_password: this is the password of the ops user
[bastion] (ansible_virtualenv) ~/
$ cd ~/ansible_playbooks/infrasecrets

[bastion] (ansible_virtualenv) ~/ansible_playbooks/infrasecrets
$ ansible-playbook BASTION_install.yml \
-i inventories/demo/ec2.py -i inventories/demo/hosts_ec2.lst \
-D --force-handlers \
-e @inventories/demo/extra_vars_terraform.yml \
-e "my_vault_ops_password=CHANGE_WITH_OPS_PASSWORD" \
-l bastion

7. Get the Terraform states back

We need to activate the Consul backend in all Terraform projects.

[bastion] (ansible_virtualenv) ~/ansible_playbooks/infrasecrets
$ cd ..
$ find . -name "backend.tf.disabled" -exec sh -c 'cp "$1" "${1%.disabled}"' _ {} \;

And then, you can test that you have all states:

[bastion] (ansible_virtualenv) ~/ansible_playbooks
$ export CONSUL_HTTP_TOKEN=XXXX-XXXX-XXXX-XXXX

[bastion] (ansible_virtualenv) ~/ansible_playbooks
$ cd echo/terraform/demo/echo/system_socat_green
$ terraform init
$ terraform apply -var-file '../../vars_network.tf' -var-file '../../vars_network_echo.tf'

Nothing should be created or destroyed with the cloud providers: only the local files should be regenerated!

8. And it’s done!