I have been messing around with Terraform and Ansible over the last few weeks, with the goal of hosting a bunch of web services on a VM while providing the ability to easily replicate environments.

So far so good, I ended up with a monorepo, leveraging Github actions and applying my terraform plans from the CI.

Given I am not working in a production environment, I decided to let the CI apply terraform plans once a day. I was not expecting this to have any impact as I do not ship any changes when I sleep and also all my jobs are applied directly on a PR merge.

This morning I saw something strange. I was working on my VM while suddenly the SSH connection was terminated. Curious … I decided to login again to the machine and discovered that the remote ssh key had changed ! This starts getting serious …

After thinking about the different possible scenarios, I decide to clean my known_hosts and to login in the machine. Some files are missing (of course) and I start to think that this is actually a new VM.

Looking at the Terraform runs, I see something strange:

╷ │ Error: OVH client seems to be misconfigured: “Error 500: "Internal server error"” │ │ │ with provider[“registry.terraform.io/ovh/ovh”], │ on provider.tf line 88, in provider “ovh”: │ 88: provider “ovh” { │ ╵

Interesting, it seems that OVH had a production issue on their side. How would that impact me ? Well looking at the next plan, things look a bit wierd

Terraform will perform the following actions:

module.bhs_prod_1.openstack_compute_instance_v2.instances[0] will be updated in-place

~ resource “openstack_compute_instance_v2” “instances” { id = “a824c582-cfcd-4f20-b56e-70e387be6abe” ~ image_name = “Image not found” -> “AlmaLinux 8” name = “bhs-prod-1” tags = [ “common”, “database”, “github_actions”, ] # (16 unchanged attributes hidden)

    # (2 unchanged blocks hidden)
}

Plan: 0 to add, 1 to change, 0 to destroy.

Terraform will perform the following actions:

module.bhs_prod_1.openstack_compute_instance_v2.instances[0] will be updated in-place

~ resource “openstack_compute_instance_v2” “instances” { id = “a824c582-cfcd-4f20-b56e-70e387be6abe” ~ image_name = “Image not found” -> “AlmaLinux 8” name = “bhs-prod-1” tags = [ “common”, “database”, “github_actions”, ] # (16 unchanged attributes hidden)

    # (2 unchanged blocks hidden)
}

Plan: 0 to add, 1 to change, 0 to destroy.

Okay, so the image is to be updated in place ? That does not sound legit. It turns out an image update in place is not at all in-place and leads to the VM being destroyed.