The snowflake problem: every server is a little different. Someone installed nginx by hand, someone else used dnf groupinstall, someone else cloned a different VM template. Six months in, "what runs on this host" is a question whose only honest answer is "go look".

The fix is not "write better docs". The fix is the host is YAML. One block in an inventory file describes what the host is. One playbook bootstraps it from nothing. The output is a running, fully configured server. The inventory is the source of truth. If you want to know what runs on a host, you read the YAML.

This post is about the shape of that bootstrap and the three pieces that make it work for any combination of OS and hypervisor. The code lives at git.archworks.co/sandwich/Ansible-Bootstrap. Eleven distros, five hypervisor backends, one playbook.

What one inventory entry looks like#

db01.example.com:
  ansible_host: 10.0.0.10
  system:
    type: virtual
    os: debian
    version: "12"
    name: db01.example.com
    id: 101
    cpus: 2
    memory: 4096
    network:
      bridge: vmbr0
      ip: 10.0.0.10
      prefix: 24
      gateway: 10.0.0.1
      dns:
        servers: [1.1.1.1, 1.0.0.1]
    disks:
      - size: 40
      - size: 120
    users:
      - name: ops
        keys: [ssh-ed25519 AAAAC3...]
        sudo: "ALL=(ALL) NOPASSWD:ALL"
  hypervisor:
    type: proxmox
    url: pve01.example.com
    node: pve01
    storage: local-lvm
  postgresql:
    version: 17
    listen: 10.0.0.10
    databases: [app, app_test]
  firewall:
    open: [22, 5432]

That is the entire definition. Run ansible-playbook provisioning/bootstrap/main.yml. Twenty minutes later there is a Debian 12 VM on Proxmox, two disks attached, network configured, an ops user with sudo, Postgres 17 listening on the right address, and nftables only permitting 22 and 5432. Run it again on the same inventory entry. Zero changes. Run it after editing the YAML. The delta applies.

Three pieces make this work.

Piece one: the variable contract#

Everything goes through two dicts.

  • hypervisor describes where the host lives. type is proxmox, vmware, libvirt, xen, or none (bare metal). The rest of the keys depend on the type.
  • system describes what the host is. OS, version, identity, CPU, RAM, disks, network, users, features.

Both merge across inventory scopes via hash_behaviour = merge. Put the hypervisor credentials in group_vars/all.yml, the OS family in group_vars/web.yml, the per-host IP in host_vars/db01.yml, and the final dict is the union. Defaults at the top, overrides at the bottom, no leaks between hosts.

The contract is written down once and CI refuses anything that breaks it. The cost is one migration when the contract first lands. The savings is every variable shape argument that never happens after.

Piece two: the universal installer#

The trick that makes this work for eleven different distros is that the installer is always the same: an Arch ISO booted on the target. Arch ships a tiny live environment with pacstrap, arch-chroot, and a kernel that boots on nearly anything. Add a few extra packages and the same live environment can install Debian (debootstrap), RHEL family (dnf --installroot), Alpine (apk), Void (xbps-install), openSUSE (zypper). The pipeline:

  1. Provision the VM on the chosen hypervisor (or unlock the bare-metal box).
  2. Boot the Arch ISO. Wait for SSH to come up on a known IP.
  3. Run the chosen distro's installer command into a chroot. Partition layout, encryption, multi-disk, all driven by the system dict.
  4. Generate /etc/fstab, install a bootloader, set the hostname, drop the user keys.
  5. Reboot into the installed system.
  6. Re-connect and run the role layer.

The trick is that step 3 is the only OS-specific step. Everything before is hypervisor-specific. Everything after is role-driven. The OS layer is one function dispatch by system.os.

So system.os: ubuntu-lts and system.os: rocky share 90% of the same code path. The remaining 10% is the right invocation of debootstrap versus dnf --installroot. Adding a new distro is "write the installer step for that distro, add a row to the support matrix". Adding a new hypervisor is "write the VM-creation tasks for that hypervisor, add a row to the support matrix". The cross product comes free.

Piece three: roles trigger on their variable#

This is the part that makes new roles cheap to add. The pattern:

# in inventory
db01.example.com:
  system: { ... }
  nginx:
    vhosts:
      - name: example.com
        upstream: 127.0.0.1:8080
  cron:
    - name: backup
      schedule: "0 3 * * *"
      command: /usr/local/bin/run-backup.sh

The playbook's role-application loop is one line:

- name: Apply service roles
  ansible.builtin.import_role:
    name: "internal.services.{{ item.key }}"
  loop: "{{ vars | dict2items }}"
  when: item.key in known_service_roles and item.value

If the inventory has nginx: { ... }, the internal.services.nginx role runs. If it has cron: [...], the internal.services.cron role runs. If neither, neither runs. Adding a new role to the catalogue is one PR: write the role, register its name in known_service_roles, and every host that wants it just declares the variable. No playbook edits. No inventory-wide flag flips. No central registry of "what runs where".

The discipline that keeps this clean: a role only reads its own top-level variable. The nginx role reads nginx.*, never firewall.*. The firewall role reads firewall.*, never nginx.*. Roles compose by reading the same shared system dict, never by reading each other's namespaces.

What that looks like running#

Same inventory entry, same playbook. Change hypervisor.type from proxmox to libvirt, the VM gets created on a libvirt host with the same system spec, the rest is identical. Change system.os from debian to rocky, the install step switches to dnf --installroot, the rest is identical. Change hypervisor.type to none, the play assumes the box is already powered on at the inventory IP, skips provisioning, runs the rest.

This is not a future plan. This is what running the bootstrap looks like today across maybe sixty hosts. Half are Debian, a third are RHEL family, the rest are Arch or Alpine. Three hypervisors. The VM-creation tasks are about 200 lines per backend. The OS-install steps are about 100 lines per distro. The role catalogue is around forty service roles. The cross product is tens of thousands of valid host configurations and exactly one playbook to drive them.

Idempotence is the contract#

A bootstrap that only works on a clean install is a one-shot installer. The whole point of declarative is the second run.

The discipline: every play in the pipeline must produce zero changes on a second run against the same inventory. CI proves it. The script is short:

ansible-playbook -i inventories/test/hosts.yml provisioning/bootstrap/main.yml
ansible-playbook -i inventories/test/hosts.yml provisioning/bootstrap/main.yml | tee /tmp/run.log
grep -q 'changed=[1-9]' /tmp/run.log && exit 1

Anything that fails this is broken, even if it produces the right end state on the first run. The reason is drift. If a role reports "9 changed, 3 ok" on a second run, those nine tasks are either misreporting (broken changed_when) or actually not converging. Both are bugs. Both block merge.

The sibling check is --check mode in production. Nightly. Same playbook, no changes applied, exit non-zero if anything would have changed. Combined with CI proving convergence, the fleet and the repo are always within one apply of each other.

The CIS hardening side-quest#

There is a third optional dict: cis. Setting system.features.cis.enabled: true triggers a hardening role that applies the relevant CIS Benchmark for the host's distro: SSH config, kernel sysctls, file permissions, audit rules, login policies. The hardening is parameterised through the cis dict so it can be tuned per host where the defaults break the workload.

It is opt-in for a reason. CIS-hardened defaults break enough things that you want it on production hosts and off dev hosts, and you want one inventory variable to flip between them. The standard pattern is cis.enabled: true in group_vars/prod.yml and nothing in group_vars/dev.yml.

What it does not solve#

It does not solve "what should run where". Capacity planning, service placement, dependency ordering across hosts are decisions made outside the bootstrap. The bootstrap is downstream of those decisions.

It does not solve secrets. Vault paths or vault-encrypted files do that. The bootstrap just reads them. The shape of the inventory is password: !vault | ... and the rest is plumbing.

It does not solve "the OS image is missing the packages I need". Roles are stateful. They install what they need. But if a role assumes a package exists at first run and the OS does not ship it, the first apply fails. The fix is the role's responsibility, not the bootstrap's.

It does not solve "live patching". Bootstrap is for provisioning and convergence, not for runtime mutations. A live patch goes through operational/actions/ playbooks, runs against an existing host, does its specific thing, exits. The bootstrap stays untouched.

Why it is worth the up-front cost#

The cost is real. Writing the contract, writing the universal installer step for each distro, writing the variable-driven role loop, wiring up the CI gate, training people to put their host in YAML before pressing buttons. It is a quarter of work for the first deploy.

The payoff: every host after is twenty minutes of YAML editing. Disaster recovery is "rebuild the inventory entry, run the bootstrap". Onboarding a new engineer is "here is the inventory, here is the role catalogue, write what you want". Auditing what runs anywhere is grep against the inventory. The host is the YAML. The YAML is the host.

I have rebuilt this pattern three times in different forms. Each time the conclusion is the same: the up-front cost is the right cost. The alternative is the snowflake problem, and the snowflake problem compounds.

Source: git.archworks.co/sandwich/Ansible-Bootstrap.