The Two-Server Problem: Auditing /etc Drift Before It Audits You

Every sysadmin has met the two-server problem. You built web01 and web02 from the same checklist. They ran the same stack, served the same app, and passed the same smoke test. Six months later, one of them accepts a connection the other refuses, and nobody remembers why. The packages match. The kernel matches. The only difference is a stray line in /etc/ssh/sshd_config that somebody added during an outage and never wrote down.

That is configuration drift, and it rarely announces itself. It accumulates through quick fixes, vendor installers, late-night troubleshooting, and the quiet confidence of sudo nano. You do not need a full configuration-management rollout to get a handle on it. You need a history, a baseline, and a short ritual that makes drift visible before it becomes a mystery.

Start with the uncomfortable inventory

The first step is admitting that /etc is not a configuration directory. It is an archaeological site. Every layer tells a story: the original install, the package upgrades, the consultant who tweaked sysctl.conf, the time you disabled IPv6 to make a game server behave. None of that is inherently bad. The problem is that the stories are undocumented.

Before you compare two machines, decide what “same” means. Package lists matter, but they are not the whole picture. Two hosts can have identical nginx packages and wildly different behavior because one has a conf.d drop-in that the other lacks. Focus on the files that change runtime behavior: SSH daemon settings, sudoers rules, mount options in fstab, sysctl tunables, systemd unit overrides, cron jobs, hostname resolution, and network configuration. Those are the usual suspects, and they are almost always plain text.

You can start with a crude comparison. From a known-good host, run a read-only diff against a suspect host over SSH, excluding volatile files and secrets. Something like rsync -navi --delete /etc/ suspect:/etc/ will give you a dry-run inventory of differences without copying anything. Treat the output as a conversation starter, not a verdict. Some differences are expected: machine-specific IDs, host keys, and anything generated at boot. The goal is to spot the unexpected ones.

etckeeper is the cheapest insurance you will ever install

If you only do one thing after reading this, install etckeeper. It puts /etc under version control, usually with Git, and commits changes automatically around package manager operations. On Debian and Ubuntu, apt install etckeeper is enough to get started. On Fedora and relatives, the package is available too, and the DNF plugin can hook into transactions.

Initialize it once per host:

sudo etckeeper init
sudo etckeeper commit "Initial import of /etc"

From then on, every package install, upgrade, and removal leaves a commit. You can also commit by hand after a deliberate edit. That matters more than the automation. A commit message like “raise sshd MaxStartups after load test” turns a mysterious one-line change into a documented decision. Six months later, that message is the difference between a five-minute fix and an afternoon of guessing.

A few practical notes. Keep /etc/.git locked down; it contains configuration history and potentially sensitive values. Do not push it to a public repository. If you want off-host backup, use a private remote or an encrypted backup target. Also decide whether you want etckeeper to commit before and after package operations. The defaults are sane, but read the config file and adjust to your comfort level.

Build a baseline that is not “whatever Bob did in 2019”

The second piece is a baseline you can trust. A baseline is not a perfect server. It is a reference point that you have reviewed and agreed on. In a homelab, that might be a tarball of /etc from a freshly configured host, stored somewhere safe. In a small production environment, it might be a Git branch or tag that represents the approved state.

With etckeeper, you already have a natural baseline mechanism. Tag a known-good state:

cd /etc
sudo git tag -a baseline-2026-11 -m "Approved baseline after quarterly review"

When a host starts acting strange, compare its current /etc against that tag. On the same host, git diff baseline-2026-11 -- . shows everything that has changed since approval. Across hosts, you can export the tag to a bare repository and diff two histories, or use a simple diff -r against a read-only copy. The tool matters less than the habit of comparing against something intentional.

Be careful with secrets and generated files. Machine IDs, SSH host keys, and some application secrets will always differ. Exclude them from cross-host comparisons, or you will drown in noise. The point is to surface meaningful drift, not to achieve byte-for-byte symmetry.

The ten-minute weekly diff ritual

A baseline is useless if you never look at it. Set aside ten minutes a week, ideally Monday morning before the ticket queue wakes up. Pick one host, or rotate through them. Run a short review:

cd /etc
sudo git log --since="1 week ago" --oneline
sudo git diff HEAD~10 -- .

You are not reading every line. You are looking for changes you cannot explain. An upgrade that touched a dozen package configs is normal. A hand-edited sshd_config with no commit message is a flag. A new cron job that appeared out of nowhere deserves a question. If you find something legitimate, commit it with a clear message. If you find something accidental, revert it and note why.

This ritual scales better than you might expect. One person can review a small fleet in under an hour. The value is not just catching mistakes. It is building a shared memory of why the infrastructure looks the way it does.

A checklist for the usual suspects

When you sit down to audit drift, run through the files that cause the most pain. Check sshd_config for authentication and access changes. Validate sudoers with visudo -c before and after edits. Review fstab for mount options that affect performance or resilience. Compare sysctl tunables, especially networking and memory settings. Inspect systemd drop-ins under /etc/systemd/system, since they override vendor defaults. Look at cron and systemd timers for jobs nobody owns. Finally, check /etc/hosts and resolver configuration, because DNS weirdness loves to hide there.

A simple diff-based baseline makes this faster. Keep a reviewed copy of those files in a Git repository, or use etckeeper tags, and compare. The checklist is not about suspicion. It is about coverage.

Drift is a decision without a record

The goal is not zero drift. Systems change, and they should. The goal is zero surprise. When a configuration changes, you want a reason, an author, and a rollback path. etckeeper gives you the first two almost for free, and Git gives you the third.

Small teams and homelabs often skip configuration management because it feels like a second job. Fair enough. You do not need Ansible, Puppet, or a platform team to start. You need version control for /etc, a baseline you actually reviewed, and a weekly habit of asking, “Did I mean to do that?” That is enough to turn a museum of bad decisions into a documented history of deliberate ones.