Before You Rebuild That Service, Query the Resolver: A DNS-First Debugging Routine

Some Linux failures announce themselves. A disk fills up, a service crashes, the kernel logs something dramatic. DNS failures are the other kind: they masquerade as slow sudo, broken package managers, flaky SSH, and web apps that work on one machine but not the next. You end up rebuilding a service that was never the problem.

This is a ladder, not a lecture. Start with the cheapest checks, move toward packet captures, and stop when the evidence points somewhere else. The goal is simple: rule out DNS before you spend a Saturday reinstalling things that were fine.

Why DNS Looks Like Everything Else

Almost every command that touches a network name eventually asks a resolver for help. That includes package managers reaching mirrors, SSH looking up host keys, sudo trying to resolve a hostname for logging, and web services connecting to a database by name. If the resolver hesitates, the symptom shows up wherever the lookup happened. A five-second DNS timeout can look like a slow application, a bad sudo configuration, or a failing repository.

In a home lab, the resolver chain is usually more complicated than people remember. Your laptop may talk to systemd-resolved, which talks to a router, which forwards to an ISP resolver, which may or may not respect your internal zone. Add a VPN, a Pi-hole, or a second network interface and the plot thickens.

Rung 1: Separate Reachability from Name Resolution

Before touching configuration, prove whether the problem is names or packets. Ping an IP address you know is reachable, then ping the hostname. Try getent hosts somehost because it uses the same resolver path as most normal programs. Then try dig somehost for a direct DNS query.

If the IP works and the name does not, you are in DNS territory. If neither works, check routing, firewall rules, and link state first. This one step saves a lot of pointless resolver archaeology.

Rung 2: Read resolv.conf Like Evidence

On many modern distributions, /etc/resolv.conf is a symlink into systemd-resolved, often pointing at a stub listener such as 127.0.0.53. Read it, but do not assume it is the whole story:

ls -l /etc/resolv.conf
cat /etc/resolv.conf
resolvectl status

Look for the search domains, the configured DNS servers, and options like timeout and attempts. A search list with several domains can turn a simple lookup into a chain of failed queries before the right one succeeds. If a hostname contains no dot, the resolver may try each search domain in order. That delay is often mistaken for a slow service.

Also check which interface owns the resolver settings. A laptop that has both Ethernet and Wi-Fi up may be sending queries to a lab DNS server on one network and a home router on another. resolvectl status shows per-link DNS configuration, which is exactly what you need when only some hosts fail.

Rung 3: Ask the Resolver Directly

Use resolvectl query to see what systemd-resolved thinks a name means. Then flush the cache and try again:

resolvectl query example.internal
resolvectl flush-caches
resolvectl statistics

The statistics output is underrated. If cache hits are high and misses are climbing during an outage, you are probably looking at a resolver problem rather than an application problem.

For a second opinion, use dig. A normal query tests the resolver you configured. dig +trace walks from the root servers down, which helps distinguish a local resolver problem from an authoritative DNS problem. If dig +trace works but ordinary lookups fail, your local resolver or its upstream path is suspect.

dig example.internal
dig +trace example.com
dig @1.1.1.1 example.com

That last query bypasses your configured resolver. If it succeeds while the default query fails, you have narrowed the problem to your resolver chain.

Rung 4: Split-Horizon and Search Domain Traps

Home labs love split-horizon DNS: internal names resolve to private addresses inside the house, while public names resolve to public addresses outside. The classic failure is a laptop that connected to a VPN or changed networks and is now asking the wrong resolver for an internal name.

Try the fully qualified name with a trailing dot, like db.lab.example.com., to prevent search-domain guessing. Check hostname -f on the host that is misbehaving; a broken domain search path can make local service discovery weird in ways that look like application bugs.

Also watch for short names that collide with public DNS. If your lab uses wiki as a hostname, some resolver somewhere may try wiki.com or a search-domain variant before your internal record. Use fully qualified names in service configs when you want predictable behavior.

Rung 5: Caching and Negative Caching

Caches are helpful until they are not. A record may have changed, but a resolver is still serving the old answer until the TTL expires. Negative caching is the sneakier version: a failed lookup gets remembered, so the name keeps failing even after you fix the zone.

Flush the local cache, then check the upstream resolver. On systemd-resolved, resolvectl flush-caches is the first move. If you run dnsmasq or Pi-hole, restart or flush that service too. Then query again with dig +noall +answer so you see the actual record without extra noise.

If the answer changes depending on which host asks, compare resolvers directly. Query the router, the Pi-hole, and a public resolver from the same machine. Differences usually point to split-horizon configuration, stale forwarding rules, or a cache that has not caught up.

Rung 6: Capture Packets Before Rebuilding

When the logs are vague, let the packets talk. On the client, capture DNS traffic while reproducing the failure:

sudo [tcpdump](https://en.wikipedia.org/wiki/Tcpdump) -ni any 'udp port 53 or tcp port 53'

You are looking for queries that go unanswered, repeated retries, SERVFAIL responses, or NXDOMAIN answers for names you expected to resolve. If you see the query leave the machine and nothing comes back, the problem is upstream. If you see no query at all, the application may be using a different resolver path, a cached answer, or no DNS at all.

Run the same capture on the resolver if you can. Comparing client and server captures tells you whether the query arrived and whether the answer came back. That is usually the moment the mystery collapses.

The Pre-Rebuild Checklist

Before you reinstall a package, rebuild a container, or blame the service, run through this:

• Ping the IP and the hostname separately.
• Check /etc/resolv.conf and resolvectl status.
• Try getent hosts, resolvectl query, and dig.
• Test a fully qualified name with a trailing dot.
• Compare your resolver with a public resolver.
• Flush caches and check for negative caching.
• Capture DNS traffic while the failure happens.
• Confirm whether the failing host uses a different network or VPN.

DNS is rarely the first thing anyone checks because it is boring, shared, and invisible when it works. That is exactly why it causes so much confusion when it breaks. Put the resolver early in your troubleshooting ladder and you will spend less time rebuilding healthy services.