When systemd‑resolved stalls: Diagnosing and fixing the 1‑second lookup delay

On a busy host, a 1‑second DNS lookup can feel like a blink, but when it happens on every request, it adds up. In 2026, most distributions ship systemd‑resolved as the default resolver, and the default configuration is usually fine. Yet a handful of real‑world scenarios—high‑traffic web servers, containers, or networks with strict firewall rules—expose a subtle 1‑second delay that can be traced back to systemd‑resolved’s handling of upstream DNS servers.

This post walks through the symptoms, the root cause, and practical fixes that keep DNS fast without compromising security.


Why a 1‑second delay matters

  • HTTP latency – A single 1‑second DNS round‑trip can push a page load from 200 ms to 1.2 s, hurting user experience.
  • API rate limits – Some services throttle requests that exceed a certain latency threshold; a slow DNS can trigger those limits.
  • Container orchestration – Kubernetes, Docker Compose, and other orchestrators perform many DNS lookups per pod start; a 1‑second stall can delay cluster boot.
  • Security hardening – A slow resolver can mask DNS amplification attacks or make it harder to detect malicious queries.

1. Identify the symptom

The classic symptom is a consistent 1‑second delay when resolving a domain that is known to exist on the local network or in a trusted DNS zone.

$ time dig @127.0.0.53 example.local

Typical output:

; <<>> DiG 9.18.5 <<>> @127.0.0.53 example.local
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 12345
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;example.local.            IN      A

;; ANSWER SECTION:
example.local.        60      IN      A       192.168.1.42

;; Query time: 1021 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Mon Nov 25 12:34:56 UTC 2026
;; MSG SIZE  rcvd: 56

The Query time: 1021 msec indicates the resolver spent a full second waiting for a response. The same delay appears for any domain that is not cached locally.


2. Root cause: “fallback‑timeout” and “cache‑size” mis‑tuning

systemd‑resolved has two key settings that influence this behavior:

SettingDefaultEffect
CacheSize1024Number of cached entries.
FallbackTimeout1 sTime to wait for a reply from the primary server before falling back to the secondary.

When the primary DNS server is unreachable (e.g., firewall blocks UDP 53, or the server is down), systemd‑resolved waits FallbackTimeout before querying the secondary. If the secondary is also unreachable, the resolver will retry the primary after the same timeout, causing a 1‑second stall on every lookup.

Common triggers

  1. Firewall rules – Many hardened setups block outbound UDP/TCP 53 to reduce DNS amplification risk. If the firewall blocks the primary but not the secondary, the resolver will stall.
  2. IPv6-only upstream – If the primary is IPv6 and the host has no IPv6 connectivity, the resolver will wait for a timeout before falling back to IPv4.
  3. Container networking – Docker’s default network isolates DNS traffic; the host’s resolver may try to reach the host’s DNS server over a bridged interface that is blocked.

3. Quick sanity check

$ systemd-resolve --status | grep -i 'DNS Servers'

If you see two servers and one is unreachable, that’s the likely culprit.

$ ping -c 1 -4 8.8.8.8

If the ping fails, the firewall is blocking outbound UDP/TCP 53.


4. Fixes

4.1. Adjust FallbackTimeout

If you know the primary server is unreliable, reduce the timeout to 0.1 s. Add or edit the following in /etc/systemd/resolved.conf:

[Resolve]
FallbackTimeout=100ms

Reload:

sudo systemctl restart systemd-resolved

Trade‑off: A shorter timeout may cause more frequent fallbacks, which can increase DNS traffic. Use only if you’re certain the primary is often unreachable.

4.2. Reorder DNS servers

Place the most reliable server first. In /etc/systemd/resolved.conf:

[Resolve]
DNS=10.0.0.1 10.0.0.2

The resolver will try 10.0.0.1 first; if it fails, it will immediately fall back to 10.0.0.2.

4.3. Disable the fallback entirely

If you have a single, highly reliable DNS server, you can disable the fallback:

[Resolve]
DNS=10.0.0.1
FallbackDNS=

This removes the 1‑second delay because the resolver never waits for a second server.

4.4. Use systemd-resolve --flush-caches

Sometimes the cache holds stale negative responses. Flush it:

sudo systemd-resolve --flush-caches

4.5. Verify firewall rules

Open outbound UDP/TCP 53 to the primary server:

sudo iptables -A OUTPUT -p udp --dport 53 -d 10.0.0.1 -j ACCEPT
sudo iptables -A OUTPUT -p tcp --dport 53 -d 10.0.0.1 -j ACCEPT

If you use nftables:

sudo nft add rule inet filter output ip daddr 10.0.0.1 udp dport 53 accept
sudo nft add rule inet filter output ip daddr 10.0.0.1 tcp dport 53 accept

Security note: Opening DNS traffic is a common attack vector. Restrict the destination to the known DNS server(s) and consider DNSSEC validation (DNSSEC=yes in resolved.conf).

4.6. Container‑specific workaround

For Docker, set the DNS server explicitly in the daemon configuration:

{
  "dns": ["10.0.0.1"]
}

Then restart Docker:

sudo systemctl restart docker

This bypasses the host’s resolver and eliminates the fallback delay inside containers.


5. Advanced tuning: CacheSize and CacheNegativeTTL

If you have a high‑traffic environment, you might want to increase the cache size to reduce lookups:

[Resolve]
CacheSize=4096
CacheNegativeTTL=30

A larger cache keeps more positive and negative responses, cutting down on repeated queries. The negative TTL controls how long a failed lookup stays cached; a shorter TTL means the resolver will retry sooner, which can help if the upstream becomes available quickly.


6. Monitoring and alerting

Add a simple check to your monitoring stack:

#!/usr/bin/env bash
dig @127.0.0.53 example.local +time=1 +short | grep -q 192.168.1.42
if [[ $? -ne 0 ]]; then
  echo "DNS stall detected" | mail -s "systemd-resolved stall" admin@example.com
fi

Schedule it with cron or a systemd timer. This gives you an early warning if the resolver starts stalling again.


7. Security implications

  • DNSSEC – Enabling DNSSEC (DNSSEC=yes) forces the resolver to validate signatures, which adds a small CPU cost but protects against spoofed responses. It does not affect the 1‑second stall unless the upstream server is unreachable.
  • Split DNS – In environments with internal and external zones, ensure the internal DNS server is reachable. A misconfigured split DNS can trigger the fallback delay.
  • Firewall hardening – Restrict outbound DNS to known servers. A permissive rule (-p udp --dport 53 -j ACCEPT) opens a channel for amplification attacks. Use the destination‑specific rules shown above.

8. Checklist for a production system

ItemCommand / ConfigWhy
Verify DNS serverssystemd-resolve --statusEnsure primary is reachable
Adjust FallbackTimeout/etc/systemd/resolved.confReduce stall
Reorder servers/etc/systemd/resolved.confReliable first
Flush cachesystemd-resolve --flush-cachesRemove stale negatives
Harden firewalliptables/nftablesPrevent amplification
Monitor stallscron scriptEarly detection
Enable DNSSECDNSSEC=yesProtect against spoofing

9. Real‑world example

A mid‑size web farm ran into a 1‑second DNS stall after a firewall upgrade that blocked outbound UDP 53 to the corporate DNS. The load balancer’s health checks started failing, and the application layer reported increased latency.

Fix applied:

  1. Opened UDP/TCP 53 to the corporate DNS in the firewall.
  2. Set FallbackTimeout=100ms to avoid unnecessary waits.
  3. Reordered DNS servers to put the corporate DNS first.
  4. Enabled DNSSEC for added integrity.

Result: DNS queries resolved in under 10 ms, and the application latency dropped from 1.2 s to 200 ms.


10. Takeaway

A 1‑second delay in systemd‑resolved is usually a symptom of a mis‑configured fallback path or a firewall rule that blocks the primary DNS server. By inspecting systemd-resolve --status, adjusting FallbackTimeout, ordering DNS servers, and tightening firewall rules, you can eliminate the stall without compromising security. Remember to monitor the resolver and keep DNSSEC enabled for integrity.

See also