How DNS Works (Recursive Resolution and Stub Resolvers)

Hi there! I'm Maneshwar. Right now, I’m building LiveAPI, a first-of-its-kind tool for helping you automatically index API endpoints across all your repositories. LiveAPI helps you discover, understand, and use APIs in large tech infrastructures with ease. Every time you type a web address or service name, DNS (Domain Name System) translates that human-readable name to an IP address. This involves several steps: your computer uses a stub resolver (part of the OS) to send a query to a recursive DNS resolver (often your ISP’s or a public DNS server like Google’s 8.8.8.8). The recursive resolver either returns a cached result or performs the full lookup: it asks a root nameserver, then a TLD (Top-Level Domain) nameserver (for “.com”, “.org”, etc.), and finally the authoritative nameserver for the domain. Once the IP is found, the recursive resolver sends it back to your computer, and may cache it for future use. The client’s stub resolver is usually built into the operating system. Key points of this process: Stub Resolver (Client Side): Software in the OS or application that issues DNS queries. It simply forwards queries to a pre-configured DNS server (the recursive resolver). Recursive Resolver: The first DNS server contacted. It performs the full lookup (root → TLD → authoritative) and returns the answer to the stub resolver. Root and TLD Servers: Known globally (13 root servers) and direct the recursive resolver to the correct DNS zones. Caching: Resolvers cache DNS answers. If a cached answer exists, they skip querying authoritative servers again. In practice, most users rely on their ISP’s recursive resolver or a public DNS service (Cloudflare, Google, OpenDNS, etc.). The stub resolver in your OS doesn’t do recursion itself—it just calls these DNS servers. System (Internal) DNS Configuration At the system level, DNS settings control which resolver(s) the stub uses. On Unix/Linux, the key file is /etc/resolv.conf, which lists nameservers and search domains. For example: nameserver 8.8.8.8 search example.com This tells the stub resolver to send queries to 8.8.8.8, and that if you query a short hostname (like host1 with no dots), it should try appending the example.com search domain (e.g. host1.example.com). The “search” option can contain up to 6 domains. By default the search list is the local domain; with search example.com, a query for host1 becomes host1.example.com first, before trying the original name. This is useful on a LAN where local services are addressed by short names. On modern Linux systems using systemd-resolved, DNS may be configured differently. Typically, /etc/resolv.conf is a symlink to a dynamic stub file (often /run/systemd/resolve/stub-resolv.conf) managed by systemd. This stub file usually contains something like: nameserver 127.0.0.53 options edns0 Here 127.0.0.53 is systemd-resolved’s local stub listener. All application lookups (and browsers) will use this IP. Behind the scenes, systemd-resolved then forwards queries to the actual DNS servers learned via DHCP or statically configured. For example, it might show your DHCP-supplied DNS or any manual DNS you set in /etc/systemd/resolved.conf. If needed, you can disable the stub listener by setting DNSStubListener=no in /etc/systemd/resolved.conf, allowing you to edit /etc/resolv.conf directly. Systemd-resolved also manages “search” and “routing” domains via its Domains= setting. In /etc/systemd/resolved.conf or a drop-in file, you might see: [Resolve] DNS=1.1.1.1 FallbackDNS=1.0.0.1 Domains=example.com This would use Cloudflare’s DNS servers and treat example.com as a search suffix for single-label names. Importantly, if a domain in Domains= is prefixed with ~, it becomes a route-only domain. For example, Domains=~consul (with a ~) tells systemd that queries for the “consul” domain should be routed to the DNS server(s) configured on that interface. In effect, ~consul means “only use the DNS servers specified here for any .consul queries”. (Without the ~, a domain is just used for searching short names.) Configs for Consul and DNS In setups where HashiCorp Consul runs locally (typically on each node), Consul provides a DNS interface for service discovery on the loopback address. By default, a Consul agent’s DNS is at 127.0.0.1:8600 and uses the top-level domain consul. To make .consul queries resolve via Consul, one common approach is to configure systemd-resolved to forward those queries to Consul. For example, a drop-in file /etc/systemd/resolved.conf.d/consul.conf might contain: [Resolve] DNS=127.0.0.1:8600 DNSSEC=false Domains=~consul This tells systemd-resolved: “For the domain consul (because of the ~), use the DNS server at 127.0.0.1 port 8600.” In other words, queries ending in .consul go to the local Consul agent’s DNS server. If you cannot specify the port (older systemd), you could instead c

Jun 11, 2025 - 07:40

How DNS Works (Recursive Resolution and Stub Resolvers)

Hi there! I'm Maneshwar. Right now, I’m building LiveAPI, a first-of-its-kind tool for helping you automatically index API endpoints across all your repositories. LiveAPI helps you discover, understand, and use APIs in large tech infrastructures with ease.

Every time you type a web address or service name, DNS (Domain Name System) translates that human-readable name to an IP address.

This involves several steps: your computer uses a stub resolver (part of the OS) to send a query to a recursive DNS resolver (often your ISP’s or a public DNS server like Google’s 8.8.8.8).

The recursive resolver either returns a cached result or performs the full lookup: it asks a root nameserver, then a TLD (Top-Level Domain) nameserver (for “.com”, “.org”, etc.), and finally the authoritative nameserver for the domain.

Once the IP is found, the recursive resolver sends it back to your computer, and may cache it for future use.

The client’s stub resolver is usually built into the operating system.

Key points of this process:

Stub Resolver (Client Side): Software in the OS or application that issues DNS queries. It simply forwards queries to a pre-configured DNS server (the recursive resolver).
Recursive Resolver: The first DNS server contacted. It performs the full lookup (root → TLD → authoritative) and returns the answer to the stub resolver.
Root and TLD Servers: Known globally (13 root servers) and direct the recursive resolver to the correct DNS zones.
Caching: Resolvers cache DNS answers. If a cached answer exists, they skip querying authoritative servers again.

In practice, most users rely on their ISP’s recursive resolver or a public DNS service (Cloudflare, Google, OpenDNS, etc.).

The stub resolver in your OS doesn’t do recursion itself—it just calls these DNS servers.

System (Internal) DNS Configuration

At the system level, DNS settings control which resolver(s) the stub uses.

On Unix/Linux, the key file is /etc/resolv.conf, which lists nameservers and search domains. For example:

nameserver 8.8.8.8
search example.com

This tells the stub resolver to send queries to 8.8.8.8, and that if you query a short hostname (like host1 with no dots), it should try appending the example.com search domain (e.g. host1.example.com).

The “search” option can contain up to 6 domains.

By default the search list is the local domain; with search example.com, a query for host1 becomes host1.example.com first, before trying the original name.

This is useful on a LAN where local services are addressed by short names.

On modern Linux systems using systemd-resolved, DNS may be configured differently.

Typically, /etc/resolv.conf is a symlink to a dynamic stub file (often /run/systemd/resolve/stub-resolv.conf) managed by systemd.

This stub file usually contains something like:

nameserver 127.0.0.53
options edns0

Here 127.0.0.53 is systemd-resolved’s local stub listener.

All application lookups (and browsers) will use this IP.

Behind the scenes, systemd-resolved then forwards queries to the actual DNS servers learned via DHCP or statically configured.

For example, it might show your DHCP-supplied DNS or any manual DNS you set in /etc/systemd/resolved.conf.

If needed, you can disable the stub listener by setting DNSStubListener=no in /etc/systemd/resolved.conf, allowing you to edit /etc/resolv.conf directly.

Systemd-resolved also manages “search” and “routing” domains via its Domains= setting.

In /etc/systemd/resolved.conf or a drop-in file, you might see:

[Resolve]
DNS=1.1.1.1
FallbackDNS=1.0.0.1
Domains=example.com

This would use Cloudflare’s DNS servers and treat example.com as a search suffix for single-label names.

Importantly, if a domain in Domains= is prefixed with ~, it becomes a route-only domain.

For example, Domains=~consul (with a ~) tells systemd that queries for the “consul” domain should be routed to the DNS server(s) configured on that interface.

In effect, ~consul means “only use the DNS servers specified here for any .consul queries”. (Without the ~, a domain is just used for searching short names.)

Configs for Consul and DNS

In setups where HashiCorp Consul runs locally (typically on each node), Consul provides a DNS interface for service discovery on the loopback address.

By default, a Consul agent’s DNS is at 127.0.0.1:8600 and uses the top-level domain consul.

To make .consul queries resolve via Consul, one common approach is to configure systemd-resolved to forward those queries to Consul.

For example, a drop-in file /etc/systemd/resolved.conf.d/consul.conf might contain:

[Resolve]
DNS=127.0.0.1:8600
DNSSEC=false
Domains=~consul

This tells systemd-resolved: “For the domain consul (because of the ~), use the DNS server at 127.0.0.1 port 8600.”

In other words, queries ending in .consul go to the local Consul agent’s DNS server.

If you cannot specify the port (older systemd), you could instead configure Consul to listen on port 53 or use iptables to forward 53→8600, but newer systemd versions allow the :8600 suffix directly.

Meanwhile, /etc/resolv.conf on such a system might still use the systemd stub (127.0.0.53) by default.

The important part is that systemd-resolved now knows where to send .consul queries.

For example, after restarting systemd-resolved, running resolvectl domain should show Global: ~consul to confirm it’s handling the consul domain.

A DNS query like resolvectl query consul.service.consul would then return the Consul server address (often 127.0.0.1 if testing).

If instead you were not using systemd-resolved or wanted to disable it, you could directly set /etc/resolv.conf to use Consul. For instance:

sudo rm /etc/resolv.conf
echo "nameserver 127.0.0.1" | sudo tee /etc/resolv.conf

Then all DNS queries (including .consul) go to localhost.

If Consul’s agent is running and listening on UDP port 8600, you would need to ensure that port 53 is forwarded to 8600 (or run Consul on port 53 with elevated privileges) so that queries on 127.0.0.1:53 reach Consul. (For example, using iptables redirection.)

This bypasses systemd-resolved entirely, sending everything to Consul first, with fallback DNS as configured.

What Is Consul and Its DNS Interface

Consul is a service networking solution from HashiCorp.

In simple terms, Consul maintains a registry of services running in your infrastructure (across VMs, containers, Kubernetes, etc.) along with health checks.

It lets services register themselves (e.g. “Service name: web, port: 80”) and other services discover them. Consul is platform-agnostic and works across clouds or on-prem systems.

Besides its DNS interface, Consul also provides an HTTP API and, in more advanced setups, a full service mesh with mutual TLS (Consul Connect) and traffic controls.

For DNS-based discovery, Consul’s DNS interface is key. Once a service (say “web”) registers in Consul, you can resolve it via DNS by querying web.service.consul.

By default, Consul’s DNS listens on 127.0.0.1:8600 and uses the domain .consul.

You can query it with tools like dig or any standard DNS library.

Consul will respond with records for healthy instances of that service.

For example, assume a service “web” has three running instances with IPs 10.0.0.1, .2, .3. A DNS query might look like:

$ dig web.service.consul @127.0.0.1 -p 8600

;; QUESTION SECTION:
;web.service.consul.        IN  A

;; ANSWER SECTION:
web.service.consul.  0   IN  A   10.0.0.3
web.service.consul.  0   IN  A   10.0.0.2
web.service.consul.  0   IN  A   10.0.0.1

This example (adapted from HashiCorp) shows the A records for web.service.consul – one for each healthy instance.

In the Additional/Authority sections (not shown), Consul may include metadata or TXT records (e.g. source of the service).

The key point is that you get one IP per service instance, letting clients load-balance simply by trying multiple addresses.

Consul also supports SRV records, which include the port number of the service. If “web” had a non-default port, you could do:

$ dig @127.0.0.1 -p 8600 web.service.consul SRV

This returns records like . For example, a Redis service might yield:

web.service.consul. 0 IN SRV 1 1 8300 foobar.node.dc1.consul.
foobar.node.dc1.consul. 0 IN A 10.1.10.12

Here SRV says “the service is on port 8300 at host foobar.node.dc1.consul.”

The A record (in Additional section) then gives foobar.node.dc1.consul → IP 10.1.10.12. (DNS tools typically append underscores for SRV queries, e.g. _web._tcp.service.consul, but Consul also supports a simplified form as above.)

In summary, Consul’s DNS interface lets any standard application do service discovery without modifying the app code – it just performs DNS lookups.

Use Cases: Consul in Microservices

Consul shines in dynamic, microservice-based environments where services come and go. For example:

Dynamic Service Discovery: When new instances of a service spin up or down, they register/deregister in Consul. DNS queries automatically see the current healthy set. No need to update config files or load balancers manually.
Cross-Data Center Discovery: Consul can replicate its catalog across datacenters. An application can discover services in other sites via DNS (e.g. service.service.dc2.consul).
Health Checks: Only healthy service instances are returned in DNS. If a service instance fails its health check, Consul drops it from DNS responses, enabling simple failover with client retries.
Multi-Cloud/Hybrid: Consul is cloud-agnostic, so services across AWS, Azure, on-prem, etc. can register and discover each other under a unified naming scheme.
Integration with Config and Mesh: Beyond DNS, Consul also offers a key-value store for config and a built-in service mesh (Connect) for mTLS. But even without that, its DNS interface alone gives a lot of flexibility.

For example, a web server in one service simply needs to connect to api.service.consul:8080 to talk to the “api” service, regardless of how many instances or where they are.

In Kubernetes, Consul can even sync Kubernetes Service objects into its registry, allowing hybrid discovery between K8s and VMs.

In short, Consul replaces hardcoded IP addresses with logical names, automates failover, and makes large service networks more manageable.

It is especially popular alongside orchestration tools (Kubernetes, Nomad, Docker Swarm) and in environments where services are frequently scaled or redeployed.

Example Consul DNS Queries

Here are some concrete examples of using Consul’s DNS on the command line (assuming Consul is running locally):

A record lookup: dig web.service.consul @127.0.0.1 -p 8600 +short might output a list of IPs for the “web” service.
SRV record lookup: dig _web._tcp.service.consul SRV @127.0.0.1 -p 8600 +short returns lines like 1 1 8300 foobar.node.dc1.consul. where you can parse the port and host.
Node address: You can resolve a node’s IP by querying e.g. node1.node.consul.

These follow the DNS naming rules documented by Consul. In all cases, prepend @127.0.0.1 -p 8600 to tell dig to query the local Consul agent. The output lines (A or SRV records) show exactly what clients will see.

LiveAPI helps you get all your backend APIs documented in a few minutes

With LiveAPI, you can quickly generate interactive API documentation that allows users to search and execute APIs directly from the browser.

If you’re tired of manually creating docs for your APIs, this tool might just make your life easier.