systemd-resolved ignores per-connection DNS split-horizon rules after NetworkManager upgrade on Ubuntu 22.04

Environment

Ubuntu 22.04 desktop with NetworkManager 1.40 (recently upgraded from 1.38), systemd-resolved acting as local stub resolver (127.0.0.53), multiple VPN and wired connections with per-connection 'search' and 'dns' settings intended to provide split-horizon for internal.example.corp.

Symptoms

Internal hostnames under internal.example.corp that previously resolved to internal IPs now resolve to public/global IPs or time out when the VPN is active. DNS resolution via systemd-resolved returns different answers depending on the system's active link order, not per-connection split-horizon rules.

Observations

resolvectl status shows multiple Link entries with DNS servers configured but 'Domains' field empty for the VPN link despite nmcli showing domain=internal.example.corp on that connection. /etc/resolv.conf still points to 127.0.0.53. dnsmasq was not in use. Reverting the NetworkManager package to the older version inside a test VM restored per-connection domain behavior. Enabling verbose NetworkManager logging shows a change in how NM writes DNS domains to systemd-resolved via the DBus API (the domain field is omitted).

Already attempted

Restarted NetworkManager and systemd-resolved services, ran 'resolvectl flush-caches', recreated the VPN connection profile with explicit 'search' and 'domain' options, added the internal domain to /etc/hosts for a single host (temporary fix), tested nslookup pointing directly to the VPN's DNS server which returned correct internal addresses, and tested downgrading NetworkManager on a nonproduction workstation which reproduced the previous, working behavior.

Question

Which reversible configuration changes or diagnostics will restore per-connection split-horizon resolution with systemd-resolved under the upgraded NetworkManager, and how to determine whether the NM upgrade changed DBus contract semantics, or if a local configuration workaround (e.g., per-link resolved configuration) is preferable until an official fix or patch is available?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 1w ago• Trouble ID

Most likely causes

NetworkManager upgrade changed DBus method or payload used to pass 'Domains' to systemd-resolved, leaving domain unset on the link. 95%

A regression in NetworkManager's parsing of VPN connection 'domain' or 'search' fields for certain VPN types. 35%

systemd-resolved updated expectations for per-link domain fields that are stricter and NetworkManager didn't adapt to. 15%

Summary

Evidence suggests the NetworkManager upgrade altered the way it communicates per-connection DNS domains to systemd-resolved (likely a DBus API change or a deliberate behavior change), resulting in systemd-resolved not associating a domain with the VPN link. Because direct queries to the VPN DNS server succeed, the DNS server itself is correct; the issue is the per-link domain assignment. Workarounds include configuring systemd-resolved with per-link settings using resolvectl or nm dispatcher scripts, or temporarily pinning NetworkManager to the previous version until an upstream fix is available.

Suggested Fixes

Diagnostic step 1

Confirm the root by running 'resolvectl status' before and after activating the VPN and compare the 'Domains' field for the VPN link. If domain is missing, this confirms per-link domain is not being set.

Diagnostic step 2

As a reversible workaround, add a NetworkManager dispatcher script (in /etc/NetworkManager/dispatcher.d/) that runs on connection-up to call 'resolvectl domain <ifindex> internal.example.corp' for the VPN link; test behavior and remove when no longer needed.

Diagnostic step 3

Open a bug report with NetworkManager including 'resolvectl status' output, NM connection profile, and verbose NM logs (sudo NM_CONTROLLED=1 dbus-send or journalctl -u NetworkManager) to help maintainers reproduce; if acceptable to your environment, temporarily pin NetworkManager to the previous package version while an upstream fix is pending.