OpenSSH has this very nice setting, VerifyHostKeyDNS, which when
enabled, will pull SSH host keys from DNS, and you no longer need to
either trust on first use, or copy host keys around out of band.
Naturally, trusting unsecured DNS is a bit scary, so this requires the
record to be signed using DNSSEC. This has worked for a long time,
but then broke, seemingly out of the blue. Running ssh -vvv gave
output similar to
debug1: found 4 insecure fingerprints in DNS
debug3: verify_host_key_dns: checking SSHFP type 1 fptype 2
debug3: verify_host_key_dns: checking SSHFP type 4 fptype 2
debug1: verify_host_key_dns: matched SSHFP type 4 fptype 2
debug3: verify_host_key_dns: checking SSHFP type 4 fptype 1
debug1: verify_host_key_dns: matched SSHFP type 4 fptype 1
debug3: verify_host_key_dns: checking SSHFP type 1 fptype 1
debug1: matching host key fingerprint found in DNS
even though the zone was signed, the resolver was checking the
signature and I even checked that the DNS response had the AD bit
set.
The fix was to add options trust-ad to /etc/resolv.conf. Without
this, glibc will discard the AD bit from any upstream DNS
servers. Note that you should only add this if you actually have a
trusted DNS resolver. I run unbound on localhost, so if somebody can
do a man-in-the-middle attack on that traffic, I have other problems.