The Curious Problem with my Pihole DNS Service
Okay, this will be a long one. It all started because I heard of a word called "DNS leaks".
I was happy with my installation of a Pihole, which allows one to create a DNS sinkhole. All DNS queries in your local network can be processed by the Pihole, which compares the queries to a ad blocklist. If a query matches, then it's simply dropped. So in place of using your ISP's DNS server, use your own DNS server, which in this case, is the Pihole. This is quite useful when you are using devices where it's hard to use a dedicated adblocker, such as Android devices or SmartTVs. Keep in mind, you still need to use an upstream DNS server in the Pihole, which in my case was Cloudflare.
The Rabbit Hole
Everything was working quite well, till I found about a site called dnsleaktest. The premise of the site is that, even if you set your own DNS server, it's possible, due to network configuration issues, you might be using an OS/ISP specified DNS server, that is, you have "leaked" your DNS queries to a 3rd-party unintentionally. It's mentioned in their site that it's normally an issue for Windows machine, so I thought "Yeah, no problem for Linux", but I still decided to run the leak test. It was shocking to see that the website reported that i'm using two DNS services; Google's and my ISPs. There was no Cloudflare in sight.
I tried to debug it using from the bottom up
- Is there a configuration issue on my Pihole?
- Is there a configuration issue on my router?
- Is there a configuration issue on my PC?
- Is it a single device issue or for all the devices?
I saw that all my devices are using the same set of DNS services. I thought "It's most probably an issue of my router".
Now, I was using an ASUS router, with its proprietary firmware. I heard it's a highly modified version of the open-source Tomato firmware, but I can't trust that it does not have it's own set of hardcoded DNS servers. In hindsight, I could have checked its internal routing table and seen that everything was fine, but I took no chances; I formatted the router with OpenWrt. Now, I should not have any DNS issue, right?
WRONG!
The problem still persisted.
At this point, I consulted with the experts in the Pihole subreddit and they said this is an unexpected issue, there was no problem on my end.
I didn't know what else can be changed.
However, OpenWrt has a big advantage over default router firmwares; you can install external plugins.
I decided to go down to packet level to investigate the issue using tcpdump.
I created two DNS capture instances; one for internal traffic and one for external traffic.
This was achieved by running the capture on different interfaces; in my case it was br-lan and eth0.
tcpdump -i <interface name> -w <capture file name> udp port 53 or tcp port 53 and dst not <pihole IP> and src not <pihole IP>
I saw that nearly all internal DNS traffic were directed to the Pihole, and all external DNS traffic were directed to Cloudflare.
The Deeper Problem
Ok, let's step back for a second. It was a bit shocking. All external traffic was going to Cloudflare but dnsleaktest is reporting that it's going to Google. How is it possible? Is the website make a mistake?
No, it wasn't. When I used a VPN, dnsleaktest told me I was using a DNS service which was outside my country.
So something was happening at the ISP level. dnsleaktest mentioned that it's possible that some ISPs can use a technique called "transparent DNS proxying", or simply DNS hijacking. It involves capturing all DNS queries in real time, stripping the IP headers, replacing them with their own DNS server IPs and forwarding them. While returning the response, the original IP fields are restored and the user is left none the wiser.
One solution for this to use a VPN, but it's just a bulky solution; throughput and latency can be affected by using a VPN. A better solution could be encrypting just the DNS queries, instead of the entire connection. That is exactly what DNS-over-HTTPS, or DoH, tries to provide. There are other alternatives, such as DNS-over-TLS (DoT) or DNS-over-QUIC (DoQ), however I decided to go with DoH. There are some legitimate concerns about DoH, but that's a separate topic.
After this realization, it simply became a matter of using the technology.
Now, some applications/devices support DoH, however, the goal is the provide DoH to all devices inside the local network in my house.
Fortunately, Cloudflare also provides the solution, called cloudeflared which is a simple daemon which connects to the Cloudflare DNS service over HTTPS.
The only change that I have to do is to set the upstream DNS provider in my Pihole to the cloudflared application.
All of it was done via Docker containers, which makes it very easy to deploy.
There is a simple one-click docker-compose file available here.
The End?
Fine, so all problems solved, right?
Well, almost.
Remember a few paragraphs ago I mentioned that NEARLY all traffic was routed to my Pihole. Turns out that few devices use hard-coded DNS entries to reach their destination. You can't change them without formatting the device.
So how do you solve this?
The solution is "transparent DNS proxying". Yes, I will use the weapon which my own ISP used against me. OpenWrt, being a versatile firmware, has this feature. You simply need to write a few firewall rules, telling the router to capture all DNS traffic which are not going to my Pihole and "repack" them.
iptables -t nat -A PREROUTING ! -s <pihole IP> -p tcp --dport 53 -j DNAT --to <pihole IP>:53
iptables -t nat -A PREROUTING ! -s <pihole IP> -p udp --dport 53 -j DNAT --to <pihole IP>:53
iptables -t nat -A POSTROUTING -j MASQUERADE
In OpenWrt terms, it's called masquerade.
Conclusion
This is was the final part of the puzzle, and with this all my DNS queries were properly served by my Pihole...for now. Just a simple word, which is "dns leak", sent me down a huge rabbit hole and taught me a lot.
You might be wondering, why go to such lengths? The answer is "Why not? I want to have complete control over how my network works". There are some philosophical aspects of the whole problem, which I intend to discuss in a separate post.