TL:DR Setting up secure and resilient Internet access in a country with sub-par infrastructure

I moved to the UK, a country that was a leader in Europe for PC adoption and early telecoms deregulation, but has since become one of the worst for the quality of its broadband through misguided laisser-faire policies. The only fixed broadband option available in my apartment is BT OpenReach’s pathetic VDSL service¹ (resold by Vodafone), which advertises 72 Mbps but I am lucky to get 40 Mbps down and 10 Mbps up.

There are several problems with this state of affairs:

• The network is very unreliable. I’ve had outages lasting 8 hours. It is so bad I wrote my own tool to track ping times and downtime.
• The consumer ISPs in the UK are anything but network-neutral, due to government regulations mandating Orwellian nanny-filters on the connection². At one point, I was unable to reach the Stack Overflow for over 2 days. It turns out for some unfathomable reason Vodafone decided to use Stack Overflow as the test site when they developed the government-mandated nanny-filter, and somehow that was deployed to production as per this highly instructive email thread.
• The IP address is dynamic. While Vodafone does not change it too often and it can be worked around using Dynamic DNS, on cellular carriers the use of Carrier-Grade NAT (CGNAT) is rife, and it makes those connections highly unsuitable for:
  • self-hosting mail servers, calendar or other services
  • working from home where I need to have long-lived SSH connections doing critical work.

Recently I found out my mobile operator, Three, offers 5G fixed broadband service. I was skeptical, their 4G service in my NIMBY-infested area³ is abysmal, I hardly ever have any signal at all on Hampstead High Street, but it turns out their 5G service is excellent, offering 500 Mbps down and 30 Mbps up, with decent ping times, probably because they managed to buy a 100Mhz contiguous allocation of 5G spectrum. Unfortunately, the service is not officially offered in my post code, so I decided to roll my own using an unlimited SIM card and an unlocked Huawei CPE Pro 2 5G router.

I have been experimenting with VPNs of late, leading to my edgewalker self-hosted VPN server, and building a VLAN on my network that thinks it is in the US using a VPN provider that shall remain nameless because it still has not been blocked by Netflix. This allows my daughter to watch her favorite US shows that are not available in the UK because of the despicable geofencing of the content cartels, who want to gouge you depending on where you live (except in this case they are not even offering the gouging, just no content).

The natural next step is to make the entire network be connected to the Internet via a WireGuard VPN. Because WireGuard, was designed for mobile connections like IPsec/IKEv2/MOBIKE, it easily adapts to shifting IP addresses (as long as one end stays put). This means it can deal with CGNAT and also fail over from 5G to DSL and back without breaking a sweat or even dropping a session.

Unfortunately there are side-effects to using a self-service VPN hosted by a cloud provider:

• Netflix, Amazon and the BBC will refuse to serve video to you. I had to work around it by creating a special VLAN for VPN-averse devices (the LG Smart TV, AppleTV 4K and any other streamers in my household). This VLAN is bridged to the Huawei in a way that stops the offensive STP packets, so it is as if they were plugged in directly into the Huawei. This is not a solution for when we want to watch video from out iDevices, however.
• The VPN encapsulation reduces the maximum data size (MTU) from the standard 1500 bytes of Ethernet to 1380 or so. Some sites have broken Path MTU Discovery (I found out the hard way DuckDuckGo is one of them), which means by blocking ICMP packets the server does not realize their large packets are not getting through, keeps retrying in vain until the browser times out in disgust. Setting the OpenBSD PF scrub (no-df) option took care of that.
• Then there is the bizarre phenomenon by which Google thinks my IP is in the United Arab Emirates. I do not know how, IP2Location thinks it is in the Netherlands, and MaxMind that it is in the UK (as it is). I tried again with some other Vultr servers and kept being located to the UAE or Saudi Arabia. My best guess is that Google builds its own IP geolocation database using GPS data from Android phones, and that some brave souls in the UAE or Saudi Arabia used a VPN service running on Vultr servers, and that caused the Vultr IPs to be associated with the those countries. The only way I found to resolve that was to keep creating virtual servers and additional IPs until I found a pair that did not locate to the UAE or Saudi Arabia. Now Google thinks I am in the US rather than the UK, I can live with that.
• Some services like Wikipedia will also block the device from edits, as it triggers a false positive for an open proxy. I sent an email to them on Saturday night and they had fixed that by next morning, whereas Google makes strenuous efforts to ensure you cannot reach a human within their organization, ever, and there is seemingly no way to prevent the defective IP geolocalisation from screwing things up (they disabled the /ncr workaround they used to have a few years ago). Tells you everything you know about the importance of customer service for a monopoly.

This is what I implemented, by replacing the too-limiting Ubiquiti Security Gateway in my UniFi switched and wireless network with an OpenBSD router that establishes a WireGuard VPN to a modified edgewalker running in the cloud with Vultr.

The configuration is quite complex because I have the following VLANs:

• The default VLAN (which is actually not even a VLAN, as Ubiquiti gear is not really Enterprise-class and does not default to VLANs). Because of the VPN compatibility issues, I am now using source IP based policy routing on the default VLAN and only servers go through the VPN.
• VLAN 2 for my office work-from-home Mac, I just do not trust the various antivirus (and other software that are required for compliance) anywhere near my personal networks
• VLAN 4 for the VPN-averse devices as mentioned above
• VLAN 7 is directly on the ISP router
• VLAN 666 for Internet of Things devices (at least those that can be operated without connecting to the rest of the LAN unlike printers)
• VLAN 1776 for my geofencing-busting freedom VPN that thinks it is in the USA
• Not a VLAN, but the Ethernet connection between my OpenBSD box and the Huawei router runs on a dedicated interface because in a bizarre effort to be “helpful” it sends a stream of Spanning Tree Protocol (STP) packets that basically cause my Ubiquiti switched network to melt down. OpenBSD can block them, but seemingly UniFi does not give you that control (so much for security, then). VLAN 4 is bridged to this.

OpenBSD has a concept of routing domains that allows you to virtualize your network stack into multiple routing tables, the way you can with VRF on a Cisco. This has proved invaluable, as has managing the configuration files in git to ensure I can always back out failed changes, and using Emacs’s TRAMP mode to edit files remotely.

It is mostly running, I have yet to move the Vodafone VDSL PPPoE circuit over from the decommissioned USG to the OpenBSD router and set up an IGP or some other routing protocol to fail over the default route to the Internet underlying WireGuard if one of the two connections fails. I am sure I will discover oddities as I go.

5G is extremely sensitive to positioning. Moving the Huawei just 20cm along the window makes the difference between 300Mbps down/10Mbps up/20ms ping and 500/30/12ms.

Not everything is perfect, of course. Ping times have risen slightly, and are more variable, as can be expected of a wireless network with layers of VPN processing latency added.

All DNS has to go through my own DNS servers, which are routed through the VPN so my ISPs cannot sniff those requests, or tamper with them. I thus get the benefits of encrypted DNS without having to trust the likes of Cloudflare or worse, Google. I do not (yet) block DoH or DoT like this gentleman does but I am planning to. My endgame is to add filtering similar to what the PiHole does, and interface the DNS server with the firewall to only allow IoT devices to connect to IP addresses that are the result of legitimate DNS lookups. I am also planning on recording DNS lookups using the dnstap interface for audit and parental control purposes.

Each VLAN has its own DHCP server instance. Known MAC addresses get a static IP in their DHCP lease, others get an address in 10.0.6.0/24 and all of their network traffic is recorded for forensics purposes using pflog. The same also applies to IoT devices.

Ubiquiti devices are prohibited by the OpenBSD firewall from accessing the Internet by the firewall, because of their history of security breaches, and to prevent any interference from the cloud.

Update (2022-06-10):

Real fiber finally reached my apartment complex in October 2021. It’s operated by a small Manchester ISP called 4th Utility. They top at 400Mbps but it is symmetrical. Annoyingly, their routers are locked down and you need to ask customer support to make any changes. I was the first one connected, but sadly for them the median age here is probably 70 and they are having a hard time signing up new subscribers.