…and I was like a kid on christmas eve! Just couldn’t want to get my hands on it to play.
BGP is a much used routing protocol on internet. A routing protocol is basically when network components starts talking to each others, announcing «hey, I know how to reach 192.168.250.16! And the other router will say «cool, I’ll remember that for future use!»
There’s basically two uses-cases for BGP:
- iBGP handles routing announcements within your network/organization
- eBGP handles routing to external partners.
iBGP is a common method to handle routing announcements so that nodes in a Kubernetes cluster know how to talk to the other nodes. The nodes will just continuously update each others about what ip addresses and network segments they have. Some of these network blocks may be allocated on the fly – for example the CNI I have chosen, Calico, will allocate smaller ranges of ip addresses from the pools given. In a multi-node setup, the Calico node pod on each node is responsible for announcing these to its peers.
I, however, am a mere hobbyist with only one node, so this use case is of no importance to me.
However, there is another announcement my Kubernetes cluster have to do: The load balancer IP addresses need to be announced on the local network. So far, I have used layer two announcements (ARP in the IPv4 world), but it’s possible to instead do it with BGP. In a multi-node setup, my load balancer IP addresses could actually be announced from either of the nodes, but with my one-node setup it will more or less just replace the layer two announcements. But there might still be some benefits to be had, as I will show later.
To start off with BGP; we need to pick an autonomous system. With an iBGP setup, you use the same AS for every peer. Much like ip addresses, there is registration entities that gives out AS numbers to organizations who need it, and if you need to peer with other organizations, you’d have to get one. However, there are AS numbers reserved for private use, from which you can just pick one if you use it purely for your internal consumption. I have chosen 64512, the first in the range of private use AS numbers. The official list of AS numbers can be found here.
Unifi uses FRR for its routing setup. There’s no GUI for it beyond «upload your written BGP configuration here», but since I am a bit old-school, I find this gives me more control anyways. So, let’s start off the Unifi BGP configuration
router bgp 64512
bgp router-id 192.168.1.1
!
This defines the AS, and also the router ID. It’s a well established practise to use the IP address here. Choosing a router-id can be a science in itself, but for my small setup I’ve just chosen the IP address on the interface which should peer with Calico.
Next, you’ll need to define some neighbors:
neighbor metallb peer-group
neighbor metallb remote-as 64512
neighbor metallb activate
neighbor metallb soft-reconfiguration inbound
neighbor 192.168.1.153 peer-group metallb
neighbor 2001:db8:393:f101::1:0 peer-group metallb
These are the IPv4 and IPv6 address of my node that I want to use for BGP communication. As you can see, we’re using the same AS in Kubernetes as on the router, since we’re using iBGP.
You’ll also want to specify some parameters for the communication, both for IPv4 and IPv6
address-family ipv4 unicast
redistribute connected
neighbor metallb next-hop-self
neighbor metallb soft-reconfiguration inbound
neighbor metallb route-map ALLOW-ALL in
neighbor metallb route-map ALLOW-NONE out
exit-address-family
!
address-family ipv6 unicast
redistribute connected
neighbor metallb activate
neighbor metallb next-hop-self
neighbor metallb soft-reconfiguration inbound
neighbor metallb route-map ALLOW-ALL in
neighbor metallb route-map ALLOW-ALL out
exit-address-family
exit
These specify that I want to announce directly connected routes to the peer – it’s probably not necessary in thise case, since the default route of the node points to this router, but it also doesn’t really harm. next-hop-self specifies that the ip address I’m using for communication with the peer should be the target of the route, and soft-reconfiguration inbound is a parameter that’s just good to have, but it’s a bit vague. There’s also some lists that’d filter what routes you’ll announce and what you’ll accept. For now, I’ve just defined an ALLOW-ALL that allows all routes, which is fine in my small and controlled setup:
route-map ALLOW-ALL permit 10
exit
!
On the Kubernetes/Calico end of this, I first need to check and possibly change some default settings. Since I’ve got Calico installed with the Tigera Operator, a pretty common setup, we’ll need to check some Tigera settings.
$ kubernetes get installation default -o yaml
You’ll see a lot of info, but you should see something like:
spec:
calicoNetwork:
bgp: Enabled
hostPorts: Enabled
ipPools:
- blockSize: 26
cidr: 10.151.0.0/16
disableBGPExport: false
encapsulation: VXLANCrossSubnet
name: default-ipv4-ippool
natOutgoing: Enabled
nodeSelector: all()
- blockSize: 26
cidr: fd00:10::/48
disableBGPExport: false
encapsulation: VXLANCrossSubnet
name: default-ipv6-ippool
natOutgoing: Enabled
nodeSelector: all()
linuxDataplane: Iptables
linuxPolicySetupTimeoutSeconds: 0
multiInterfaceMode: None
nodeAddressAutodetectionV4:
interface: eno1
nodeAddressAutodetectionV6:
interface: eno1
windowsDataplane: Disabled
Hopefully, you’ll see BGP enabled, else you should change that.
The nodeAddressAutodetection is likely to be different. For me, it was easiest just to decide that whatever IP address I have on eno1 (primary network interface) is where I want to annonunce my routes to. This is basically the same kind of decision as next-hop-self was on the router.
Next, we’ll want to adjust our our BGPConfiguration:
$ kubectl get bgpconfigurations.projectcalico.org -o yamml
error: unable to match a printer suitable for the output format "yamml", allowed formats are: custom-columns,custom-columns-file,go-template,go-template-file,json,jsonpath,jsonpath-as-json,jsonpath-file,name,template,templatefile,wide,yaml
hassio% kubectl get bgpconfigurations.projectcalico.org -o yaml
apiVersion: v1
items:
- apiVersion: projectcalico.org/v3
kind: BGPConfiguration
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"projectcalico.org/v3","kind":"BGPConfiguration","metadata":{"annotations":{},"name":"default"},"spec":{"serviceLoadBalancerIPs":[{"cidr":"192.168.250.0/24"},{"cidr":"192.168.251.0/24"},{"cidr":"2001:db8:393:f1e1:2::/80"},{"cidr":"2001:db8:393:f1e1:3::/80"},{"cidr":"2001:db8:393:f1e2:2::/80"}]}}
creationTimestamp: "2025-03-20T18:27:28Z"
name: default
resourceVersion: "6521440"
uid: f3b9f934-c394-43d8-89d8-b003716b5596
spec:
ignoredInterfaces:
- kdmzlink
- kdmz01
- kdmz02
serviceLoadBalancerIPs:
- cidr: 192.168.250.0/24
- cidr: 192.168.251.0/24
- cidr: 2001:db8:393:f1e1:2::/80
- cidr: 2001:db8:393:f1e1:3::/80
- cidr: 2001:db8:393:f1e2:2::/80
Here, I have specified that I’m going to ignore some interfaces (those I used previously for layer two announcements). I also specify the ranges where my load balancers end up. These ranges need not match exactly, but you’d probably not want to have it overlap something else on your network.
Now, we’re ready to try to connect to the world. We’ll use a couple of bgpeers entries for that. They roughly correspond to the addressfamily entries we did on the router. I’m marked the key information in bold.
$ kubectl get bgppeers.projectcalico.org -o yaml
apiVersion: v1
items:
- apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"projectcalico.org/v3","kind":"BGPPeer","metadata":{"annotations":{},"name":"unifi"},"spec":{"asNumber":64512,"peerIP":"192.168.1.1"}}
creationTimestamp: "2025-03-20T18:19:47Z"
name: unifi
resourceVersion: "6521131"
uid: 61dbedd7-9061-4a2a-ba57-e17e77f2cfef
spec:
asNumber: 64512
peerIP: 192.168.1.1
- apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"projectcalico.org/v3","kind":"BGPPeer","metadata":{"annotations":{},"name":"unifi-ipv6"},"spec":{"asNumber":64512,"peerIP":"2001:db8:393:f101::1"}}
creationTimestamp: "2025-03-20T19:57:18Z"
name: unifi-ipv6
resourceVersion: "7188129"
uid: 9b1f0770-e308-4ec5-a968-9051c14d7387
spec:
asNumber: 64512
peerIP: 2001:db8:393:f101::1
kind: List
metadata:
resourceVersion: ""
Once this is done, if everything is correct, information should start to flow.
On the router, we have a configuration interface called vtysh. This is part of the FRR suite, so chance is you’ll also have it.
frr# sh bgp ipv4 summary
IPv4 Unicast Summary (VRF default):
BGP router identifier 192.168.1.5, local AS number 64512 vrf-id 0
BGP table version 68
RIB entries 70, using 13 KiB of memory
Peers 2, using 2892 KiB of memory
Peer groups 1, using 128 bytes of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
192.168.1.153 4 64512 2489 2181 0 0 0 1d12h11m 15 0 N/A
2001:db8:f101::1:0 4 64512 2462 2175 0 0 0 1d11h49m NoNeg NoNeg N/A
Total number of neighbors 2
frr#
(Note: this is slightly edited, I actually have more peers at the time I write this).
As you can see, there are some messages being passed back and forth, and Calico has announced 15 prefixes (routes)!
Let’s look at them:
frr# sh bgp ipv4 neighbors 192.168.1.153 received-routes
BGP table version is 68, local router ID is 192.168.1.5, vrf id 0
Default local pref 100, local AS 64512
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
*> 10.151.253.192/26
192.168.1.153 100 0 i
*> 10.152.253.192/26
192.168.1.153 100 0 i
*> 10.152.253.192/26
192.168.1.153 100 0 i
*> 192.168.250.0/24 192.168.1.153 100 0 i
*> 192.168.250.5/32 192.168.1.153 100 0 i
*> 192.168.250.6/32 192.168.1.153 100 0 i
*> 192.168.250.7/32 192.168.1.153 100 0 i
*> 192.168.251.0/24 192.168.1.153 100 0 i
*> 192.168.251.10/32
192.168.1.153 100 0 i
*> 192.168.251.11/32
192.168.1.153 100 0 i
*> 192.168.251.12/32
192.168.1.153 100 0 i
*> 192.168.251.13/32
192.168.1.153 100 0 i
*> 192.168.251.14/32
192.168.1.153 100 0 i
*> 192.168.251.15/32
192.168.1.153 100 0 i
*> 192.168.251.16/32
192.168.1.153 100 0 i
Total number of prefixes 15
frr#
As you can see, a lot of /32a are announced. These are single IPs for my load balancers, and they point to the IPv4 address of my node.
For ipv6:
frr# sh bgp ipv6 neighbors 2001:db8:393:f101::1:0 received-routes
BGP table version is 44, local router ID is 192.168.1.5, vrf id 0
Default local pref 100, local AS 64512
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
*> 2001:db8:393:f1e1:2::/80
2001:db8:393:f101::1:0
100 0 i
*> 2001:db8:393:f1e1:2::/128
2001:db8:393:f101::1:0
100 0 i
*> 2001:db8:393:f1e1:2::1/128
2001:db8:393:f101::1:0
100 0 i
*> 2001:db8:393:f1e1:2::2/128
2001:db8:393:f101::1:0
100 0 i
*> 2001:db8:393:f1e1:2::3/128
2001:db8:393:f101::1:0
100 0 i
*> 2001:db8:393:f1e1:2::4/128
2001:db8:393:f101::1:0
100 0 i
*> 2001:db8:393:f1e1:2::5/128
2001:db8:393:f101::1:0
100 0 i
*> 2001:db8:393:f1e1:2::6/128
2001:db8:393:f101::1:0
100 0 i
*> 2001:db8:393:f1e1:3::/80
2001:db8:393:f101::1:0
100 0 i
*> 2001:db8:393:f1e2:2::/80
2001:db8:393:f101::1:0
100 0 i
*> 2001:db8:393:f1e2:2::1/128
2001:db8:393:f101::1:0
100 0 i
*> 2001:db8:393:f1e2:2::2/128
2001:db8:393:f101::1:0
100 0 i
*> 2001:db8:393:f1e2:2::3/128
2001:db8:393:f101::1:0
100 0 i
*> 2001:db8:393:f1e2:2::4/128
2001:db8:393:f101::1:0
100 0 i
*> 2001:db8:393:f1e2:2::5/128
2001:db8:393:f101::1:0
100 0 i
*> 2001:db8:393:f1e2:2::6/128
2001:db8:393:f101::1:0
100 0 i
*> 2001:db8:393:f1e2:2::7/128
2001:db8:393:f101::1:0
100 0 i
*> fd00::b8f3:e78:1a73:9c40/122
2001:db8:393:f101::1:0
100 0 i
Again a lot of routes pointing to our nodes ipv6 address.
If it doesn’t work yet, you’ll need to go over your configuration again. I can’t speak for other kind of BGP routing software, but vtysh also have a ton of logging and debugging possibilties. In vtysh, try the following:
frr# terminal monitor
frr# debug bgp updates
frr# debug bgp neighbor-events 192.168.1.153
frr# show bgp peer-group metallb
frr# show ip bgp next-hop
If the answer to the last ones doesn’t show them as valid, there’s something wrong with the routing towards them. The possibility for this in an iBGP setup where we’re working with physical devices and network ports are probably not too great, but it’s still a good command to have in your toolbox. This is just a few hints to get you going, if you’re into this as a learning experience, you should of course look at all settings and debug-info you can find in vtysh! I did.
There’s also routes for the POD network allocations that Calico has, so it’s even possible to talk directly to the PODs, even if you didn’t previously know that range outside of Kubernetes.
Now, I’m actually ready to remove my previous layer two advertisement configurations that I created in this blog post. The observant leader might also note that I have opted for totally new networks to expose my load balancers on – metallb’s ipaddresspools will of course have to have ranges from those.
I can also delete my 802.1q interfaces on the node – the traffic will no longer come in there, they will be routed directly to the nodes ip address. In fact, I don’t really care that the ip addresses belong on any network defined in my router, but the ones exposed to the internet will of course have to belong to a network that is routed toward me. I have opted to still have an internal and external DNS created in my router, with these subnets defined. This is because when I create new networks in Unifi, the router will propose a suitable range (the next one) to assign to the network, and I just want to avoid it proposing these addresses somewhere else.
But even if these networks are assigned to a different network, my BGP announced routes would work. They are in no way tied to a specific VLAN id on layer two anymore, either. But DHCP might happily assign my load balancer IPs to other hosts, so it’s just best to just use separate networks – there’s less chance of problems down the road.
So, as a home user, is this really needed? No, layer two announcements would probably be sufficient. But nerds do nerd stuff, and BGP sure has some nerd value. Having dabbled in networking professionally ages ago, I admit my knowledge of some of this was familiar from before, so it was definitely nice with a refresh of routing protocols! You never know when you’ll need them.
My next post will be about eBGP. As a home user, you’ll again probably not have much use for it, but maybe like me you have (or can rent for the purpose) a virtual server in the cloud somewhere that you can try getting network-wise integrated more on your home network!
All real IP addresses in this blog post is replaced with IP addresses from RFC5737 and RFC3849