How to Load Balance Traffic Across Multiple VXCs

For those wanting to maximise the use of multiple network connections for redundancy and optimised performance.

Load balancing traffic across your network connections allows you to maximise the use of multiple network paths when routing to the same destination networks. This strategy allows for increased throughput and redundancy. We’ll talk you through the process of setting up load balancing using a Megaport-connected network architecture.

When there are multiple paths available to reach the same network, BGP uses a best path selection process to choose the optimum path based on a range of criteria, mark the chosen path as ‘active’, and install the next hop in the forwarding table. This results in traffic to a given destination flowing only over one connection despite having ‘active/active’ BGP sessions. The desired behavior of spreading traffic across both BGP sessions is called Equal Cost Multi-Path (ECMP).

Take the scenario below into account: we have a router connected to Megaport in the Tierpoint Chicago data centre in Franklin Park, Illinois. We have two VXCs connected to AWS at diverse edge nodes for redundancy. Both VXCs are connected to the same Direct Connect Gateway (DGW) to access our VPC in US-East-2.

In this scenario, we’re going to step through configuring this on a Juniper MX router running Junos 18.3R1.9. Similar capabilities are also available from Cisco, Arista, Palo Alto, and others. Each networking vendor has slight nuances in how they handle best path selection, as well as configure multipath to install multiple entries in the forwarding table for a given network. Before starting the process yourself, make sure you reference the vendor documentation for your particular router.

When we look at just the BGP relationship between the various components, we can simplify the topology.

We have RTR1 with two VXCs. The first VXC has an IP address of 10.100.25.1/30 with its BGP peer, AWS, as 10.100.25.2. The second VXC has an IP address of 10.100.24.1/30 and its BGP peer as 10.100.24.2. The VPC is advertising the same prefix of 192.168.100.0/22 over both BGP sessions.

When we look at the route table of the RTR1, we’re learning the route for 192.168.100.0/22 from two different BGP peers, 10.100.25.2 and 10.100.24.2.

However, the BGP best path selection process chooses only one of these as the best path, marking the route ‘active’ and putting it in the forwarding table. If we take another look at the output of show route 192.168.100.0, we see that the route being learned from 10.100.25.2 is marked as ‘active’.

We can then verify this by looking in the forwarding table of the router to see exactly where packets destined for 192.168.100.0/22 are being sent.

Here we verify that the router is sending all packets destined for 192.168.100.0/22 across VXC1 only without load balancing across both VXCs. We’re going to use both VXCs by leveraging BGP multipath. If you aren’t familiar with BGP multipath, we encourage you to read Juniper’s docs for a more in-depth understanding prior to implementation: Understanding BGP Multipath

The first step is to configure a load balancing policy which will then be applied to the router. We’re going to call this policy ecmp; you can name it anything you like.

With Juniper, the syntax can be a little counterintuitive as the configuration is titled ‘load-balance per-packet’ however, the functionality is similar to other vendors’ per-flow load balancing based on source and destination IP address. We have configured the policy to load balance across both paths for any networks being received. Refer to Juniper’s documentation for applying a policy that targets more specific networks.

The next step will be to apply the policy to the router. This is done under the ‘routing options’ stanza.

Now that we have the load balancing policy applied, we need to enable multipath for BGP. There are a few ways in which this can be done, the first is to apply the configuration at the protocol hierarchy which would be applicable to all BGP peers on the router. The second is to specifically target a BGP group or neighbor. We’re going to configure multipath on our BGP group connecting us to AWS at AS64544.

First, we look at the existing BGP configuration:

Now, we’ll configure multipath under the BGP group to-AS64544

Now that we have added the configuration, we see the keyword ‘multipath’
.

Once we have performed a commit on the configuration, we can check the forwarding table again. 

The router is now forwarding packets destined to the VPC in AWS across both VXCs. We’ve successfully enabled BGP multipath. Microsoft Azure and Google Cloud also both support ECMP by default. In addition, if you want the router to take TCP ports in to account for a more granular ECMP algorithm, you can do that too. Please reference the section on ‘hash-key’ on Juniper’s Per Packet Load Balancing configuration guide.

Cisco has a great explanation of the BGP best path selection process available:  [https://www.cisco.com/c/en/us/support/docs/ip/border-gateway-protocol-bgp/13753-25.html

](https://www.cisco.com/c/en/us/support/docs/ip/border-gateway-protocol-bgp/13753-25.html)If you need to understand how to implement BGP multipath on a Cisco router, this is done via the maximum-paths command. You can search maximum-paths in their configuration guide or get a more in-depth look here.

If you’d like to know more about load balancing on a Megaport-connected network architecture, feel free to reach out to our team.