The Hidden Cost of Running Cloud-Hosted SD-WAN for IaaS

By Henry Wagner, Chief Marketing Officer

In an industry driven by SD-WAN, there are three common ways to connect your branch locations to the cloud. We break down the benefits and limitations of each.

Many enterprises are executing a strategy of cloud migration first, followed by application modernization – including SD-WAN. Moving towards a cloud-native architecture using containerization, microservices, or serverless architectures like SD-WAN can lower costs over time, increase scalability, reduce development cycles, and speed up time to market.

But the next challenge is making sure that branch sites can reach this now modernized, public cloud-based application. But what if there are still a number of legacy applications hosted in the private cloud or services hosted in second or third public cloud providers?

With the popularity of SD-WAN and an increasing number of enterprise WAN migrations from MPLS to broadband-based underlays, business-critical traffic now travels across best effort, public internet-based connections. It’s clear that this new shift to the “cloud-powered branch” requires a redesign from an enterprise architecture perspective.

Weigh the similarities, differences, and benefits of SD-WAN and MPLS with our blog post.

One option is to bring cloud-bound traffic from the branch sites back to the data center then outwards to the cloud, ideally over private connectivity. This can work well, but depending on the geographic location of the enterprise’s data center, this can also introduce hairpinning, which significantly increases latency and reduces application performance. The second option is to allow branch sites to communicate directly to the cloud.

There are various methods to achieve this, specifically:

• Cloud Networking service provider VPN gateways (AWS VPG, Azure VNG, Google Cloud VPN)
• Cloud Networking-based SD-WAN (including AWS Transit VPC, Azure Virtual WAN, Google Cloud NCC)
• NaaS (Megaport Virtual Edge)

Standard branch to cloud with VPN tunnels

Let’s consider a medium-sized retail enterprise with around 40 branch sites spread across the United States that require access to a point of sale/inventory management platform hosted in the cloud. One quick and reliable solution would be to utilize the standard Site-to-Site VPN gateway offered by the cloud providers.

In an AWS environment, this would mean building IPsec tunnels across the public internet to a virtual private gateway connected to a single virtual private cloud (VPC). The drawback to this design is the one-to-one rule which also applies to Azure, as their equivalent (VPN Gateway) can also only be attached to a single VNet. Since a virtual network is limited to only one gateway, this means that if the enterprise builds a second virtual network – this would require a second VGW and new IPsec tunnels to all 40 branch sites. The cloud providers have a maximum number of branch site tunnels that attach to a single VPN gateway. For Azure, this number is 30, and for AWS, it is 10 (but an increase can be requested).

There is also a ceiling on the network speeds that can be achieved. A common misconception is that IPsec tunnels to the cloud are limited to 1.25 Gbps — in fact, that is the total throughput of the gateway. So if you configure 40 branch sites, they will each share that 1.25 Gbps resulting in about 30 Mbps of throughput per site. It’s also worth considering that the branch site’s 30 Mbps will be travelling 100% across the public internet and likely face unpredictable levels of jitter, packet loss, and latency. Finally, since the cloud-to-branch site traffic is travelling via the public internet, it will be subject to standard cloud provider egress or data transfer out (DTO) fees of approximately 10 cents for every gigabit that leaves your virtual network.

Cloud-hosted SD-WAN edge

To overcome the aforementioned challenges of a standard VPN tunnel-based design, the cloud providers and traditional network hardware vendors (such as Megaport partners Cisco and Fortinet) have built solutions that can integrate into an enterprise SD-WAN.

Fortinet Secure SD-WAN is now enabled on Megaport Virtual Edge (MVE). Read the blog here.

SD-WAN is a virtualized WAN architecture that utilizes a mix of underlying transport methods such as MPLS, broadband, and LTE to connect branch sites to applications hosted in the private and public cloud. One of the key advantages of SD-WAN is the centralization of control and routing functions which direct traffic paths across the overlay WAN.

Although there are slight variations, fundamentally this architecture extends the SD-WAN fabric into the public cloud using Infrastructure as a Service (IaaS), then loading SD-WAN vendor software from the cloud provider’s marketplace (BYOL, or bring your own license). This allows for very fast deployment, high levels of automation, and simplified operations using the SD-WAN vendor’s management console.

Firstly, let’s look at an IaaS-based solution in AWS and its key components. The first is the Transit VPC, which acts as the central hub for all traffic to and from the branch sites, and as the conduit to the application virtual private clouds (VPCs).

Two virtual machines are spun up in the Transit VPC and an SD-WAN vendor’s image is then loaded to create two virtual routers which can be managed by Cisco vManage, FortiManager, or an equivalent. The hourly charge for the underlying AWS VMs will depend on the specifications selected during the build.

In the diagram above, you can see virtual routers have northbound IPsec tunnel connections to the production, development, and test VPCs via their respective virtual private gateways (VPGs). Both virtual routers also have southbound IPsec internet-based connections to each branch site in the SD-WAN. For redundancy, it is recommended that each branch site builds a tunnel to each virtual router. Using a BGP-style routing protocol, the private subnets in the application’s VPC will be advertised to the virtual routers, which in turn will re-advertise those subnets to the 40 branch sites (for clarity, the diagram only shows five branch sites).

Need to spin up branch-to-Azure connectivity on Cisco SD-WAN? Watch our step-by-step video here.

As the number of SD-WAN branch sites grows, additional virtual machines (routers) may be required as there are CPU and bandwidth limitations due to the amount of IPsec tunnel encryption/decryption needed.

This SD-WAN solution still utilizes the public internet so the standard egress fees will apply but one additional hidden cost to be aware of is the potential for double egress fees. Since the northbound connections are using internet IPsec tunnels, there will be a fee charged for each gigabyte that leaves the application’s VPCs toward the Transit VPC. If the traffic is destined for a branch site, then there will be an additional per gigabyte charge as it leaves the Transit VPC southward. Effectively the customer is getting charged double egress fees for each gigabyte reaching a branch site.

Depending on workloads and traffic flow, the IaaS-based SD-WAN solution can work extremely well for many enterprises. Let’s consider what happens when a second cloud provider is introduced to avoid vendor lock-in, plan for business continuity, or use a new internal application that works best inside, for example, Microsoft Azure. The architecture gets significantly more complex, as now all 40 branch sites require access to both AWS VPCs and the new Azure VNets.

In an Azure environment, a Virtual WAN would be deployed and a virtual hub VNet would be created (similar, in many ways, to an AWS Transit VPC). Again, dual VMs would be spun up in this hub and SD-WAN software loaded on top to create a Network Virtual Appliance (NVA). An existing branch site will then connect via IPsec tunnels to both NVAs in the Azure virtual hub (in addition to the existing two tunnels to AWS). As you can see from the diagram, this now means there are hundreds of IPsec tunnels.

So far we have only considered multicloud in the branch-to-cloud flow, but there could also be a requirement for cloud-to-cloud connectivity; an example might be cross-cloud data replication. The options here are either to hairpin the traffic back to the on-premises data center (which introduces higher latency) or build new IPsec tunnels between the Azure virtual hub and the AWS Transit VPC which, of course, will incur internet egress fees and have tunnel speed limitations.

Megaport Virtual Edge (MVE)

Megaport Virtual Edge (MVE) combined with private layer 2 cloud connectivity, such as AWS Direct Connect, Azure ExpressRoute, or Google Cloud Interconnect, addresses many of the pain points identified in the first two solutions above.

MVE enhances your existing enterprise SD-WAN platform by giving you the ability to strategically build optimal pathways to critical applications wherever they reside. Essentially, MVE enables businesses to build their own regional PoPs within minutes and extend their WAN reach by leveraging Megaport’s global Software Defined Network (SDN). There are currently 22 MVE metro regions globally, each with at least two data centers and internet transit providers supporting MVE availability zones to provide end-to-end WAN resiliency.

Let’s look under the hood of MVE to understand its core elements and capabilities.

MVE is a virtualized, on-demand SD-WAN edge device that comes in three sizes based on combinations of vCPU, IP transit, and the number of branch sites that will be attached. The SD-WAN vendor software can be chosen by the customer; MVE currently supports Cisco and Fortinet with more planned through the rest of 2021. From this point onwards, the MVE is effectively managed like a regular SD-WAN appliance within the enterprise WAN fabric and can be configured from Cisco vManage, FortiManager, or an equivalent.

The branch traffic arrives at the MVE via first-mile local internet, typically less than ~10ms depending on proximity. From there, the MVE has access to the global Megaport fabric which includes over 235 public cloud on-ramps enabling private layer 2 connections (such as AWS Direct Connect, Azure ExpressRoute, Google Cloud Interconnect, and others).

How it works

SD-WAN branch sites can build IPsec attachments to the MVE via the included redundant, Tier 1 IP transit with Megaport-provided public IP addresses. After the branch traffic arrives at the MVE, enterprises can aggregate their branch site traffic at a centralized MVE location, transition it out of the public internet IPsec tunnels quickly, and reroute that traffic to the major cloud providers over the Megaport backbone, which provides a number of benefits.

Benefits

1. Application performance: Branches that previously accessed cloud-based applications via the public internet will see improved performance. Now only the “first mile” will traverse the internet while the majority of the path will be via the Megaport backbone, reducing jitter, latency, and packet loss.
2. Access to the Megaport fabric: MVE provides access to Megaport’s ecosystem of more than 360 service providers and over 700 enabled data centers globally, which includes 235+ cloud on-ramps, the most of any neutral Network as a Service (NaaS) provider.
3. Opex reduction: By using the hyperscaler’s private connection instead of the internet IPsec tunnels, egress fees can drop by 70 to 80%. In North America, this means a reduction from an average of 10 cents per gigabyte to around two cents per gigabyte. Aside from egress fees, if we compare MVE to an IaaS SD-WAN transit solution using our baseline of 40 branch sites, with dual AWS VMs (C5 large) pulling down 10TB a month from the cloud to the branches, the MVE solution will cost 15% less on a monthly basis.
4. Network simplification: In the IaaS solution, again using a 40-site example, this would require at least four IPsec tunnels per site which means a minimum of 160 total tunnels. These will each require /30 IP addresses, link state notifications, and will eat up valuable CPU resources at both the branch and in the cloud-based virtual routers. By moving to MVE, an enterprise can massively reduce the complexity of their network architecture.
5. SD-WAN efficiency: Integrating your SD-WAN platform with MVE allows you to build strategic middle-mile transports throughout your WAN using Megaport’s Virtual Cross Connects (VXCs). SD-WAN traffic-shaping services will immediately leverage these more efficient pathways for end-to-end application flows.
6. On-demand: There is no equipment to purchase or install in a data center, or cross connects to run. MVE (like the IaaS solutions) can be provisioned in real time and be ready for use within minutes.
7. Cloud to cloud: Lastly, since the MVE has private layer 2 connectivity to both AWS and Azure, any cloud-to-cloud traffic can also utilize these connections and avoid a costly IPsec tunnel or being forced to hairpin traffic back to your on-premises router.

Many initial SD-WAN deployments are motivated by a long term goal to replace expensive branch MPLS connections with more cost-effective solutions such as broadband or LTE. In parallel, these same organizations are following cloud migration and application modernization strategies. Megaport Virtual Edge enables all these objectives by acting as the bridge between SD-WAN and the cloud (including private, public, hybrid and multiclouds).

Learn how SD-WAN can elevate your enterprise networking.

SD-WAN technology is extremely powerful. Its core value is the ability to shape application traffic and steer it across multiple WAN transports. Inserting Megaport Virtual Edge into your enterprise WAN fabric gives businesses more flexibility and control of their WAN to optimize their applications (which is not always achievable with a cloud hosted SD-WAN edge). MVE can also save you a huge amount of money in egress fees, all the while significantly simplifying your network.

Megaport Virtual Edge can host virtual instances of your SD-WAN edge routers in 22 locations across North America, Asia-Pacific, and Europe. To learn more, click here.