A Tale of Relationships and BGP

The loss of some relationships might sadden us, but some might raise interesting questions in regards to Border Gateway Protocol (BGP)…

Network Separation of Two Companies
Imagine this, two companies that were once a single entity have decided to part ways.
Employees will be split between the companies, with financial and other matters to address. However, as network engineers, we are mostly interested on who will keep the old network they once shared.

A few months ago, we faced this question when one of our customers have decided to split its network to two as part of the separation of its company.
From a single network, two networks needed to be born, one as an ISP (Internet Service Provider) and the other as a Mobile SP (Service Provider).
Both networks shared a variety of services, including Radius, SBC, DNS, and more. Additionally, they shared public IPs and a Public AS.

A Mobile Service Provider needed a new network
This blog focuses on a Mobile SP network — specifically, a standard MPLS network connecting two sites in different cities, each capable of backing up the other. For this network, a new AS number was acquired along with new Public IP addresses for subscribers.
Allow me to share with you some of the technical challenges that needed to be addressed before the separation:

1. Should both sites be migrated simultaneously, or should it be done in stages?
2. How will MP-BGP will function over EBGP Peering within our topology?
3. How can we avoid AS-LOOPS, and when might they occur?
4. Register the acquired public IPs with the new ASN, taking into account also IP location and RPKI considerations.

The Power of POC
A POC is a great way to explore all the dilemmas we mentioned above without disrupting the network’s functionality.
We were able to replicate the customer’s network using a single Juniper router (thanks, Juniper, for creating the logical-systems) and conducted some tests on this replica. Now let’s answer some questions:

1. Migrating both sites at the same time is too risky, there are too many services to consider, too many intertwined functions on both networks (the SP and ISP). Also, on mobile networks, we have the option to move subscribers from one site to the other easily so why not use it?
2. Many tests were conducted, and it turned out to work just fine. We based our solution over the ‘Inter AS Option B’ design which is usually done by using P2P addresses between ASBRs for the EBGP peers and enabling MPLS on the inter-AS link. In our case, it’s a bit different as we are using the loopback addresses for the BGP peers with the ‘multihop’ option, and we also retained the IGP, LDP, RSVP and MPLS configuration we had before the change.

• • No issues were observed here. Both ASBRs are exchanging labels with EBGP and we are able to get routes under VRFs to be advertised between the two different ASs.
  • Remember that there will be a transition state where one site is configured with the new ASN, and the other site is configured with the old ASN. That’s the reason it was important to ensure this design is working properly.

3.To understand the answer to this question, firstly, you need to understand the problem we faced. We have a large number of providers connected to the routers on each site, and we can’t schedule all of them to work with us on the same MW (maintenance window). To solve this, against our providers, we have configured the ‘local-as 200 loops 2 alias’ command. Now that you know that, you also need to be aware of the as-path-domain concept that Juniper implemented on their Junos OS. Basically each ‘domain’ contains ASs that are configured on the router, this includes all ASs from all routing-instances.

• Let’s explain this concept through an example:
  Our router belongs to AS100. However, as mentioned earlier, we have configured the ‘local-as 200 loops 2 alias’ command against our providers’s BGP peers. Let’s break down this command
  • ‘local-as’ essentially means that we don’t need to inform our providers that we are changing our AS or schedule the maintenance window with them because the AS they will see remains the old AS. On the as-path-domain level, although this command is configured under a specific peer, it actually affects the entire router. This makes AS200 part of the as-path-domain and, consequently, part of the router’s ASs.
  • ‘loops 2’ allows us to receive routes with as-path that consist AS200 up to twice. We needed this beacuse 200 is the old AS of site A and the current AS of site B. So when the site B sends an update to site A, the router would normally consider it as an AS-LOOP and not install the route. Allowing ‘loops 2’ addresses this issue.
  • ‘alias’ means that we will hide AS200 from the routes the router receives from the peer and advertise it to the rest of Operator’s network.

Take a look on the following output from the router:
oasis@PE-Mobile-SP-A-1> show as-path domain
Domain: 2 Primary: 65301
References: 1 Paths: 8
Local AS: 65301 Loops: 1
Domain: 3 Primary: 100
References: 18 Paths: 9945
Flags: Master
Local AS: 200 Loops: 2
Local AS: 100 Loops: 1
The output above simply tells us that routes that are coming into the router are checked against AS200 and AS100 no matter on which VRF the routes should enter to. The AS65301 is an independent-domain and thus belong to a different domain.

If for example the router PE-Mobile-SP-A-1 will receive a route that looks like this:
10.10.10.10/32 *[BGP/170] 00:00:01, localpref 100, from 1.1.1.2
AS path: 200 I, validation-state: unverified
> to 10.11.12.13 via lt-0/0/0.3108, label-switched-path PE-Mobile-SP-A-1_TO_PE-Mobile-SP-B-2
It will be accepted even if AS200 appears twice (it doesn’t look like that, but once you see the ‘I’ in the as-path it indicates the path origin, which AS200 is part of this path).

Another example:
10.10.10.10/32 [BGP ] 00:00:21, from 1.1.1.1
AS path: 200 200 I, validation-state: unverified
> to 10.11.12.13 via lt-0/0/0.3108, label-switched-path PE-Mobile-SP-A-1_TO_PE-Mobile-SP-B-1
In this case, the route will be rejected due to as-loop (as we enabled only twice the AS200) and here it appears 3 times.

4. ISPs around the world rely on the RIPE Database to adjust their routing tables, aligning their import/export policies with the information provided in RIPE registrations. The Mobile Operator has acquired IP addresses previously assigned to a different company (ASN) and location, and now requires these addresses to be re-registered under the Operator’s new ASN.

• Using RIPE, we can update the Operator’s IPs and ASN, informing the internet routing table about who can advertise the Operato’s IPs and establishing rules for which ASs can advertise/receive the Operator’s new IPs.
  • Additionally, we needed to migrate some IP ranges that were in use by the ISP (the Internet Service Provider’s network), so a change in the RIPE registration of the ISP was necessary.
• Concerning IP location, numerous databases on the internet are used by different applications to determine if a specific IP, based on its location, can access their services. For example, CloudFlare uses these databases to identify the locations of IPs accessing their applications.
• Updating the RPKI Database is crucial. RPKI is a tool used by ISPs to prevent route hijacking. It serves as a validator for IPs to determine if the AS that advertised them truly belongs to that AS.

Summary
Performing network separation in a service provider network is a complex project, mainly due to the need to perform it on a live network, to get to know both the network, the relevant protocols, and the possibilities to wrinkle them a little.
Despite the challenges, the network migration went smoothly, leaving both entities as distinct networks in the internet landscape. This achievement marked a crucial step in the complex separation process.

Join Our Team:
If you find these networking challenges exciting, we’re hiring talented network engineers at Oasis. *Contact us* to be part of our passionate team.

Nir Gal
Net&Sec Solution Architect

Linkedin