This is a story about troubleshooting routing flaps, which ultimately led to encountering a historic Junos design limitation and introducing new Junos commands to overcome it.
Recently, we added Cisco NCS540 devices running SR-MPLS to our network, using ISIS as the IGP. These two NCS devices form ISIS adjacencies with three Juniper MX routers, which function as BNGs, as well as with several other devices.
It’s important to note that these MX routers also run OSPF with other devices, as we are in the process of migrating from OSPF (without MPLS) to SR-MPLS with ISIS as the IGP.
Since the MX routers have a limited number of interfaces, the connection is indirect. We use two QFX switches between the MX routers and the NCS devices. Each MX establishes four ISIS adjacencies over two separate LAG interfaces, with each LAG connecting to a different QFX switch. Each NCS device is also connected to both QFX switches.
After setting it up, we began noticing intermittent ISIS flaps. The most concerning issue was that, at times, one of the MX routers would lose all four of its ISIS adjacencies simultaneously.
BNG-MX rpd[30288]: RPD_ISIS_ADJDOWN: IS-IS lost L2 adjacency to ncs540-2-cust on ae31.723, reason: Aged out
BNG-MX rpd[30288]: RPD_ISIS_ADJDOWN: IS-IS lost L2 adjacency to ncs540-1-cust on ae32.753, reason: Aged out
BNG-MX rpd[30288]: RPD_ISIS_ADJDOWN: IS-IS lost L2 adjacency to ncs540-1-cust on ae31.713, reason: Aged out
BNG-MX rpd[30288]: RPD_ISIS_ADJUP: IS-IS new L2 adjacency to ncs540-2-cust on ae31.723
BNG-MX rpd[30288]: RPD_ISIS_ADJUP: IS-IS new L2 adjacency to ncs540-1-cust on ae32.753
BNG-MX rpd[30288]: RPD_ISIS_ADJUP: IS-IS new L2 adjacency to ncs540-1-cust on ae31.713
BNG-MX rpd[30288]: RPD_ISIS_ADJDOWN: IS-IS lost L2 adjacency to ncs540-2-cust on ae32.763, reason: Aged out
BNG-MX rpd[30288]: RPD_ISIS_ADJUP: IS-IS new L2 adjacency to ncs540-2-cust on ae32.763
Since the Juniper MX is responsible for terminating PPPoE sessions over EVPN-MPLS VPWS (as described in the following JVD: Juniper Metro Fabric and Broadband Edge), a simultaneous ISIS flap results in the loss of all subscribers. This made the issue extremely critical.
At the same time, other ISIS adjacencies on the NCS and OSPF neighbor relationships on the MX remained stable.
Later, we added another ISIS adjacency between the MX and a Juniper ACX, and that connection also remained unaffected.
Since the issue appeared to be specific to the Juniper MX, we began investigating it with JTAC.
We started by enabling traceoptions under ISIS, and as shown below, we discovered that at certain points, the MX was not receiving ISIS hello packets from the NCS540.
14:39:14.112581 ISIS L2 hello from ncs5402-cust interface ae11.723 absorbed 14:39:18.918981 ISIS L2 hello from ncs5402-cust interface ae11.723 absorbed 14:39:23.610983 ISIS L2 hello from ncs5402-cust interface ae11.723 absorbed 14:39:27.844182 ISIS L2 hello from ncs5402-cust interface ae11.723 absorbed ... 14:39:42.845688 interface ae31.723, L2 neighbor ncs540-2-cust 14:39:42.845695 adjacency state change state Up -> Down 14:39:42.845709 Adjacency state change, ncs5402-cust, state Up -> Down 14:39:42.845717 interface ae31.723, level 2 14:39:42.845859 task_timer_reset: reset IS-IS_Adjacency Garbage Collect ... 14:40:04.388475 ISIS L2 hello from ncs5402-cust interface ae11.723 absorbed 14:40:16.870090 ISIS L2 hello from ncs5402-cust interface ae11.723 absorbed
JTAC initially suspected the issue was related to Control Plane Policing (CoPP), known as ddos-protection in the Juniper CLI. However, I demonstrated that this was not the case, as there were no DDoS events occurring at the time, nor any drops related to ISIS or BFD.
At this point, we started capturing packets using monitor traffic on both the MX and the QFX. Even though the QFX operates at Layer 2 only, it punts ISIS packets to its control plane, allowing us to capture them there.
From the captures, we observed that the QFX received all ISIS packets, but the MX did not see all of them in its control plane—where monitor traffic … captures packets.
Additionally, we noticed that the Cisco devices were sending jumbo packets, while the Juniper devices were transmitting smaller packets.
This issue occurs because Cisco enables hello padding by default. Since ISIS adjacencies between Juniper devices were stable, we changed the hello padding setting on one VLAN to “hello-padding sometimes” on the Cisco side. As a result, that VLAN stopped flapping.
By now, we could have considered the issue resolved and moved on, but we decided to continue troubleshooting to determine why the packets were being dropped.
To investigate further, we compared the ISIS packet counters at the Packet Forwarding Engine (PFE) using show ddos-protection protocols isis statistics against the counters at the Routing Engine (RE) using show isis statistics. After clearing both counters simultaneously, we noticed a discrepancy, confirming that some ISIS hello packets were being dropped in the host path between the PFE and the RE.
user@BNG-MX> show isis statistics | match iih IIH 76634 4 0 82136 0 user@BNG-MX> show ddos-protection protocols statistics brief | match "Protocol|isis-hello" Protocol Packet Received Dropped Rate Violation State isis isis-hello 76643 0 0 0 ok
JTAC requested additional command outputs for further analysis, so I created a pyez script to collect the necessary data and save the output to a CSV file.
At some point, we began gathering statistics from the command “show system statistics ip” and noticed that there were drops specifically related to fragmented packets.
user@BNG-MX> show system statistics ip | match frag | match drop 5566528 fragments dropped (dup or out of space) 1063421 fragment sessions dropped (queue overflow) 148041098 fragments dropped after timeout
It’s important to understand how a Juniper router is structured. As we know, it consists of a data plane (PFE) and a control plane (RE). But how are packets forwarded from the PFE to the RE—for example, ISIS or BGP packets?
There is an Ethernet interface between them (and in chassis-based systems, a switch as well). Juniper uses a proprietary protocol called TTP (IP protocol 84) over this interface, this protocol has a 26-byte header overhead that encapsulates all traffic between the PFE and the RE.
For historical reasons (From M40 in 1998) I don’t know, the MTU on this interface is set to 1500. As a result, any packet larger than this gets fragmented.
For example, the following packet is an ISIS hello with hello-padding. The destination MAC of the ISIS hello (09:00:2b:00:00:05) appears within the packet data.
At this point, we identified that the drops were caused by fragmentation on the interface. To address this, we had to find a way to resolve the issue.
To troubleshoot, we added a filter using the statsfilter command on FreeBSD to collect statistics from the kernel for the specific address pair used by Juniper for TTP connections (128.0.0.16 -> 128.0.0.1). We then compared the statistics with the output from netstat -p ip.
From the comparison of the data we collected every 5 minutes, we observed that the fragmentation drops occurred within the TTP connection. We found a significant number of “fragments dropped after timeout” and “fragment sessions dropped (queue overflow),” indicating that some fragments did not arrive at all, causing the queue used to reassemble the fragments to fill up, leading to fragment drops.
To solve the issue, Juniper engineers suggested lowering the sysctl parameter net.inet.ip.fragreassemblytimeout from 60 to 10. This is particularly relevant for a BNG that handles numerous subscribers and experiences bursty control plane traffic, resulting in fragmented packets queuing up in the kernel fragmentation queue.
By reducing the timeout in the kernel, the queue was flushed faster, preventing it from building up and reducing the chance of three consecutive ISIS hello packets timing out.
To apply this fix via normal CLI commands, Juniper introduced two new commands in Junos 22.4R3-S6.
set system internet-options ip6-reassembly-timeout
set system internet-options ip-reassembly-timeout
They were not brave enough to increase the MTU on the internal interface between the RE and the PFE, and I can understand why—especially given that this interface has over 20 years of code history. Changing such a fundamental parameter could introduce unforeseen complications, so they opted for the safer solution of adjusting the fragmentation timeout instead.
Nitzan Tzelniker
Solutions Architect at Oasis Communication Technologies
Leave your details below and we’ll get back to you shortly
