Friday, July 2, 2010

Fragmentation in Controller Architectures

*Updated Dec. 2010 - Added CAPWAP information.

Why IP Fragmentation is a Concern on Wireless Networks
A big concern with controller-based architectures is reduced performance due to IP fragmentation due to tunnel overhead between the AP and controller. Typically, most lightweight APs tunnel all client traffic back to the controller through a layer 3 tunnel. This tunnel adds overhead, as the AP has to encapsulate the original client frame within a new IP packet.

Almost all vendors now offer options to locally switch (or bridge) the client frames into the wired network from the AP, without tunneling the frame back to the controller. Lookup Cisco H-REAP, Aruba RAP, Ruckus does this by default on all APs, Aerohive doesn’t even have a physical controller, you get my point.

Tunnel overhead reduces the amount of data available in the packet payload that can carry client data. If the client sends frames that are above a certain threshold, then adding the additional tunnel headers increases the packet size beyond the Ethernet MTU (1500 bytes unless using jumbo frames), resulting in IP fragmentation. Ultimately, this results in increased network latency and jitter, and decreased throughput.

Bad for data, worse for voice!

LWAPP Overhead
In Layer 3 LWAPP mode, additional IP, UDP, and LWAPP headers are added to encapsulate the original client 802.11 frame between the access point and controller.

LWAPP overhead for tunneled client frames is 52 bytes due to these additional frame headers which encapsulate the original frame taken out of the air:

§  18 – Ethernet header and checksum
§  20 – IP header (without IP options)
§  8 – UDP header
§  6 – LWAPP header (CAPWAP header is 8 bytes)

The contents of the LWAPP frame also contain the original 802.11 client data frame, with a few notable portions that are stripped out. Let’s add these into the tunneled frame contents:

§  24 – Original 802.11 MAC header, without QoS, encryption, or checksum fields
§  8 – 802.2 LLC and SNAP headers (3 LLC, 5 SNAP)
§  20 – Original IP header
§  Variable – Transport layer header (20 TCP, 8 UDP, 8 ICMP, etc.) and data payload

Notice that I said the whole 802.11 frame, not just the client’s data payload, get’s tunneled. The 802.11 MAC header is tunneled in order for the controller to be able to de-multiplex the original client frame, determine source and destination MAC addresses, forward the frame to appropriate destination host, and perform the non-real-time 802.11 MAC functions in the Split-MAC architecture.

Split-MAC Responsibilities
In Cisco’s LWAPP architecture, split-MAC is the term coined to differentiate what MAC functions get handled by the controller versus the access point.

Since the controller needs to handle some MAC functions, it needs the client’s 802.11 MAC header tunneled back to it to perform its responsibilities. Since the AP handles real-time MAC functions, such as QoS and encryption/decryption, those fields are stripped out of the header before tunneling the packet to the controller.

Controller Responsibilities:
§  Security management (policy enforcement, rogue detection, etc.)
§  Configuration and firmware management
§  Northbound management interfaces
§  Non real-time 802.11 MAC functions
o   Association, dis-association, re-association
o   802.11e/WMM resource reservation (CAC, TSPEC, etc.)
o   802.1x/EAP authentication
o   Key management
§  802.11 distribution services
§  Wired and wireless integration services

Lightweight Access Point Responsibilities:
§  Real-time 802.11 MAC Functions
o   Beacon generation
o   Probe responses
o   Informs WLC of probe requests via LWAPP control channel
o   Power management and packet buffering
o   802.11e/WMM scheduling and queuing
o   MAC layer data encryption and decryption
o   802.11 control messages (ACK, RTS/CTS)
§  Data encapsulation and de-capsulation to LWAPP
§  Fragmentation and re-assembly
§  RF spectral analysis
§  WLAN IDS signature analysis

LWAPP Fragmentation
Let’s take a look at how Cisco’s LWAPP protocol handles IP fragmentation in code versions prior to 5.2, when the switch to CAPWAP is made.

Layer 3 LWAPP is able to perform full fragmentation and reassembly of tunnel packets using standard IP fragmentation, thereby allowing client traffic to make use of a full 1500 byte MTU and not to have to adjust for any tunnel overhead. This simplifies administration of wireless clients, since you don’t have to touch them to modify MTU settings for the wireless adapter. Clients simply assume their IP packets have an MTU of 1500 bytes and move happily along. It’s not until the frame reaches the AP that fragmentation needs to occur because of the tunnel overhead.

* Note - CAPWAP includes fragmentation and re-assembly support and does not rely on the standard IP fragmentation. Therefore, in fragmented CAPWAP tunnel packets, fragmentation will be performed on the CAPWAP data, with each fragment duplicating the outer IP, UDP, and CAPWAP headers.

Adding up the tunnel overhead, we find that the threshold for the original client’s IP packet that causes fragmentation of the LWAPP IP packet on an Ethernet segment with a 1500 byte MTU is at a size of 1434 bytes. We get this by adding the tunnel headers (IP, UDP, LWAPP) and the tunneled 802.11 frame headers (802.11, LLC, SNAP), and subtracting those amounts from the 1500 byte MTU. This leaves only the original client IP packet which must be under a 1434 byte threshold to prevent fragmentation. The outer Ethernet header is not included because the MTU size does not consider the layer 2 MAC header, which is appended later.

(Threshold is 1432 bytes for CAPWAP encapsulated frames)

Here is a packet capture to illustrate this fragmentation. Remember, the outer IP header for the tunnel will handle the fragmentation, so the UDP and LWAPP headers will only be found in the first fragment.

Testing LWAPP Fragmentation
An easy way to test LWAPP fragmentation is to connect to the wireless network and ping the subnet gateway above and below the maximum ICMP payload size of 1406 bytes assuming a 1500 byte MTU. How we get to 1406 bytes for this test:

1500 byte MTU
     Subtract 34 bytes for LWAPP encapsulation (subtract 36 bytes for CAPWAP)
     Subtract 24 bytes for the tunneled 802.11 header
     Subtract 8 bytes for the tunneled 802.2 LLC/SNAP header
     Subtract 20 bytes for the tunneled IP header (without options)
     Subtract 8 bytes for the ICMP header
= 1406 bytes remaining for ICMP payload (use 1404 bytes for CAPWAP)

Setup a tap or port mirror on the switch port attached to the access point. Connect a workstation running a protocol analyzer to the mirrored port or tap, filter on port UDP 12,222 (LWAPP Data) or UDP 5247 (CAPWAP Data), and start capturing packets. 

From the wireless client, try out these commands and watch the results in the protocol analyzer.

Test 1
ping –f –l 1406 ip-address

Result will be successful, no fragmentation.

Test 2
ping –f –l 1407 ip-address

Result will be successful, but LWAPP tunnel packets will be fragmented. LWAPP tunnel packets do not carry over the original packet’s don’t fragment flag.

Test 3
ping –f –l 1473 ip-address

Result will fail, as original IP packet needs to be fragmented by the client but the don’t fragment flag is set.

Many VPN client software packages will change the default interface MTU to a lower value. Verify the interface MTU to adjust test payload size properly. For example the Cisco Systems VPN client software for Windows XP modifies the MTU from 1500 bytes to 1300 bytes.

To view the current interface MTU:
§  Windows XP regedit

(if the key does not exist, it uses the default MTU of 1500 bytes)

§  Windows Vista/7 view interface MTU command is “netsh interfaces ipv4 show subinterfaces”

Preventing Fragmentation
Unfortunately, there is no method to prevent LWAPP fragmentation in the controller or access points until code version 6.0 with the CAPWAP protocol.

To avoid fragmentation of client packets enable TCP Maximum Segment Size (MSS) control on the controller and lightweight access points in code versions 6.0 and later.

This feature allows connected CAPWAP access points to re-write client TCP SYN packets with the configured maximum segment size to optimize large data transfers and prevent fragmentation when CAPWAP encapsulated. The valid range is from 536 to 1363 bytes, and it is recommended to use a 1363 byte value.

Configure it from the CLI using this command:
config ap tcp-adjust-mss { enable | disable } { all | ap-name } tcp_mss_value

- Andrew


  1. Andrew,
    Super great post, thanks for sharing your testing results !

  2. Interesting post. So, do you see H-REAP as a valid alternative? I've been tossing around the idea of moving all of our local APs to H-REAP to provide a consistent level of capabilities at our campuses and remote sites, and this only helps to bolster that idea.

  3. H-REAP is an alternative for remote sites and corporate sites alike. Administrators just need to be aware of key operational differences and limitations, and understand how they affect their specific environment. One other key nugget of info on H-REAP, it's supposed to boost 802.11n performance by 5-10% on locally switched WLANs! (see #9).

  4. Great post thanks!
    I am not sure I fully understand the "worse for voice part" though... can you explain which voice codec uses packets larger than 1406 bytes?

  5. Hi Leon,
    First, thanks for reading my blog. The "worse for voice" part applies if there is a WAN MTU that is smaller than the typical Ethernet MTU, say around 500 bytes. In that case, voice packets tunneled inside LWAPP or CAPWAP frames would get fragmented as well and suffer from increased latency from the fragmentation and re-assembly process, as well as possibly out of order packet arrival.


  6. This is a brilliant blog Andy thanks pal

  7. Hopefully this tutorial may save my ass... having trouble with a remote location which is connected via Cisco DMVPN Connection, and the capwap traffic (hreap ap in location) fragments, and there are strange issues, though wan bandwidth is big (15Mbit/s and RTT of 50ms).
    Will try out to adjust tcp MSS on ap.
    But as our DMVPN is hardcoded at MTU 1300 on Tunnel interface, i think i should adjust MSS on AP to 1200 or something lower than 1300?

    1. Since the CAPWAP recommended TCP-MSS is 1,363 bytes on a standard 1,500 byte Ethernet MTU, then I would recommend dropping it by the same amount for a 1,300 byte MTU tunnel interface to around 1,163 byte TCP-MSS.

      Give that a try.


  8. Command is netsh interface ipv4 show subinterfaces, not netsh interfaces ipv4 show subinterfaces

  9. Any word on the DMVPN question above? I'm deploying DMVPN on my network soon and would like to know if I will be able to use CAPWAP for our smaller sites.

    1. Hi Aaron,
      See my comment above. I would follow Cisco's recommendation for CAPWAP TCP-MSS adjustment. Since the recommended value for a 1500 byte MTU is 1363 byte TCP-MSS, then I would subtract the same amount from the lowest link MTU. So for a DMVPN link with a 1300 byte MTU, try a 1163 byte TCP-MSS.


    2. Hi Andrew, this was a great read. Would be nice to see a similar article for the anchoring EtherIP overhead and how best to resolve that. It turns out using the same WLC adjust-mss command can help with that too, but then you are potentially punishing your enterprise wireless at the same time. Interestingly, I tried using an IOS interface command "ip tcp adjust-mss 1420" on a point upstream from the guest subnet (thus upstream from the encapsulation, on the path to the Internet), and yet, my packet captures from the wireless client show MSS 1460. So either the command is doing nothing (which I'll have to test by plugging directly into the router). Is it possible the AP resets it to 1460 by default (i.e., when no MSS Adjust is configured on the WLC)?

  10. Very Nice Post. I have a doubt if you can clarify pls. version 6.0 or after are we stopping fragmentation on IP packet or LWAPP packet which is encapsulated inside IP packet. I meant like for example, router on internet would see the traversing packet as IP packet and will do fragmentation if MTU is > than allowed on next link. Does LWAPP header comes in picture in this cases?
    Is there any chance port 12222, 12223 for LWAPP would be blocked on Internet? If so how do we know?