QoS LLQ Queue Starvation

In the LLQ lesson, I explained that priority queue traffic is transmitted right away before dealing with any other CBWFQ queues. Here’s what it looks like:

What if you receive packets non-stop for the priority queue? Will this starve out other queues? Technically, it would be because the scheduler serves packets in the priority queue before dealing with any of the CBWFQ queues. The priority queue has a bandwidth limit to prevent this from happening, and it has to be policed. This is how it works:

When packets are in the priority queue, a policer checks whether they exceed the bandwidth. If not, we can send those packets to the transmit ring, and they are transmitted. If the packet in the priority queue does exceed the bandwidth, we discard the packet and continue with the CBWFQ queues.

We can test and verify this.

Configuration

I’ll use the following topology:

We need only two Ubuntu hosts connected to a router. I’ll use H1 as an iPerf client and H2 as an iPerf server.

Configurations

Want to take a look for yourself? Here, you will find the startup configuration of R1.

hostname R1
!
interface GigabitEthernet0/0/0
 ip address 192.168.1.254 255.255.255.0
 negotiation auto
!
interface GigabitEthernet0/0/1
 ip address 192.168.2.254 255.255.255.0
 speed 100
 no negotiation auto
!
end

I’m using a Cisco ISR4331 router running Cisco IOS XE Version 15.5(2)S3, RELEASE SOFTWARE (fc2).

I also reduced the speed of one of the router interfaces to create a bottleneck:

R1#show running-config interface GigabitEthernet 0/0/1
Building configuration...

Current configuration : 110 bytes
!
interface GigabitEthernet0/0/1
 ip address 192.168.2.254 255.255.255.0
 speed 100
 no negotiation auto

Let’s start the iPerf server on H2. I’ll start it on two different ports:

ubuntu@H2:~$ iperf3 -s -p 5201
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------

ubuntu@H2:~$ iperf3 -s -p 5202
-----------------------------------------------------------
Server listening on 5202
-----------------------------------------------------------

That’s all we have to do on the server. It will listen on these two ports.

Without Priority Queue

Let’s run some tests. We’ll start with a scenario where we don’t have QoS configured.

Without Congestion

First, we’ll see what kind of bitrate we can get. I’ll use this on the client:

ubuntu@H1:~$ iperf3 -c 192.168.2.2 -p 5202 -u -b 150M -t 10
Connecting to host 192.168.2.2, port 5202
[  5] local 192.168.1.1 port 51045 connected to 192.168.2.2 port 5202
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec  17.9 MBytes   150 Mbits/sec  12938  
[  5]   1.00-2.00   sec  17.9 MBytes   150 Mbits/sec  12949  
[  5]   2.00-3.00   sec  17.9 MBytes   150 Mbits/sec  12948  
[  5]   3.00-4.00   sec  17.9 MBytes   150 Mbits/sec  12949  
[  5]   4.00-5.00   sec  17.9 MBytes   150 Mbits/sec  12950  
[  5]   5.00-6.00   sec  17.9 MBytes   150 Mbits/sec  12948  
[  5]   6.00-7.00   sec  17.9 MBytes   150 Mbits/sec  12949  
[  5]   7.00-8.00   sec  17.9 MBytes   150 Mbits/sec  12949  
[  5]   8.00-9.00   sec  17.9 MBytes   150 Mbits/sec  12949  
[  5]   9.00-10.00  sec  17.9 MBytes   150 Mbits/sec  12949  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec   179 MBytes   150 Mbits/sec  0.000 ms  0/129478 (0%)  sender
[  5]   0.00-10.11  sec   115 MBytes  95.2 Mbits/sec  0.190 ms  46396/129472 (36%)  receiver

Let me explain those parameters:

-c 192.168.2.2: This is the IP address of our server (H2).
-p 5202: The port number of the server.
-u: we want to use UDP instead of TCP.
-b 150M: generate 150 Mbits/sec of traffic.
-t 10: transmit for 10 seconds.

The client transmits 150 Mbits/sec, and the server receives 95.2 Mbits/sec. That makes sense because our bottleneck is that 100 Mbits/sec interface.

Let’s try generating 30 Mbits/sec traffic where packets are marked with DSCP EF:

ubuntu@H1:~$ iperf3 -c 192.168.2.2 -p 5201 -u -b 30M --tos 184 -t 10
Connecting to host 192.168.2.2, port 5201
[  5] local 192.168.1.1 port 42278 connected to 192.168.2.2 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec  3.57 MBytes  30.0 Mbits/sec  2588  
[  5]   1.00-2.00   sec  3.58 MBytes  30.0 Mbits/sec  2590  
[  5]   2.00-3.00   sec  3.58 MBytes  30.0 Mbits/sec  2589  
[  5]   3.00-4.00   sec  3.58 MBytes  30.0 Mbits/sec  2590  
[  5]   4.00-5.00   sec  3.58 MBytes  30.0 Mbits/sec  2590  
[  5]   5.00-6.00   sec  3.58 MBytes  30.0 Mbits/sec  2590  
[  5]   6.00-7.00   sec  3.58 MBytes  30.0 Mbits/sec  2590  
[  5]   7.00-8.00   sec  3.58 MBytes  30.0 Mbits/sec  2589  
[  5]   8.00-9.00   sec  3.58 MBytes  30.0 Mbits/sec  2590  
[  5]   9.00-10.00  sec  3.58 MBytes  30.0 Mbits/sec  2590  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  35.8 MBytes  30.0 Mbits/sec  0.000 ms  0/25896 (0%)  sender
[  5]   0.00-10.04  sec  35.8 MBytes  29.9 Mbits/sec  0.218 ms  0/25896 (0%)  receiver

By using --tos 184 we set the ToS byte to 184 (decimal), which equals DSCP EF. Because there is no congestion, all packets make it to the server. Let’s also try this with 80 Mbits/sec of unmarked traffic:

ubuntu@H1:~$ iperf3 -c 192.168.2.2 -p 5202 -u -b 80M -t 10
Connecting to host 192.168.2.2, port 5202
[  5] local 192.168.1.1 port 33149 connected to 192.168.2.2 port 5202
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec  9.53 MBytes  79.9 Mbits/sec  6900  
[  5]   1.00-2.00   sec  9.54 MBytes  80.0 Mbits/sec  6906  
[  5]   2.00-3.00   sec  9.54 MBytes  80.0 Mbits/sec  6906  
[  5]   3.00-4.00   sec  9.54 MBytes  80.0 Mbits/sec  6907  
[  5]   4.00-5.00   sec  9.54 MBytes  80.0 Mbits/sec  6906  
[  5]   5.00-6.00   sec  9.54 MBytes  80.0 Mbits/sec  6906  
[  5]   6.00-7.00   sec  9.54 MBytes  80.0 Mbits/sec  6906  
[  5]   7.00-8.00   sec  9.54 MBytes  80.0 Mbits/sec  6906  
[  5]   8.00-9.00   sec  9.54 MBytes  80.0 Mbits/sec  6906  
[  5]   9.00-10.00  sec  9.54 MBytes  80.0 Mbits/sec  6906  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  95.4 MBytes  80.0 Mbits/sec  0.000 ms  0/69055 (0%)  sender
[  5]   0.00-10.04  sec  95.3 MBytes  79.6 Mbits/sec  0.204 ms  10/69055 (0.014%)  receiver

There is no congestion, so all packets make it.

With Congestion

Let’s see what happens when there is congestion. I’ll generate 30 Mbits/sec of DSCP EF traffic and 80 Mbits/sec of unmarked traffic. I’ll run both commands at the same time:

ubuntu@H1:~$ iperf3 -c 192.168.2.2 -p 5201 -u -b 30M --tos 184 -t 10
Connecting to host 192.168.2.2, port 5201
[  5] local 192.168.1.1 port 48526 connected to 192.168.2.2 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec  3.57 MBytes  30.0 Mbits/sec  2588  
[  5]   1.00-2.00   sec  3.58 MBytes  30.0 Mbits/sec  2590  
[  5]   2.00-3.00   sec  3.58 MBytes  30.0 Mbits/sec  2589  
[  5]   3.00-4.00   sec  3.58 MBytes  30.0 Mbits/sec  2590  
[  5]   4.00-5.00   sec  3.58 MBytes  30.0 Mbits/sec  2590  
[  5]   5.00-6.00   sec  3.58 MBytes  30.0 Mbits/sec  2590  
[  5]   6.00-7.00   sec  3.58 MBytes  30.0 Mbits/sec  2589  
[  5]   7.00-8.00   sec  3.58 MBytes  30.0 Mbits/sec  2590  
[  5]   8.00-9.00   sec  3.58 MBytes  30.0 Mbits/sec  2590  
[  5]   9.00-10.00  sec  3.58 MBytes  30.0 Mbits/sec  2590  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  35.8 MBytes  30.0 Mbits/sec  0.000 ms  0/25896 (0%)  sender
[  5]   0.00-10.11  sec  26.2 MBytes  21.7 Mbits/sec  0.156 ms  6943/25896 (27%)  receiver

ubuntu@H1:~$ iperf3 -c 192.168.2.2 -p 5202 -u -b 80M -t 10
Connecting to host 192.168.2.2, port 5202
[  5] local 192.168.1.1 port 58261 connected to 192.168.2.2 port 5202
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec  9.53 MBytes  79.9 Mbits/sec  6900  
[  5]   1.00-2.00   sec  9.54 MBytes  80.0 Mbits/sec  6907  
[  5]   2.00-3.00   sec  9.54 MBytes  80.0 Mbits/sec  6905  
[  5]   3.00-4.00   sec  9.54 MBytes  80.0 Mbits/sec  6907  
[  5]   4.00-5.00   sec  9.54 MBytes  80.0 Mbits/sec  6906  
[  5]   5.00-6.00   sec  9.54 MBytes  80.0 Mbits/sec  6906  
[  5]   6.00-7.00   sec  9.54 MBytes  80.0 Mbits/sec  6906  
[  5]   7.00-8.00   sec  9.54 MBytes  80.0 Mbits/sec  6906  
[  5]   8.00-9.00   sec  9.54 MBytes  80.0 Mbits/sec  6906  
[  5]   9.00-10.00  sec  9.54 MBytes  80.0 Mbits/sec  6906  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  95.4 MBytes  80.0 Mbits/sec  0.000 ms  0/69055 (0%)  sender
[  5]   0.00-10.06  sec  89.1 MBytes  74.3 Mbits/sec  0.193 ms  4538/69054 (6.6%)  receiver

Now we are trying to squeeze 30 Mbits and 80 Mbits of traffic through a 100 Mbit interface which results in packet loss. We see 21.7 Mbits/sec of DSCP EF traffic and 74.3 Mbits/sec of unmarked traffic. Those two add up to 96 Mbits/sec, which is about the maximum bitrate we can get.

With Priority Queue

Let’s configure a priority queue to see how it impacts our bitrates and packet drops. I’ll configure a class-map that matches DSCP EF and a policy map with a priority queue for 20 Mbits/sec:

R1(config)#class-map DSCP_EF
R1(config-cmap)#match dscp ef

R1(config)#policy-map LLQ
R1(config-pmap)#class DSCP_EF
R1(config-pmap-c)#priority 20000

R1(config)#interface GigabitEthernet 0/0/1
R1(config-if)#service-policy output LLQ

Here’s what it looks like:

R1#show policy-map interface GigabitEthernet 0/0/1
 GigabitEthernet0/0/1 

  Service-policy output: LLQ

    queue stats for all priority classes:
      Queueing
      queue limit 512 packets
      (queue depth/total drops/no-buffer drops) 0/0/0
      (pkts output/bytes output) 0/0

    Class-map: DSCP_EF (match-all)  
      0 packets, 0 bytes
      5 minute offered rate 0000 bps, drop rate 0000 bps
      Match:  dscp ef (46)
      Priority: 20000 kbps, burst bytes 500000, b/w exceed drops: 0
      

    Class-map: class-default (match-any)  
      0 packets, 0 bytes
      5 minute offered rate 0000 bps, drop rate 0000 bps
      Match: any 
      
      queue limit 416 packets
      (queue depth/total drops/no-buffer drops) 0/0/0
      (pkts output/bytes output) 0/0

We now have a 20 Mbps/sec priority queue for DSCP EF marked traffic, and everything else ends up in the class-default class.

Without Congestion

Let’s see what happens when there is no congestion. I’ll generate 30 Mbits/sec DSCP EF marked traffic:

ubuntu@H1:~$ iperf3 -c 192.168.2.2 -p 5201 -u -b 30M --tos 184 -t 10
Connecting to host 192.168.2.2, port 5201
[  5] local 192.168.1.1 port 37116 connected to 192.168.2.2 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec  3.57 MBytes  30.0 Mbits/sec  2588  
[  5]   1.00-2.00   sec  3.58 MBytes  30.0 Mbits/sec  2590  
[  5]   2.00-3.00   sec  3.58 MBytes  30.0 Mbits/sec  2589  
[  5]   3.00-4.00   sec  3.58 MBytes  30.0 Mbits/sec  2590  
[  5]   4.00-5.00   sec  3.58 MBytes  30.0 Mbits/sec  2590  
[  5]   5.00-6.00   sec  3.58 MBytes  30.0 Mbits/sec  2590  
[  5]   6.00-7.00   sec  3.58 MBytes  30.0 Mbits/sec  2590  
[  5]   7.00-8.00   sec  3.58 MBytes  30.0 Mbits/sec  2589  
[  5]   8.00-9.00   sec  3.58 MBytes  30.0 Mbits/sec  2590  
[  5]   9.00-10.00  sec  3.58 MBytes  30.0 Mbits/sec  2590  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  35.8 MBytes  30.0 Mbits/sec  0.000 ms  0/25896 (0%)  sender
[  5]   0.00-10.04  sec  35.8 MBytes  29.9 Mbits/sec  0.217 ms  0/25896 (0%)  receiver

There is no congestion, so we can get a higher bitrate than what we configured with the priority command. Let’s check the router output:

R1#show policy-map interface GigabitEthernet 0/0/1
 GigabitEthernet0/0/1 

  Service-policy output: LLQ

    queue stats for all priority classes:
      Queueing
      queue limit 512 packets
      (queue depth/total drops/no-buffer drops) 0/0/0
      (pkts output/bytes output) 25896/38585040

    Class-map: DSCP_EF (match-all)  
      25896 packets, 38585040 bytes
      5 minute offered rate 981000 bps, drop rate 0000 bps
      Match:  dscp ef (46)
      Priority: 20000 kbps, burst bytes 500000, b/w exceed drops: 0
      

    Class-map: class-default (match-any)  
      20 packets, 2094 bytes
      5 minute offered rate 0000 bps, drop rate 0000 bps
      Match: any 
      
      queue limit 416 packets
      (queue depth/total drops/no-buffer drops) 0/0/0
      (pkts output/bytes output) 19/1677

We see many packets that match the DSCP_EF class map and no packet drops because they exceeded the bandwidth.

With Congestion

Now, let’s see what happens when there is congestion. I’ll start those two packet streams of 30 Mbits/sec DSCP EF traffic and 80 Mbits/sec unmarked traffic at the same time:

ubuntu@H1:~$ iperf3 -c 192.168.2.2 -p 5201 -u -b 30M --tos 184 -t 10
Connecting to host 192.168.2.2, port 5201
[  5] local 192.168.1.1 port 59719 connected to 192.168.2.2 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec  3.57 MBytes  30.0 Mbits/sec  2588  
[  5]   1.00-2.00   sec  3.58 MBytes  30.0 Mbits/sec  2590  
[  5]   2.00-3.00   sec  3.58 MBytes  30.0 Mbits/sec  2589  
[  5]   3.00-4.00   sec  3.58 MBytes  30.0 Mbits/sec  2590  
[  5]   4.00-5.00   sec  3.58 MBytes  30.0 Mbits/sec  2590  
[  5]   5.00-6.00   sec  3.58 MBytes  30.0 Mbits/sec  2590  
[  5]   6.00-7.00   sec  3.58 MBytes  30.0 Mbits/sec  2590  
[  5]   7.00-8.00   sec  3.58 MBytes  30.0 Mbits/sec  2589  
[  5]   8.00-9.00   sec  3.58 MBytes  30.0 Mbits/sec  2590  
[  5]   9.00-10.00  sec  3.58 MBytes  30.0 Mbits/sec  2590  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  35.8 MBytes  30.0 Mbits/sec  0.000 ms  0/25896 (0%)  sender
[  5]   0.00-10.28  sec  24.2 MBytes  19.7 Mbits/sec  0.113 ms  8406/25895 (32%)  receiver

ubuntu@H1:~$ iperf3 -c 192.168.2.2 -p 5202 -u -b 80M -t 10
Connecting to host 192.168.2.2, port 5202
[  5] local 192.168.1.1 port 51758 connected to 192.168.2.2 port 5202
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec  9.53 MBytes  79.9 Mbits/sec  6900  
[  5]   1.00-2.00   sec  9.54 MBytes  80.0 Mbits/sec  6906  
[  5]   2.00-3.00   sec  9.54 MBytes  80.0 Mbits/sec  6906  
[  5]   3.00-4.00   sec  9.54 MBytes  80.0 Mbits/sec  6906  
[  5]   4.00-5.00   sec  9.54 MBytes  80.0 Mbits/sec  6907  
[  5]   5.00-6.00   sec  9.54 MBytes  80.0 Mbits/sec  6906  
[  5]   6.00-7.00   sec  9.54 MBytes  80.0 Mbits/sec  6906  
[  5]   7.00-8.00   sec  9.54 MBytes  80.0 Mbits/sec  6906  
[  5]   8.00-9.00   sec  9.54 MBytes  80.0 Mbits/sec  6906  
[  5]   9.00-10.00  sec  9.54 MBytes  80.0 Mbits/sec  6906  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  95.4 MBytes  80.0 Mbits/sec  0.000 ms  0/69055 (0%)  sender
[  5]   0.00-10.05  sec  91.2 MBytes  76.2 Mbits/sec  0.232 ms  2982/69053 (4.3%)  receiver

Now, we see that the priority queue has a bitrate of 19.7 Mbits/sec, with 32% packet loss. Our unmarked traffic has a bitrate of 76.2 Mbits/sec, with some packet loss.

This looks good, we have a priority queue but it’s limited to 20 Mbits/sec and it doesn’t starve our CBWFQ class default queue. You can see the DSCP EF packet drops here:

R1#show policy-map interface GigabitEthernet 0/0/1
 GigabitEthernet0/0/1 

  Service-policy output: LLQ

    queue stats for all priority classes:
      Queueing
      queue limit 512 packets
      (queue depth/total drops/no-buffer drops) 0/10404/0
      (pkts output/bytes output) 23209/34581410

    Class-map: DSCP_EF (match-all)  
      33613 packets, 50083370 bytes
      5 minute offered rate 991000 bps, drop rate 306000 bps
      Match:  dscp ef (46)
      Priority: 20000 kbps, burst bytes 500000, b/w exceed drops: 10404
      

    Class-map: class-default (match-any)  
      102994 packets, 153419717 bytes
      5 minute offered rate 3058000 bps, drop rate 108000 bps
      Match: any 
      
      queue limit 416 packets
      (queue depth/total drops/no-buffer drops) 0/3705/0
      (pkts output/bytes output) 99286/147902287

Because of congestion, our DSCP EF priority queue traffic has packet drops. These are packets that exceeded the 20 Mbits/sec priority queue bitrate.

We’ll do one more test. Let’s turn it up a notch and increase DSCP EF traffic to 300 Mbit/sec while simultaneously sending 80 Mbits/sec of unmarked traffic:

Source link

case studies

See More Case Studies

Networking

Threats delivered over encrypted channels continue to rise

Encryption is the default for online communication, with nearly all web traffic protected by secure protocols like TLS/SSL. Yet, as encryption becomes more ubiquitous by

Learn more

Networking

Cisco Turns 40 and Redefines its Purpose Goals

Cisco turns 40 this month and is celebrating by doing what Cisco has always done: renewing its commitment to innovation, inclusion, and sustainability. I’ve been

Learn more

Cyber Security

Lazarus Group Spotted Targeting Nuclear Engineers with CookiePlus Malware

The Lazarus Group, an infamous threat actor linked to the Democratic People’s Republic of Korea (DPRK), has been observed leveraging a “complex infection chain” targeting

Learn more

Partner with Us for Comprehensive IT

We’re happy to answer any questions you may have and help you determine which of our services best fit your needs.

Your benefits:

What happens next?

We Schedule a call at your convenience

We do a discovery and consulting meting

We prepare a proposal

Schedule a Free Consultation

First name

Last name

Company / Organization

Company email

Phone

How Can We Help You?

Message

QoS LLQ Queue Starvation

Configuration

Without Priority Queue

Without Congestion

With Congestion

With Priority Queue

Without Congestion

With Congestion

See More Case Studies

Threats delivered over encrypted channels continue to rise

Cisco Turns 40 and Redefines its Purpose Goals

Lazarus Group Spotted Targeting Nuclear Engineers with CookiePlus Malware

Partner with Us for Comprehensive IT

Your benefits:

What happens next?

Schedule a Free Consultation

Solutions

Company

LinkedIn

TikTok

Twitter

Facebook

Youtube

Simplifying IT
for a complex world.

Platform partnerships

Services

Business Challenges

Digital Transformation

Security

Automation

Gaining Efficiency

Industry Focus

QoS LLQ Queue Starvation

Configuration

Without Priority Queue

Without Congestion

With Congestion

With Priority Queue

Without Congestion

With Congestion

See More Case Studies

Partner with Us for Comprehensive IT

Your benefits:

What happens next?

Schedule a Free Consultation

Simplifying IT for a complex world.

Platform partnerships

Services

Business Challenges

Digital Transformation

Security

Automation

Gaining Efficiency

Industry Focus

Simplifying IT
for a complex world.