Within current distributions of Linux, there is a kernel component called netem that can be used to test and simulate the type of issues one would see over a Wide Area Network. This component is managed with a tool called traffic controller.
This can be helpful during testing/troubleshooting to emulate the network latency or packet loss that might occur from a remote client endpoint.
Prerequisites
To test client scenarios in this article, we will use the standard ping as well as Apache workbench to measure performance against a URL.
# ubuntu sudo apt-get install apache2-utils # rhel sudo yum install httpd-tools
Baseline performance
The first thing we want to do is measure our baseline performance. First we will use ping, then Apache workbench.
$ ping www.apache.org PING www.apache.org (40.79.78.1) 56(84) bytes of data. 64 bytes from 40.79.78.1: icmp_seq=1 ttl=49 time=14.3 ms 64 bytes from 40.79.78.1: icmp_seq=2 ttl=49 time=14.5 ms 64 bytes from 40.79.78.1: icmp_seq=3 ttl=49 time=15.8 ms
Pings yield around 15ms responses. Then pull the main Apache website.
$ ab -n 20 -c 5 -f TLS1.2 https://httpd.apache.org/
...
Connection Times (ms)
min mean[+/-sd] median max
...
Total: 420 445 11.8 443 471
Percentage of the requests served within a certain time (ms)
50% 443
66% 448
75% 456
80% 458
90% 462
95% 471
98% 471
99% 471
100% 471 (longest request)
I’ve shown an abbreviated output to highlight that total connection median time is 443ms and a 80% majority of the requests are served in 458ms.
Introduce latency
Now let’s introduce an egress latency of 250ms (something you might see on a poor connection from Australia to North America). My network interface is ‘eth0’, this will vary based on your network interfaces.
# introduce latency on outbound network interface sudo tc qdisc change dev eth0 root netem delay 250ms # verify addition tc -s qdisc | grep netem
And ping the site again.
$ ping www.apache.org PING www.apache.org (95.216.24.32) 56(84) bytes of data. 64 bytes from tlp-he-fi.apache.org (95.216.24.32): icmp_seq=1 ttl=51 time=353 ms 64 bytes from tlp-he-fi.apache.org (95.216.24.32): icmp_seq=2 ttl=51 time=352 ms 64 bytes from tlp-he-fi.apache.org (95.216.24.32): icmp_seq=3 ttl=51 time=352 ms
Ping is consistently taking 350ms+. Let’s try Apache workbench to pull the apache site.
$ ab -n 20 -c 5 -f TLS1.2 https://httpd.apache.org/ ... Connection Times (ms) min mean[+/-sd] median max ... Total: 1062 1072 9.7 1068 1097 Percentage of the requests served within a certain time (ms) 50% 1068 66% 1072 75% 1075 80% 1079 90% 1091 95% 1097 98% 1097 99% 1097 100% 1097 (longest request)
Now the total connection median time is at 1068ms, and 80% percent of requests are served in 1079ms. This is about double the times recorded in the baseline, performance has eroded as we expected.
Both latency and packet loss
Now let’s introduce both latency and packet loss.
# latency and packet loss sudo tc qdisc change dev eth0 root netem delay 250ms loss 20%
And ping the site again. You might see a slight increase of a few milliseconds, but the introduction of light packet loss should not affect the ping times.
But let’s run Apache workbench again.
$ ab -n 20 -c 5 -f TLS1.2 https://httpd.apache.org/ ... Connection Times (ms) min mean[+/-sd] median max ... Total: 1064 1861 771.8 1674 4076 Percentage of the requests served within a certain time (ms) 50% 1674 66% 2065 75% 2084 80% 2262 90% 3472 95% 4076 98% 4076 99% 4076 100% 4076 (longest request)
Now the total connection median time is at 1674ms, and 80% percent of requests are served in 2262ms. This would be a significant penalty to end user response times and browser rendering.
Remove latency and packet loss
The changes can be removed by changing delay and loss to 0, or deleting the settings as shown below.
# remove delay and loss sudo tc qdisc change dev eth0 root netem delay 0ms loss 0% # OR sudo tc qdisc del dev eth0 root netem # verify removal tc -s qdisc | grep netem
REFERENCES
openeye, emulating network latency and packet loss