This article presents a simple and inexpensive methodology for predicting the performance of a client/server application over a wide area network. A network emulator, placed between the client and server, is used to vary key network properties, such as latency, bandwidth and packet loss. This method is not meant to replace extensive network modeling tools such as OPNET or Load Runner, however, it can provide developers with a simple way to explore the behavior of applications over a wide area network before deployment. For example, developers will be able to determine performance over a dial-up line or low-speed frame relay circuit.
The emulator uses the Linux operating system on a PC with built-in emulation and queuing utilities. The PC is configured to operate in "bridge" mode, which eliminates IP readdressing requirements. Performance of the emulator was validated using tests detailed in the sidebar "Network Emulation Details and Validation."
Bandwidth
Bandwidth provides a measure of the link capacity and is usually specified as bits per second (bps) or bytes per second (Bps). Increased bandwidth allows more data to flow over a link in the same amount of time.
When circuits are added in a series, the bottleneck is the link with the slowest bandwidth. For example, if the network path consists of a 128Kbps circuit, linked to an OC-3 (155MBps) backbone, and finally a DS-3 (45Mbps), the network bottleneck is the 128Kbps circuit - meaning it's not possible to push data through the total circuit faster than 128Kbps.
The bandwidth of the network emulator represents the slowest possible link in the circuit path. Since most circuit paths within the ESN will be symmetrical, identical bandwidth parameters will be applied to incoming and outgoing packets. Different bandwidths could be applied to simulate links such as asymmetrical digital subscriber lines (ADSLs).
Network Latency
Latency is the measure of the time for a packet to traverse the circuit path, and is usually expressed in terms of milliseconds. Each circuit, router, switch and other equipment within a circuit path increases the latency. Wide area network circuits account for the largest part of the network delay. The equipment usually adds less than a millisecond per device.
Table 1 shows typical latency values for each component. The values for applications (A and B) vary widely and are application dependent. This test scenario does not include these values, although it is recognized that network latency could have an impact on the overall performance. For example, if the latency exceeds an application retry timeout threshold, the application may request unnecessary retries, which could result in increased traffic and poorer performance overall.
From Table 1, it's clear that the wide area network links contribute the largest delay in the total latency. The network emulator applies the total latency (expect A and B) to the circuit path.
The "ping" utility measures round-trip delay, which is the total time for a request and answer to travel a circuit (i.e., out and back time). But path delays may not be symmetrical. Tools, such as the "one way ping" utility from the Internet end-to-end initiative, can be used to measure the link in one direction, but they require additional equipment and software.
ESN uses Cisco's proxy ping utility to measure round-trip delay. Infovista retrieves, stores, and trends this information, which creates the performance measuring baseline. The network emulator can apply delay to both incoming and outgoing packets to simulate asymmetrical delays. To simplify testing, identical delays will be used.
Bandwidth Delay Product
Application performance over a wide area network is limited by both latency (delay) and throughput (bandwidth). For example, the link may have low latency, but not enough capacity to support the application. Similarly, the link may have a lot of bandwidth (e.g., satellite circuits), but too much latency. (This second problem is also known as the "long, fat pipe" problem, which affects TCP performance.)
Each application may require different bandwidth delay products (BDP) (See Table 2). For example, an interactive, console-based application requires very little bandwidth and the limiting factor is the latency. As the interaction between the client and server increase, both factors may become important. Bandwidth Delay Product is measured in terms of (b/s x s) bits. In order to fully utilize a circuit, the TCP buffer should be at least the BDP.
Packet Loss
Not all packets reach their destinations. TCP can detect dropped packets and ask for a retry. However, this process slows performance. Packet loss ratio should be less than .1%. The network emulator can implement packet loss algorithms, but there are no plans to map application behavior over various packet loss ratios.
Testing Architecture
Figure 1 depicts the test architecture and is equivalent to Figure 2.
There are three zones that are referenced in the following discussions.
The following describes the components for each zone:
Zone 1: Remote Server End
- WAN Circuit: Connects remote site to the network backbone
- LAN (switches, hubs): Local area network
- Equipment: Routers, firewalls, security devices
- Application, server (computer, application): Varies greatly
- WAN: Network "cloud" that includes
- Backbone transport
- Associated WAN equipment (routers, switches, etc.)
- Based on service level agreement
- WAN Circuit: Connects remote site to the network backbone
- LAN (switches, hubs): Local area network
- Equipment: Routers, firewalls, security devices
- Application, client (computer, application): Varies greatly
Testing Methodology
Testing is straightforward, so detailed instructions are not provided. Testing the client/server performance is very subjective and should be thought of in terms of:
- Acceptable: Test does not significantly affect the application
- Marginal: Some degradation detected, but probably okay to use
- Unacceptable: Absolutely can't be deployed under these conditions
Testing
- Insert the emulator as indicated in Figure 1.
- Configure the PC for bridge mode.
- Verify client/server connectivity.
- Enable QoS (See appendix A for details).
- Run client/server tests. Repeat.
- Vary latency and bandwidth to obtain values for each cell in Table 3.
- Vary packet loss using the values .01%, .1%, 1% and 3%
Netem is used to emulate the properties of wide area networks such as delay, bandwidth restrictions, and packet loss. The code is available in recent Linux kernels; version 2.6 and higher will suffice. Netem was used to implement network delay (e.g., 10ms latency) on a laboratory PC running Fedora Core 4 Linux distribution with 100Mbps LAN ports.
To restrict bandwidth, Quality of Service (QoS) controls were implemented using the hierarchical token bucket. Extensive documentation is available on the Web at the following locations:
- http://linux-net.osdl.org/index.php/Netem - the netem wiki
- http://lartc.org/howto/lartc.qdisc.html - Linux advanced routing
- http://luxik.cdi.cz/~devik/qos/htb/ - HTB home page
The emulator was configured as a bridge to eliminate the need for changing IP addresses on the client. Queuing parameters were applied to both interfaces (eth1 and eth0) to shape both incoming and outgoing traffic. Eth0 is the upstream interface, which is connected to the wide area network. Eth1 is the downstream interface, which is connected to the client.
Commands
# Set up Ethernet 1
tc qdisc del dev eth1 root
# Use netem to implement packet delay
tc qdisc add dev eth1 root handle 1: netem delay 16ms
# Now, set up the bandwidth limits
tc qdisc add dev eth1 parent 1: handle 10: htb deault 1
tc class add dev eth1 parent 10: classid 10:1 htb rate 768kbit ceil 768kbit
# Do the same thing for Ethernet 0
tc qdisc del dev eth0 root
tc qdisc add dev eth0 root handle 1: netem delay 16ms
tc qdisc add dev eth0 parent 1: handle 10: htb deault 1
tc class add dev eth0 parent 10: classid 10:1 htb rate 128kbit ceil 128kbit
trpr - Trace Plot Real-Time
This program
calculates flow rates using tcpdump capture files and outputs the
results to a file suitable for graphing using gnuplot. Trpr was written
by the Naval Research Laboratory.
Collect data using tcpdump, then calculate the rates with trpr and display the results with gnuplot. Trpr requires the hex output. When working with a slower computer, two steps are needed: capture the binary data, then write the hex file for trpr.
tcpdump -l -n -w /tmp/test.dmp
tcpdump -n -x host 192.168.10.102 -r /tmp/test.dmp | trpr \
auto X output /tmp/plot.plt
gnuplot -persist /tmp/plot.plt
Several experiments were run to validate the performance of the emulator. The tests are by no means exhaustive, but they do show that the emulator performs as expected. Figure 3 shows the result of an iperf (bandwidth test) session over a 100Mbps LAN link, using two Pentium 400s with the Linux OS. The plot was produced using the tools described above: tcpdump, trpr, and gnuplot. The transfer rate is about 80 Mbps.
Next, two plots are made with the simulator between the sending and receiving station. The first plot (See Figure 4) shows the results of setting the emulator at 128Kbps (in/out), and the second plot (See Figure 5) shows the results for a T1 (1.54Mbps). The latency is 80ms.
Lab-grade Testing
We used a lab-grade (Spirent)
network testing device to verify operations of a PC-based emulator. The
tests showed the latency and speed emulations were acceptable for
qualitative testing. Assigned values differed slightly from measured
values. The PC used in this test was a highly portable Mini-ITX system
with 10Mbps LAN ports using a Fedora Core 4 Linux operating system.
One-way latency was measured as the "netem" latency parameters were varied from 0ms to 1000ms. The results are listed in Table 3.
Except for 0ms latency, the actual performance was about 11ms slower than the delay parameter as set. For example, the "netem" delay parameter was set to 20ms and the measured response was 30.8ms.
Throughput was measured to determine if the set queuing speed matched. Using the same test configuration as above, measurements were taken at T1 speeds and 128Kbps speeds. (Figure 6)
At T1 speed (1.544M bits per second), the measured throughput was 194 frames per second. Each frame was 1000 bytes long, which is equal to a rate of 1.552Mbps [1000 bytes X 194 frames X 8 bits/byte = 1552000 bits per second].
When the emulator was set for 128Kbps, the measured results were 15 frames per second, which corresponds to 120Kbps. (Figure 7) Again, this is probably close enough for qualitative measurements. (Packet sizes could be varied to see if finer resolution may be obtained.)
A packet sniffer captured traffic through a bridge (i.e., br0), and there seemed to be a discrepancy between the number of packets measured versus the number of packets sent by the Spirent test equipment. Ethereal (sniffer) indicated the packet was 996 bytes long, not 1000, as configured on the Spirent test equipment. The Ethernet header was 14 bytes, the IP header was 20, and the rest of the packet was 942. This is moot for the purposes of this test, but further work should be conducted to account for the "missing" 4 bytes.
References