IPERF for Windows

To test available bandwidth on the circuit we need two computers, running IPERF for Windows on both ends. One computer acts a a server, listening for incoming packets; another one works as a client, connecting to the server and generating test traffic.

To measure maximum bandwidth, it's better to use UDP instead of TCP, to avoid TCP's intelligence with windowing, back off, and retransmissions.  IPERF UDP client just blasts the maximum traffic you order, and the circuit either carries or drops it. IPERF UDP server reports the total bandwidth of the traffic received, lost packets, and jitter.

It's better to begin testing for maximum throughput from very low target bandwidth (-b key), to ensure you get zero packet loss. Then you gradually increase target bandwidth, watching packet loss and CPU usage. On powerful computers with high quality gigabit network cards, ~900Mbits/sec throughput is possible!

There is a -w (window) parameter, which is in reality just the buffer size. Increasing it does not increase the amount of generated traffic. What actually increases it is the -l (packet length/size) parameter, which can be up to 64kbytes (the maximum size of a UDP packet.) In reality, numbers higher than 65500 just hang the program. Of course, 64k packets don't get transmitted on your network directly; if it's ethernet, they'll be fragmented to 1470 bytes.

To ensure that the circuit doesn't override the DSCP value you assigned, run a packet sniffer like Wireshark. It seriously affects performance, so you should first check that DSCP values are okay under light load, and then run full-load tests.

Start IPERF in server mode, say on IP address 10.0.0.1:

iperf.exe -s -u -p 5123 -i 5 -l 65500 -q 48

where:

-s	means server mode
-u	use UDP
-p 5123	listen on port 5123 instead of default 5001 (any large port number can be used, if it's open on firewalls)
-i 5	print a report every 5 seconds (more frequent ones aren't very representative)
-l 65500	use close to maximum UDP packet size (65536) to try to maximize throughput
-q 48	assign DSCP = 48 to server-to-client packets

IPERF server starts and waits for incoming traffic.

Then start IPERF in client mode:

iperf.exe -c 10.0.0.1 -u -p 5123 -b 5000m -i 5 -t 20 -l 65500 -q 46

where:

-c 10.0.0.1	means client mode, connecting to server 10.0.0.1
-u	use UDP
-p 5123	transmit to port 5123 instead of default 5001, it should match server port above
-b 5000m	target bandwidth to attempt is 5Gbit/sec (start low, say 100m, then increase until you get packet loss)
-i 5	print a report every 5 sec (match server setting, to avoid confusion)
-t 20	run for 20 seconds (with reports every 5 seconds)
-l 65500	use close to maximum UDP packet size (65536) to try to maximize throughput
-q 46	assign DSCP = 46 to client-to-server packets

and watch IPERF output together with Windows Task Manager's "Network" tab. The bandwidth used should match. By default, Task Manager summs sent and received traffic together, but can be configured to show them separately.

When your circuit is close to saturation, it is possible for some communication between server and client to get lost. When this happens, sometimes no final server report gets printed on client. That's normal behaviour, and is difficult to avoid since inter-IPERF communication occurs over data path being stress tested. One way around lost reports, is to run a Windows Terminal Session to the remote computer. It persists through heavy transient network loads, and will show you IPERF results on the remote machine. (That's how the movies on the home page were shot.)

Hide ^

IPERF can generate multicast traffic. In the simplest configuration one computer is the sender, and multiple others are receivers of multicast traffic. Although it's possible to run this with just one receiver, you need two receivers to make it interesting. With multiple receivers, IPERF and Task Manager will show equal traffic on the sender and each of the receivers, which is impossible with unicast.

To be a receiver of multicast traffic, IPERF has to be in server mode. To generate the multicast traffic, IPERF has to be in client mode.

Start IPERF as a receiver, in server mode, subscribed to the multicast address 224.1.1.1:

iperf.exe -s -u -B 224.1.1.1 -i 5 -l 65500 -q 48

where:

-s	means server mode
-u	use UDP
-B 224.1.1.1	listen to, or subscribe to, the multicast group 224.1.1.1
-i 5	print a report every 5 seconds (more frequent ones aren't very representative)
-l 65500	use close to maximum UDP packet size ( 65536) to try to maximize throughput
-q 48	assign DSCP = 48 to server-to-client packets

IPERF server starts and waits for incoming traffic.

Then start IPERF as a sender, in client mode, transmitting to the multicast address 224.1.1.1:

iperf.exe -c 224.1.1.1 -u -T 1 -b 5000m -i 5 -t 20 -l 65500 -q 46

where:

-c 10.0.0.1	means client, transmitting or publishing to the multicast group 224.1.1.1
-u	use UDP
-T 1	number of hops (routers) for multicast traffic to cross (Caution: do not put a large number here on production networks!)
-b 5000m	target bandwidth to try is 5Gbit/sec (start from, say 100m, then increase until get packet loss)
-i 5	print a report every 5 sec (match server setting, to avoid confusion)
-t 20	run for 20 seconds (with reports every 5 seconds)
-l 65500	use close to maximum UDP packet size ( 65536) to try to maximize throughput
-q 46	assign DSCP = 46 to client-to-server packets

and watch IPERF output together with Windows Task Manager's "Network" tab. Repeat tests with different DSCP values to see the difference, if any.

Hide ^

If your provider supports IPv6 addressing, that circuit can be tested by IPERF using IPv6 addresses. Configure computers on both ends of the circuit with IPv6 addreses (see instructions below), ping each other, then run IPERF.

Start IPERF in server mode, listening on IPv6 addresses:

iperf.exe -s -u -p 5123 -i 5 -l 65500 -V

where:

-s	means server mode
-u	use UDP
-p 5123	listen on port 5123 instead of default 5001 (any large port number can be used, if it's open on firewalls)
-i 5	print report every 5 sec (more often is not very representative)
-l 65500	use close to maximum UDP packet size ( 65536) to try to maximize throughput
-V	use IPv6 addresses

IPERF server starts and waits for incoming traffic.

Then start IPERF in client mode:

iperf.exe -c 3ffe:1900:4545:3:200:f8ff:fe21:67ce -V -u -p 5123 -b 5000m -i 5 -t 20 -l 65500

where:

-c <address>	connect to and send traffic to IPv6 address 3ffe:1900:4545:3:200:f8ff:fe21:67ce
-V	use IPv6 addresses
-u	use UDP
-p 5123	transmit to port 5123 instead of default 5001, it should match server port above
-b 5000m	target bandwidth to try is 5Gbit/sec (start low, at say 100m, then increase until you get packet loss)
-i 5	print a report every 5 sec (match server setting, to avoid confusion)
-t 20	run for 20 sec (with reports every 5 sec)
-l 65500	use close to maximum UDP packet size ( 65536) to try to maximize throughput

and watch IPERF output together with Windows Task Manager's "Network" tab. 

Hide ^

There is always the question: how to test circuit for maximum bandwidth - with UDP or TCP traffic? They are very different, and IPERF implements them differently. As it was mentioned above for UDP, -l parameter (buffer size) can not be larger 64k - the maximum size of UDP packet. Well, for TCP it works differently. Running IPERF TCP test with default parameters between two computers which Gigabit cards are connected back to back, gave only 35Mbits/sec... Increasing -l parameter improved throughput significantly:

On the left we have remote desktop of Windows 7, running IPERF in server mode, on the right Windows XP with IPERF client. It looks like for this circuit the value 550Mbits/sec is close to the maximum. Maybe it's because TCP window is large: 2.5M? Trying without it:

Using TCP window either default 8Kbytes or 2.5Mbytes doesn't make a difference at all - with large IPERF buffer size. Again - these results are valid only for IPERF implementation on Windows. There is a well-known fact, that to get better perforance on decent circuits it's worth to increase TCP window. Maybe it's not the case with IPERF on Gigabit speed, where -l parameter makes real difference instead.
There is another well-known fact, that to get real circuit performance on TCP, someone should use multiple TCP sessions. Even when testing Internet privider circuit with FTP file upload, it's worth to try running multiple FTP sessions at once. IPERF provides similar functionality with -P parameter:

Apparently in this test, two TCP streams didn't push higher bandwidth, probably because circuit is already working to its maximum. At lower speed, multiple TCP streams really help to feel what the circuit is capable for.

Thus, with high-bandwidth TCP tests with IPERF, not only TCP window size but also buffer size and number of streams should be used.

Hide ^

To enable IPERF QoS setting, the Windows operating systems of the machines must allow it. This is accomplished with the following Registry editing. Without it IPERF tests will be running with DSCP value 0 no matter what you assign in -q parameter! This key

HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\DisableUserTOSSetting = 0

of type REG_DWORD must be created and computer rebooted. This is how it looks like in regedit:

You can also copy the following lines into a text file, name it "IPERF WinXP registry fix.cmd" and run it:

reg.exe add HKLM\SYSTEM\CURRENTCONTROLSET\SERVICES\TCPIP\PARAMETERS /v DisableUserTOSSetting /t REG_DWORD /d 0 /f
@echo+
@echo Done. Reboot the computer now.
@echo+
@pause

To be certain, check with Wireshark that packets are really being assigned the DSCP value you want.

Hide ^

To enable IPERF QoS settings in Windows 7, more steps are required.
First of all, the same DisableUserTOSSetting key as in Windows XP must be configured:

Secondly, Windows 7 QoS settings must be allowed by domain policies. To disable that, additional REG_SZ key must be configured:

HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\QoS\"Do not use NLA"="1"

Note that it is of REG_SZ type, not REG_DWORD:

You can copy the following lines into a text file, name it "IPERF Win7 registry fix.cmd" and run it:

reg.exe add HKLM\SYSTEM\CURRENTCONTROLSET\SERVICES\TCPIP\QoS /f
reg.exe add HKLM\SYSTEM\CURRENTCONTROLSET\SERVICES\TCPIP\QoS /v "Do not use NLA" /t REG_SZ /d "1" /f
reg.exe add HKLM\SYSTEM\CURRENTCONTROLSET\SERVICES\TCPIP\PARAMETERS /v DisableUserTOSSetting /t REG_DWORD /d 0 /f
@echo+
@echo Done. Reboot the computer now.
@echo+
@pause

Thus, you escape the domain policy... The third and last step, is to override the local group policy.

Bear in mind that it is not available in Windows 7 Home edition. Start Microsoft Management Console (click Start, in Start Search box type mmc, then double-click mmc program found). Console Root is probably empty. On the File menu, click Add/Remove Snap-in. In the Add or Remove Snap-ins dialog box, click Group Policy Object Editor, and then click Add:

 Now in the newly added Local Computer Policy, under Windows Settings, find Policy-Based QoS. Right-click on it and choose Advanced QoS Settings. In DSCP Marking Override tab, check Control DSCP marking... checkbox and choose Allowed to allow applications to set DSCP values independently of QoS policies:

We're almost done, reboot just in case. Wireshark will show that this setting doesn't work - packets are still sent with DSCP = 0 not matter what IPERF is trying to set.

We need to configure IPERF-specific policy. Right-click Policy-based QoS and choose Create new policy. Call it what you wish, for example IPERF QoS Policy. Specify the DSCP value you want to assign:

Insert the full path to the IPERF executable:

Specify any source or destination IP addresses:

Set protocols to both TCP and UDP:

and now new IPERF policy is completed:

Wireshark should now show it working, without a reboot!

Hide ^

By default, an IPv6 stack is not installed in Windows XP. In Windows 7, the network adapter already has "Internet Protocol Version 6", which Properties you can open and configure IPv6 address:

In Windows XP the network adapter has only "Internet Protocol" which is v4. To install IPv6, run this command in the command line:

netsh int ipv6 install

This will enable IPv6, and you can see it in network adapter properties:

However, the button to configure an IPv6 address is disabled... So it must be configured via command line:

netsh int ipv6 set address interface="LanGig" address=3ffe:1900:4545:3:200:f8ff:fe21:67cf

where "LanGig" is the name of interface you see in "ipconfig" output. Then, check that an IPv6 address is really configured:

netsh int ipv6 show address interface="LanGig"
Interface 6: LanGig
Unicast Address : 3ffe:1900:4545:3:200:f8ff:fe21:67cf
Scope : Global

or in classic way:

ipconfig
Windows IP Configuration
Ethernet adapter LanGig:
Connection-specific DNS Suffix . :
IP Address. . . . . . . . . . . . : 10.0.0.2
Subnet Mask . . . . . . . . . . . : 255.255.255.0
IP Address. . . . . . . . . . . . : 3ffe:1900:4545:3:200:f8ff:fe21:67cf
IP Address. . . . . . . . . . . . : fe80::211:d8ff:fe29:a5f6%6

As well, check if you can ping another IPv6 computer (Windows 7 in this case):

ping 3ffe:1900:4545:3:200:f8ff:fe21:67ce
Pinging 3ffe:1900:4545:3:200:f8ff:fe21:67ce with 32 bytes of data:
Reply from 3ffe:1900:4545:3:200:f8ff:fe21:67ce: time<1ms
Reply from 3ffe:1900:4545:3:200:f8ff:fe21:67ce: time<1ms
Reply from 3ffe:1900:4545:3:200:f8ff:fe21:67ce: time<1ms
Reply from 3ffe:1900:4545:3:200:f8ff:fe21:67ce: time<1ms
Ping statistics for 3ffe:1900:4545:3:200:f8ff:fe21:67ce:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 0ms, Average = 0ms

That Windows 7 machine has this IPv6 address configured:

ipconfig
Windows IP Configuration
Ethernet adapter LanGig:
IPv6 Address. . . . . . . . . . . : 3ffe:1900:4545:3:200:f8ff:fe21:67ce
Link-local IPv6 Address . . . . . : fe80::9d63:4580:a415:d3f8%11
IPv4 Address. . . . . . . . . . . : 10.0.0.1
Subnet Mask . . . . . . . . . . . : 255.255.255.0

and can ping the Windows XP IPv6 address:

ping 3ffe:1900:4545:3:200:f8ff:fe21:67cf
Pinging 3ffe:1900:4545:3:200:f8ff:fe21:67cf with 32 bytes of data:
Reply from 3ffe:1900:4545:3:200:f8ff:fe21:67cf: time<1ms
Reply from 3ffe:1900:4545:3:200:f8ff:fe21:67cf: time<1ms
Reply from 3ffe:1900:4545:3:200:f8ff:fe21:67cf: time<1ms
Reply from 3ffe:1900:4545:3:200:f8ff:fe21:67cf: time<1ms
Ping statistics for 3ffe:1900:4545:3:200:f8ff:fe21:67cf:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 0ms, Average = 0ms

so everything works with IPv6... Well, almost. Windows XP Task Manager does not show IPv6 traffic, but rest assured that's just a bug, and traffic is actually hitting the network.

Hide ^