Mailing List virtualized_ecs_users@2rosenthals.com Archived Message #71

From: "Lewis G Rosenthal" <virtualized_ecs_users@2rosenthals.com> Full Headers
Undecoded message
Subject: Re: [Virtualized eCS] TAP driver killing TCP/IP bandwidth
Date: Fri, 11 Dec 2009 21:23:28 -0500
To: Virtualized eCS Users Mailing List <virtualized_ecs_users@2rosenthals.com>

Hi, Doug...

[long post warning: this message contains a number of stats in the body]

On 12/10/09 04:55 pm, Doug Bissett thus wrote :
On 2009-12-09, at 18:28:29, Lewis G Rosenthal wrote:
  
Hi, Doug...
    

Hi Lewis...

  
On 12/09/09 05:35 pm, Doug Bissett thus wrote :
    
On 2009-12-09, at 14:01:33, Lewis G Rosenthal wrote:
  
...snip...
  
      
I'll have to fashion myself a CAT-6 crossover to eliminate the switch from the equation.)
            
Most modern switches, and NICs, will automatically reverse the wiring,
if they think you might be using the wrong cable. Just try a normal
cable.

  
      
Indeed, I thought of the auto-sensing circuitry, but don't know if the T43 or the new Intel chip (still trying to get GenMAC to work with it) - an 82567LF-2 - do auto-sense. The switch surely does, which is why I no longer stock crossover cables. :-)
    

I did my tests with a standard cable (probably CAT5) that is about 5
feet long. No trouble with not using a crossover cable.

  
I was able to successfully connect a CAT-6 straight through between a workstation and the ThinkPad. The interesting discovery I made was that even though I have power management at a bare minimum, evidently, while on battery power, my wired NIC is throttled down. The best I could get (no TAP driver) was about 480Mbps on battery. Plugged into AC, however, I was able to eek out >830Mbps.

There was only a minor difference when going through the switch (actually, a well functioning switch *should* improve overall performance - and indeed, with the NDIS2 driver, it really did for me), however, the TAP driver was a killer. With an active VM, it got even worse (see below)...
Has anyone else seen this type of performance hit cuased by the TAP driver?
            
No, but my switch, and router, only run at 100 Mbs, so I may not have
noticed a speed reduction. I will see if I can hook it direct from my
desktop to my T43, and see what happens (may be tomorrow, before I get
a chance).
  
      
Thanks. I'm going to test on a couple other networks and systems available to me, as well. I, too, am normally sitting behind a 100Mbps secondary switch on this segment of the LAN or going wireless. However, the native B57 NDIS2 driver dropped off to under 500bps (that's bits, not Bytes!) vs GenMAC running the NDIS5 driver (without IJFW and IPX on the wire).

    
My tests using the B57 driver were just as abysmal under AC as they were under battery power (no TAP driver loaded). That said, on the ThinkPad I got similar results to what I saw on the server, so I suspect it is just a poorly written device driver base from Broadcom (different versions, but similar lack of optimization). Does anyone have an NDIS2 driver running at close to 1Gbps?

<snip>
Okay, first some information:
Test boxes:
 Asus M3A78-EM motherboard, using GENMAC with a Realtek 8168 NIC 10EC:8168.
 IBM ThinkPad T43 (1871-W8M), using GENMAC with a Broadcom NIC
14E4:167D. This machine also has an Intel wireless  8086:4224, which
was radio off for the testing.

Both systems are running eCS 2.0 Silver, using NETBIOS over TCP/IP. The
T43 doesn't have the TAP driver installed. The M3A78-EM did not have
VBOX installed when I started, and the T43 has never had it installed.
Both systems also have Virtual PC installed, with the bridge driver. I
never actually ran VBOX on the M3A78-EM, before doing the tests. I am
using NETIO126.

  
<snip>

I tested my T43 running eCS 1.2R against an openSuSE 11.2 box. The Linux machine has an Intel DP45SG motherboard, 2GB DDR3 1333MHz DRAM, and on onboard Intel 82567LF-2 ethernet controller (whcih I cannot - yet - get to run under GenMAC...grrrrr...).

My PROTOCOL.INI has a bit less than yours (see notes, below).
Test results:
=======================
Test 1: No TAP$ on either side.
TCP connection established.
Packet size  1k bytes:  8558 KByte/s Tx,  8507 KByte/s Rx.
Packet size  2k bytes:  11468 KByte/s Tx,  11505 KByte/s Rx.
Packet size  4k bytes:  11478 KByte/s Tx,  11378 KByte/s Rx.
Packet size  8k bytes:  11483 KByte/s Tx,  11505 KByte/s Rx.
Packet size 16k bytes:  11483 KByte/s Tx,  11161 KByte/s Rx.
Packet size 32k bytes:  11486 KByte/s Tx,  11380 KByte/s Rx.

NetBIOS connection established.
Packet size  1k bytes:  2664 KByte/s Tx,  3664 KByte/s Rx.
Packet size  2k bytes:  8592 KByte/s Tx,  11090 KByte/s Rx.
Packet size  4k bytes:  11252 KByte/s Tx,  11130 KByte/s Rx.
Packet size  8k bytes:  11313 KByte/s Tx,  11341 KByte/s Rx.
Packet size 16k bytes:  11397 KByte/s Tx,  11399 KByte/s Rx.
Packet size 32k bytes:  11425 KByte/s Tx,  11432 KByte/s Rx.

UDP connection established.
Packet size  1k bytes:  9204 Byte/s (99%) Tx,  8752 KByte/s (0%) Rx.
Packet size  2k bytes:  148 KByte/s (99%) Tx,  8994 KByte/s (0%) Rx.
Packet size  4k bytes:  58843 Byte/s (99%) Tx,  0 Byte/s (100%) Rx.
Packet size  8k bytes:  0 Byte/s (100%) Tx,  0 Byte/s (100%) Rx.
Packet size 16k bytes:  0 Byte/s (100%) Tx,  0 Byte/s (100%) Rx.
Packet size 32k bytes:  0 Byte/s (100%) Tx,  0 Byte/s (100%) Rx.


Test 2: Tap$ on client side (M3A78-EM) only.
TCP connection established.
Packet size  1k bytes:  7762 KByte/s Tx,  8236 KByte/s Rx.
Packet size  2k bytes:  7881 KByte/s Tx,  8653 KByte/s Rx.
Packet size  4k bytes:  7959 KByte/s Tx,  9176 KByte/s Rx.
Packet size  8k bytes:  7997 KByte/s Tx,  10743 KByte/s Rx.
Packet size 16k bytes:  7954 KByte/s Tx,  11393 KByte/s Rx.
Packet size 32k bytes:  7980 KByte/s Tx,  11400 KByte/s Rx.

NetBIOS connection established.
Packet size  1k bytes:  2705 KByte/s Tx,  3731 KByte/s Rx.
Packet size  2k bytes:  6244 KByte/s Tx,  7519 KByte/s Rx.
Packet size  4k bytes:  6527 KByte/s Tx,  8182 KByte/s Rx.
Packet size  8k bytes:  7325 KByte/s Tx,  10064 KByte/s Rx.
Packet size 16k bytes:  7836 KByte/s Tx,  11270 KByte/s Rx.
Packet size 32k bytes:  7904 KByte/s Tx,  11174 KByte/s Rx.

UDP connection established.
Packet size  1k bytes:  7656 KByte/s (7%) Tx,  4532 Byte/s (99%) Rx.
Packet size  2k bytes:  8328 KByte/s (0%) Tx,  12619 Byte/s (99%) Rx.
Packet size  4k bytes:  11039 KByte/s (0%) Tx,  66999 Byte/s (99%) Rx.
Packet size  8k bytes:  11097 KByte/s (0%) Tx,  62 KByte/s (99%) Rx.
Packet size 16k bytes:  0 Byte/s (100%) Tx,  0 Byte/s (100%) Rx.
Packet size 32k bytes: send(): No buffer space available
===============================

For some reason, UDP seems to have a problem, so I think you can
discount what that test says, in both cases.

The really interesting numbers are in TCP, and NETBIOS, with and
without the TAP driver installed. Sending seems to be limited to what I
see with my 100 Mbs hub in the network. With a direct connection, and
no TAP driver, send and receive seem to be about the same. The
interesting thing is that receiving doesn't seem to be affected by
whatever is slowing down sending.

  
If you used a CAT-5 cable vs a CAT-5e or a CAT-6, this might account for the results. Also, without locking the NICs down to 100Mbps, and with no 100Mbps switch in between, they might be trying to shift up to 1000Mbps, which will likely fail over a CAT-5 connection, causing them to resend and/or renegotiate. It's hard to tell without seeing the actual packets.

I didn't test UDP; only TCP:


Drivers loaded:  B57.OS2
Power:           battery
Media:           CAT-6 cable (straight through)

TCP connection established.
Packet size  1k bytes:  18315 KByte/s Tx,  18109 KByte/s Rx.
Packet size  2k bytes:  22313 KByte/s Tx,  19018 KByte/s Rx.
Packet size  4k bytes:  24580 KByte/s Tx,  18875 KByte/s Rx.
Packet size  8k bytes:  25565 KByte/s Tx,  18839 KByte/s Rx.
Packet size 16k bytes:  26026 KByte/s Tx,  19284 KByte/s Rx.
Packet size 32k bytes:  25903 KByte/s Tx,  19293 KByte/s Rx.
Done.

25903 KByte/s Tx,  19293 KByte/s Rx = 202Mbps up / 151Mbps down
------------------------------------------------------------

Protocols bound: TCP/IP, IPX
Drivers loaded:  GenMAC (10.35.0.0), IJFW
Power:           battery
Media:           CAT-6 cable (straight through)

TCP connection established.
Packet size  1k bytes:  47224 KByte/s Tx,  41324 KByte/s Rx.
Packet size  2k bytes:  55751 KByte/s Tx,  42746 KByte/s Rx.
Packet size  4k bytes:  58283 KByte/s Tx,  42976 KByte/s Rx.
Packet size  8k bytes:  61433 KByte/s Tx,  43984 KByte/s Rx.
Packet size 16k bytes:  60853 KByte/s Tx,  43926 KByte/s Rx.
Packet size 32k bytes:  61342 KByte/s Tx,  44411 KByte/s Rx.
Done.

61342 KByte/s Tx,  44411 KByte/s Rx = 479Mbps up / 347Mbps down
------------------------------------------------------------

Protocols bound: TCP/IP, IPX
Drivers loaded:  GenMAC (10.35.0.0), IJFW
Power:           AC
Media:           CAT-6 cable (straight through)

TCP connection established.
Packet size  1k bytes:  104552 KByte/s Tx,  111982 KByte/s Rx.
Packet size  2k bytes:  105074 KByte/s Tx,  109605 KByte/s Rx.
Packet size  4k bytes:  105488 KByte/s Tx,  112969 KByte/s Rx.
Packet size  8k bytes:  103970 KByte/s Tx,  110719 KByte/s Rx.
Packet size 16k bytes:  104466 KByte/s Tx,  113184 KByte/s Rx.
Packet size 32k bytes:  106235 KByte/s Tx,  112873 KByte/s Rx.
Done.

106235 KByte/s Tx,  112873 KByte/s Rx = 830Mbps up / 882Mbps down
------------------------------------------------------------

Protocols bound: TCP/IP
Drivers loaded:  B57.OS2
Power:           AC
Media:           10/100/1000 switch

TCP connection established.
Packet size  1k bytes:  50032 KByte/s Tx,  41183 KByte/s Rx.
Packet size  2k bytes:  55746 KByte/s Tx,  42862 KByte/s Rx.
Packet size  4k bytes:  57041 KByte/s Tx,  43749 KByte/s Rx.
Packet size  8k bytes:  59871 KByte/s Tx,  44213 KByte/s Rx.
Packet size 16k bytes:  60384 KByte/s Tx,  43990 KByte/s Rx.
Packet size 32k bytes:  60622 KByte/s Tx,  43979 KByte/s Rx.
Done.

60622 KByte/s Tx,  43979 KByte/s Rx = 474Mbps up / 344Mbps down
------------------------------------------------------------

Protocols bound: TCP/IP
Drivers loaded:  B57.OS2
Power:           AC
Media:           CAT-6 cable (straight through)

TCP connection established.
Packet size  1k bytes:  21829 KByte/s Tx,  20908 KByte/s Rx.
Packet size  2k bytes:  23709 KByte/s Tx,  20911 KByte/s Rx.
Packet size  4k bytes:  25073 KByte/s Tx,  20914 KByte/s Rx.
Packet size  8k bytes:  25576 KByte/s Tx,  21054 KByte/s Rx.
Packet size 16k bytes:  25126 KByte/s Tx,  20862 KByte/s Rx.
Packet size 32k bytes:  25012 KByte/s Tx,  20982 KByte/s Rx.
Done.

25012 KByte/s Tx,  20982 KByte/s Rx = 195Mbps up / 164Mbps down
------------------------------------------------------------

Protocols bound: TCP/IP, IPX
Drivers loaded:  GenMAC (8.39.1.0.A), IJFW
Power:           AC
Media:           10/100/1000 switch

TCP connection established.
Packet size  1k bytes:  86129 KByte/s Tx,  100503 KByte/s Rx.
Packet size  2k bytes:  91500 KByte/s Tx,  103650 KByte/s Rx.
Packet size  4k bytes:  94792 KByte/s Tx,  113484 KByte/s Rx.
Packet size  8k bytes:  101372 KByte/s Tx,  113454 KByte/s Rx.
Packet size 16k bytes:  106365 KByte/s Tx,  112734 KByte/s Rx.
Packet size 32k bytes:  108754 KByte/s Tx,  113398 KByte/s Rx.
Done.

108754 KByte/s Tx,  113398 KByte/s Rx = 850Mbps up / 886Mbps down
------------------------------------------------------------

Protocols bound: TCP/IP, IPX
Drivers loaded:  GenMAC (10.35.0.0), IJFW
Power:           AC
Media:           10/100/1000 switch

TCP connection established.
Packet size  1k bytes:  94540 KByte/s Tx,  106908 KByte/s Rx.
Packet size  2k bytes:  107416 KByte/s Tx,  113212 KByte/s Rx.
Packet size  4k bytes:  107070 KByte/s Tx,  112550 KByte/s Rx.
Packet size  8k bytes:  106078 KByte/s Tx,  112666 KByte/s Rx.
Packet size 16k bytes:  107330 KByte/s Tx,  112944 KByte/s Rx.
Packet size 32k bytes:  107954 KByte/s Tx,  112284 KByte/s Rx.
Done.

107954 KByte/s Tx,  112284 KByte/s Rx = 843Mbps up / 877Mbps down
------------------------------------------------------------

Protocols bound: TCP/IP, IPX
Drivers loaded:  GenMAC (10.35.0.0), IJFW, TAP
Power:           AC
Media:           10/100/1000 switch

TCP connection established.
Packet size  1k bytes:  10390 KByte/s Tx,  9279 KByte/s Rx.
Packet size  2k bytes:  11263 KByte/s Tx,  9401 KByte/s Rx.
Packet size  4k bytes:  12361 KByte/s Tx,  9364 KByte/s Rx.
Packet size  8k bytes:  12856 KByte/s Tx,  9267 KByte/s Rx.
Packet size 16k bytes:  13199 KByte/s Tx,  9464 KByte/s Rx.
Packet size 32k bytes:  13205 KByte/s Tx,  9463 KByte/s Rx.
Done.

13205 KByte/s Tx,  9463 KByte/s Rx = 103Mbps up / 74Mbps down
------------------------------------------------------------

Protocols bound: TCP/IP, IPX
Drivers loaded:  GenMAC (10.35.0.0), IJFW, TAP
Power:           AC
Media:           10/100/1000 switch
VBox (QT):       Win XP guest; 512MB; PCnet-FAST III

TCP connection established.
Packet size  1k bytes:  1400 KByte/s Tx,  4979 KByte/s Rx.
Packet size  2k bytes:  3722 KByte/s Tx,  2073 KByte/s Rx.
Packet size  4k bytes:  6438 KByte/s Tx,  4187 KByte/s Rx.
Packet size  8k bytes:  7597 KByte/s Tx,  4318 KByte/s Rx.
Packet size 16k bytes:  7840 KByte/s Tx,  5765 KByte/s Rx.
Packet size 32k bytes:  7893 KByte/s Tx,  5790 KByte/s Rx.
Done.

7893 KByte/s Tx,  5790 KByte/s Rx = 62Mbps up / 45Mbps down
------------------------------------------------------------

Protocols bound: TCP/IP, IPX
Drivers loaded:  GenMAC (10.35.0.0), IJFW, TAP
Power:           AC
Media:           10/100/1000 switch
VBox (QT):       Win XP guest; 512MB; PCnet-FAST III (disabled)

TCP connection established.
Packet size  1k bytes:  6149 KByte/s Tx,  5710 KByte/s Rx.
Packet size  2k bytes:  6342 KByte/s Tx,  5780 KByte/s Rx.
Packet size  4k bytes:  7286 KByte/s Tx,  5812 KByte/s Rx.
Packet size  8k bytes:  7557 KByte/s Tx,  5764 KByte/s Rx.
Packet size 16k bytes:  7826 KByte/s Tx,  5472 KByte/s Rx.
Packet size 32k bytes:  7877 KByte/s Tx,  5727 KByte/s Rx.
Done.

7877 KByte/s Tx,  5727 KByte/s Rx = 62Mbps up / 45Mbps down
------------------------------------------------------------

Win XP Guest to eCS 1.2 host:

Pass 1:

TCP connection established.
Packet size  1k bytes:  3542 KByte/s Tx,  342 KByte/s Rx.
Packet size  2k bytes:  2002 KByte/s Tx,  1149 KByte/s Rx.
Packet size  4k bytes:  4509 KByte/s Tx,  57535 Byte/s Rx.
Packet size  8k bytes:  4515 KByte/s Tx,  1247 KByte/s Rx.
Packet size 16k bytes:  6101 KByte/s Tx,  3546 KByte/s Rx.
Packet size 32k bytes:  6727 KByte/s Tx,  2162 KByte/s Rx.
Done.

Best up: 32k @ 6727 KBs = 53Mbps
Best down: 16k @ 3546 KBs = 28Mbps

Pass 2:

TCP connection established.
Packet size  1k bytes:  3575 KByte/s Tx,  614 KByte/s Rx.
Packet size  2k bytes:  3705 KByte/s Tx,  852 KByte/s Rx.
Packet size  4k bytes:  3862 KByte/s Tx,  559 KByte/s Rx.
Packet size  8k bytes:  4154 KByte/s Tx,  472 KByte/s Rx.
Packet size 16k bytes:  5219 KByte/s Tx,  3177 KByte/s Rx.
Packet size 32k bytes:  6735 KByte/s Tx,  2903 KByte/s Rx.
Done.

Best up: 32k @ 6735 KBs = 53Mbps
Best down: 16k @ 3177 KBs = 25Mbps

----------------------------------------------------------

Win XP Guest to openSuSE 11.2 remote:

Pass 1:

TCP connection established.
Packet size  1k bytes:  2459 KByte/s Tx,  3510 KByte/s Rx.
Packet size  2k bytes:  3784 KByte/s Tx,  3338 KByte/s Rx.
Packet size  4k bytes:  4114 KByte/s Tx,  3556 KByte/s Rx.
Packet size  8k bytes:  4329 KByte/s Tx,  3496 KByte/s Rx.
Packet size 16k bytes:  4139 KByte/s Tx,  3569 KByte/s Rx.
Packet size 32k bytes:  4389 KByte/s Tx,  3615 KByte/s Rx.
Done.

Best up: 32k @ 4389 KBs = 34Mbps
Best down: 32k @ 3615 KBs = 28Mbps

Pass 2:

TCP connection established.
Packet size  1k bytes:  2459 KByte/s Tx,  3500 KByte/s Rx.
Packet size  2k bytes:  4010 KByte/s Tx,  3561 KByte/s Rx.
Packet size  4k bytes:  3996 KByte/s Tx,  3525 KByte/s Rx.
Packet size  8k bytes:  4335 KByte/s Tx,  3543 KByte/s Rx.
Packet size 16k bytes:  4105 KByte/s Tx,  3578 KByte/s Rx.
Packet size 32k bytes:  4343 KByte/s Tx,  3618 KByte/s Rx.
Done.

Best up: 32k @ 4343 KBs = 34Mbps
Best down: 32k @ 3618 KBs = 28Mbps
----------------------------------------------------------

Ugh... Perhaps some RAM tuning for the guest might tweak the numbers a bit; I didn't try. Even though I have 2GB in the T43, I can't take the guest up to 1GB, due to memory fragmentation and lack of memory in the shared arena. I would expect, however, that slightly more RAM devoted to the host might improve performance here, rather than vice-versa.

As an aside, I discovered that the MPTS in eCS 2.0 Silver has added a
feature to install LLAeCS.EXE into \MPTN\BIN\MPTSTART.CMD, every time
it runs (even if you don't change anything). This took me on a wild
goose chase for a while, since LLAeCS.EXE hangs at boot, unless I start
it much later than in MPTSTART.CMD.

  
For the record, I *hate* that M$ LLA nonsense. If I have no static address and no DHCP server, then by g-d, I don't want a fake IP address. If I get a bunch of machines together to chatter amongst themselves, they're either statically assigned or one of them is running DHCP. IMO LLA is more trouble than it's worth, and a real hassle for our stack to deal with, on top of everything else.
This is somewhat limited testing, but I think that it shows that the
TAP driver does limit send speed to something near 100 Mbs, even when
using a gigabit NIC. In different configurations, the results could be
different.

  
Note mine... :-(
I never tried wireless, but I will do that, when I find some time to
play with it again.
  
Good luck getting the TAP driver to bind to the wireless NIC. The trick which Andy showed me (and this was under VPC, before we had VBox) was to keep it bound to the wired NIC, and enter a static route to allow traffic to bounce back and forth off of the wired interface to the wireless. To get this right, you really need to "think like a packet," as a friend of mine likes to say. Also, this arrangement obviously doesn't work in a bridged environment (all of my tests were bridged). We might want to try a NAT configuration, I guess...

Cheers/2, and thanks for testing this with me. Much appreciated.

--
Lewis
-------------------------------------------------------------
Lewis G Rosenthal, CNA, CLP, CLE
Rosenthal & Rosenthal, LLC                www.2rosenthals.com
Need a managed Wi-Fi hotspot?                www.hautspot.com
Secure, stable, operating system          www.ecomstation.com
-------------------------------------------------------------


Subscribe: Feed, Digest, Index.
Unsubscribe
Mail to ListMaster