Return to BSD News archive
Path: euryale.cc.adfa.oz.au!newshost.anu.edu.au!harbinger.cc.monash.edu.au!munnari.OZ.AU!news.ecn.uoknor.edu!news.eng.convex.com!newshost.convex.com!newsgate.duke.edu!news.mathworks.com!newsfeed.internetmci.com!newsxfer2.itd.umich.edu!uunet!inXS.uu.net!nntp.inet.fi!news.funet.fi!news.helsinki.fi!news
From: torvalds@linux.cs.Helsinki.FI (Linus Torvalds)
Newsgroups: comp.os.linux.networking,comp.unix.bsd.netbsd.misc,comp.unix.bsd.freebsd.misc
Subject: Re: TCP latency
Date: 4 Jul 1996 08:25:02 +0300
Organization: A Red Hat Commercial Linux Site
Lines: 107
Message-ID: <4rfkje$am5@linux.cs.Helsinki.FI>
References: <4paedl$4bm@engnews2.Eng.Sun.COM> <4qaui4$o5k@fido.asd.sgi.com> <4qc60n$d8m@verdi.nethelp.no> <31D2F0C6.167EB0E7@inuxs.att.com>
NNTP-Posting-Host: linux.cs.helsinki.fi
Mime-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit
Xref: euryale.cc.adfa.oz.au comp.os.linux.networking:44021 comp.unix.bsd.netbsd.misc:3926 comp.unix.bsd.freebsd.misc:22836
In article <31D2F0C6.167EB0E7@inuxs.att.com>,
John S. Dyson <dyson@inuxs.att.com> wrote:
>Steinar Haug wrote:
>
>> Pentium local 250 usec
>> AMD Linux local 330 usec
>> AMD FreeBSD local 350 usec
>> AMD Linux -> Pentium 420 usec
>> AMD FreeBSD -> Pentium 520 usec
>>
>> So the difference is quite noticeable. Wish I had another P133 here to
>> test with, but unfortunately I don't.
>>
>All this TCP latency discussion is interesting, but how does this
>significantly impact performance when streaming data through the
>connection? Isn't TCP a streaming protocol?
No. TCP is a _stream_ protocol, but that doesn't mean that it is
necessarily a _streamING_ protocol.
Latency and bandwidth are both crucial, and they are almost totally
distinct (ie the correlation is by no means very strong). A high
bandwidth is important for the kinds of applications that use TCP for
just large writes and no synchronization (ftp is the obvious example),
and that's where you see the "streaming" behaviour.
But many applications don't really care about bandwith past a certain
point (they might need only a few kB/s), but latency can be supremely
important. The TCP connection might be used for some kind of interactive
protocol, where you send lots of small request/reply packets back and
forth (*).
(*) Now somebody is bound to bring up UDP, but no, UDP is _not_ a good
protocol for many of these things. If you need good performance over
wildly different network connections, and require in-order and reliable
connections, UDP sucks. You'd end up doing all the work TCP does in user
space instead, and TCP would be a lot more efficient. UDP is fine for
_certain_ applications, but if you think you should always use UDP for
request/reply, you're wrong.
Of course, a lot of applications need _both_ good throughput and low
latency. Their behaviour might even change depending on what you do. For
X, for example, you normally need low latency for good performance, but
if you're sending large bitmaps you need throughput (generally, this is
one of the advantages of TCP over UDP: it doesn't set any size limits
and it's efficient under different circumstances).
The particular benchmark here (lmbench's "lat_tcp") is just sending a
single byte ping-pong over a TCP connection, just to get the latency
number for a simple kind of connected protocol. I think Larry did the
benchmark because he noticed that this latency was one of the deciding
factors for Oracle performance..
> Data
>througput starts overshadowing connection latency quickly.
Definitely not. It depends on the application, and neither is "more
important". However, getting good throughput is usually a lot easier
than getting good latency - often you can just increase buffer sizes
(increase the TCP window, increase the MTU). Getting lower latency is a
lot harder: you have to actually fix things. That's why I personally
think latency numbers are a lot more indicative of system performance.
Oh, btw, think of memory system performance - the issues are very
similar. You obviously want good throughput ("memcpy speed") for lots
of things, but for other things latencies ("linked list speed") are more
important. The two are totally different (there is a correlation, but
it's actually pretty small, and can even be negative - you may sacrifice
latency for better throughput or vice versa).
For example, for good memory throughput you want long cache-lines and a
wide memory path (and if you want to be fancy you do readahead etc).
This is an "easy" issue, the same way TCP bandwidth is "easy": you can
just throw more hardware at it. It's not fundamentally hard to get good
throughput on SMP systems, for example.
Contrast that to low latency, which is usyally a "hard" problem: instead
of just throwing more hardware at it, you have to actually _design_ the
hardware (add a cache, or just plain get _faster_ hardware - it's
essentially a quality vs quantity thing). As a result, low latency
memory is a lot harder for SMP, for example.
Again, it depends on the application whether low latency is more
important than high bandwidth. Low latency is _critical_ for pointer
jumping (and gcc is one example of a program that does a lot of pointer
jumping), while high throughput is critical for a lot of scientific
number crunching.
>Interesting data point, but really doesn't appear to impact system
>performance much if at all.
>
>Isn't meaningless benchmarking fun!!!
Wrong. TCP latency is very important indeed. If you think otherwise,
you're probably using TCP just for ftp.
NOTE! I want to make it abundantly clear that TCP throughput is _also_
important, and that's actually what you see for a lot of "traditional"
TCP applications. And Linux gets very good throughput as well, I'm not
trying to make excuses here.
Think Quality (latency) vs Quantity (throughput). Both are important,
and depending on what you need, you may want to prioritize one or the
other (and you obviously want both, but that can sometimes be
prohibitively expensive).
Linus