Return to BSD News archive
Newsgroups: news.software.nntp,comp.unix.bsd.freebsd.misc Path: euryale.cc.adfa.oz.au!olive.mil.adfa.oz.au!navmat.navy.gov.au!posgate.acis.com.au!warrane.connect.com.au!news.syd.connect.com.au!news.mel.connect.com.au!harbinger.cc.monash.edu.au!news.mira.net.au!Germany.EU.net!howland.reston.ans.net!gatech!newsfeed.internetmci.com!in2.uu.net!twwells!bill From: bill@twwells.com (T. William Wells) Subject: Poor performance with INN and FreeBSD. Organization: None, Mt. Laurel, NJ Message-ID: <DMu8B6.6Jn@twwells.com> References: <311F8C62.4BC4@pluto.njcc.com> Date: Thu, 15 Feb 1996 22:11:29 GMT Lines: 120 Xref: euryale.cc.adfa.oz.au news.software.nntp:19900 comp.unix.bsd.freebsd.misc:13822 I have a FreeBSD2.1 box running INN (and related processes) and our primary nameserver and nothing else; it has the following: 100 MHz Pentium 64M RAM PCI Adaptec 2940 SCSI controller Fujitsu M1603-S (system disk) 2 x Conner CFP4207S (spool disks) Digital DC21041 ethernet This ought to be able to keep up with our two incoming full feeds, two outgoing partial feeds, and at most a dozen nnrp processes. Heck, *half* this machine should be able to do that! It doesn't. I put in a bunch of timing code into the server. Here are the results. All the times I mention are averages over hourly periods, except where they are averages over the entire data set. Here's a typical transaction: uunet sends ihave network transmits ihave 20 ms inn receives ihave inn check for article 11 ms inn sends 335 network transmits 335 20 ms uunet receives 335 uunet retrieves article ? uunet starts sending article network transmits start of article 20 ms network transmits article 400 ms inn receives end of article inn processes article 315 ms inn sends 235 network transmits 235 20 ms uunet receives 235 uunet retrieves next id to send ? I think I have, somewhere, the ? times, but I'm not going to try to dig them out; for my purposes, their exact values don't matter. Guessing them each as 11ms (essentially, one disk seek time), the total time comes to 828 ms. (You may be wondering about the 400ms article time. Well, if you're getting the binary groups, the average article is like 12K. Uunet manages to transmit the data to us at about 30K/sec....) I think you can see where I'm heading....but just for completeess, with those numbers, I can expect only 1.2 articles per second. If the 20ms times are replaced with the 50ms times that are the estimate from comparing real numbers (specifically, uunet's measured ihave-to-335 times are 110 ms, my measured check time is 11ms, the 99ms difference gets split into two network transits) as opposed to ping times, we're talking 1.05 articles/second. (And the uunet processing times are underestimates...by quite a lot.) I have a second feed, so I get almost twice that. In fact, my measured speed varies from 1.5 to 1.9, though I haven't a clue as to what's causing the variance...network load I'd expect. Thus: if you have any sort of network latency you are totally, absolutely fucked (obscenity courtesy of the CDA). If your system takes any appreciable amount of time to process the article you're fucked anally, with a sulfuric acid lubricant. Even if your system manages *zero* processing time, you still are only going to get 2 articles per second. If you are lucky. The bottom line is that network latency is going to kill you and no matter how you beef up your system, you just aren't going to be able to keep up unless you get rid of the network latency. Multiple feeds will help. Some. You end up overlapping some of the network delays on the multiple feeds. But even with the two feeds I have, I'm still not keeping up....so that's not the answer, either. Other scenarios: I could get a faster link. In theory, I get better than double the speed from my sprintlink connection. That's because their news machine happens to be just the other side of my link to them. And the average ihave-ihave time reflects this. The uunet time is 1052ms; the sprintlink time is 566ms. Reasonable, given that the 4 round trip times are much smaller and the bulk transmission rate is much higher. There are three sets of times involved here. There's time at uunet, which I'm generously estimating at 22ms. There's local processing time, of 326ms. There's network time, for uunet either 480ms or 600ms depending on which set of numbers you believe. The numbers for sprintlink would be 210 or 263 ms. No idea, of course, what the sprintlink processing time is, but I'd bet uunet's is better...still, for this swag, I'll use the small number. 210 ms plus 326 ms plus 22 ms....558ms/article. No *way* that's going to keep up with today's full newsfeed. Never mind tomorrow's. However, at this point, the big number is the local processing time...and that's subject to local fixes. 315ms *is* excessive. I could damned near file them manually in that time. :-) So no matter how you look at it, "things have to change". You are going to have to have a feed with low latency and INN is going to have to process articles faster. The former, I understand, is addressed by the streaming mode for NNTP. The latter.... I just started working on this. However, I've just spotted a *really* obvious place for optimization. The very, very largest times in my kernel traces are when returning from an open call and that's the time from the namei to the return. Thus, if you have directories that have, for whatever reason, become very large, you are going to spend a lot of time doing opens. Alas, I've done just that -- my junk directory is a quarter megabyte and I file a fair amount under it. Time to shrink that directory and keep it small.... Now, I have a question for the FreeBSD gurus out there -- I set the fragment size to 512 bytes in order to cut down on disk space waste. I have this sinking feeling that this also ups my directory read time. What do you think?