Return to BSD News archive
Path: euryale.cc.adfa.oz.au!newshost.carno.net.au!harbinger.cc.monash.edu.au!munnari.OZ.AU!news.Hawaii.Edu!news.uoregon.edu!hunter.premier.net!www.nntp.primenet.com!nntp.primenet.com!howland.erols.net!newsfeed.internetmci.com!news-in2.uu.net!twwells!twwells!not-for-mail From: bill@twwells.com (T. William Wells) Newsgroups: comp.unix.bsd.freebsd.misc Subject: Re: FreeBSD as news-server?? Date: 20 Oct 1996 17:28:06 -0400 Organization: None, Mt. Laurel, NJ Lines: 104 Message-ID: <54e5h6$p6@twwells.com> References: <537ddl$3cc@amd40.wecs.org> <544nas$b5h@flash.noc.best.net> <5462it$r37@twwells.com> <5467p6$bl4@flash.noc.best.net> NNTP-Posting-Host: twwells.com In article <5467p6$bl4@flash.noc.best.net>, Matthew Dillon <dillon@best.com> wrote: : :>Even if you do this, I think the the nonstreaming feeds will : :>still get crunched. Innd processes the stuff from each stream all : :>at once, so this introduces latency in all the other feeds. The : :>is a *real* problem for nonstreaming feeds because any latency : :>above the network latency directly slows the nonstreaming feed. : :>(This is, in fact, the problem that streaming was invented to : :>solve....) : : I can see the logic to that, but I think reasonable history : file caching pretty much fixes the problem. Not so. The reason is that once you start introducing processing latency from other channels that is greater than the network latency on the nonstreaming channel, the average latency on the nonstreaming channel goes up. Do the math yourself. Or run a simulation. Be that as it may, this is an observed phenomenon.... : :>: :>Alas, this is only true if your feeds are all so close to "real : :>: :>time" that things remain in the cache. Otherwise, caching doesn't : :>: :>do anything for you. (In my system, I solve this problem with a : :>: :>message id daemon, which eliminates most redundant history : :>: :>lookups.) : :>: : :>: You can cache a *lot* of history file. Sure, the cache will not be : :>: as optimal, but it will still be there, and it will be a disk read : :>: rather then a file create. : :> : :>Well maybe. There's an awful lot of disk activity going on on a : :>news server and most of it isn't in the history file. This one, I : :>suspect, isn't going to be answered without tools that can : :>examine the buffering directly. : : The history file has its own partition. And, as I said, you : need memory for filesystem caches. But if you set it up right, : there isn't a problem. Buffer caching doesn't give a damn about partitions. Lots of reader activity pretty much guarantees that there's little history in the cache. (This is why, on a machine that doesn't do swapping, putting the history index into innd improves its performance relative to nnrps and other processes.) : overchan should never get behind. If it does, one has other : problems to deal with it. I've run two busy news machines for : two years now. Overchan has not ever, not once gotten behind. : The rest of the system could be dying and overchan still wouldn't : get behind. Repeating a thing doesn't make it relevant. And, as for experience, I've been running news systems since before there was C news or INN. In my twenty odd years of programming experience, I've *written* operating systems, device drivers, disk management software, and so on. I'll thank you to keep to the facts as opposed to repeating your personal experience as if it were the whole of reality. On *your* system overchan might never get behind. On other peoples' with different disks, memory, CPU, user loads, etc., overchan does get behind. And the correct, documented, solution is to move overchan into its own directories. : Also, keep in mind my original comment... I said that if you : had three or more spool disks, that you would not have to : separate the overview files. I rather doubt this, for those systems where it matters at all. Splitting the disks lowers the *relative* disk activity -- but if you're doing that much spool, you've got a lot *more* disk activity, in the form of directory entry creation, happening on those disks. (Never mind that on such a system, you also have reader disk activity on the articles themselves to contend with.) Another factor to consider: when the directories are being updated by inn, they are *locked*. Nothing can access them during those periods. So, the high activity groups spend a fair amount of time with the directory locked, introducing delays into overchan (and other) processing. Hm. Now I think about it, I suspect your using the B flag is also relevant, precisely because of that locking. That, plus fast disks, is probably why you aren't having any problem with overchan. : :>OK, then *you* tell *me* what EAGAIN from fork means. :-) When I : :>checked the kernel code, it looked like nothing short of a swap : :>shortage would cause it. (Well, running out of slots for child : :>processes could, too, but I don't think that's the case here. : :>Eveen at peak times, I've still got about 50% leeway in my process : :>slots before I hit the limit.) : : I think you jumped to conclusions as to what exactly was being : duplicated. Well, no. Actually, what I ASSumed, since I'd covered all the other resource exhaustion possibilities in fork, was that even though I didn't believe it was allocating swap, it was anyway. I've a different opinion now: on another machine I'm running, I'm getting approximately daily crashes. (Not hangs -- processes that just use CPU keep running, disk activity continues, and I can break into ddb.) Everything points to a VM bug of some sort. Time will tell, as I do my debugging.