Return to BSD News archive
Path: euryale.cc.adfa.oz.au!newshost.anu.edu.au!harbinger.cc.monash.edu.au!news.mira.net.au!Germany.EU.net!news.dfn.de!hookup!news.mathworks.com!uunet!not-for-mail From: kls@MAILBOX.SLAC.Stanford.EDU (Karl L. Swartz) Newsgroups: comp.unix.bsd.freebsd.misc,news.admin.technical,news.software.nntp Subject: Re: Poor performance with INN and FreeBSD. Date: 20 Mar 1996 00:52:34 -0500 Organization: Stanford Linear Accelerator Center Lines: 75 Sender: zorch@ftp.UU.NET Approved: zorch@uunet.UU.NET Distribution: world Message-ID: <DoJ1Js.H2t@unixhub.SLAC.Stanford.EDU> References: <311F8C62.4BC4@pluto.njcc.com> <4go23v$8hp@fozzie.sun3.iaf.nl> NNTP-Posting-Host: ftp.uu.net Xref: euryale.cc.adfa.oz.au comp.unix.bsd.freebsd.misc:15628 news.admin.technical:1352 news.software.nntp:20746 In article <199602292319.SAA01686@po_box.cig.mot.com>, Kevin Kadow <kadow@komondor.cig.mot.com> wrote: >I don't know- I've been contemplating rewriting the entire operating system >just for news servers- perhaps this is a market for the makers of the FASwerver >dedicated NFS fileserver to get in to... An entire news server is pretty heavy-weight, but Network Appliance, the makers of the FAServer aka toaster, is in the news market in a sense -- their machines make an excellent /var/spool/news. We're in the process of converting to such a system at SLAC, and I know of at least two other major news sites that use them in a similar role. Frankly, I was pretty skeptical of the idea of having an NFS-based /var/spool/news for my news server, figuring the access over the network would be sure death. The appeal of having an enormous filesystem was sufficiently tempting for me to at least give it a try, and I set up some tests to compare a local disk with a model 1400 "toaster." The tests consisted of a mixture of unbatching a bunch of news and deleting selected files in random order, with other activity running "off the clock." Unbatching was two to three times faster using than toaster than a local disk, while the deletes were nearly an order of magnitude faster. One place where you probably lose (I haven't tested it yet) is with INN and outgoing feeds which can keep up in real time. In this case, as long as articles stay in cache, the local disk is better, though not because of anything to do with disk performance. Hopefully, I'll have a paper with all the details at LISA this year. >Of course, I was thinking more along the lines of a log filesystem with the >overviews stored in a separate SQL-oriented database, indexed by message-id, >starting sector, and group:article#. I talked with Margo Seltzer a bit about news after one of her talks on the BSD 4.4 log-structured filesystem and she thought news would be an excellent application for LFS. I'd like to give it a try, but so far the code is still supposed to be pretty buggy and I haven't had the time to fiddle with it. I've never given the NOV part of it much thought since that's never seemed to be the bottleneck, though maybe integrating it to handle the directory portion of the FS would be interesting. >Once the filesystem has been filled, the oldest articles are >automatically overwritten as new messages are added to the spool. Nice idea, but it doesn't address varying expiration times, not to mention seasonally varying loads. >Want to add more capacity? Stripe the data in one 'newsfs' across multiple >drives. This is one of the very nice features of using a "toaster" for the job. >Need to have different expiration times for different group hierarchies? >break the spool into separate 'newsfs's by group. That's awfully coarse. What about certain subsets of hierarchies which you want to have different expiration times, perhaps because your readers care more about those particular groups? There are also varying expiration periods built into news in the form of cancelation and superseded articles (shorter expire times) and expiration dates on individual articles (potentially much longer expire times, such as for FAQs). Separate filesystems also mean you have to invest more time in system management, looking not just at how much space you expect to need overall, but at which pockets you think you'll need it in. One of the attractions to the "toaster" is that you just throw more disk at it when necessary, without worrying about where exactly it should go. With disks being cheap and reduction in staffing being plentiful, that's very attractive. -- Karl Swartz |INet kls@unixhub.slac.stanford.edu SLAC Computing Services | or kls@chicago.com 1-415/926-3630 |UUCP uunet!lll-winken!unixhub!kls -or- ditka!kls (SLAC and the US Dept. of Energy don't necessarily agree with my opinions.)