Return to BSD News archive
Path: euryale.cc.adfa.oz.au!newshost.anu.edu.au!harbinger.cc.monash.edu.au!news.mira.net.au!vic.news.telstra.net!act.news.telstra.net!imci3!imci4!newsfeed.internetmci.com!news.kei.com!news.texas.net!news1.best.com!nntp1.best.com!shellx.best.com!not-for-mail From: rcarter@shellx.best.com (Russell Carter) Newsgroups: comp.unix.bsd.freebsd.misc Subject: Re: My recent comments on the Baker/ Lai paper at USENIX Date: 12 Apr 1996 17:31:12 -0700 Organization: Best Internet Communications Lines: 55 Message-ID: <4kmskg$f1r@shellx.best.com> References: <316999D7.167EB0E7@FreeBSD.org> <4kdoa2$nc@dyson.iquest.net> NNTP-Posting-Host: shellx.best.com In article <4kdoa2$nc@dyson.iquest.net>, John S. Dyson <root@dyson.iquest.net> wrote: >In article <316999D7.167EB0E7@FreeBSD.org>, >Jordan K. Hubbard <jkh@FreeBSD.org> wrote: >> >>I do think that benchmarking is important and that many types of useful >>"real world" data can be derived from them - our very own John Dyson >>puts significant amounts of time and effort into running benchmarks with >>Linux and other operating systems so as to "keep FreeBSD honest" in our >>own performance, looking for areas where they've made improvements which >>we have not. >> >I would like to re-iterate Jordan's comment in perhaps different words. The >primary (and probably sole) purpose of my benchmarking is indeed to make sure >that we are not falling behind anywhere. I do not normally disclose my >results except to people who ask (I have fed some info to Linus for example) or >need to know (perhaps once in a while I do post info just for general interest.) >The purpose is not marketing, and in fact the benchmark results are very >difficult to interpret. The old adage "liars figure and figures lie" describes >a situation that I try to avoid, especially since the benchmarking is part of >the feedback mechanism that I use to make sure that the kernel and shared >library support developers aren't making things worse (and hopefully are >making things better)... If I distort the results of my tests for marketing >reasons, it would destroy the advantage of doing the benchmarks for the FreeBSD >project. > >A good example of something that is sorely lacking in existing benchmark >suites is that many of them are best run during idle system conditions. >Almost NO scalability under load is measured. There are some very very >weak exceptions to this though (lat_ctx in lmbench, or the random seek >benchmark in bonnie.) This sort of testing has been done effectively in the past, but only at rather large expense. Typically three or five people work at it more or less full time for a couple of years. The amount of system time chewed up is quite impressive. About 3 orders of decimal magnitude than any PC OS is ever going to get allocated, for instance. Interestingly, I think, based on facts of life as taught to us by the Intel Orion PCI chipset group, the Stanford OS group that Jordan is so transparently sucking up to, and the sheer swelling mass of Windows NT thundering down into Sand Creek, one could deduce: performance, ultimately, doesn't matter more than a single paramecium on a dried out student's slide at the end of lab at a second rate community college in anycity USA. Later, Russell >John Dyson >dyson@freebsd.org >