Return to BSD News archive
Newsgroups: comp.unix.bsd.freebsd.misc Path: euryale.cc.adfa.oz.au!newshost.anu.edu.au!newshost.telstra.net!act.news.telstra.net!vic.news.telstra.net!news.mira.net.au!news.vbc.net!alpha.sky.net!news.sprintlink.net!news.up.net!news.mtu.edu!newsxfer.itd.umich.edu!news.mathworks.com!gatech!swrinde!howland.reston.ans.net!vixen.cso.uiuc.edu!uwm.edu!news.inap.net!news1!not-for-mail From: root@dyson.iquest.net (John S. Dyson) Subject: Re: My recent comments on the Baker/ Lai paper at USENIX X-Nntp-Posting-Host: dyson.iquest.net Message-ID: <4kdoa2$nc@dyson.iquest.net> Sender: news@iquest.net (News Admin) Organization: John S. Dyson's Machine References: <316999D7.167EB0E7@FreeBSD.org> Date: Tue, 9 Apr 1996 13:22:10 GMT Lines: 51 In article <316999D7.167EB0E7@FreeBSD.org>, Jordan K. Hubbard <jkh@FreeBSD.org> wrote: > >I do think that benchmarking is important and that many types of useful >"real world" data can be derived from them - our very own John Dyson >puts significant amounts of time and effort into running benchmarks with >Linux and other operating systems so as to "keep FreeBSD honest" in our >own performance, looking for areas where they've made improvements which >we have not. > I would like to re-iterate Jordan's comment in perhaps different words. The primary (and probably sole) purpose of my benchmarking is indeed to make sure that we are not falling behind anywhere. I do not normally disclose my results except to people who ask (I have fed some info to Linus for example) or need to know (perhaps once in a while I do post info just for general interest.) The purpose is not marketing, and in fact the benchmark results are very difficult to interpret. The old adage "liars figure and figures lie" describes a situation that I try to avoid, especially since the benchmarking is part of the feedback mechanism that I use to make sure that the kernel and shared library support developers aren't making things worse (and hopefully are making things better)... If I distort the results of my tests for marketing reasons, it would destroy the advantage of doing the benchmarks for the FreeBSD project. A good example of something that is sorely lacking in existing benchmark suites is that many of them are best run during idle system conditions. Almost NO scalability under load is measured. There are some very very weak exceptions to this though (lat_ctx in lmbench, or the random seek benchmark in bonnie.) BTW, if anyone working on FreeBSD would like to submit ideas for macro-level performance testing, let me know!!! I will try to add them. I generally keep things quiet (for reasons that I have tried to describe above), but IMO these measurements are an important part of the QC procedure on FreeBSD. I don't have an organized, up-to-date set of results right now (I have been comparing FreeBSD vs. NetBSD lately -- so I can't boot Linux today), but my recent measurments show that FreeBSD is not falling behind in performance. A cute example of how bogus benchmarks misinform -- the lat_mmap benchmark for FreeBSD vs. Linux shows that FreeBSD is very very much slower than Linux... Until one realizes that the cost of the FreeBSD mmap is very quickly made up by the almost total elimination of pagefaults after the mapping (esp. in the case of read-only segments, like .text). There is an aggregate performance increase by the tradeoff that we made. The above is one reason that the adage previously mentioned applies to some interpretations of certain benchmark results. John Dyson dyson@freebsd.org