Return to BSD News archive
Path: euryale.cc.adfa.oz.au!newshost.carno.net.au!harbinger.cc.monash.edu.au!news.cs.su.oz.au!metro!metro!munnari.OZ.AU!spool.mu.edu!uwm.edu!cs.utexas.edu!math.ohio-state.edu!howland.erols.net!panix!news.panix.com!not-for-mail From: tls@panix.com (Thor Lancelot Simon) Newsgroups: comp.unix.solaris,comp.unix.bsd.misc,comp.unix.internals Subject: Re: Solaris 2.6 Date: 26 Nov 1996 15:09:20 -0500 Organization: Panix Lines: 287 Distribution: inet Message-ID: <57fipg$q7j@panix2.panix.com> References: <32986299.AC7@mail.esrin.esa.it> <casper.329abb76@mail.fwi.uva.nl> <57ej3a$7ij@panix2.panix.com> <casper.329ae8f2@mail.fwi.uva.nl> Reply-To: tls@rek.tjls.com NNTP-Posting-Host: panix2.panix.com Xref: euryale.cc.adfa.oz.au comp.unix.solaris:90536 comp.unix.bsd.misc:1601 comp.unix.internals:11337 In article <casper.329ae8f2@mail.fwi.uva.nl>, Casper H.S. Dik <casper@fwi.uva.nl> wrote: >tls@panix.com (Thor Lancelot Simon) writes: > >[ I *really* hate it when people send me copies of replies thru email; > invariably this I end up replying twice ] Oh, sorry. Thought you'd notice the Newsgroups: header. >>>tls@panix.com (Thor Lancelot Simon) writes: > >>Oh? And why can't you? This works just fine under 4.4BSD, without causing >>any applications compiled for the old size to lose. See below. > >What about old .o files? Old dynamic libraries? Old archive libraries? If you link together old .o files and new .o files, you may in fact have difficulty. However, other commercial operating systems have expanded the size of other basic types without severe difficulty, notably SCO and HP/UX. > >How do you compile using those? > >You *can't* do it in the way you describe. You need new external >symbols. No, you don't. If you absolutely have to support old .o files, keep an old set of headers and an old libc around until you can rebuild all of the old applications. This works -- it's not tremendously elegant, but it works -- and it's a lot cleaner than having two exposed system calls, and requiring some new programs to know about them! There are plenty of examples out there of commercial vendors doing just exactly the above. >I know that free-unixes don't really care that much about binary compatibility >(you should only run applications you ahve source for, right?) Nonsense. You're attributing to me an attitude which I don't have, and libelling a number of whole projects besides. The 4.4BSD-derived systems have gone to great pains to ensure that not only can they run native precompiled software across OS revisions -- including the one which changed the size of off_t -- they can run precompiled software for other operating systems as well. In fact, for a long time NetBSD/sparc not only ran its own binaries and 4.4BSD/sparc binaries quite happily, it ran quite a few SunOS 4 binaries that you folks somehow didn't quite get working under SunOS 5. Still runs those, and many Solaris binaries, too. I note that while managing to sling a lot of mud around almost randomly, you unfortunately missed BSDI. They did the right thing, too, and they're not a "free-unix", whatever that means. >>As for newly compiled applications, this is *why* there are all of those >>typedefs in the header files in the first place -- so that correct code which >>uses off_t, dev_t, pid_t, &c & so forth won't lose when the type size is >>changed. > >>And no, naive code won't lose -- that's what prototypes are for. > >No, but code that uses "long" to calculate with off_t's will need to >be fixed; it won't compile or just gives a warning and an erronous >computation. Code that uses "long" to calculate with off_t's is *broken* -- but it won't lose unless lseek's argument is explicitly cast to "long", or its return value stored in a "long" -- because the header files take care of the former, and the compiler can and does warn about the latter. >You can claim that such code has a bug, but you also have the >C-standard claiming that lng is the largest integral type. And what happens to your snappy response on a machine where "long" is 64 bits? For compatibility on machines where "long" is 32 bits, you have to provide a larger type, yes. However, in case you didn't notice, the filesystem code _already_ requires this -- and has since 4.2BSD -- so since you can run UFS we can safely presume that you can do this; you have quad_t, implemented in _some_ contiguous fashion. >>>All you really need is -D_FILE_OFFSET_BITS=64 on the compile >>>command line. The new API is for internal use of libraries and include >>>files only, except for code where a distinction between the two types >>>of open's needs to be made, such as in libraries. > >>There shouldn't be two types of open, period. The 32-bit open (lseek, etc) >>should remain implemented under its old syscall number, but should vanish >>from all libraries and include files entirely. Problem solved -- *without* >>two types of open and all the concomitant lossage, not to mention inelegance. > >The dynamic libraries will need to retain the old "open" entry point for >use by old applications. No! You bump the library major number and the old applications get the old library. I'm telling you, we already wrestled this tiger -- and it turned out to be a kitten. >>Basically, you guys are ignoring plenty of preexising real-world experience >>with this issue, and doing something really dumb just because some other >>vendors are doing so. Would your development organization jump off a bridge >>if HP's did? I didn't think so. :-) > >I think you guys are being casual about supporting preexisting software >in binary form; apart from a lot of broken code being sold that still must >work after we go to large files, there's also a lot of the following >around: > > - vendors shipping archive/dynamic libraries for users > to link against (All major DB vendors, for one) Those "major DB vendors" may be a lot less dumb than you think. They can already handle 64 bit off_t at the source level -- have to, for the Alpha and various other systems -- and certainly _they_ can recompile. In fact, their installation schemes are so incredibly system-dependent that one generally has to obtain a new set of media for the database when performing an OS upgrade anyway -- for example, between Solaris/x86 2.5 and 2.5.1, I had to get new Oracle media, because 7.2.3 would no loner install. Certainly, with warning, they can just type "make". And if you really don't believe that they can, as a temporary expedient you can ship a "compatibility" set of headers and libraries for them to link against. They're already relinking during the install process, so they're about as special a case as you'll ever see -- and there are still clearly ways to handle it. > - libraries created/maintained locally > >You can't solve that without keeping the old symbols and sizes >available. Libraries created/maintained locally can be recompiled; think that one over for a moment. > >You can say, well just bump the libc.so. revision number? >Well, it wouldn't work; you'd get your users into a complex mess >requiring them to link either 32bit file only objects or >64bit file only objects. Since "open/lseek/stat" etc are already >taken the "open64" etc calls are used (invisibly) as entry points >for 64bit compiled objects. Yes, objects with different ideas of the size of basic types should not be linked together. This is pretty elementary. However, this is a pretty small fraction of the total number of precompiled programs out there, and there _are_ methods of dealing with it; if you have to, you can ship a 32-bit-offset toolchain _temporarily_ instead of gunking up interfaces and libraries _forever_. Requiring a compile-time option and gunking up the libraries with two versions of "open" is so ugly I have trouble understanding how you can have talked yourself into thinking that it could ever really be the best option -- remember, you're stuck with this from now on always! >>Mind you, the commercial vendors, as usual, didn't bother to examine actual >>experience with the research systems before going their merry way; they >>evidently preferred, as usual, to theorize about it -- incorrectly. > >I think that, as usual, free Unixes ignore an important part of >what binary compatibility consists of *for commercial unixes*. >The market is different, the requirements are different. There are commercial operating systems based on 4.4BSD. In fact, the last time I looked, despite Sun's touting itself a bigtime player in the Internet/intranet server market, it was _losing_ market share there to 4.4-based systems, although of course everyone loses market share to Linux, too. >Tell me, can you take 4.3 BSD .o files and link them to a 4.4 BSD program> No, but guess what? In the timespan between 4.3BSD and 4.4BSD, Sun did that _twice_. Tell me, can you take SunOS 3 .o files and link them to a SunOS 5 program? Same span of years. Pot-kettle-black. >>>Anotehr important requirement is that non-64 bit offset aware programs >>>should not be able to manipulate or modify files with sizes overflowing their >>>idea of off_t. You can't tell in advance whether they're gong to seek or >>>not, but we don't want them to corrupt data if they do. > >>I'm not following you here. POSIX indicates that lseek takes an off_t as an >>argument. You seem to be suggesting either that the same syscall number's to >>used for lseek() by old and new binaries, or that code which explicitly >>casts the argument of lseek to long (usually by using the constant 0L, in my >>experience) isn't broken. > >No, I'm suggesting that 32bit off_t programs may corrupt >32bit files because >of truncation that happens when seek pointers are obtained and later used. > >There's still the unsolvable problem of large fds inherited by non largefile >savvy programs. Well, yes and no. In the first place, if you do this the 4.4 way, there _is_ no such thing as a "large" fd. Now, obviously, a 32-bit program can't seek off the end of a long file; the old lseek() won't let it. The only problem you really have is with programs which explicitly do an ftruncate() without checking to see if they're at the EOF first. There aren't many of these. If you make sure you change all the system utilities, nobody's ever likely to get bit here -- almost nothing actually ever creates long files, and the applications which do do so, like databases, handle their files with their own tools, so when the database engine goes 64-bit its tools will, too. >>Again, actual experience porting hundreds of large programs to 4.4BSD >>indicates that this is essentially a nonissue -- the few erroneous casts are >>easily enough taken care of, and correct prototypes in the header files >>ensure that naive programs get lseek()'s seek argument promoted to 64 bits -- >>and anyone writing new code which explicitly casts lseek's argument to long >>should, frankly, be taken out back and shot. > >Sure, but we'd have gone to some traumatic source transformations before. Right, but it's not "before". The systems which have already changed off_t to 64 bits (and > sizeof(long)) have removed most of the opposition for you; you ought to follow the already-blazed path. >I don't believe we can sell "sure you must rebuild all your .o files and >libraries" to customers upgrading to 2.6. I think you can. What most people hated so much about the SunOS4->SunOS5 transition was that a large fraction of already-compiled, linked programs just wouldn't run any more. I think you're letting the enormity of that mistake unduly color your judgements. There's always the two-step approach that SCO seems to take to many of these kinds of issues (and they really have more expeience with binary compatibility than anyone): just change it, but ship an old set of the libraries and includes that differ, and make sure that everyone knows that these will stick around for _one_ release, except as a consulting special. In NetBSD, we've found great utility to the GNU toolchain's __warn_references extension; you might consider something similar. >>And the idea that programs should ever _need_ to be "64 bit offset aware" is >>an abomination all of its own. What happens when you want 128 bit offsets? I >>guess you go through this all *again*? > >That's really only needed for the components of the ssytem that need to >distinguish, such as the kernel and the libraries. >It's not for use in programs. It's not something application >developers have to go through (and again, and again, etc) Even exporting this difference to the libraries is a huge lose. And I can't use non-broken 32-bit programs with the new long files without explicitly performing some kind of conversion on the fd before handing it to them, right? That's dumb. You've made a mountain out of a molehill. And it's going to happen _again_ next time you resize off_t. You've just taken Unix, which had this nice consistent "one-kind-of-data: byte stream" model, and broken it so that you have a "two-kinds-of-data: byte stream and longer byte stream" model. Ick! >>*Sigh* once upon a time, Sun was one of the "good" Unix vendors, actually >>paying attention to the results which came out of research systems and acting >>accordingly. Unfortunately, these days, though Sun does plenty of good >>research themselves, they seem to suffer from severe (if typical) >>commercial-vendor NIH. > > >Nothing you said has convinced me one bit that it is possible to >redefine open() and get the backward compatibility exepected of a commercial >OS vendor. Well, you can set an arbitrarily high target if you like and at that point I won't be able to hit it. However, perhaps the above recitation of real-world experience with this issue (as opposed to untested hypothesis) may help show why I dislike Sun's position so much. I am hardly the only person who thinks so. A number of pretty darned eminent people in the Unix community seem to share my position. Binary compatibility is good. It is _not_ the _only_ good, and I suspect that if y'all hadn't blown it so badly with SunOS 5 and SunOS 4 applications, you would have a bit more perspective. You can't spend forever atoning for your past sins! >There's one simple option to recompile your code (if it doesn't >mixup off_t's and longs and if it doesn't define it's own types/ >prototypes) to get largefiles. I don't see a way to do it differently, >given the constraints commercial OS vendors operate under. I think we're going to have to agree to disagree, then. I respect your reasons, I just think you're evaluating the balance of facts incorrectly. -- Thor Lancelot Simon tls@panix.COM There's nothing like rancid bear fat to keep you happy. -Perry Metzger