The massive growth in in the installed base of public domain tools, and especially operating systems, has seen Linux acquire cult status and the reputation, in the eyes of non-Unix initiates, as being the THE public domain Unix. Whilst this is a frequent source of bemusement for the hard core of the Unix systems programming community, it masks a more interesting reality - if you are looking for professional quality public domain Unix, other alternatives to Linux do indeed exist.

While Linux has been lapping up the limelight, the family of BSD offshoots has been quietly expanding and consolidating. In this AUUG special, we will take a look at the evolution of BSD and discuss the currently available offerings, FreeBSD, NetBSD and OpenBSD.

The Taxonomy of BSD

The FreeBSD, NetBSD and OpenBSD operating systems share one attribute which is unseen in almost any other commercial or public domain operating system available today. This is an unbroken evolutionary lineage which can be traced back to the earliest days of Unix.

In practical terms this translates into a level of maturity, and thus both stability and integrity, which is almost impossible to find anywhere else. Commercial Unix has moved to variants of the much younger SVR4, and Linux is but a young pup in this sense. Why is older and more mature of such importance ? The answer is very simple - bugginess and speed. Immature software will nearly always have more bugs than mature software since it has been through a smaller number of bugfix iterations. Speed is a function of how carefully the operating system has been designed, and how many iterations of code tuning have been performed on it. Maturity translates into higher speed since the inner architecture will have been refined more carefully to eliminate bottlenecks, and those pieces of code which have been found to be bottlenecks will have been tweaked to do better.

Indeed the monstrous size and miserable performance of certain proprietary operating systems hosted on desktop machines is a direct reflection of these two realities. In a market which is driven by gadgetry to increase sales to computing novices, bugginess and woeful performance are a direct byproduct of code cobbled together to meet marketing driven deadlines first and foremost.

BSD Unix is the product of almost two decades of systematic and directed evolution, where squeezing every clock cycle out of the processor was often the top priority in the development process. It would not be unfair to say that BSD today is probably the fastest general purpose operating system in existence, with the caveat that better speed is usually only found in specialised real time systems.

Historically the genesis of BSD Unix lies in the PDP-11 period, when a group of Academics at Berkeley in California decided that the AT&T Sixth Edition (V6) Unix could be improved upon. In those days AT&T would licence Unix to Universities on very favourable terms indeed and provided source code as well. Thus 1BSD was born.

As time progressed BSD evolved. 3BSD saw the introduction of virtual memory and demand paging which were features unseen in the basic Unix offering of the period. Virtual memory allowed the programmer the freedom of not having to sweat over memory management in applications, the address space was sufficiently large to render the issue mostly irrelevant. BSD also became a testbed for other interesting ideas and tools, such as the C shell and curses. Many of the standard features and tools we know and love in Unix today started their existence as research projects for CS postgraduates at Berkeley.

The big break for BSD Unix came around 1980, when DARPA, the research arm of the US DoD was searching for an operating system which could be adopted as a standard for defence contractors and research establishments working on defence contracts. The Berkeley CSRG (Computer Systems Research Group) was funded to integrate the the TCP/IP protocol suite into BSD Unix. The resulting product was the 4BSD operating system, which introduced the now very much standard socket, and many of the basic maintenance protocols and tools we see today. The remote shell toolset, comprising rsh, rcp, rlogin, rwho and their siblings, was an offshoot of this development. The now ubiquitous NFS, developed by Sun Microsystems, also cam into existence on the 4BSD operating system.

Sun's relationship with BSD was what catapulted it into the pre-eminent position in the first generation of Unix workstation products. Adopting 4BSD as the basis of SunOS was a bold yet calculated move, aimed at capturing the University and research market which was then wedded to the PDP-11, and very fond of any product which could be tinkered with. With DARPA's drive to standardise on BSD for their new toy, the Internet, Sun simply rode on the wave of growth delivering the product then best able to plug into the embryonic open systems market.

BSD became the standard operating system used by almost every vendor, as the startups and proprietary vendors jumped on the open systems bandwagon. 4.0BSD was followed by 4.1BSD, and then 4.2BSD which was probably the most widely used variant. The 4.3BSD release saw incremental changes, mainly under the skin, and formed the basis of the last variant of SunOS, 4.1.

The Unix wars at the beginning of the nineties spelled doom for commercial large scale BSD as a standard. AT&T USL began to seriously promote their System V Unix as the would be industry standard, while then giants in the proprietary market, DEC and IBM, set out to design their new "industry standard", OSF/1. Curiously enough, OSF/1 used the Mach kernel and built up the rest of the system using a mix of proprietary features, and ideas grabbed from BSD. In the end, OSF/1 was only ever used by DEC (now Compaq), while IBM returned to their trusty AIX, itself saturated with ideas and features borrowed from BSD.

The outcome of the fruitless Unix wars was the players who remained in the market as major suppliers, Sun, HP, SGI all went the route of AT&T Unix, together with SCO, while the two then biggest proprietary vendors, IBM and DEC, stuck to their own creations. In hindsight, the Unix wars were a disaster for the industry, and damaged the market share of all of the players, since many disaffected users turned to Microsoft.

With Sun abandoning BSD as its flagship operating system, funding began to dry up for the CSRG developers. By this time Richard Stallman's Free Software Foundation had set the trend in the research community, and the very late "Net 2" release of 4.3BSD had been purged of virtually all USL code. The Net2 4.3BSD became the basis of the first Intel port, 386BSD, and led ultimately to the public domain 4.4BSD releases we see today.

The final CSRG release was 4.4BSD which incorporated a number of very advanced design features:

There were many other additional features added under the skin, and thus 4.4BSD was arguably one of the hottest operating systems of the day, from a technical perspective. These features were added on to the most mature kernel of the period, producing an unusual hybrid of very new code and very mature code.

The Unix community in the US very quickly and enthusiastically adopted 4.4BSD as the public domain "real Unix", this in a period when most Linux installations still ran on top of DOS FAT filesystems and could not be trusted to remain up for more than hours at a time.

Clearly 4.4BSD was a seen as a serious threat to the established Unix vendors, who recognised that a publicly available operating system with the pedigree of BSD was technically competitive against their variants of SVR4, especially in the bottom end of the workstation market. The real commercial threat in the marketplace, Microsoft, was ignored.

Thus began the foolish affair of the AT&T vs BSD lawsuit, which effectively paralysed public domain 4.4BSD in the critical 2 year period when it was competing head-to-head against Linux. The lawsuit saw AT&T allege the use of their proprietary source code in the design of 4.4BSD, and ended up bogging down UCB, BSDI in Colorado who were commercialising 4.4BSD, and the various vendors of 4.4BSD CD-ROM releases. In the end 4.4BSD had to be withdrawn, and re-released with various bits of the kernel rewritten, as 4.4BSD Lite, a play on Miller's Lite.

Needless to say BSD's loss was AT&T/Novell's gain, insofar as BSD's large scale entry into the public domain market was significantly slowed and the much less potent Linux gained the lion's share of installations, as many users shied away from the BSD offering for fear of having it disappear from under them. Thus the SVR4 vendor community has able to say "public domain Unix is mostly Linux which is not as robust as out products". The real consequence is that many sites who may have opted for BSD on Intel, instead opted for NT. And software vendors shied away from BSD for fear of entanglement. As has so often been the case commercial Unix vendors damaged the collective interest of the Unix community by witless infighting over market share.

Delayed into the "market", 4.4BSD has nevertheless thrived amongst the entrenched fans of classical Unix. The operating system has spawned four basic derivatives, the commercial BSD/OS, Intel hosted FreeBSD, now being ported to Alpha and SPARC, and the multi-platform NetBSD and OpenBSD.

The multi-platform NetBSD and to a lesser degree, OpenBSD, have found a very curious niche in which to exist - replacement of unsupported Unix variants on older Unix workstations. With the demise of BSD and some proprietary Unix's on a range of proprietary machines, the vendors have progressively abandoned their users. Sites like Universities, which have large inventories of older machines which are still useful as Xterminals and teaching platforms, are frequently confronted with the problems of what to do with them, once the supplied X11 server has gone "stale" and other tools fall out of sync with current standards. Often such machines have high quality fixed frequency 19", 20" or 21" monitors which are too good to throw out, but are difficult to transplant on to PC hardware, given poor support by the vendors of fixed frequency PCI graphics cards.

Retention of the existing investment is cheaper than replacement, and public domain 4.4BSD enables that. On some platforms the 4.4BSD runtime environment has been adapted to handle emulation libraries, so that native 4.3BSD binaries supplied with the original vendor operating system can be run under the 4.4BSD port. This also means that existing investments in commercial software can be retained - porting is often simply not required.

FreeBSD, the "specialised" Intel variant of 4.4BSD, has become a popular platform in the ISP community, mainly because its network protocol stack is much more efficient than that in SVR4, or Linux.

Emulation libraries for Linux and SCO mean that commercial software written for either can be run under 4.4BSD. A frequently heard anecdote is that on any given Intel box, the tool is likely to run much faster under 4.4BSD with the emulation library, than native Linux ! Judging from experience running Wolfram Mathematica, StarOffice, Acroread and other tools in their Linux incarnations, this is almost certainly true !

The three variants all offer Intel ports, and while some measure of competitiveness exists between the three groups, the relationship is also symbiotic in that pieces of source code do drift across between the three variants. Whether we see a genuine long term effort to fully standardise and merge the three remains to be seen. More than likely, we will see similar fragmentation to that seen in the Linux development community. Interestingly enough, some Linux code, especially device drivers, has also found its way into 4.4BSD, albeit very substantially rewritten due to the different kernel interfaces.

FreeBSD

FreeBSD (http://www.freebsd.org) is the most widely used and best known of the 4.4BSD variants, largely due to the projects primary focus upon the Intel PC as a platform. The history of FreeBSD can be broadly split into 3 periods.

The early period was the life cycle of the original 386BSD/4.3BSD based FreeBSD 1.0-1.1 variants. These were quite demanding to install and set up and best compared to doing an original 4.xBSD install off a Berkeley tape on to a VAX. Plenty of loose ends and a nightmare for a non BSD literate.

The " 4.4BSD Lite Release 2" period followed soon after, and saw the system very quickly mature into a package which was close to commercial quality, if not better in some areas. I have run 1.1.5, 2.0.5 and 2.2.2 on my home system and can honestly not fault 2.2.2 in any area, performance is superb, reliability exemplary and setup relatively undemanding. Tools which are not present in the comprehensive CD of precompiled binaries are usually very easy to compile up, the usual trick is to use the defaults for SunOS.

This year has finally seen the new FreeBSD 3 hit the streets in quantity as part of the "stable" series of releases, as compared to the "developmental" code releases. The aches and pains of unstable OS components are bypassed in the FreeBSD world by the maintenance of two discrete code streams, stable being intended for commercial or reliability sensitive users.

FreeBSD 3.2 is impressive, with bundled features such as a symmetric multiprocessing kernel, and Vinum Volume Manager, inspired by the Veritas LVM used by HP, and providing a number of RAID modes (RAID 5 being commercial and not bundled). Therefore it is now feasible to set up a large dual CPU Pentium with RAID or striped disks using straight-off-the-CD-ROM FreeBSD 3.2.

The currently active development projects list for FreeBSD covers several pages and is simply too big to reproduce. Interesting items include support for mounting NTFS filesystems, the Coda distributed filesystem, the Tertiary Disk project to aimed at creating huge filesystems over multiple machines, the V9FS memory filesystem using the Plan-9 model, the Drawbridge firewall, DHCP, a revised driver framework for the OS, ATM support, integrated ISDN support, the CAM SCSI interface, Token Ring support, a USB driver and many more.

Ports for the SPARC and Alpha are now under way, overlapping with the NetBSD/OpenBSD effort. An SVR4 emulator is now available, intended to run Solaris/x86 binaries. The Linux Threads under FreeBSD scheme will allow the running of applications compiled under the Linux threads environment. Embedded developers will be most interested in the PicoBSD system, aimed at tiny system configurations and bootable on an 8 MB 386SX from a 1.44MB floppy.

The PAO laptop package is also now maturing (reviewed in our recent feature on laptop Unix).

X11 support is provided with the XFree86 server, although commercial XiG or Xsuse servers are also available for it.

FreeBSD is the native platform for the Apache server, is used by Yahoo as their web server platform, and ftp.cdrom.com claim to hold the record for a sustained FTP transfer performed on their FreeBSD server. At least 1500 organisations employ FreeBSD as web servers or primary routers.

NetBSD and OpenBSD

NetBSD (http://www.netbsd.org) and OpenBSD (http://www.openbsd.org) developed in parallel with FreeBSD, and are essentially splinters of the one original project. Both are based on the 4.4BSD Lite source tree, with a range of additions. Therefore OpenBSD may be regarded to be a variant of NetBSD, maintained and distributed mostly from Canada and therefore legally exportable with strong encryption, which is not permitted from the US.

While FreeBSD focussed on the Intel 86 architecture, NetBSD from the outset aimed to be a multiplatform BSD port. At this time a very broad range of diverse machines is supported by NetBSD 1.4 and OpenBSD 2.x:

These mainstream machines are not the only ports available, since NetBSD or OpenBSD, or both, are available for 68k Amiga, arm32 Acorn RiscPC/A7000, 68k Atari TT030, Falcon, Hades, PowerPC BeBox, MIPS Sony NEWS 3400, NS 32k PC532, MIPS Acer Pica, and 68k Sharp X680x0.

A veritable menagerie of machines !

Understandably, as a volunteer effort, not all of these ports will be rock solid, or complete, so owners of older machines with oddball or less common peripherals and I/O boards may be left out in the cold, but nevertheless this is by any measure an impressive accomplishment.

I cannot help but chuckle at the thought that in a few years a bunch of part time volunteers have managed to achieve what our vendor community has never been able to do - true standardisation across a huge range of platforms.

This raises interesting issues across the board, but perhaps the most important one is basic agendas. The BSD volunteer developers aim for standardisation since they are part of the user community, and reflect the user community's interests. The vendor community have failed since their agenda has always been differentiation, to aid in marketing against similar but competitive hardware.

The Future of BSD

As with anything in the computing world, predicting the future is fraught with dangers. Can we discern a trend with BSD ? At this point in time it is clear that BSD is just beginning to pick up momentum, after a fairly slow start.

What BSD does offer is the potential for is a standard Unix across all mainstream industry architectures, offering users a truly homogeneous multi-platform computing environment, something which has eluded the user community to date. All based on a high performance, mature and robust kernel and standardised public domain toolset.

Will the vendor community cooperate or resist ? Apple's Darwin has been crafted around substantial chunks of the BSD source tree, but one vendor maketh not a trend. With the enormous investment of manhours many vendors have in existing SVR4 or other proprietary Unix variants, they are unlikely to budge unless it becomes clear that they will lose sales.

Can we expect convergence between NetBSD, OpenBSD and FreeBSD in time ? This is another excellent question. Will Linux swamp BSD out of existence, as it matures and becomes more reliable (even if its reliability already beats certain proprietary operating systems) ?

Will BSD and Linux be able to challenge the existing desktop monopoly, as they mature ? This is yet another excellent question.

As an eternal optimist I hope to one day see the "open source mammals" triumph over the "proprietary dinosaurs". The question is - what kind of comet will it take to make this happen ?