Hello,
I have recently written an updated version of my Linux 2.2 article which
lists the features and whatnot of Linux 2.4. (Or rather, Linux 2.3. But
just try and explain that to many journalists...) I would very much
appreciate your input, as deveolopers, to the writing process. I'm not
as in the loop as I used to be, unfortunately. If anyone can provide any
input, I would appreciate it muchly.
Now, the last time I did this, I got flames when I sent the file to the
list and flames when I didn't. It's a lose-lose situation for me, but I
did nicely preface this whole thing with "offtopic" so I hope that too
many people whp pay by the byte don't try and download it.
I'm particularly lacking in input from the sound, video4linux, scsi, and
network side of things. But if you know of any major feature that I left
out, don't hesitate to let me know. Just be nice about it. :) Also, if
you would respond just to me or trim off the real article in any
replaies, it would save a lot of people bandwidth.
Thanks so much for any input that you might have,
Joe
--
Wonderful World of Linux 2.4 (7/31/99) "Making it right II"
A long time ago, in a galaxy not to far from this one, I wrote an
article.
It wasn't much, just a laundry list of the new and improved features of
Linux 2.2. That was a long time ago and the newness of Linux 2.2 is
beginning to wear off; it's even lost that new kernel smell. And so with
anxious eyes, we set our eyes towards the future of Linux: Linux 2.4.
When
is it due? When it is ready. (But between you and me, we're shooting on
sometime in late fall.)
Before we move into the meat and bones of the situation, I always like
to
step back for a moment and say a few things. Linux 2.2 marked a major
milestone in Linux's evolution: it was the first kernel release which
truly
caught the eye of the all-seeing media. No longer would just obscure
computer publications trumpet the coming of the open source revolution,
but
mainstream publications as well. With the attention has come the
inevitable
FUD (Fear, Uncertainty, and Doubt) that we prayed would never come, but
out
of this media molasses we have emerged as a stronger community. While
many
of Linux's weak points were trumpeted across the sky and many members of
the Linux community despaired, there were still the legions of kernel
hackers that worked day and night to take those weaknesses, acknowledge
them, and work to fix them. I think they've done wonderful job.
As a side effect of the increased media attention, many members of the
media chose to use my original article as a guide, a reference to the
new
changes and features in Linux 2.2 in the absence of any official press
release. To be honest, I was flattered. I do ask however that you cite
the
source whenever possible (both myself and the community who made this
document possible) and if you require any clarifications, please do not
hesitate to email me at the address below with your questions. If you
would
like to reprint or translate this article in full or in part in any
medium,
please email me at the address below so that I can be aware of your work
before you do so. I will make a list of translations that I am aware of
available.
Now, without further ado, Linux 2.4. This draft of the WWoL2.4 is
subtitled
the "making it right edition" and will be definitely followed by a
number
of later drafts, at least 3 of which I will term "final". We all have
different notions of what changes are important, if I don't mention a
particular update that you feel is important, by all means please let me
know.
Joe - [email protected]
< Many thanks to linuxtoday for providing me with the spiffy email addy.
>
--
The Heart - Linux Internals
Linux 2.2 was a great improvement over Linux 2.0. It supported many new
file systems, it supported a completely rethought caching system for
file
names, and it was much more scalable than Linux 2.0. Linux 2.4 will
build
on the great advancements provided under Linux 2.2 to become an even
better
platform for desktop, server, and embedded tasks. However, it is the
intent
of the Linux kernel developers to get Linux 2.4 into the hands of the
end
users much more quickly than Linux 2.2 did. To meet this goal, Linux 2.4
will understandably not be as different from Linux 2.2 as Linux 2.2 was
from Linux 2.0. I think you will agree however that the advancements in
Linux 2.4 will be just as noteworthy as previous versions. (Or else I
wouldn't need to be writing this!)
What, at the core, is Linux? Linux is much more than just a collection
of
assorted device drivers, as any operating system must certainly be. It's
what binds these drivers together into a cohesive unit that matters.
It's
the scheduler, the resource allocator, the virtual filesystem layer,
memory
management, and so many other unsung features that are the real heroes
of
the Linux world. These are the portions of the Linux operating system
that
really define what is Linux because on every platform that Linux has
been
ported to from i386 (Intel-compatible PC), to ARM (embedded devices), to
Sparc64 (high-end servers) this code is the same. In many ways, the
"heart"
of Linux 2.4 is different than Linux 2.2's and most of the subsystems
that
I just listed have been changed in one way or another.
Linux 2.2 and earlier Linux's included a base resource management system
which was used rather bluntly to allocate and keep track of IO ports and
IRQ lines and the other limited niceties of computer architectures.
Unfortunately it was deficient in a number of important ways which are
crucial to modern desktop operating systems: it lacked the ability to
cleanly handle nested resources (where we handle one device's memory as
a
subset of another, for example a PCI bus), it included a hardcoded list
of
resource types, and other deficiencies. In the interests of better
resource
management, the old code was scrapped in favor of new code which
resolved
the issues that I just described. While it has not been put to this use
yet, it has been suggested that the new resource allocator code may put
us
one step closer to real PnP support in the kernel. (For a more detailed
discussion of PnP, please check the section on buses, below.)
The Linux model of network sockets is one which is standard across most
UNIX variants. Unfortunately however, the standard does have some
deficiencies but these deficiencies can be corrected without breaking
the
standard altogether. Under Linux 2.2 and previous versions, if you have
a
number of processes all waiting on an event from a network socket (a web
server, for instance), they will all be woken up when activity if
detected.
So, for every web page request received, Linux would wake up a number of
processes which would each try and get at the request. As it does not
make
sense for multiple processes to serve the same request, only one will
get
to the data; the remainder will notice that it doesn't have anything to
process and fall back asleep. Linux is quite efficient at making this
all
happen as quickly as possible, however it is still very inefficient...
but
there is a better way. Linux 2.4 includes changes which implement "wake
one" under Linux which will allow us to completely remove the "stampede
effect". In short, "wake one" does exactly as its name indicates: wakes
up
only one process in the case of activity. This will allow applications
such
as Apache to be even more efficient and make Linux an even better choice
as
a web server.
Linux includes other changes that will make it a better choice for
higher-end systems in particular. Linux 2.2 scaled to multiple
processors
much better than did Linux 2.0 (the first version of Linux to support
multiple-processors at all.) This was primarily achieved by removing a
global kernel lock and replacing it with finer granularity locks which
would allow more work to be done on multiple processors at once. Linux
2.4
takes this advancement even further as many subsystems' locks have been
replaced or rethought to allow finer control. In fact, several key
subsystems including TCP/IP (IPv4) and the virtual filesystem layer
(VFS)
have been completely rewritten with the intent to provide more
scalability.
8-way scalability under Linux is now a reality! (Linux can still scale
to
as many as 64 processors on some high level machines, however certain
kernel activities would have to have complete control of the machine.
What
these changes allow us to do is spend considerably less time in routines
that require complete control and let the rest of the system go faster.)
The virtual filesystem layer, as previously mentioned, has been
optimized
for multi-processing, but that is not the only change in that layer.
Linux
2.2 featured a number of wonderful changes to the filesystem layer to
support better caching, however it still used a split buffer system that
kept the reading and writing buffer separate. Linux 2.4 brings this wall
down with a complete rewrite of the page caching layer leading to a
subsequent reduction in overall memory needs and removing a number of
cases
where Linux had to jump through hoops to keep the two caches in synch.
Overall, this is a more efficient system allowing for many operations,
including synching, to happen much more rapidly than could be managed
previously. There were other changes to made to this layer in the
process,
including the removal of race conditions (when multiple processes can
manage to interfere with each other because they "race" for access to
variables), the speeding up of large writes to multiple disks at once,
and
the dynamic allocation of file descriptors (which allow more files to be
open at once, in some cases.) These changes led to some necessary evils
(described in the filesystem section) but overall I feel that this is a
total win for Linux.
One common problem with Linux 2.2 that interfered with high-end (Intel?)
machines was its process limitations. Linux 2.2 only allowed you to have
1024 processes or threads running at once. With high-end systems with
many
thousands of users, this could become a problem very quickly. Linux 2.4
has
gotten rid of this relic and implemented a scalable limit which can be
configured at run time. The number of simultaneous processes is limited
of
course to the amount of RAM you have in the system. On high-end servers
with as little as half a gigabyte of RAM installed, it is easily
possible
to support as many as 16 thousand processes at once. Other users have
reported being able to run many more than that on their specific
systems.
This was one of the major bottlenecks that kept Linux out of the
Enterprise
markets.
In terms of memory consumption, Linux 2.4 will require approximately the
same amount of memory as does Linux 2.2. Some subsystems have been added
or
expanded and some have been streamlined. Some obsolete code has been
removed. There are even certain cases where some systems will require
less
memory than Linux 2.2!
Processors and Ports
While infrastructure is the heart of the Linux operating system, it is
the
parts of the operating system that are specific to the individual
flavors
of Linux that are most obvious to the end users. These "arms and legs"
of
the Linux operating system include all of the architecture dependent and
independent driver code which controls the processors, disk drives,
ports,
and everything else that provides real access to the computer. For the
purposes of this article, I will be concentrating on i386 Linux which is
the Linux that I use most often at home. That is not to say that there
have
not been great advancements in other areas, however they are beyond my
direct experience. If there is enough demand, and someone is interested
in
helping, I may be willing to write appendixes which describe some of the
big improvements in other areas. There have been no new ports entered
into
mainstream Linux since Linux 2.2 although this may change.
On Intel-compatible hardware, Linux 2.4 includes the same excellent
support
for processors as did Linux 2.2. This includes optimizations for 386,
486,
586 (Pentium), and 686 (Pentium Pro / Pentium II / Pentium III)
processors, as well as "compatible" counterparts such as those made by
AMD
and Cyrix. Additionally, Linux 2.4 will include additional support for
hardware present with modern chips. While Linux 2.2 includes support for
Intel's Memory Type Range Registers (MTRRs) to raise performance to some
kinds of high-bandwidth devices, Linux 2.4 has taken this support even
further by supporting variants common to the compatible chips. This
includes the double MTRRs present with AMD K7 processors and MCR variant
preferred by Cyrix. Linux 2.2 also included support for the IO-APIC
which
allowed interrupts to be spread across multiple processors in a
multi-processing system. Linux 2.4 will, as expected, take this to the
next
level and support some high-end systems which actually contain multiple
IO-APIC controllers; this will increase performance on these machines.
To my knowledge, the only multi-processor systems which we still do not
completely support are some very old 486 ones that mix 486DX and 486SX
chips in the same system. This is mainly because the SX chips did not
contain math coprocessors and there is some difficulty in making sure
that
applications that need floating point math get to work on the right one.
As
you can imagine, there isn't much call for this feature.
In terms of what types of programs it can run, Linux 2.4 is very much
the
same as Linux 2.2. The only notable exception to this rule is that
support
has been removed to run Java applications directly, it has been replaced
by
a more modular system which allows for arbitrary associations, not
unlike
Windows's own associations. This change is not new to Linux 2.4, however
you will have to configure this before your Java applications will work.
The removal of this feature will save memory on some configurations,
albeit
not much.
Buses - ISA, PCI, USB, MCA, etc.
Processes however are just a small part of the guts of a computer.
Equally
important to its operation is its bus architecture because it is this
component of the system that is responsible for (or irresponsible
towards,
as the case may be) internal devices. Linux 2.4 has not yet touched much
on
the internal workings of many of the supported busses, including (E)ISA,
VLB, PCI, and MCA except to begin to work them into the new resource
management subsystem. Plug-and-Play, the somewhat misguided attempt to
support device configuration and detection on the ISA bus, is still not
supported at the kernel level. However, PnP devices can still be used
somewhat easily by configuring them from userspace with the isapnp
utilities package available with nearly every distribution. Once
configured, PnP devices can be loaded using modules just like any other
device.
There is exciting news from this front however. Universal Serial Bus, a
new
bus type just now coming into prominence for devices such as keyboards,
mice, sound systems, and scanners is now supported in the Linux kernel.
At
the time of this writing, the support is not 100% and many individual
and
common USB devices are not supported or not completely supported. I
would
be confident however that the number of devices which are supported will
only rise over time, just as we observed a similar rise in the number of
framebuffer devices that are now supported. (The framebuffer was a new
feature to Linux 2.2, see below.) Currently, keyboards and mice are
working
mostly as you would expect. Support for sound systems is coming along
rapidly. Other devices, such as modems and network cards, already have
preliminary support however their drivers are not complete.
In addition to USB, I2O device (Intelligent Input/Output) support, an
extension of PCI, has been added in Linux 2.4. In theory, this will
allow
for more operating system independent devices and drivers to exist. Many
I2O devices are already functioning and more will be added before Linux
2.4.
PCMCIA support is still not provided in the kernel as shipped by Linus.
However the code is generally available and provided built-in to most
distributions. This is done by mutual agreement.
Block Devices - Disk Drives, RAID Controllers, etc.
Now, just having a bus and a processor does not' make a computer any
more
than a skeleton and heart would make any other organism. To spread this
analogy even further into the realm of not actually being applicable,
you
need a brain. No, not to think with, that's handled in our heart.
(Imagine
for a moment that we still subscribe to the classical theory of
anatomy.)
Instead, we need a brain to hold our data and actually do meaningful
things.
Linux 2.2 already supported the most common types of storage media
including RAID controllers, IDE and SCSI disks, and many other
advancements. Linux 2.4 has built on this by providing more of the same:
more drivers for more cards and more stability for the ones that already
exist. Linux 2.4 has also made it easier for larger systems at this
level
by doubling the number of IDE controllers supported internally in the
kernel to 8. If you have one of these beasts, you will now be able to
use
all of its interfaces. There have also been some kernel changes to allow
for better support of IDE CD changers although support for having
multiple
CDs in the changer mounted at once is not provided. (For obvious
reasons:
the latency on a multi-user machine would be terrible!) And finally,
Linux
2.4 is much better at identifying buggy or special IDE hardware and
enabling workarounds or speed enhancements.
SCSI has received less advancements in Linux 2.4, except for the
touching
down of a number of new SCSI controller drivers.
It's long been traditional for Linux developers to take ideas and
concepts
that "make sense" from one operating system and apply it to Linux.
Rarely
however do these ideas emerge onto the Linux scene without an incredible
amount of thought and digestion, usually resulting in a better final
idea.
One particular idea was adapted from mainstream UNIX: raw device i/o,
the
ability to access a low-level block device directly and without any
intervening cache layer. Few doubt that this idea had its merits,
however
the traditional implementation of this idea (the creation of "raw"
devices
corresponding to each and every block device) was unacceptable to Linux
developers. Eventually, a solution was developed that even Linus could
agree on: the ability to associate a raw character device with any
arbitrary block device without having to preallocate many unnecessary
device nodes. While applications to this system are not obvious to the
casual desktop user, this system allows greater control to applications
that want it, including and especially database applications which are
smart enough to do their own disk management..
Filesystems
Having a block device is only really useful if you can have some data on
it. But having data is useless unless it is in some prescribed format
which
allows for convenient storage and retrieval. By this point, we have
completely broken my analogy. Oh well. It should also be mentioned that
these filesystems (as well as nearly everything else) will work with all
versions of Linux and are not only applicable to i386 Linux.
Linux 2.4 includes all of the new filesystems present in Linux 2.2.
These
filesystems include FAT (for MSDOS), NTFS (for Windows NT/2000), VFAT
and
FAT32 (for Windows 9x), HFS (for Macintoshes), and many, many others.
All
of these filesystems have been rewritten to some extent to support the
new
page caching system and will be more efficient because of it. On the
flip
side however, binary-only filesystem modules designed for Linux 2.2 will
not work with Linux 2.4. (Unlike some software firms, Linux does not
generally provide for back-compatibility at the module level. Generally,
open source modules can adapt quickly enough and binary module providers
are expected to do the same or release the code.)
Some users will however notice major improvements to allow for better
compatibility with other systems. OS/2 users will finally be able to
both
read and write to their disks under Linux. (This change is a long time
in
coming.) Linux 2.4 will also include a couple of improvements designed
to
make it interoperate better with other UNIX-like operating systems. Key
to
this is Linux 2.4's upcoming support for the IRIX efs filesystem and the
IRIX disklabel (partition table) format. Also, support for NextStep has
also improved as the UFS driver now supports its CDROMs.
Users who mount Windows shared drives via SMB (Server Message Block
protocol) will be pleased that there will no longer be a compile time
option for enabling workarounds for (released broken) Win9x systems.
Instead, Linux will be able to detect what kind of system it is
connecting
to and enable bug fixes as needed. This will make Linux a considerably
better option for heterogeneous networks.
Of special importance to many Linux users is Linux's ability to mount
the
shared drives of UNIX operating systems. Linux 2.4 includes for the
first
time the ability to access NFS shares which conform to version 3 of the
NFS
protocol. NFS version 3 includes many advantages over previous versions
and
it has been one of Linux's most often requested features.
There are still some pieces of support that is currently lacking in
Linux
2.4. There is no support for journalizing filesystems, for instance. Due
to
the relatively low fsck times and the ease of data recovery journalizing
filesystems support, this is considered by many to be an entrance
requirement to the enterprise. It is hoped that this and other "missing"
features will be completed before 2.4 is ready for release.
Input Devices - Keyboards and Mice
Now, storage is important and the bus is important, but if you can't do
anything with the machine yourself then it really is not any use to you.
To
desktop users, the keyboard and mouse are the input devices of choice.
It
is fitting then that Linux 2.4 includes some improvements to this arena.
The biggest news on this front is that Linux 2.4 will support for the
first
time keyboards and mice attached to the Universal Serial Bus. When
plugged
in, these input device will behave just as if they were "normal"
keyboards
and mice. Additionally, Linux will now work on more systems, including
broken (or specially embedded) ones where the keyboard is not
pre-initialized by the BIOS. Also, better support is provided for
machines
without keyboards in some cases. (Mostly for buggy machines that don't
handle the lack of a keyboard as well as one would like.)
There are some places where some people feel that Linux 2.4 could
improve,
of course. With the addition of USB we have the chance to have multiple
keyboards and mice attached to the same bus. Linux 2.4 however does not
have internal multi-heading of these devices; you cannot assign one
keyboard and one mouse to one terminal and another set to a different
terminal. Support for this is provided in the GGI project, however this
project's code has not yet (and may never be) synched into the
mainstream
kernel. (It is however a good place to check if you need this
functionality.)
Video: Console, Frame-buffers
Now, you have ways of getting data into your Linux box, but that's
really
of no use unless you can get data out. That's pretty pointless! Linux
2.2
included some of the most revolutionary features to hit mainstream
Linux's
console support since the introduction of virtual consoles. (which was
well
before v1.0) Primarily, these changes were in the modularity of the
console
subsystem and the inclusion of frame-buffer drivers into the kernel,
with a
virtual frame-buffer console. (This support had existed previously in
many
Linuxes, but was never made portable enough to be of any use to i386
Linux
users.) In theory, this may someday allow for every Linux user to use a
single, smaller, X server and keep video support in the kernel. In the
end
however the option to use a frame-buffer is just that, an option, and
while
many people may not agree in the extent of its usage, no one can doubt
the
important role that it plays in systems which do not have a built-in
text
mode.
Linux 2.4 expands on frame-buffer support in many ways. Many new drivers
have been added for video cards including better support for VGA
displays
and even really old (16 color) VGA. As promised, we are seeing Linux
gradually support a wider and wider range of hardware in this capacity.
Linux 2.4 also includes support, useful primarily for servers, to
redirect
console output (including kernel and debug messages) to a device on a
parallel port (a printer, for example.) Like many features, this is just
an
option however it has already proven useful in a number of server and
debugging configurations.
Ports - Serial, Parallel, Infra-Red, etc.
Linux has always had excellent support for ports, such as serial ports,
that allows one to get around many of the drawbacks currently in the
keyboard and video layers. Linux really does have excellent support for
consoles, even system consoles with boot messages, on the serial ports.
It
is fitting that Linux 2.4 has improved support for many of these port
types
that are in common use today.
Serial support for Linux 2.4 has not changed much and many of the same
limitations still apply. (In particular, setting module options is
generally done with an external utility rather than the standard
parameters
usually passed to loadable modules.) Later versions of Linux 2.2 and all
versions of Linux 2.2 will allow one to share IRQs on PCI serial boards;
previously this was only allowed on ISA cards and on-board serial ports.
Some other pieces of multiport hardware will be better supported under
Linux 2.2. More updates and new drivers are flowing in regularly.
In contrast, the parallel port subsystem has undergone some major
overhauls
since 2.2. There is now a generic parallel port driver for abstracted
communication with "unknown" types of parallel devices. This could be
used,
for example, by programs that want to poll the parallel port for
Plug-and-Play information. (This is completely unrelated to the
Plug-and-Play capabilities of ISA cards.) Additionally, this has the
side
effect of allowing the root console to be spit out to a parallel port:
you
can configure this to cause console messages to be written out to a
printer. (This is especially useful for debugging in cases when copying
from the screen just won't do.) Also, Linux 2.4 supports writing to the
parallel ports using DMA, if supported in the hardware. This will speed
up
access to printers and other parallel devices.
Infra-red support has progressed since Linux 2.2 and there have been
many
changes in this area, including better network support.
In a separate department, there has been some work since 2.2 on
supporting
so-called "WinModems". These are modems which exist largely in software
and
whose drivers are only provided by the manufacturer for Windows. (Hence
the
name.) While no code has been submitted to Linus for the support of
these
beasts, it is possible that we may see some support for them before 3.0.
One major obstacle here is that each and every WinModem is different; it
is
unlikely that a driver for one would be applicable to another and the
sheer
number of different types of WinModems would make this difficult or
impossible to ever get a decent selection of hardware supported.
Impossible
odds have never phased open source developers however and I for one will
not be surprised when the first driver makes it into the kernel,
someday.
Networking and Protocols
Even further distant from the simplicity of keyboards and mice,
networking
and network hardware is one of the major areas where Linux has always
excelled. Linux 2.2 included many improvements to this layer including
new
drivers. Linux 2.4 has expanded on the already great amount of supported
hardware by providing bug fixes and new drivers for a variety of cards
including and especially ISDN devices.
Linux 2.4 also includes a completely rewritten networking layer. In
fact,
it has been made an unserialized as possible so that it will scale far
better than any previous version of Linux. In addition, it contains many
optimizations to allow it to work with the particular quirks of the
networking stacks in use in many common operating systems, including
Windows. It should also be mentioned at this point that Linux is still
the
only operating system completely compatible with the IPv4 specification.
We
can be sure to see many more improvements in this area before the Linux
2.4
release date is reached.
Next to the new network layer, the next most important improvement in
the
Linux 2.4 network layer is the addition of code to handle the DECNet
protocols. This allows for better interoperation with specialized
Digital/Compaq systems.
For the low-end desktop users, PPP is an important part of day to day
life.
Linux 2.4 includes some major rewrites and modularization of much of the
code, including a long awaited combination of the PPP layers from the
ISDN
layer and the PPP layer used on serial devices, such as modems.
Multimedia: Sound, TV, Radio, etc.
And gradually, we have worked our way from the essential components of
the
system to the fluffy components, the ones that may or may not be useful
to
your individual needs. Linux, in its emerging role as a desktop
platform,
tries very hard to support these devices, including sound cards, TV and
radio tuners, and other sound and video output devices. To be honest,
Linux
2.4 does not include as many ground-breaking changes as Linux 2.2 did in
this respect. (Yet...) Linux 2.4 does however include updates and new
drivers for a variety of sound and video cards, including full duplex
support. Linux 2.4 and some later versions of Linux 2.2 also include
code
which will allow some sound devices to more easily allocate memory in
the
ranges needed by some sound cards; this should make the configuration
and
use of these cards much easier.
Linux 2.4 (and later versions of Linux 2.2) now supports the DoubleTalk
speech synthesizer card. This is the first piece of hardware of this
type
which is supported under Linux, maybe we'll see more of these in the
future. (Really, all the "cool" hardware types are already supported.
We'll
take what we can get. :) )
--
I appreciate any comments that you may have, especially changes that you
feel are important that I missed. For the purpose of keeping my email
sorted into nice, neat piles, please respond to
[email protected].
(Yes, .com. Watch the .com.)