Hello,
The new kbuild architecture included as part of Linux 2.5 has a number of
problems that I would consider 2.5 show-stoppers and which must be resolved.
I believe the issues listed below are all regressions from 2.4; some are
easier to fix than others.
All of the issues relate to the building of kernel modules external to the
kernel source tree. Using kbuild to construct kernel modules instead of using
Makefile "hacks" has been rasied by the kbuild developers repeatedly on this
list and is, I believe, the preferred way to do it.
In my opinion, for kbuild to be a viable method of constructing kernel
modules external to the shipped source tree, the following conditions must
be met:
o The state of kbuild in shipped (distribution) kernels must be such that the
construction of external modules can be done without having to modify the
shipped kernel-source package.
This can be seen in almost all of today's distributions. Generally, you find
that the kernel-source package ships with a .config and the tree has had make
dep done. Effectively, the kernel source package matches the installed
binary. In the case of kbuild 2.4, this is excellent, as kbuild is in a ready
state to compile external modules against this kernel-source tree, and
perhaps more importantly, against the running kernel.
However, in 2.5, conditions are different depending on whether you are
building against a CONFIG_MODVERSIONS kernel or not. The preparation of the
tree in either case is also more complicated, though this does not seem
unreasonable.
I have determined that the kernel tree must have the following things done to
it before modules can be built against it (without modification to the tree).
make mrproper # ensure tree is completely tidied
cp -a /etc/kernelops-2.5 .config # copy of your default config
make oldconfig # configure tree to work with this config
So far, this matches the behaviour in 2.4. However, in 2.4 you need only do a
"make dep" (and, I believe, some distros also touch a couple of other files).
kbuild 2.4 is now ready to build kernel modules against the running kernel,
irrespective of kernel configuration. Module symbol CRCs are computed as
demanded by CONFIG_MODVERSIONS, so this is not a problem under 2.4.
In 2.5, I determined that the steps below were required instead of make dep.
make prepare # builds supporting utilities, links asm
# directory, creates vermagic.h and version.h
make scripts # seems to be required, not sure why (should
# already be called by prepare, right?)
This tree now seems prepared to build external modules against it, as the
headers are now complete and the compile-time utilities are built. When
CONFIG_MODVERSIONS is set, this even builds genksyms for us.
So far, so good. In fact, just doing this much allows traditional Makefile
"hacks" like NVIDIA's driver Makefile to build quite happily against this
tree. However, trying to use kbuild 2.5 now presents the following problems:
/bin/sh: line 1: .tmp_version/foo.mod: Permission denied
And later:
/bin/sh: line 1: ./.__modpost.cmd: Permission denied
These issues are *only* apparent when you attempt to build external modules
as an unprivileged user. Building as the user owning the source tree (in
distributions, I'd imagine this is ordinarily user "src" or "rpm" or even
"root", in certain cases) causes no such errors.
o KBUILD 2.5 creates temporary files inside the kernel source tree.
- .__modpost.cmd is hardcoded by Makefile and cannot be redirected;
- .tmp_version seems also to be hardcoded, but I am not sure.
- there is no obvious (or documented) way to manipulate the $srctree or
$objtree variables.
Doing a simple:
$ mkdir -p .tmp_versions;
$ touch .__modpost.cmd
$ chmod 1777 .tmp_versions .__modpost.cmd
Allows me to build external modules against a kernel tree as an unprivileged
user. I only need read-access to the tree.
Again, under 2.4, kbuild does not require such hacks. It does not seem to
create ANY temporary files in the source tree and works fine for building as
a non-privileged user.
The second major issue is that under 2.4, CONFIG_MODVERSIONS did not require
vmlinux to have been built in order to generate symbol CRCs. Building driver
foo against a kbuild 2.4 tree can even utilise modversions, simply by
including modversions.h.
Under 2.5, considerable improvements to the module system has made it so that
CONFIG_MODVERSIONS can only be utilised by modules in or out of the kernel
tree after vmlinux has been constructed. In the kbuild output I clearly see:
scripts/modpost vmlinux <list of all modules> /blah/dir/here/foo.o
Which correctly adds the required symbol CRCs to the module. A quick look at
scripts/modpost.c shows me that int modversions is not set to 1 unless
vmlinux is passed into it. Indeed, this makes sense, but it does mean that if
I build driver foo against an uncompiled kernel tree (as ordinarily would be
shipped by a distribution), the resulting driver does not have symbol CRCs
added to it, as was the case in 2.4.
To summarise, for kbuild 2.5 to be a viable platform for building external
modules in distribution kernels, the following improvements must be made:
o If it is considered important that users should be able to compile kernel
drivers (not install them) as an unprivileged user, the tree must desist
from creating temporary files and directories in the $srctree. Instead, the
utilisation of /tmp where suitable would be better;
o If distribution trees are to be shipped with CONFIG_MODVERSIONS set, kbuild
2.5 needs to have some way other than vmlinux of providing symbol CRCs. I
certainly am not sufficiently versed in the intricacies of the new module
system to comment on a solution, but I'm sure somebody here is.
As a developer, these concerns do not affect me at all. A portable kbuild
solution surpasses the inherent inflexibility of Makefile hacks and I can
already use it with moderate success on 2.4 and 2.5 platforms, as I always
build and install all modules as root, and always have a vmlinux hanging
around.
I do not use a linux "distribution", but I appreciate that the end user
typically will, and ultimately these kbuild solutions will be utilised by the
end user, for building both fully free drivers and proprietary kits including
kernel glue (e.g., the NVIDIA driver).
As some of you may be aware, I have been working with Christian Zander for
some time now, maintaining the NVIDIA driver (through UNOFFICIAL patches)
against the linux 2.5 mainline. Recently I rewrote NVIDIA's Makefile to use
kbuild on 2.4 and 2.5, and with the exception of the noted issues, it seems
to work properly for the majority of people. You can see these patches at
http://minion.de/ , Christian's site.
For your convenience, I have attached a sample Makefile which could easily be
adapted to be the module foo I have mentioned. In reality, this is a Makefile
for the proprietary NVIDIA module, although I am sure sensible people will
appreciate the global ramifications of these issues and how they will affect
the building of any module, free or non-free.
I'm open to suggestions if there are solutions or workarounds I have not
considered. If there are, however, it might be sensible to document these.
Cheers,
Alistair Strachan.
On Fri, 2003-07-11 at 19:40, Alistair J Strachan wrote:
> o The state of kbuild in shipped (distribution) kernels must be such that the
> construction of external modules can be done without having to modify the
> shipped kernel-source package.
that is actually not hard; I just did this in a RH rpm like way last
week.
> So far, this matches the behaviour in 2.4. However, in 2.4 you need only do a
> "make dep" (and, I believe, some distros also touch a couple of other files).
you never ever should need to do make dep in distro trees for building
external modules.
On Friday 11 July 2003 18:47, Arjan van de Ven wrote:
> On Fri, 2003-07-11 at 19:40, Alistair J Strachan wrote:
> > o The state of kbuild in shipped (distribution) kernels must be such that
> > the construction of external modules can be done without having to modify
> > the shipped kernel-source package.
>
> that is actually not hard; I just did this in a RH rpm like way last
> week.
I cannot see how you can make modversions modules without first building
vmlinux. This "RPM" presumably does not ship with vmlinux constructed, and
modpost depends on vmlinux to extract dependency symbols (as far as I can
tell.. though a KBUILD run on an uncompiled modversions tree appears to work,
you'll find the modversions section is not added to the kernel module, note
the size difference).
Try it with the NVIDIA driver, or any other preexisting driver using kbuild
and modversions but lacking a prebuilt vmlinux. I don't use RH, but
presumably the distributed 2.4 kernels did not have to be built before you
could include modversions.h. Assuming modpost is now a replacement for
modversions.h, and it does require vmlinux to include symbol CRCs, no, I
would say there was a problem.
> > So far, this matches the behaviour in 2.4. However, in 2.4 you need only
> > do a "make dep" (and, I believe, some distros also touch a couple of
> > other files).
>
> you never ever should need to do make dep in distro trees for building
> external modules.
Probably not, but what I meant was that supporting files (such as autoconf.h
and other "generated" headers that ARE required for compilation) would be
made during such a stage in 2.4 and do not have to be rebuilt. This isn't
actually the main issue, but I did notice that modpost is sometimes rebuilt
when you call the kernel makefile with SUBDIRS=/to/external/directory, and I
think it's obvious that this will require write access to the tree.
Cheers,
Alistair.
On Friday 11 July 2003 19:01, Arjan van de Ven wrote:
> On Fri, Jul 11, 2003 at 06:56:53PM +0100, Alistair J Strachan wrote:
> > On Friday 11 July 2003 18:47, Arjan van de Ven wrote:
> > > On Fri, 2003-07-11 at 19:40, Alistair J Strachan wrote:
> > > > o The state of kbuild in shipped (distribution) kernels must be such
> > > > that the construction of external modules can be done without having
> > > > to modify the shipped kernel-source package.
> > >
> > > that is actually not hard; I just did this in a RH rpm like way last
> > > week.
> >
> > I cannot see how you can make modversions modules without first building
> > vmlinux. This "RPM" presumably does not ship with vmlinux constructed
>
> It does actually.
Ah. In that case, I suppose it's all moot and won't end up being an issue. It
just strikes me that vmlinux would not have to be included in a distro 2.4
kernel, because it is not a "dependency" of the build system. If this is how
distros will operate, then just forget about it.
>
> > Try it with the NVIDIA driver
>
> no think you I prefer not to touch that with a 10 foot pole
Interesting snip of my original sentence.
Cheers,
Alistair.
I am sorry to reopen this thread but as we hit the same issues in Mandrake
I hope someone can advice.
> On Friday 11 July 2003 19:01, Arjan van de Ven wrote:
> > On Fri, Jul 11, 2003 at 06:56:53PM +0100, Alistair J Strachan wrote:
> > > On Friday 11 July 2003 18:47, Arjan van de Ven wrote:
> > > > On Fri, 2003-07-11 at 19:40, Alistair J Strachan wrote:
> > > > > o The state of kbuild in shipped (distribution) kernels must be such
> > > > > that the construction of external modules can be done without having
> > > > > to modify the shipped kernel-source package.
> > > >
> > > > > that is actually not hard; I just did this in a RH rpm like way last
> > > > week.
> > >
> > > I cannot see how you can make modversions modules without first building
> > > vmlinux. This "RPM" presumably does not ship with vmlinux constructed
> >
> > It does actually.
But is it really enough? modpost needs all modules to extract symbol versions
so it means kernel-source should actually ship all module.o files.
> Ah. In that case, I suppose it's all moot and won't end up being an issue. It
> just strikes me that vmlinux would not have to be included in a distro 2.4
> kernel, because it is not a "dependency" of the build system. If this is how
> distros will operate, then just forget about it.
Mandrake and AFAIK RedHat ship single 2.4 kernel source that allos building
external module for any currently running distribution kernel. While this
could be tweaked using symlinks it means that kernel-source needs to be
shipped with copies of vmlinux and *all modules.o for every kernel flavour
in distribution. As of this writing Mandrake includes 6 or 7 kernel versions.
It is rather too much :(
So it is still an issue. I am thinking about some way to direct modpost
to extract versions out of /boot/vmlinuz and /lib/modules if it detects
pristine distribution sources
To recap - currently under Mandrake and RH it is possible to do
rpm -i kernel-source
cd /external/module/src
make
and it will automatically create module for currently running kernel as long
as kernel is distribution kernel without any extra configuration.
It appears that to support same functionality on 2.6 modversions need to be disabled ...
TIA
-andrey
On Wed, Nov 05, 2003 at 10:10:14AM +0300, "Andrey Borzenkov" wrote:
> Mandrake and AFAIK RedHat ship single 2.4 kernel source that allos building
the 2.6 rpms I'm working on don't do this anymore. What Dave Jones will do
in his RPMs I don't know.
On Wed, 2003-11-05 at 10:23 +0100, Arjan van de Ven wrote:
> On Wed, Nov 05, 2003 at 10:10:14AM +0300, "Andrey Borzenkov" wrote:
>
> > Mandrake and AFAIK RedHat ship single 2.4 kernel source that allos building
>
> the 2.6 rpms I'm working on don't do this anymore. What Dave Jones will do
> in his RPMs I don't know.
Hopefully the out-of-source-tree build will allow this to be a lot nicer
than it was in 2.4.
--
dwmw2
> So it is still an issue. I am thinking about some way to direct modpost
> to extract versions out of /boot/vmlinuz and /lib/modules if it detects
> pristine distribution sources
>
> To recap - currently under Mandrake and RH it is possible to do
>
> rpm -i kernel-source
> cd /external/module/src
> make
>
> and it will automatically create module for currently running kernel as long
> as kernel is distribution kernel without any extra configuration.
I am in the process of addressing this issue.
What I plan to do is to provide a script solely for the purpose of building
modules. This script will be simple but allow us to change the build process,
without changing the way modules are build.
In order to use module versioning the distributor needs to ship all
modules. Otherwise the build process cannot look up a symbol exported
in module A, used by module B.
I have the basics implemented. Building a module with src located
away from the kernel src, and output files located a third place.
I simply need some spare time to do the rest....
The idea comes from Kai Germaschewski - but I like it and will try to
implement it.
Sam
On Thu, 6 Nov 2003, Sam Ravnborg wrote:
>
> I am in the process of addressing this issue.
> What I plan to do is to provide a script solely for the purpose of building
> modules. This script will be simple but allow us to change the build process,
> without changing the way modules are build.
> In order to use module versioning the distributor needs to ship all
> modules. Otherwise the build process cannot look up a symbol exported
> in module A, used by module B.
Do you mean all modules that module A refers to.
--
,-._|\ Ian Kent
/ \ Perth, Western Australia
*_.--._/ E-mail: [email protected]
v Web: http://themaw.net/
On Thursday 06 November 2003 20:23, Sam Ravnborg wrote:
> > So it is still an issue. I am thinking about some way to direct modpost
> > to extract versions out of /boot/vmlinuz and /lib/modules if it detects
> > pristine distribution sources
> >
> > To recap - currently under Mandrake and RH it is possible to do
> >
> > rpm -i kernel-source
> > cd /external/module/src
> > make
> >
> > and it will automatically create module for currently running kernel as
> > long as kernel is distribution kernel without any extra configuration.
>
> I am in the process of addressing this issue.
> What I plan to do is to provide a script solely for the purpose of building
> modules. This script will be simple but allow us to change the build
> process, without changing the way modules are build.
> In order to use module versioning the distributor needs to ship all
> modules. Otherwise the build process cannot look up a symbol exported
> in module A, used by module B.
>
sure. But distributor already has shipped all the modules as part of kernel
package. Given the number of different kernels in Mandrake currently shipping
yet another copy of binary modules for kernel sources would easily fill up
the whole CD alone.
besides it is not the only problem. Even without MODVERSIONS build process
attempts to write in temporary directory in kernel build tree making it
impossible for non-root to build module.
> I have the basics implemented. Building a module with src located
> away from the kernel src, and output files located a third place.
> I simply need some spare time to do the rest....
>
> The idea comes from Kai Germaschewski - but I like it and will try to
> implement it.
>
Could you point to posts that describe this idea? Also we could join efforts
:)
Thank you
-andrey
On Sun, Nov 09, 2003 at 01:12:56PM +0300, Andrey Borzenkov wrote:
>
> sure. But distributor already has shipped all the modules as part of kernel
> package. Given the number of different kernels in Mandrake currently shipping
> yet another copy of binary modules for kernel sources would easily fill up
> the whole CD alone.
in addition at least for the rpm's I'm working on, it's no
longer kernel-source you need or can use for building external modules
against.