We've been discussing splitting the kernel headers into userspace API headers
and kernel internal headers and deprecating the __KERNEL__ macro. This will
permit a cleaner interface between the kernel and userspace; and one that's
easier to keep up to date.
What we've come up with is this:
(1) Create new directories in the linux sources to shadow existing include
directories:
NEW DIRECTORY DIRECTORY SHADOWED
============= ==================
include/user/ include/linux/
include/user-*/ include/asm-*/
Note that this doesn't take account of the other directories under
include/, but I don't think they're relevant.
(2) Take each file from the shadowed directory. If it has any userspace
relevant stuff, then:
(a) Transfer this stuff into a file of the same name in the new
directory. So, for example, the syscall number list from
include/asm-i386/unistd.h will be transferred to
include/user-i386/unistd.h.
(b) Make kernel file #include the user file. So:
[include/asm-i386/unistd.h]
...
#include <user-i386/unistd.h>
...
(c) Where a user header file requires something from another header file
(such as a type), that file should include a suitable user header file
directly:
[include/user-i386/termio.h]
...
#include <user/types.h>
...
(d) stdint types should be used where possible.
[include/user-i386/termios.h]
struct winsize {
uint16_t ws_row;
uint16_t ws_col;
uint16_t ws_xpixel;
uint16_t ws_ypixel;
};
(e) These header files should be bounded with __USER_XXXXX_H conditionals:
[include/user-i386/termios.h]
#ifndef __USER_I386_TERMIOS_H
#define __USER_I386_TERMIOS_H
...
#endif /* __USER_I386_TERMIOS_H */
(3) Remove all #if(n)def __KERNEL__ clauses.
(4) Remove the -D__KERNEL__ from the master kernel Makefile.
(5) For userspace use (such as for glibc), the appropriate include/user*/
directories should be selected and installed in /usr/include/ or wherever,
and symlinks made. For example, on i386 arch boxes, you might find:
SOURCE INSTALLED AS
====================== ============
include/user/ /usr/include/user/
include/user-i386/ /usr/include/user-i386/
/usr/include/linux -> user
/usr/include/asm -> user-i386
(6) On multi-arch archs (such as ppc64 which can also support ppc), you might
find:
SOURCE INSTALLED AS
====================== ============
include/user/ /usr/include/user/
include/user-ppc/ /usr/include/user-ppc/
include/user-ppc64/ /usr/include/user-ppc64/
/usr/include/linux -> user
/usr/include/asm-ppc -> user-ppc
/usr/include/asm-ppc64 -> user-ppc64
And then /usr/include/asm/ might contain files that do arch-size dependent
switching between user-ppc and user-ppc64.
David
On Thu, Nov 25, 2004 at 04:20:06PM -0200, Alexandre Oliva wrote:
> This means these headers shouldn't reference each other as
> linux/user/something.h, but rather as linux/something.h, such that
> they still work when installed in /usr/include/linux. This may
> require headers include/linux/something.h to include
> linux/user/something.h, but that's already part of the proposal.
That's going to take severe brain-ache to get right ... and worse,
keep right. These headers aren't going to get tested outside the kernel
tree often. So we'll have missing includes and files that only work if
the <linux/> they're including is a kernel one rather than a user one.
I'm not particularly stuck on the <user/> namespace. We could invent
a better name. How about <kern/> and <arch/> to replace <linux/>
and <asm/>? Obviously keeping linux/ and asm/ symlinks for backwards
compatibility.
--
"Next the statesmen will invent cheap lies, putting the blame upon
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince
himself that the war is just, and will thank God for the better sleep
he enjoys after this process of grotesque self-deception." -- Mark Twain
On Thu, Nov 25, 2004 at 03:13:12PM +0000, David Howells wrote:
> (2) Take each file from the shadowed directory. If it has any userspace
> relevant stuff, then:
>
> (b) Make kernel file #include the user file. So:
>
> [include/asm-i386/unistd.h]
> ...
> #include <user-i386/unistd.h>
> ...
We may also want a user-asm symlink pointing to user-$ARCH.
If <linux/foo.h> wants a definition from <user-$ARCH/foo.h> then it has
to include <asm/foo.h> which includes <user-$ARCH/foo.h>. If asm/foo.h
is empty other than the include, then it'd be nice to delete it and have
<linux/foo.h> include <user-asm/foo.h> directly.
> (c) Where a user header file requires something from another header file
> (such as a type), that file should include a suitable user header file
> directly:
>
> [include/user-i386/termio.h]
> ...
> #include <user/types.h>
> ...
It's occasionally been on my mind that the transition from linux -> asm
should be one way and that asm files should not include linux files.
I'm not sure this is necessarily a worthy goal, but it seems worth
mentioning.
--
"Next the statesmen will invent cheap lies, putting the blame upon
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince
himself that the war is just, and will thank God for the better sleep
he enjoys after this process of grotesque self-deception." -- Mark Twain
On Thu, 2004-11-25 at 21:01 +0000, Matthew Wilcox wrote:
> I'm not particularly stuck on the <user/> namespace. We could invent
> a better name. How about <kern/> and <arch/> to replace <linux/>
> and <asm/>? Obviously keeping linux/ and asm/ symlinks for backwards
> compatibility.
There's no _real_ need to keep them. All we need to fix is a handful of
libc implementations; anything else using them was broken anyway.
--
dwmw2
On Thu, 2004-11-25 at 15:13 +0000, David Howells wrote:
> We've been discussing splitting the kernel headers into userspace API headers
> and kernel internal headers and deprecating the __KERNEL__ macro. This will
> permit a cleaner interface between the kernel and userspace; and one that's
> easier to keep up to date.
>
> What we've come up with is this:
I've already done something vaguely along those lines, extracting
user-visible bits from include/linux/mtd/*.h, putting them into files in
include/mtd/*.h and including them from there (except for jffs2.h which
currently does it backwards).
--
dwmw2
David Woodhouse wrote:
> On Thu, 2004-11-25 at 21:01 +0000, Matthew Wilcox wrote:
>
>>I'm not particularly stuck on the <user/> namespace. We could invent
>>a better name. How about <kern/> and <arch/> to replace <linux/>
>>and <asm/>? Obviously keeping linux/ and asm/ symlinks for backwards
>>compatibility.
>
>
> There's no _real_ need to keep them. All we need to fix is a handful of
> libc implementations; anything else using them was broken anyway.
>
<rant mode>
Cute idea. So any new definition which is supposed to be used in
userspace will take at least a year to propagate - well, I know, one can
always look for the definition in the kernel headers, copy it, ...
Now this is no solution.
And how do you want to deal with the fact that up to now all the
netfilter headers required for userspace programming live in the kernel
include tree? Now this has been like this for quite some years. Shall
one no longer use netfilter?
It is easy to be happy with the status quo and to reject any attempt to
change things for the better. If you really wanted to fix (g)libc you
would have taken action long ago. You're around for long enough.
</rant mode>
--
Andreas Steinmetz SPAMmers use [email protected]
On Sat, 2004-11-27 at 01:13 +0100, Andreas Steinmetz wrote:
> And how do you want to deal with the fact that up to now all the
> netfilter headers required for userspace programming live in the kernel
> include tree? Now this has been like this for quite some years. Shall
> one no longer use netfilter?
Don't get me started on fucking netfilter. Have you tried running 32-bit
userspace on a 64-bit kernel recently? Throw away the fucking netfilter
shite and burn it.
--
dwmw2
David Woodhouse wrote:
> On Sat, 2004-11-27 at 01:13 +0100, Andreas Steinmetz wrote:
>
>>And how do you want to deal with the fact that up to now all the
>>netfilter headers required for userspace programming live in the kernel
>>include tree? Now this has been like this for quite some years. Shall
>>one no longer use netfilter?
>
>
> Don't get me started on fucking netfilter. Have you tried running 32-bit
> userspace on a 64-bit kernel recently? Throw away the fucking netfilter
> shite and burn it.
>
I know very well about this problem. Still, you can easily use 64-bit
applications with regards to netfilter. If you are so annoyed about this
problem I would suggest you contact the netfilter developers to find a
solution :-)
--
Andreas Steinmetz SPAMmers use [email protected]
On Sat, 2004-11-27 at 01:30 +0100, Andreas Steinmetz wrote:
> I know very well about this problem. Still, you can easily use 64-bit
> applications with regards to netfilter. If you are so annoyed about this
> problem I would suggest you contact the netfilter developers to find a
> solution :-)
It's a shame _they_ didn't know well about this problem five or so years
ago when the rest of us did. Oh well, it's on the list of things to
attempt when I can actually stomach it.
In the meantime, don't even think about claiming that netfilter is an
example of _anything_ which you can hold up as an example when we're
talking about cleaning stuff up. Breaking it in its current state would
be a feature, not a bug.
--
dwmw2
On Thu, 25 Nov 2004, David Howells wrote:
> (b) Make kernel file #include the user file.
Does kernel really need to include user headers? When it is definition of
some const then it should be defined in one file (to be sure it has only
one definition). But user headers may have some compatibility hacks that
kernel do not need (and even maybe does not want) to have.
How you will handle that?
Thanks,
Grzegorz Kulewski
Grzegorz Kulewski wrote:
> On Thu, 25 Nov 2004, David Howells wrote:
>
>> (b) Make kernel file #include the user file.
>
>
> Does kernel really need to include user headers? When it is definition
> of some const then it should be defined in one file (to be sure it has
> only one definition).
You have do define a interface between the kernel and the userspace..
you either include kernel headers from userspace (with a lot of
__KERNEL__ in them) or you make separate headers with the definitions
and include them in both kernel and userspace (better).
BTW, these are not userspace headers like the ones in /usr/include,
those are just special headers preparated so that they can be included
both from the kernel and userspace.
> But user headers may have some compatibility hacks
> that kernel do not need (and even maybe does not want) to have.
About the compatibility hacks.. now it's time to remove them, together
with this change. I don't think this will happen before 2.7/2.8 and
until then all should have changed their code.
If you announce these changes soon enough and the developers have enough
time to change their code, I don't see any problems.
Maybe you also could wrap these definitions in some #ifdef's and mark
them as deprecated and write somewhere that they'll be removed in the
next stable tree (2.8). So you could check if a library compiles with
the new headers or if it still uses some old definitions.
tom
On Thu, 2004-11-25 at 16:54 +0000, Matthew Wilcox wrote:
> > Note that this doesn't take account of the other directories under
> > include/, but I don't think they're relevant.
>
> I think they may be. If they are, they can be accommodated. For example:
>
> include/user-scsi/ include/scsi/
> include/user-net/ include/net/
> include/user-rxrpc/ include/rxrpc/
>
> etcetera. This only creates a conflict if someone creates a directory
> foo that also clashes with asm-foo. And if someone does propose that,
> we just kill them. Easy.
Agreed.
> > (d) stdint types should be used where possible.
> >
> > [include/user-i386/termios.h]
> > struct winsize {
> > uint16_t ws_row;
> > uint16_t ws_col;
> > uint16_t ws_xpixel;
> > uint16_t ws_ypixel;
> > };
>
> I really hate stdint. Can't we use __u16 instead?
We're trying to clean all this crap up. I think we'd need a better
justification than 'I hate stdint' to use anything other than the
standard types which the language provides.
You'll get over it, I promise. I hated stdint too for a whole week or so
after I admitted it made sense to switch JFFS2 to it, because I actually
wanted it to be usable other than in the Linux kernel.
> This proposal doesn't address the asm-generic problem directly. Can I
> presume that you intend to also create linux/include/user-generic, install
> it as /usr/include/user-generic and create an asm-generic symlink that
> points to user-generic? A good problem file to be dealt with would
> be asm/errno.h
That makes sense.
> ppc might want to consider following the lead of parisc, s390 and mips
> and unify at least their header files, if not their arch directory.
That makes a certain amount of sense, although ppc64 currently lacks the
mess of platform-specific stuff that ppc32 has for various embedded
platforms. I suspect it would take a fair amount of work to produce a
single directory which everyone's happy with.
--
dwmw2
On Fri, Nov 26, 2004 at 12:00:43PM +0000, David Woodhouse wrote:
> On Fri, 2004-11-26 at 11:58 +0000, David Howells wrote:
> > How about calling the interface headers "kapi*/" instead of "user*/". In case
> > you haven't guessed, "kapi" would be short for "kernel-api".
>
> I don't think that change really makes any difference. The nomenclature
> really isn't _that_ important.
Indeed. We could also make this transparent to userspace by using a script
to copy the user-* headers to /usr/include. Something like this:
install_user_headers:
$(srctree)/scripts/install_user_headers.sh
#!/bin/bash
rm -rf /usr/include/asm /usr/include/linux /usr/include/asm-generic
mkdir /usr/include/asm /usr/include/linux /usr/include/asm-generic
to_user() {
sed -e 's,<user/,<linux/, \
-e 's,<user-generic/,<asm-generic,' \
-e 's,<user-[^/]*/,<asm,' \
<$1 >$2
}
for src in include/user/*; do
to_user $src /usr/include/linux/`basename $src`
done
for src in include/user-$ARCH/*; do
to_user $src /usr/include/asm/`basename $src`
done
for src in include/user-generic/*; do
to_user $src /usr/include/asm-generic/`basename $src`
done
If we really wanted to get fancy, we could also sed __u32 to uint32_t.
But that would probably cause more pain, confusion, hurt and bad feeling
than I'd ever want to be responsible for.
--
"Next the statesmen will invent cheap lies, putting the blame upon
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince
himself that the war is just, and will thank God for the better sleep
he enjoys after this process of grotesque self-deception." -- Mark Twain
On Fri, 2004-11-26 at 14:19 +0000, Matthew Wilcox wrote:
> On Fri, Nov 26, 2004 at 12:00:43PM +0000, David Woodhouse wrote:
> > On Fri, 2004-11-26 at 11:58 +0000, David Howells wrote:
> > > How about calling the interface headers "kapi*/" instead of "user*/". In case
> > > you haven't guessed, "kapi" would be short for "kernel-api".
> >
> > I don't think that change really makes any difference. The nomenclature
> > really isn't _that_ important.
>
> Indeed. We could also make this transparent to userspace by using a script
> to copy the user-* headers to /usr/include. Something like this:
Indeed.
> If we really wanted to get fancy, we could also sed __u32 to uint32_t.
> But that would probably cause more pain, confusion, hurt and bad feeling
> than I'd ever want to be responsible for.
Also true. Let's just use the standard types in the first place and not
screw around with having to fix it up later.
--
dwmw2
On Thu, Nov 25, 2004 at 03:13:12PM +0000, David Howells wrote:
> We've been discussing splitting the kernel headers into userspace API headers
> and kernel internal headers and deprecating the __KERNEL__ macro. This will
> permit a cleaner interface between the kernel and userspace; and one that's
> easier to keep up to date.
I am fully in support of both this idea and the current implementation
you have described below.
> What we've come up with is this:
>
> (1) Create new directories in the linux sources to shadow existing include
> directories:
>
> NEW DIRECTORY DIRECTORY SHADOWED
> ============= ==================
> include/user/ include/linux/
> include/user-*/ include/asm-*/
>
> Note that this doesn't take account of the other directories under
> include/, but I don't think they're relevant.
I think they may be. If they are, they can be accommodated. For example:
include/user-scsi/ include/scsi/
include/user-net/ include/net/
include/user-rxrpc/ include/rxrpc/
etcetera. This only creates a conflict if someone creates a directory
foo that also clashes with asm-foo. And if someone does propose that,
we just kill them. Easy.
> (d) stdint types should be used where possible.
>
> [include/user-i386/termios.h]
> struct winsize {
> uint16_t ws_row;
> uint16_t ws_col;
> uint16_t ws_xpixel;
> uint16_t ws_ypixel;
> };
I really hate stdint. Can't we use __u16 instead?
> (e) These header files should be bounded with __USER_XXXXX_H conditionals:
>
> [include/user-i386/termios.h]
> #ifndef __USER_I386_TERMIOS_H
> #define __USER_I386_TERMIOS_H
> ...
> #endif /* __USER_I386_TERMIOS_H */
>
> (3) Remove all #if(n)def __KERNEL__ clauses.
>
> (4) Remove the -D__KERNEL__ from the master kernel Makefile.
>
> (5) For userspace use (such as for glibc), the appropriate include/user*/
> directories should be selected and installed in /usr/include/ or wherever,
> and symlinks made. For example, on i386 arch boxes, you might find:
>
> SOURCE INSTALLED AS
> ====================== ============
> include/user/ /usr/include/user/
> include/user-i386/ /usr/include/user-i386/
> /usr/include/linux -> user
> /usr/include/asm -> user-i386
This proposal doesn't address the asm-generic problem directly. Can I
presume that you intend to also create linux/include/user-generic, install
it as /usr/include/user-generic and create an asm-generic symlink that
points to user-generic? A good problem file to be dealt with would
be asm/errno.h
> (6) On multi-arch archs (such as ppc64 which can also support ppc), you might
> find:
>
> SOURCE INSTALLED AS
> ====================== ============
> include/user/ /usr/include/user/
> include/user-ppc/ /usr/include/user-ppc/
> include/user-ppc64/ /usr/include/user-ppc64/
> /usr/include/linux -> user
> /usr/include/asm-ppc -> user-ppc
> /usr/include/asm-ppc64 -> user-ppc64
>
> And then /usr/include/asm/ might contain files that do arch-size dependent
> switching between user-ppc and user-ppc64.
ppc might want to consider following the lead of parisc, s390 and mips
and unify at least their header files, if not their arch directory.
--
"Next the statesmen will invent cheap lies, putting the blame upon
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince
himself that the war is just, and will thank God for the better sleep
he enjoys after this process of grotesque self-deception." -- Mark Twain
Grzegorz Kulewski wrote:
> Ok, so maybe do it in this way:
> 1. common headers (included by 2. and 3.)
> 2. kernel headers (things only for kernel + included 1.)
> 3. userspace headers (things only for userspace + included 1.)
Are there really any definitions that belong _only_ to userspace?
AFAIK all that is in the kernel sources is used by the kernel, so
there are no headers that would fit into your category 3. There is
no reason to put such things into the kernel sources.
> Are you talking about breaking userspace (API and ABI) compatibility?
> And possibly breaking compatibility with older versions of standards? I
> do not think it could happen. (Well at least not for common widely-used
> APIs).
>
> Instead we can place such userspace only hacks in 3.
The API can be changed, I don't see any problems with the API
compatibility, if you document all changes then it's no problem for
the developers to change their code.
And the ABI? Well, I compile my kernel by myself, as I do all my
userspace (gentoo). And the distributions still could define
__DEPRECATED__ (or something similar).
I mean.. come on.. how hard is it to change some lines of code,
some structure names, some function arguments.
I know that there's a problem if you have binary only files, but
I doubt that anyone is running a exe that was compiled for against
kernel 2.0 and every big company compiles their projects against each
stable kernel line.
Oh.. and I forgot, you usually don't include the kernel headers in your
'hello world' program, most of the applications use (g)libc, one big
library, so you'd need to change only very few projects, although I
admit that can be rather big projects, but still...
BTW, how much smaller would the kernel be if we removed all the
backwards compatibility? I mean the size of the compiled kernel
image.
tom
On Nov 25, 2004, at 10:13, David Howells wrote:
> We've been discussing splitting the kernel headers into userspace API
> headers
> and kernel internal headers and deprecating the __KERNEL__ macro. This
> will
> permit a cleaner interface between the kernel and userspace; and one
> that's
> easier to keep up to date.
Yay!!! Finally!!!
> NEW DIRECTORY DIRECTORY SHADOWED
> ============= ==================
> include/user/ include/linux/
> include/user-*/ include/asm-*/
How about include/abi for the platform-independent headers and
include/arch-*
for the platform-specific headers? Then include/arch would be
symlinked to the
correct arch-* directory.
> Note that this doesn't take account of the other directories under
> include/, but I don't think they're relevant.
Perhaps such directories should either be broken out into
abi/foo<->linux/foo
pieces, or put into their own top-level directories.
> (2) Take each file from the shadowed directory. If it has any
> userspace
> relevant stuff, then:
[...snip...]
Something like this would warn about incorrect uses of the linux/
headers, yet
still allow for a period of backwards compatibility. Once such period
has passed,
then linux/* would not even be installed, only abi/* and arch/*
include/linux/foo.h:
#ifndef _LINUX_FOO_H
#define _LINUX_FOO_H 1
#ifndef __KERNEL__
# warning "Directly including <kernel/foo.h> is deprecated!!!"
# warning "Please change your program to include <abi/foo.h> instead!!!"
#endif
#include <abi/foo.h>
#ifdef __KERNEL__
/* Old __KERNEL__ contents of include/linux/foo.h */
#endif
#endif /* not _LINUX_FOO_H */
include/abi/foo.h:
#ifndef _ABI_FOO_H
#define_ABI_FOO_H 1
#include <abi/bar.h>
#include <arch/baz.h>
/* Old non-__KERNEL__ contents of include/linux/foo.h */
#endif /* not _ABI_FOO_H */
Cheers,
Kyle Moffett
-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCM/CS/IT/U d- s++: a17 C++++>$ UB/L/X/*++++(+)>$ P+++(++++)>$
L++++(+++) E W++(+) N+++(++) o? K? w--- O? M++ V? PS+() PE+(-) Y+
PGP+++ t+(+++) 5 X R? tv-(--) b++++(++) DI+ D+ G e->++++$ h!*()>++$ r
!y?(-)
------END GEEK CODE BLOCK------
On Sat, 27 Nov 2004, Tomas Carnecky wrote:
> Grzegorz Kulewski wrote:
>> On Thu, 25 Nov 2004, David Howells wrote:
>>
>>> (b) Make kernel file #include the user file.
>>
>>
>> Does kernel really need to include user headers? When it is definition
>> of some const then it should be defined in one file (to be sure it has
>> only one definition).
>
> You have do define a interface between the kernel and the userspace..
> you either include kernel headers from userspace (with a lot of __KERNEL__ in
> them) or you make separate headers with the definitions and include them in
> both kernel and userspace (better).
> BTW, these are not userspace headers like the ones in /usr/include, those are
> just special headers preparated so that they can be included both from the
> kernel and userspace.
Ok, so maybe do it in this way:
1. common headers (included by 2. and 3.)
2. kernel headers (things only for kernel + included 1.)
3. userspace headers (things only for userspace + included 1.)
This way we will have no ifdefs, one definition per thing, user code in
userspace and kernel code in kernelspace only.
>> But user headers may have some compatibility hacks
>> that kernel do not need (and even maybe does not want) to have.
>
> About the compatibility hacks.. now it's time to remove them, together with
> this change. I don't think this will happen before 2.7/2.8 and until then all
> should have changed their code.
> If you announce these changes soon enough and the developers have enough time
> to change their code, I don't see any problems.
> Maybe you also could wrap these definitions in some #ifdef's and mark them as
> deprecated and write somewhere that they'll be removed in the next stable
> tree (2.8). So you could check if a library compiles with the new headers or
> if it still uses some old definitions.
Are you talking about breaking userspace (API and ABI) compatibility? And
possibly breaking compatibility with older versions of standards? I do not
think it could happen. (Well at least not for common widely-used APIs).
Instead we can place such userspace only hacks in 3.
Thanks,
Grzegorz Kulewski
On Thu, Nov 25, 2004 at 06:17:42PM +0000, David Woodhouse wrote:
> On Thu, 2004-11-25 at 16:54 +0000, Matthew Wilcox wrote:
> > > (d) stdint types should be used where possible.
> > >
> > > [include/user-i386/termios.h]
> > > struct winsize {
> > > uint16_t ws_row;
> > > uint16_t ws_col;
> > > uint16_t ws_xpixel;
> > > uint16_t ws_ypixel;
> > > };
> >
> > I really hate stdint. Can't we use __u16 instead?
>
> We're trying to clean all this crap up. I think we'd need a better
> justification than 'I hate stdint' to use anything other than the
> standard types which the language provides.
The justification is that it doesn't properly describe the variable size
(think userspace 32 bit variable vs. kernelspace 32 bit variable.) We
need to stick with the proper __u32 type variables for data that crosses
the userspace/kernelspace boundry.
See Linus's posts about this topic on lkml in the past for more details
about this.
thanks,
greg k-h
Salut,
On Thu, Nov 25, 2004 at 03:13:12PM +0000, David Howells wrote:
> We've been discussing splitting the kernel headers into userspace API headers
> and kernel internal headers and deprecating the __KERNEL__ macro. This will
> permit a cleaner interface between the kernel and userspace; and one that's
> easier to keep up to date.
Fnord alert: you're trying to change the API and the ABI of Linux. The thing
you fail to see is that Linux is not the only operating system in the world.
Big companies are very prone of using their compatibility code to make their
program run in any odd way they like without actually having to change
anything. Also, you're talking about assumptions that have been true for
decades.
Big companies are already eyeing Linux with fear because our kernels don't
actually work and because we break a lot of compatibility in the stable
series. If you now come up with something that breaks basically everything
so they need a new libc and everything, it's rather likely that they'll run
away.
I just want you to consider that. Linux needs to get to a stable state of
affairs at some time.
Tonnerre
> I'm not particularly stuck on the <user/> namespace. We could invent
> a better name. How about <kern/> and <arch/> to replace <linux/>
> and <asm/>? Obviously keeping linux/ and asm/ symlinks for backwards
> compatibility.
I'd rather not change linux and asm within the kernel itself. We'd have to
patch pretty much every file and move all the include files around. That's a
lot of patch.
How about calling the interface headers "kapi*/" instead of "user*/". In case
you haven't guessed, "kapi" would be short for "kernel-api".
Then we'd have in the kernel:
NEW DIRECTORY DIRECTORY SHADOWED
============= ==================
include/kapi/ include/linux/
include/kapi-*/ include/asm-*/
include/kapi-rxrpc/ include/rxrpc/
include/kapi-net/ include/net/
And in userspace:
SOURCE INSTALLED AS
====================== ============
include/kapi/ /usr/include/kapi/
include/kapi-i386/ /usr/include/kapi-i386/
include/kapi-generic/ /usr/include/kapi-generic/
/usr/include/linux -> kapi
/usr/include/asm -> kapi-i386
/usr/include/asm-generic -> kapi-generic (?)
David
On Nov 25, 2004, David Howells <[email protected]> wrote:
> SOURCE INSTALLED AS
> ====================== ============
> include/user/ /usr/include/user/
> include/user-i386/ /usr/include/user-i386/
> /usr/include/linux -> user
> /usr/include/asm -> user-i386
Although user/ and user-* make a lot of sense within the kernel source
tree, I don't think these names would be very clear in /usr/include.
I'd rather use names in /usr/include that more clearly associate them
with the kernel. Heck, even /usr/include/asm is inappropriate, but
it's been there for so long that we really shouldn't try to get rid of
it.
If I had it my way, we'd have, in the kernel tree, userland-aimed
headers in include/linux/user and include/asm-<machine>/user, and have
them installed in /usr/include/linux and /usr/include/asm-<machine>.
This means these headers shouldn't reference each other as
linux/user/something.h, but rather as linux/something.h, such that
they still work when installed in /usr/include/linux. This may
require headers include/linux/something.h to include
linux/user/something.h, but that's already part of the proposal.
--
Alexandre Oliva http://www.ic.unicamp.br/~oliva/
Red Hat Compiler Engineer aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist oliva@{lsd.ic.unicamp.br, gnu.org}
Tonnerre wrote:
> Salut,
>
> On Thu, Nov 25, 2004 at 03:13:12PM +0000, David Howells wrote:
>
>>We've been discussing splitting the kernel headers into userspace API headers
>>and kernel internal headers and deprecating the __KERNEL__ macro. This will
>>permit a cleaner interface between the kernel and userspace; and one that's
>>easier to keep up to date.
>
>
> Fnord alert: you're trying to change the API and the ABI of Linux. The thing
> you fail to see is that Linux is not the only operating system in the world.
> Big companies are very prone of using their compatibility code to make their
> program run in any odd way they like without actually having to change
> anything. Also, you're talking about assumptions that have been true for
> decades.
>
> Big companies are already eyeing Linux with fear because our kernels don't
> actually work and because we break a lot of compatibility in the stable
> series. If you now come up with something that breaks basically everything
> so they need a new libc and everything, it's rather likely that they'll run
> away.
>
I didn't say to change this now, not tomorrow...
Just tell them to change their code because, somewhere in the future,
the API might change. Maybe the new kernel APIs are better, faster, more
secure, so perhaps there is a good reason as why it would be better to
use the new API.
And for those who want to use the old API, let's give them the ability,
but also make it possible to compile the kernel with _only_ the new API.
The different distributions will take care of it.
And about the compatibility layer.. the companies have to make a
different one for each stable series anyways.
tom
Tonnerre wrote:
> Fnord alert: you're trying to change the API and the ABI of Linux.
No he's not. He's changing the location where the API is defined, and the ABI
doesn't change a bit.
Existing binaries will continue to run without modification.
Chris
On Fri, 2004-11-26 at 11:58 +0000, David Howells wrote:
> How about calling the interface headers "kapi*/" instead of "user*/". In case
> you haven't guessed, "kapi" would be short for "kernel-api".
I don't think that change really makes any difference. The nomenclature
really isn't _that_ important.
--
dwmw2
> This proposal doesn't address the asm-generic problem directly. Can I
> presume that you intend to also create linux/include/user-generic, install
> it as /usr/include/user-generic and create an asm-generic symlink that
> points to user-generic? A good problem file to be dealt with would
> be asm/errno.h
Agreed. I'd forgotten about that.
David
On Nov 25, 2004, David Woodhouse <[email protected]> wrote:
> There's no _real_ need to keep them. All we need to fix is a handful of
> libc implementations; anything else using them was broken anyway.
It's not as simple as *fixing* them. They'd have to go through a
transition period in which they support both formats, which probably
implies a mess of ifdefs or, worse, macro expansion in #include
directives.
--
Alexandre Oliva http://www.ic.unicamp.br/~oliva/
Red Hat Compiler Engineer aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist oliva@{lsd.ic.unicamp.br, gnu.org}
On Fri, 2004-11-26 at 09:47 -0200, Alexandre Oliva wrote:
> On Nov 25, 2004, Matthew Wilcox <[email protected]> wrote:
>
> > On Thu, Nov 25, 2004 at 04:20:06PM -0200, Alexandre Oliva wrote:
> >> This means these headers shouldn't reference each other as
> >> linux/user/something.h, but rather as linux/something.h, such that
> >> they still work when installed in /usr/include/linux. This may
> >> require headers include/linux/something.h to include
> >> linux/user/something.h, but that's already part of the proposal.
>
> > That's going to take severe brain-ache to get right ... and worse,
> > keep right. These headers aren't going to get tested outside the kernel
> > tree often. So we'll have missing includes and files that only work if
> > the <linux/> they're including is a kernel one rather than a user one.
>
> How about moving the internals (i.e., what's not to be exported to
> userland) from linux and asm elsewhere, then?
>
> Sure, it means significantly more churn in the kernel, but there's
> going to be a lot of moving stuff around one way or the other.
Not really. The kernel code was what was _allowed_ to be using those
headers with those names in the first place; it was userspace which
shouldn't necessarily have been doing so.
> While at that, we could also split what's kernel internal for real and
> what's to be visible to external kernel modules as well. So we'd have
> 3 layers of headers, instead of two. I'm not sure this actually makes
> any sense though, since there might be lots of dependencies of headers
> for modules on internal headers.
All modules will be entirely dependent on the full set of kernel
headers. Let's not go there.
--
dwmw2
On Sat, Nov 27, 2004 at 05:29:42AM +0100, Tonnerre wrote:
> Fnord alert: you're trying to change the API and the ABI of Linux. The thing
No we aren't. We're just trying to make the kernel interface more
explicit. Nothing should change as far as userspace programs are
concerned. It might need some coordination with the various libcs.
And enough with the doom-and-gloom big-companies-are-scared approach, OK?
It's (a) untrue, (b) irrelevant, and (c) many of us work for big companies
that are doing Linux already. We don't need to be lectured by you.
--
"Next the statesmen will invent cheap lies, putting the blame upon
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince
himself that the war is just, and will thank God for the better sleep
he enjoys after this process of grotesque self-deception." -- Mark Twain
On Nov 25, 2004, Matthew Wilcox <[email protected]> wrote:
> On Thu, Nov 25, 2004 at 04:20:06PM -0200, Alexandre Oliva wrote:
>> This means these headers shouldn't reference each other as
>> linux/user/something.h, but rather as linux/something.h, such that
>> they still work when installed in /usr/include/linux. This may
>> require headers include/linux/something.h to include
>> linux/user/something.h, but that's already part of the proposal.
> That's going to take severe brain-ache to get right ... and worse,
> keep right. These headers aren't going to get tested outside the kernel
> tree often. So we'll have missing includes and files that only work if
> the <linux/> they're including is a kernel one rather than a user one.
How about moving the internals (i.e., what's not to be exported to
userland) from linux and asm elsewhere, then?
Sure, it means significantly more churn in the kernel, but there's
going to be a lot of moving stuff around one way or the other.
While at that, we could also split what's kernel internal for real and
what's to be visible to external kernel modules as well. So we'd have
3 layers of headers, instead of two. I'm not sure this actually makes
any sense though, since there might be lots of dependencies of headers
for modules on internal headers.
--
Alexandre Oliva http://www.ic.unicamp.br/~oliva/
Red Hat Compiler Engineer aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist oliva@{lsd.ic.unicamp.br, gnu.org}
On Fri, 26 Nov 2004, Matthew Wilcox wrote:
> #!/bin/bash
<nitpick>
This is not a bash script.
</nitpick>
> Matthew Wilcox wrote:
> > Indeed. We could also make this transparent to userspace by using a script
> > to copy the user-* headers to /usr/include. Something like this:
some administrator would like to only create symlinks for saving disk space.
others would like a true "install" with copying files so that they can delete
the kernel sources anytime they do want.
for me i would install all those kernel realted files into
the well known /lib/modules/<kernel-version>.
so its a no question that a single kernel install
can be cleaned up by deleting those dir recursively.
the only part remaining somewhere else in the filesystem
would then be /boot/bzImage since booting is a special case.
there are several architectures where the bzImage is be able
to reside in /lib/modules/... but there are others archs as well.
David Woodhouse wrote:
> Matthew Wilcox wrote:
> > If we really wanted to get fancy, we could also sed __u32 to uint32_t.
> > But that would probably cause more pain, confusion, hurt and bad feeling
> > than I'd ever want to be responsible for.
>
> Also true. Let's just use the standard types in the first place and not
> screw around with having to fix it up later.
uint32_t is the type you should get when using the standarized statement
#include <stdint.h>
Am i right? Does that header belong to the international C99 Standard?
So you are suggesting the kernel interface should stick to that standard?
Changing it that way would hit a noticeable amount of non kernel code.
Lets consider the opposite point of view. If we do postpone such a
change to the standards for some 3 to 10 more years, then i expect
the impact on software and problem for getting it solve is bigger.
Maybe that sort of change is something qualified for beeing done
inside a true development kernel tree like Linux 2.7.xx which might
then beeing qualified for getting a Linux 3.0.0 release kernel tree.
-Alex.
Matthew Wilcox wrote:
> On Sat, Nov 27, 2004 at 05:29:42AM +0100, Tonnerre wrote:
>
>>Fnord alert: you're trying to change the API and the ABI of Linux. The thing
>
>
> No we aren't. We're just trying to make the kernel interface more
> explicit. Nothing should change as far as userspace programs are
> concerned. It might need some coordination with the various libcs.
Speaking of more explicit, there are various asm-ARCH header
files that do or do not hide (via __KERNEL__) interfaces
such as: get_unaligned()
and the atomic operations.
So are these Linux kernel exported APIs, or do they belong
in some library?
We'll need to decide on a consistent remedy for these,
and such changes could be made at any time IMO (i.e., soon).
> And enough with the doom-and-gloom big-companies-are-scared approach, OK?
> It's (a) untrue, (b) irrelevant, and (c) many of us work for big companies
> that are doing Linux already. We don't need to be lectured by you.
--
~Randy
Alexandre Oliva <[email protected]> writes:
> How about moving the internals (i.e., what's not to be exported to
> userland) from linux and asm elsewhere, then?
That's what I proposed several {months,weeks} ago. From a technical
point of view it seems like a superior solution:
- /usr/include/{linux,asm} -> /usr/src/linux/include/{linux,asm} = no
problems with kernel and userspace includes.
- while Linux is quite a big program, it is _one_ program = no need to
patch the whole universe, only the kernel would need changing
(of course broken userspace code would need fixing, but that's
due to their brokeness and not due to the change).
> Sure, it means significantly more churn in the kernel, but there's
> going to be a lot of moving stuff around one way or the other.
No doubt. Still, quite simple automatic operation, we've seen much larger
patches.
API backward-compatibility (including C libs) is IMHO much more
important than just some kernel patch size (I realize ABI compatibility
would be preserved in all cases).
And we can do it anytime - no need for waiting for some libc change,
no need for waiting for "2.7/2.8".
> While at that, we could also split what's kernel internal for real and
> what's to be visible to external kernel modules as well.
Can't see benefits.
--
Krzysztof Halasa
On Sat, 2004-11-27 at 08:47 -0800, Randy.Dunlap wrote:
> Speaking of more explicit, there are various asm-ARCH header
> files that do or do not hide (via __KERNEL__) interfaces
> such as: get_unaligned()
> and the atomic operations.
>
> So are these Linux kernel exported APIs, or do they belong
> in some library?
Both of those are kernel-private and should not be visible.
--
dwmw2
On Thu, Nov 25, 2004 at 04:54:33PM +0000, Matthew Wilcox wrote:
> > (5) For userspace use (such as for glibc), the appropriate include/user*/
> > directories should be selected and installed in /usr/include/ or wherever,
> > and symlinks made. For example, on i386 arch boxes, you might find:
> >
> > SOURCE INSTALLED AS
> > ====================== ============
> > include/user/ /usr/include/user/
> > include/user-i386/ /usr/include/user-i386/
> > /usr/include/linux -> user
> > /usr/include/asm -> user-i386
>
> This proposal doesn't address the asm-generic problem directly. Can I
> presume that you intend to also create linux/include/user-generic, install
> it as /usr/include/user-generic and create an asm-generic symlink that
> points to user-generic? A good problem file to be dealt with would
> be asm/errno.h
Do not allow asm-generic to be used direct by userspace.
The example with errno.h is handled by user-i386/errno.h which includes
asm-generic/errno.h.
Would that do the trick - yes?
Sam
Alexander Stohr wrote:
>> Matthew Wilcox wrote:
>> > Indeed. We could also make this transparent to userspace by using a script
>> > to copy the user-* headers to /usr/include. Something like this:
>
> some administrator would like to only create symlinks for saving disk space.
>
> others would like a true "install" with copying files so that they can delete
> the kernel sources anytime they do want.
>
> for me i would install all those kernel realted files into
> the well known /lib/modules/<kernel-version>.
IMHO the script should let you install the headers
wherever you like. In particular, in crosstool,
I would like to install the headers somewhere like
/opt/crosstool/$TARGET/include rather than /usr/include.
- Dan
--
Trying to get a job as a c++ developer? See http://kegel.com/academy/getting-hired.html
On Thu, Nov 25, 2004 at 03:13:12PM +0000, David Howells wrote:
>
> We've been discussing splitting the kernel headers into userspace API headers
> and kernel internal headers and deprecating the __KERNEL__ macro. This will
> permit a cleaner interface between the kernel and userspace; and one that's
> easier to keep up to date.
>
> What we've come up with is this:
>
> (1) Create new directories in the linux sources to shadow existing include
> directories:
>
> NEW DIRECTORY DIRECTORY SHADOWED
> ============= ==================
> include/user/ include/linux/
> include/user-*/ include/asm-*/
>
> Note that this doesn't take account of the other directories under
> include/, but I don't think they're relevant.
If we go for some resturcturing of include/ then we should get rid of
the annoying asm symlink. Following layout deals with that:
include/<arch>/asm <= Files from include/asm-<arch>
include/<arch>/mach* <= Files from include/mach-*
This layout solve the symlink issue in an elegant way.
We need to do trivial changes to compiler options to make it work. Changing
compiler options is much more relaible than using symlinks.
Then the userspace part would then be located in:
include/<arch>/user-asm
Userspace shall not be 'allowed' to include files from asm-generic direct,
but asm-generic shall be part of the userspace api.
> (2) Take each file from the shadowed directory. If it has any userspace
> relevant stuff, then:
>
> (b) Make kernel file #include the user file. So:
>
> [include/asm-i386/unistd.h]
> ...
> #include <user-i386/unistd.h>
In example above it would become:
#include <user-asm/unistd.h>
With repsect to consistency of the userspace header files then we could define
a few simple rules:
1) Files in include/user must be able to compile standalone - so they include
all files they need.
2) They are not allowed to use any files outside the kernel
3) Must not use __KERNEL__ in any place
Writing a small script / Makefile that does these checks should be trivial.
J?rn wrote something similar for include/linux at one point in time.
If it is fast then we just let it be part of the standard compile so it is
checked often.
Sam
On Sat, Nov 27, 2004 at 11:43:06AM -0800, Dan Kegel wrote:
> >
> >for me i would install all those kernel realted files into
> >the well known /lib/modules/<kernel-version>.
>
> IMHO the script should let you install the headers
> wherever you like. In particular, in crosstool,
> I would like to install the headers somewhere like
> /opt/crosstool/$TARGET/include rather than /usr/include.
Agreed and trivial to do.
One should recall that for current kernel the following path is valid:
/lib/modules/`uname -r`/source/include/*
This is valid for 2.6.7 or something.
Sam
Sam Ravnborg wrote:
> On Thu, Nov 25, 2004 at 03:13:12PM +0000, David Howells wrote:
>
>>We've been discussing splitting the kernel headers into userspace API headers
>>and kernel internal headers and deprecating the __KERNEL__ macro. This will
>>permit a cleaner interface between the kernel and userspace; and one that's
>>easier to keep up to date.
>>
>>What we've come up with is this:
>>
>> (1) Create new directories in the linux sources to shadow existing include
>> directories:
>>
>> NEW DIRECTORY DIRECTORY SHADOWED
>> ============= ==================
>> include/user/ include/linux/
>> include/user-*/ include/asm-*/
>>
>> Note that this doesn't take account of the other directories under
>> include/, but I don't think they're relevant.
>
>
> If we go for some resturcturing of include/ then we should get rid of
> the annoying asm symlink. Following layout deals with that:
>
> include/<arch>/asm <= Files from include/asm-<arch>
> include/<arch>/mach* <= Files from include/mach-*
>
> This layout solve the symlink issue in an elegant way.
> We need to do trivial changes to compiler options to make it work. Changing
> compiler options is much more relaible than using symlinks.
>
> Then the userspace part would then be located in:
> include/<arch>/user-asm
>
This complicates things for bi-arch architectures like x86_64 where one
can use a dispatcher directory instead of a symlink to suit include/asm
for 32bit as well as 64bit.
--
Andreas Steinmetz SPAMmers use [email protected]
On Sat, Nov 27, 2004 at 10:11:43PM +0100, Andreas Steinmetz wrote:
> >If we go for some resturcturing of include/ then we should get rid of
> >the annoying asm symlink. Following layout deals with that:
> >
> >include/<arch>/asm <= Files from include/asm-<arch>
> >include/<arch>/mach* <= Files from include/mach-*
> >
> >This layout solve the symlink issue in an elegant way.
> >We need to do trivial changes to compiler options to make it work. Changing
> >compiler options is much more relaible than using symlinks.
> >
> >Then the userspace part would then be located in:
> >include/<arch>/user-asm
> >
>
> This complicates things for bi-arch architectures like x86_64 where one
> can use a dispatcher directory instead of a symlink to suit include/asm
> for 32bit as well as 64bit.
X86_64 does not create any special symlinks todays so I do not see the point?
And still a symlink is just the wrong way to solve the problem.
Adjusting options to the compiler is the way to do it.
Sam
Sam Ravnborg wrote:
> On Sat, Nov 27, 2004 at 10:11:43PM +0100, Andreas Steinmetz wrote:
>
>>>If we go for some resturcturing of include/ then we should get rid of
>>>the annoying asm symlink. Following layout deals with that:
>>>
>>>include/<arch>/asm <= Files from include/asm-<arch>
>>>include/<arch>/mach* <= Files from include/mach-*
>>>
>>>This layout solve the symlink issue in an elegant way.
>>>We need to do trivial changes to compiler options to make it work. Changing
>>>compiler options is much more relaible than using symlinks.
>>>
>>>Then the userspace part would then be located in:
>>>include/<arch>/user-asm
>>>
>>
>>This complicates things for bi-arch architectures like x86_64 where one
>>can use a dispatcher directory instead of a symlink to suit include/asm
>>for 32bit as well as 64bit.
>
> X86_64 does not create any special symlinks todays so I do not see the point?
> And still a symlink is just the wrong way to solve the problem.
> Adjusting options to the compiler is the way to do it.
Ok, this is not in the kernel, but think of the following: an
application needs and include/asm header. The application may be
compiled as 32 bit and as 64 bit. Then you could use a real asm
directory instead of a symlink and use, e.g. for asm/byteorder.h:
#ifdef __x86_64__
#include <asm-x86_64/byteorder.h>
#else
#include <asm-i386/byteorder.h>
#endif
This way both archs are satisfied from a userspace point of view without
having to take care if the source is compiled on 32bit or 64bit. If
you kill the symlink this neat dispatcher can't really be done anymore
that easy.
It is not necessary to do anything in the kernel tree here except to
keep the symlink to allow for this kind of userspace tweaks for certain
architectures.
This is especially important when there's need to use a include/linux
header which itself needs an include/asm header as in this case with the
the little tweak always the right asm header is included.
--
Andreas Steinmetz SPAMmers use [email protected]
On S?nnavend 27 November 2004 18:16, you wrote:
> On Sat, 2004-11-27 at 08:47 -0800, Randy.Dunlap wrote:
> > Speaking of more explicit, there are various asm-ARCH header
> > files that do or do not hide (via __KERNEL__) interfaces
> > such as:??????get_unaligned()
> > and the atomic operations.
> >
> > So are these Linux kernel exported APIs, or do they belong
> > in some library?
>
> Both of those are kernel-private and should not be visible.
The problem with these (atomic.h, bitops.h, byteorder.h, div64.h,
list.h, spinlock.h, unaligned.h and xor.h) is that they provide
functionality that is needed by many user application but not
provided by the compiler or libc.
While I agree that it is an absolutely evil concept to include
them from the kernel headers, we have to face that by not installing
them, lots of this existing evil user code will be broken even
more and someone has to pick up the pieces.
Arnd <><
On Sat, 2004-11-27 at 23:53 +0100, Arnd Bergmann wrote:
> The problem with these (atomic.h, bitops.h, byteorder.h, div64.h,
> list.h, spinlock.h, unaligned.h and xor.h) is that they provide
> functionality that is needed by many user application but not
> provided by the compiler or libc.
>
> While I agree that it is an absolutely evil concept to include
> them from the kernel headers, we have to face that by not installing
> them, lots of this existing evil user code will be broken even
> more and someone has to pick up the pieces.
The existing evil user code is broken already. Any of the above can be
implemented in a such a way that it only works in the kernel anyway.
We don't need to preserve it any more than we need to preserve evil
broken code which defines __KERNEL__ before including certain headers.
We're supposed to be cleaning this crap up. If we want every evil broken
hack that's currently possible to _remain_ possible, it's never going to
work.
--
dwmw2
On S?nnavend 27 November 2004 22:49, Andreas Steinmetz wrote:
> Ok, this is not in the kernel, but think of the following: an
> application needs and include/asm header. The application may be
> compiled as 32 bit and as 64 bit. Then you could use a real asm
> directory instead of a symlink and use, e.g. for asm/byteorder.h:
>
> #ifdef __x86_64__
> #include <asm-x86_64/byteorder.h>
> #else
> #include <asm-i386/byteorder.h>
> #endif
>
I think we can get rid of this hack when we move to split kernel headers.
parisc, s390 and mips already have combined headers, and it should not be
too hard to combine the user ABI headers for sparc, ppc and x86_64 as well,
without having to merge the complete architecture and kernel header trees
for them.
Building for ppc64 would then use something like 'gcc -Iinclude
-Iinclude/ppc64 -Iinclude/user -Iinclude/user/ppc/' for a layout
that has files in
include/linux/foo.h
include/$arch/asm/foo.h
include/generic/asm/foo.h
include/user/linux/foo.h (installed as <linux/foo.h>)
include/user/$arch32/asm/foo.h (installed as <asm/foo.h>
include/user/asm-generic/foo.h (installed as <asm-generic/foo.h>
[Note how the first three become optional in this scheme. For files
that need an internal version an a user version, ppc64/asm/foo.h
can use #include_next <asm/foo.h> to get to generic/asm/foo.h and
user/ppc/asm/foo.h.]
For ia64, the headers should probably stay separate because I would expect
them to differ more from i386/x86_64 than the other dual 32/64 architectures.
I don't think that is a problem because you need separate tool chains for
ia64 and i386 anyway, while the others can share the libc headers and the
compiler backend already.
Arnd <><
Arnd Bergmann wrote:
> On S?nnavend 27 November 2004 18:16, you wrote:
>
>>On Sat, 2004-11-27 at 08:47 -0800, Randy.Dunlap wrote:
>>
>>>Speaking of more explicit, there are various asm-ARCH header
>>>files that do or do not hide (via __KERNEL__) interfaces
>>>such as: get_unaligned()
>>>and the atomic operations.
>>>
>>>So are these Linux kernel exported APIs, or do they belong
>>>in some library?
>>
>>Both of those are kernel-private and should not be visible.
>
>
> The problem with these (atomic.h, bitops.h, byteorder.h, div64.h,
> list.h, spinlock.h, unaligned.h and xor.h) is that they provide
> functionality that is needed by many user application but not
> provided by the compiler or libc.
>
> While I agree that it is an absolutely evil concept to include
> them from the kernel headers, we have to face that by not installing
> them, lots of this existing evil user code will be broken even
> more and someone has to pick up the pieces.
That's addressing a different problem. I agree with
David W. that we need to clean the kernel headers up.
Let libc or libxyz provide the missing functionality.
The borken programs were stealing something that wasn't
promised to them AFAIK.
IOW, let's decide the right thing to do and implement it.
--
~Randy
On Sat, 2004-11-27 at 15:27 -0800, Randy.Dunlap wrote:
> That's addressing a different problem. I agree with
> David W. that we need to clean the kernel headers up.
> Let libc or libxyz provide the missing functionality.
> The borken programs were stealing something that wasn't
> promised to them AFAIK.
Not only wasn't it promised; it wasn't even working on some
architectures anyway.
--
dwmw2
On S?nndag 28 November 2004 00:32, David Woodhouse wrote:
> On Sat, 2004-11-27 at 15:27 -0800, Randy.Dunlap wrote:
> > That's addressing a different problem. I agree with
> > David W. that we need to clean the kernel headers up.
> > Let libc or libxyz provide the missing functionality.
> > The borken programs were stealing something that wasn't
> > promised to them AFAIK.
>
> Not only wasn't it promised; it wasn't even working on some
> architectures anyway.
>
Well, at least on s390 some effort was spent on making it work
(see e.g. asm-s390/spinlock.h). I don't really mind removing
that functionality, but I'd prefer still installing some files
into /usr/include/asm/spinlock.h et.al that contain an #error
and a pointer to an explanation, so users can complain to the
right people (i.e. not me ;-).
Arnd <><
> I think we can get rid of this hack when we move to split kernel headers.
> parisc, s390 and mips already have combined headers, and it should not be
> too hard to combine the user ABI headers for sparc, ppc and x86_64 as well,
> without having to merge the complete architecture and kernel header trees
> for them.
Please don't do that. x86_64 are not really synchronized in development
(unlike s390/s390x) and I don't really want too coordinate too much
with the i386 people when I change something in asm. There are also
significant differences in some places already.
Keep it separate is imho far better.
-Andi
> The problem with these (atomic.h
that is a very non portable header and there are several good
alternatives (see the apr library for example). In fact. atomic.h is
*dangerous* in userspace, it is only atomic if CONFIG_SMP is set, so if
you compile your app on a machine without that set and then run it on an
smp machine, you are not atomic.
>
> , bitops.h
again not portable
>
> , byteorder.h,
there are perfectly good alternatives in glibc
>
> div64.h,
huh? what is wrong with "/" in C
> list.h
this one I can see
>
> , spinlock.h
EHHHH????? Spinlocks in userland? You got to be kidding.
> , unaligned.h
weird
> and xor.h)
xor.h is very raid specific (and GPL with lots of code, so a license
trap)
> The problem with these (atomic.h, bitops.h, byteorder.h, div64.h,
> list.h, spinlock.h, unaligned.h and xor.h) is that they provide
> functionality that is needed by many user application but not
> provided by the compiler or libc.
It's already broken on many architectures and just happens to work
on some.
Ok, I've looked for places where someone actually tried using
the kernel headers by googling for /usr/include/asm/foo.h.
The good news is that marking these files broken in
glibc-kernheaders has already pointed most authors to the
source of the problem.
On S?nndag 28 November 2004 08:13, Arjan van de Ven wrote:
>
> > The problem with these (atomic.h
>
> that is a very non portable header and there are several good
> alternatives (see the apr library for example). In fact. atomic.h is
> *dangerous* in userspace, it is only atomic if CONFIG_SMP is set, so if
> you compile your app on a machine without that set and then run it on an
> smp machine, you are not atomic.
Yes, it appears that most people who attempted to use this
have already been bitten so hard that they stopped trying.
> >
> > , bitops.h
>
> again not portable
netbase-ping6 tried to use it, along with some other applications that
I had not heard of.
> >
> > , byteorder.h,
>
> there are perfectly good alternatives in glibc
Google found abuses of byteorder.h in kdeedu, dbootstrap and netatalk,
I would expect to find many more if I kept looking.
> >
> > div64.h,
>
> huh? what is wrong with "/" in C
Ok, probably nobody has tried to do funny things with this one.
> > list.h
>
> this one I can see
Surprisingly few abuses. It is already been protected with
#ifdef __KERNEL__ for some time.
> >
> > , spinlock.h
>
> EHHHH????? Spinlocks in userland? You got to be kidding.
I don't think it's that uncommon to use spinlocks in user
space, IIRC samba (tdb) and some databases implement their
own version of user spinlocks. The kernel implementation
is used at least in "Open Runtime Platform" and in "Chrony".
Chrony even has an FAQ entry on how to circumvent the
"#ifndef __KERNEL__" that was added in Red Hat Linux...
> > , unaligned.h
>
> weird
At least one application that I use every day (vdr) tried including
asm/unaligned.h, but it's rather uncommon otherwise.
> > and xor.h)
>
> xor.h is very raid specific (and GPL with lots of code, so a license
> trap)
Right, this one has never been used from user space AFAICS.
Arnd <><
> > The problem with these (atomic.h
> >
> > that is a very non portable header and there are several good
> > alternatives (see the apr library for example). In fact. atomic.h is
> > *dangerous* in userspace, it is only atomic if CONFIG_SMP is set, so if
> > you compile your app on a machine without that set and then run it on an
> > smp machine, you are not atomic.
>
> Yes, it appears that most people who attempted to use this
> have already been bitten so hard that they stopped trying.
and for people who'd be tempted for new code it should at least warn if
not error even right now
> > > , byteorder.h,
> >
> > there are perfectly good alternatives in glibc
>
> Google found abuses of byteorder.h in kdeedu, dbootstrap and netatalk,
> I would expect to find many more if I kept looking.
it's still very much abuse, and quite nasty for dual endian platforms ;)
> > > , spinlock.h
> >
> > EHHHH????? Spinlocks in userland? You got to be kidding.
>
> I don't think it's that uncommon to use spinlocks in user
> space, IIRC samba (tdb) and some databases implement their
> own version of user spinlocks.
implementing your own is still evil, but glibc provides similar
constructs already. busy waiting in userspace is evil anyway (use futex
:) and.... just as with atomic.h, spinlock.h only gets compiled in for
CONFIG_SMP so the same caveats apply
> The kernel implementation
> is used at least in "Open Runtime Platform" and in "Chrony".
> Chrony even has an FAQ entry on how to circumvent the
> "#ifndef __KERNEL__" that was added in Red Hat Linux...
Chrony then needs to make their own (by copying and cleaning up for all
I care).
Previously Arnd Bergmann wrote:
> Ok, I've looked for places where someone actually tried using
> the kernel headers by googling for /usr/include/asm/foo.h.
> The good news is that marking these files broken in
> glibc-kernheaders has already pointed most authors to the
> source of the problem.
If you want a challenge try if you can get strace to compile with
glibc-kernheaders.
Wichert.
--
Wichert Akkerman <[email protected]> It is simple to make things.
http://www.wiggy.net/ It is hard to make things simple.
On Sun, 2004-11-28 at 13:28 +0100, Wichert Akkerman wrote:
> Previously Arnd Bergmann wrote:
> > Ok, I've looked for places where someone actually tried using
> > the kernel headers by googling for /usr/include/asm/foo.h.
> > The good news is that marking these files broken in
> > glibc-kernheaders has already pointed most authors to the
> > source of the problem.
>
> If you want a challenge try if you can get strace to compile with
> glibc-kernheaders.
the rh strace package seems to cope....
and while glibc-kernheaders hides the kernel internal structures, it
does expose all the external ones...
Arnd Bergmann wrote:
>>#ifdef __x86_64__
>>#include <asm-x86_64/byteorder.h>
>>#else
>>#include <asm-i386/byteorder.h>
>>#endif
>>
>
>
> I think we can get rid of this hack when we move to split kernel headers.
> parisc, s390 and mips already have combined headers, and it should not be
> too hard to combine the user ABI headers for sparc, ppc and x86_64 as well,
> without having to merge the complete architecture and kernel header trees
> for them.
>
If you can produce merged x86/x86_64 ABI headers this would be a nice
solution.
--
Andreas Steinmetz SPAMmers use [email protected]
On S?nndag 28 November 2004 01:55, [email protected] wrote:
> Please don't do that. x86_64 are not really synchronized in development
> (unlike s390/s390x) and I don't really want too coordinate too much
> with the i386 people when I change something in asm. There are also
> significant differences in some places already.
> Keep it separate is imho far better.
>
The ABI is actually a very small part of the asm/ files [1] and typically
one that does not see many changes. I found 5 files (fcntl.h ioctl.h
siginfo.h statfs.h types.h) that have trivial differences between i386
and x86_64 and can be merged, as well as 6 files (posix_types.h ptrace.h
sigcontext.h signal.h stat.h unistd.h) that are so different that it
makes sense to keep them completely separate files with an extra file
to choose between them.
The rest is either not part of the ABI or already identical.
Arnd <><
[1] A quick browse over the files suggests we need only:
errno.h fcntl.h ioctl.h ioctls.h ipc.h ipcbuf.h msgbuf.h
param.h poll.h posix_types.h ptrace.h resource.h sembuf.h
shmbuf.h shmparam.h sigcontext.h siginfo.h signal.h
socket.h sockios.h stat.h statfs.h termbits.h termios.h
types.h unistd.h
If anyone has a better list, please post it.
On S?nndag 28 November 2004 14:24, Andreas Steinmetz wrote:
> > I think we can get rid of this hack when we move to split kernel headers.
> > parisc, s390 and mips already have combined headers, and it should not be
> > too hard to combine the user ABI headers for sparc, ppc and x86_64 as well,
> > without having to merge the complete architecture and kernel header trees
> > for them.
> >
>
> If you can produce merged x86/x86_64 ABI headers this would be a nice
> solution.
I can do that (and the same for ppc/ppc64) as soon as we have
a) A patch that first splits out the ABI headers for i386
b) persuaded ak that he will look at what I come up with
Arnd <><
On Sun, Nov 28, 2004 at 08:13:39AM +0100, Arjan van de Ven wrote:
> > , spinlock.h
>
> EHHHH????? Spinlocks in userland? You got to be kidding.
There is pthread_spin_{init,lock,unlock,trylock,destroy}
in libpthread.so.
Jakub
Arjan van de Ven wrote:
> implementing your own is still evil, but glibc provides similar
> constructs already. busy waiting in userspace is evil anyway (use futex
> :) and.... just as with atomic.h, spinlock.h only gets compiled in for
> CONFIG_SMP so the same caveats apply
Yeah, this is on it's way out now that futexes exist, but KDE (for one) used
to use spinlocks for a thread-pooled malloc (spin-sleep-spin-sleep though,
calling either sched_yield or nanosleep if the lock-grab failed - so it
wasn't a pure busy-wait). It didn't reuse spinlock.h from the kernel, but
it did use the general concept, since the old pre-futex posix mutexes were
crazy-slow for a lock which was heavily used but rarely contended.
Greg KH writes:
> The justification is that it doesn't properly describe the variable size
> (think userspace 32 bit variable vs. kernelspace 32 bit variable.) We
> need to stick with the proper __u32 type variables for data that crosses
> the userspace/kernelspace boundry.
uint32_t is defined to be exactly 32 bits wide, so where's the problem
in using it instead of __u32 in the headers that describe the
user/kernel interface? (Ditto for uint{8,16,64}_t, of course.)
Paul.
On Mon, 29 Nov 2004, Paul Mackerras wrote:
>
> uint32_t is defined to be exactly 32 bits wide, so where's the problem
> in using it instead of __u32 in the headers that describe the
> user/kernel interface? (Ditto for uint{8,16,64}_t, of course.
Ok, this discussion has gone on for too long anyway, but let's make it
easier for everybody. The kernel uses u8/u16/u32 because:
- the kernel should not depend on, or pollute user-space naming.
YOU MUST NOT USE "uint32_t" when that may not be defined, and
user-space rules for when it is defined are arcane and totally
arbitrary.
- since the kernel cannot use those types for anything that is
visible to user space anyway, there has to be alternate names.
The tradition is to prepend two underscores, so the kernel would
have to use "__uint32_t" etc for its header files.
- at that point, there's no longer any valid argument that it's a
"standard type" (it ain't), and I personally find it a lot more
readable to just use the types that the kernel has always used:
__u8/__u16/__u32. For stuff that is only used for the kernel,
the shorter "u8/u16/u32" versions may be used.
In short: having the kernel use the same names as user space is ACTIVELY
BAD, exactly because those names have standards-defined visibility, which
means that the kernel _cannot_ use them in all places anyway. So don't
even _try_.
On the bigger question of what to do with kernel headers in general, let's
just make one thing clear: the kernel headers are for the kernel, and big
and painful re-organizations that don't help _existing_ user space are not
going to happen.
In particular, any re-organization that breaks _existing_ uses is totally
pointless. If you break existing uses, you might as well _not_ re-
organize, since if you consider kernel headers to be purely kernel-
internal (like they should be, but hey, reality trumps any wishes we might
have), then the current organization is perfectly fine.
So I think this whole discussion has been largely pointless. We undeniably
have existing users of kernel headers. That's just a fact. If we break
them, it doesn't _matter_ how the kernel headers look, and then "existing
practice" is about as good an organization as anything, and breaking
things just to break things is not something I'm in the least interested
in. "Beauty" comes secondary to "usefulness".
So I would encourage people to think about ways to clean up some of the
worst warts, but take into account:
- testing it out with whatever random collection of old distributions and
special applications is almost impossible. So every single step of the
way should be (a) small and (b) obviously not break any current users.
- cleanup just for the sake of cleanup always needs to take pain into
account. If you cannot make each small step "worth it", then just don't
do it. If the "cleanup" just adds another file and doesn't actually
_help_ anything that you can point to, it's not a cleanup, it's just an
exercise in wanking.
IOW, I seriously doubt any "let's reorganize header files just to make it
look good" _ever_ accomplished anything. But if there are _specific_
header files that have _specific_ problems, let's look at maybe solving
those. And if you cannot point to a specific problem with a suggested
specific solution, please don't cc me.
Linus
Linus Torvalds wrote:
> In short: having the kernel use the same names as user space is ACTIVELY
> BAD, exactly because those names have standards-defined visibility, which
> means that the kernel _cannot_ use them in all places anyway. So don't
> even _try_.
>
> On the bigger question of what to do with kernel headers in general, let's
> just make one thing clear: the kernel headers are for the kernel, and big
> and painful re-organizations that don't help _existing_ user space are not
> going to happen.
>
> In particular, any re-organization that breaks _existing_ uses is totally
> pointless. If you break existing uses, you might as well _not_ re-
> organize, since if you consider kernel headers to be purely kernel-
> internal (like they should be, but hey, reality trumps any wishes we might
> have), then the current organization is perfectly fine.
I don't think any drastic reorganization is even necessary.
Mariusz Mazur <[email protected]> updates
http://ep09.pld-linux.org/~mmazur/linux-libc-headers/ for each 2.6.x
kernel release. linux-libc-headers are the kernel headers, with all the
kernel-specific stuff stripped out. i.e. userland ABI only. Not sure
how many distros have started picking that up yet... I think Arjan said
Fedora Core had, or would.
If people want to go beyond that, IMHO it would be simple and easy to
start putting new kernel headers in include/kernel (or somesuch). That
way there are no massive reorganizations; kernel-specific stuff gets
slowly migrated to a kernel-specific area.
Jeff
On Sun, Nov 28, 2004 at 11:36:22PM -0500, Jeff Garzik wrote:
> If people want to go beyond that, IMHO it would be simple and easy to
> start putting new kernel headers in include/kernel (or somesuch). That
> way there are no massive reorganizations; kernel-specific stuff gets
> slowly migrated to a kernel-specific area.
ITYM "to areas where it actually gets used". A _lot_ in include/* is
used only by a couple of drivers and should've been sitting in
drivers/*/* instead.
On Mon, Nov 29, 2004 at 04:57:06AM +0000, Al Viro wrote:
> ITYM "to areas where it actually gets used". A _lot_ in include/* is
> used only by a couple of drivers and should've been sitting in
> drivers/*/* instead.
Amen to that! Just because it is a header doesn't mean it belongs in
include. Just because it is a #define or a struct definition does not
mean it belongs in a header.
So many people don't get the idea of exposed vs private interfaces.
On Sun, Nov 28, 2004 at 11:36:22PM -0500, Jeff Garzik wrote:
> >In particular, any re-organization that breaks _existing_ uses is totally
> >pointless. If you break existing uses, you might as well _not_ re-
> >organize, since if you consider kernel headers to be purely kernel-
> >internal (like they should be, but hey, reality trumps any wishes we might
> >have), then the current organization is perfectly fine.
>
>
> I don't think any drastic reorganization is even necessary.
Well, we want things to be split. However a split doesn't mean a
reorganisation as far as userspace visibility is concerned; the filenames
can still be the same, and the typenames etc etc, you just do the kernel
internal additions in a different dir/header
Not breaking userland ever is a dream only, anytime we touch any header,
some userland breaks (they use our spinlock code even!)
> kernel-specific stuff stripped out. i.e. userland ABI only. Not sure
> how many distros have started picking that up yet... I think Arjan said
> Fedora Core had, or would.
nope.
Fedora has it's own set of cleaned up headers. Cleaned up in the sense that
tehre's no #ifdef KERNEL anymore, no inlines (because that's a license trap,
and in addition, all inlines were kernel specific anyway) and all structs
which had spinlocks / semaphores and other kernel private structures in are
removed as well (because they are clearly kernel internal).
On Sun, 2004-11-28 at 17:28 -0800, Linus Torvalds wrote:
> In particular, any re-organization that breaks _existing_ uses is totally
> pointless. If you break existing uses, you might as well _not_ re-
> organize, since if you consider kernel headers to be purely kernel-
> internal (like they should be, but hey, reality trumps any wishes we might
> have), then the current organization is perfectly fine.
Agreed, with the proviso that the inclusion of bitops.h, spinlock.h, and
other kernel-private stuff like that should not be considered 'existing
uses'. It's _already_ broken. It won't compile on some architectures,
and will silently do the wrong thing on others. Causing the compile to
fail consistently would be a _feature_.
> So I think this whole discussion has been largely pointless. We undeniably
> have existing users of kernel headers. That's just a fact. If we break
> them, it doesn't _matter_ how the kernel headers look, and then "existing
> practice" is about as good an organization as anything, and breaking
> things just to break things is not something I'm in the least interested
> in. "Beauty" comes secondary to "usefulness".
Usefulness is what we're after. Preventing the naïve user from including
spinlock.h and thinking that it'll give useful spinlocks would be
useful.
I've lost track of the number of times things have broken because of
incorrect use of kernel headers from userspace. That's what we're trying
to fix -- by putting only the bits which are _supposed_ to be visible
into files which userspace sees, where we know they define part of the
userspace API and hence we can be extremely careful when editing them.
I don't think it makes sense at this point for us to bury our collective
heads in the sand and pretend there isn't a problem here that's worth
fixing.
I agree that it should be obviously correct though -- and that's why
we're trying to end up with a structure that in the first pass would
give us in userspace essentially what we already have in the various
glibc-kernheaders packages, but without the constant and unnecessary
need for some poor sod to keep those up to date by hand.
--
dwmw2
Linus Torvalds writes:
> In particular, any re-organization that breaks _existing_ uses is totally
> pointless. If you break existing uses, you might as well _not_ re-
> organize, since if you consider kernel headers to be purely kernel-
> internal (like they should be, but hey, reality trumps any wishes we might
> have), then the current organization is perfectly fine.
Recently I had some users complaining because their userspace program
that includes <asm/atomic.h> compiles OK as a ppc64 program but not as
a ppc32 program. The reason for that is that asm-ppc/atomic.h has
#ifdef __KERNEL__ around the inline definitions of atomic_inc et al.,
but asm-ppc64/atomic.h doesn't.
My response was that they shouldn't be using <asm/atomic.h> because
(a) it's an internal kernel header, not part of the kernel ABI, (b)
it's not portable (it happens to work on some popular architectures,
but won't work on all) and (c) they're really only entitled to use
those particular inline function definitions in GPL'd programs anyway.
To which the obvious reply is "well then why is it in /usr/include?"
It's because the glibc build process copies all the kernel headers to
/usr/include, but that's not an answer from a user's point of view.
I'd be interested to hear your take on this. Should we try to make
our atomics easy and safe for userspace to use (including putting them
under the LGPL)? Or can you see a better solution?
Paul.
On Mon, 2004-11-29 at 20:53 +1100, Paul Mackerras wrote:
> Linus Torvalds writes:
>
> > In particular, any re-organization that breaks _existing_ uses is totally
> > pointless. If you break existing uses, you might as well _not_ re-
> > organize, since if you consider kernel headers to be purely kernel-
> > internal (like they should be, but hey, reality trumps any wishes we might
> > have), then the current organization is perfectly fine.
>
> Recently I had some users complaining because their userspace program
> that includes <asm/atomic.h> compiles OK as a ppc64 program but not as
> a ppc32 program. The reason for that is that asm-ppc/atomic.h has
> #ifdef __KERNEL__ around the inline definitions of atomic_inc et al.,
> but asm-ppc64/atomic.h doesn't.
>
> My response was that they shouldn't be using <asm/atomic.h> because
> (a) it's an internal kernel header, not part of the kernel ABI, (b)
> it's not portable (it happens to work on some popular architectures,
> but won't work on all) and (c) they're really only entitled to use
> those particular inline function definitions in GPL'd programs anyway.
and (d) on x86 at least the "atomic" part of the atomic ops are #ifdef
CONFIG_SMP, which might well not be set in userland... so you get a
false sense of atomicity.
> I'd be interested to hear your take on this. Should we try to make
> our atomics easy and safe for userspace to use (including putting them
> under the LGPL)? Or can you see a better solution?
it's not the kernel's job to provide this functionality imo.
glibc could. glib does. apr does. there's several options already.
apr is BSD licensed for exanpmle, glib LGPL.
On Mon, 2004-11-29 at 10:57 +0100, Arjan van de Ven wrote:
> > I'd be interested to hear your take on this. Should we try to make
> > our atomics easy and safe for userspace to use (including putting them
> > under the LGPL)? Or can you see a better solution?
>
> it's not the kernel's job to provide this functionality imo.
Sometimes the functionality would have to be entirely reimplemented to
make it viable for userspace. For FR-V we set aside a condition register
for use in the kernel, so we can emulate ll/sc with multiple-issue
instructions. That can never work in userspace.
We can't just pretend there isn't a problem here.
--
dwmw2
Grzegorz Kulewski <[email protected]>:
> > (b) Make kernel file #include the user file.
>
> Does kernel really need to include user headers? When it is definition of
> some const then it should be defined in one file (to be sure it has only
> one definition). But user headers may have some compatibility hacks that
> kernel do not need (and even maybe does not want) to have.
>
> How you will handle that?
I must have mis-explained it. I meant:
> > (b) Make kernel file #include the user/ file.
I don't actually mean that the kernel should in anyway touch userspace
headers. I was referring to the bits split out into user-xxxx/ directories.
David
On poniedzia?ek 29 listopad 2004 02:28, Linus Torvalds wrote:
> So I would encourage people to think about ways to clean up some of the
> worst warts, but take into account:
> - testing it out with whatever random collection of old distributions and
> special applications is almost impossible. So every single step of the
> way should be (a) small and (b) obviously not break any current users.
If I understand correctly the above means "include/* userland breakage
problems open for fixing" and the "kernel includes aren't for userland" part
is obsolete?
> - cleanup just for the sake of cleanup always needs to take pain into
> account. If you cannot make each small step "worth it", then just don't
> do it. If the "cleanup" just adds another file and doesn't actually
> _help_ anything that you can point to, it's not a cleanup, it's just an
> exercise in wanking.
>
> IOW, I seriously doubt any "let's reorganize header files just to make it
> look good" _ever_ accomplished anything. But if there are _specific_
> header files that have _specific_ problems, let's look at maybe solving
> those. And if you cannot point to a specific problem with a suggested
> specific solution, please don't cc me.
Potential breakage is mostly a non issue - since vanilla kernel headers are
mostly useless for userland anyway, one must either patch them or use my
headers (fedora's?). As for my headers - I can always provide a compatibility
layer that will look like the old tree but in fact use the new tree.* And
that also means application specific hacks not to break some app (I'm doing
that now anyway, so no difference).
I'm *really* for fixing the header problem once and for all one way (lifting
the 'kernel includes aren't for userland' ban) or the other (making a
separate, userland available part that glibc and other apps can use cleanly).
Maintaining a separate tree of those headers ain't much fun and it would be
nice if I could quit doing that some time in the future.
* Fluent and (mostly) painless migration guaranteed or your money back.
--
In the year eighty five ten
God is gonna shake his mighty head
He'll either say,
"I'm pleased where man has been"
Or tear it down, and start again
On Nov 26, 2004, David Howells <[email protected]> wrote:
> How about calling the interface headers "kapi*/" instead of "user*/". In case
> you haven't guessed, "kapi" would be short for "kernel-api".
I've seen kapi being used to reference the api exposed by the kernel
to modules, but not the abi exposed to userland. ukabi sounds more
appropriate for the latter, although many people might wonder what's
with this United Kingdom ABI :-)
--
Alexandre Oliva http://www.ic.unicamp.br/~oliva/
Red Hat Compiler Engineer aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist oliva@{lsd.ic.unicamp.br, gnu.org}
On Nov 28, 2004, Linus Torvalds <[email protected]> wrote:
> On the bigger question of what to do with kernel headers in general, let's
> just make one thing clear: the kernel headers are for the kernel, and big
> and painful re-organizations that don't help _existing_ user space are not
> going to happen.
So, of the following two options, which one you think everybody should
pick?
- Linux gets to define the ABI between kernel and userland, and
userland must duplicate the contents of headers in which the kernel
defines the kernel<->userland ABI, tracking changes in them in the
hope that nothing falls through the cracks
- we move the kernel<->userland to a separate package, where it's
maintained such that it can be used by both kernel and userland, and
Linux will only build when given a pointer to the location of this
package.
I don't think you can have it both ways. As long as the kernel wants
freedom to set the ABI, either let people use the headers where you
define it, or maintain the ABI separately and let people use them.
Forcing others to maintain duplicates of your headers will just get
most people to use your headers for purposes they weren't meant to be
used; maintaining two similar but far from identical sets of headers
in sync is a major pain, that's why few people do it, and even fewer
people do it properly.
Refusing to help in the effort to maintain them for wishes of freedom
to tinker or allergy to external dependencies haven't worked as
excuses in the past, and I think they never will. Duplicating the
effort is just stupid, and requiring others to do so is arrogant. Why
shouldn't we just get together and have something that works for both?
Or how about we turn the table around and define an ABI that the
kernel must implement, and then have the kernel have to keep track of
the changes in such external ABI definition, instead of the way things
(fail to) work now? How comfortable would you feel with this
arrangement?
My usual reply when I hear this argument that the kernel headers that
define the ABI with userland are to be used by the kernel only, and
someone else is supposed to create the headers for the same purpose to
be used by userland, is that you generally won't find the headers that
expose the interface of a library in a separate package (think
libz.{a,so} and zlib.h). They're part of the same package, built out
of the same sources, and there's a reason for that: if they get out of
sync, it won't serve its purpose. It makes far more sense to maintain
them together.
You won't find zlib.h in packages that just want to use zlib, unless
they contain zlib in its entirety, just in case it's not
pre-installed. I think this makes perfect sense. Requiring libc or a
separate package used by libc to contain copies of headers that define
the API/ABI exported by the kernel to userland is just the same sort
of nonsense, with the difference that it's a way bigger set of headers
we're talking about, which renders the problem even worse.
Sure you can take the easy way out and claim you told people they
shouldn't do what they're doing when they run into problems because of
including kernel headers from userland. But as soon as you reject
proposals to clean things up such that we would have a set of headers
that both kernel and userland will be able to use, in a way that would
have the headers still conveniently (for you) sitting in the kernel
source tree, I can't see that you could say it's not your problem with
a straight face any more.
--
Alexandre Oliva http://www.ic.unicamp.br/~oliva/
Red Hat Compiler Engineer aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist oliva@{lsd.ic.unicamp.br, gnu.org}
On Mon, 29 Nov 2004, Alexandre Oliva wrote:
>
> > On the bigger question of what to do with kernel headers in general, let's
> > just make one thing clear: the kernel headers are for the kernel, and big
> > and painful re-organizations that don't help _existing_ user space are not
> > going to happen.
>
> So, of the following two options, which one you think everybody should
> pick?
Ok, that's pretty straightforward:
> - Linux gets to define the ABI between kernel and userland, and
> userland must duplicate the contents of headers in which the kernel
> defines the kernel<->userland ABI, tracking changes in them in the
> hope that nothing falls through the cracks
This is unquestionably true. The kernel obviously _does_ define the ABI,
and userland just lives with it. At some point you have to track things,
just because new features etc just can't be sanely handled any other way.
That said, I think we can make the tracking _easier_.
> - we move the kernel<->userland to a separate package, where it's
> maintained such that it can be used by both kernel and userland, and
> Linux will only build when given a pointer to the location of this
> package.
No. Quite frankly, I absolutely hate projects that do that. I gave up on
following several interesting projects (mostly media decoding) just
because it was just too damn painful to try to make sure that five
different packages were all in sync, and you couldn't find one site to
just download it all from.
So I want the kernel to be as stand-alone as humanly possible. There are a
few dependencies I can't avoid, notably the compiler, and there the kernel
triest to be as permissive as humanly reasonable, and we've always tried
to keep the required tools down to a manageable level (ie even when people
were clamorign for graphical configurations etc, the kernel always made
that very optional indeed).
***
Now, on trying to make the tracking _easier_, I would not mind at all to
move (well-defined) things around a bit to make it clearer what is
actually exported to user space. But on the other hand, I don't think it's
actively wrong either to just "mark" them in the headers some way, and
have tools to extract it automatically _without_ having to separate them
into some magic location.
In fact, in many ways I'd prefer to have source-level annotations like
"this is exported to user space" over trying to gather things in one
place. It would need to have some automated checking ability (that
probably most people wouldn't run, and would break every once in a while,
but hey, that's nothing new. That's how "checkconfig" etc works).
But it doesn't have to be "one or the other". I think we could annotate
some things that are nasty to export (and quite frankly, it's not like
this is something "new"; __KERNEL__ is really nothing but a stupid sort of
an "annotation" already, except it's likely the wrong way around: we
should not annotate the stuff that is kernel-only, we should annotate the
stuff that is user-visible, since that should be the exception rather than
the rule).
But even if we decide to have part of the ABI annotated, part of it could
be split out better. For example, I think the "posix_types.h" split ended
up working really quite well, and it cleaned up a _huge_ number of type
issues that people have probably already forgotten because that split was
done so long ago.
So we _have_ had great success with the "split out specific things"
before, and I think we should do it whenever there is an obvious chance,
and something is abstracted out and well-defined enough that we can do so.
But in general, I do kind of like the explicit marking. The same way we
explicitly mark the functions inside the kernel that we expose to modules,
we could try to mark the data structures and values that we expose to user
space. That tends to "work".
Linus
Followup to: <[email protected]>
By author: David Howells <[email protected]>
In newsgroup: linux.dev.kernel
>
> What we've come up with is this:
>
> (1) Create new directories in the linux sources to shadow existing include
> directories:
>
> NEW DIRECTORY DIRECTORY SHADOWED
> ============= ==================
> include/user/ include/linux/
> include/user-*/ include/asm-*/
>
> Note that this doesn't take account of the other directories under
> include/, but I don't think they're relevant.
>
I'm not sure if user is a good choice, and user-* is even worse. Most
people seem to have suggested include/linux-abi for this; I would
personally prefer include/linux-abi/arch for the second.
> (2) Take each file from the shadowed directory. If it has any userspace
> relevant stuff, then:
>
> (a) Transfer this stuff into a file of the same name in the new
> directory. So, for example, the syscall number list from
> include/asm-i386/unistd.h will be transferred to
> include/user-i386/unistd.h.
I'm not sure you can do such a 1:1 mapping. In fact, there are cases
where you definitely don't want to, because the current structure
doesn't make much sense.
>
> (b) Make kernel file #include the user file. So:
>
> [include/asm-i386/unistd.h]
> ...
> #include <user-i386/unistd.h>
> ...
Good...
> (c) Where a user header file requires something from another header file
> (such as a type), that file should include a suitable user header file
> directly:
>
> [include/user-i386/termio.h]
> ...
> #include <user/types.h>
> ...
Good...
> (d) stdint types should be used where possible.
>
> [include/user-i386/termios.h]
> struct winsize {
> uint16_t ws_row;
> uint16_t ws_col;
> uint16_t ws_xpixel;
> uint16_t ws_ypixel;
> };
Good, except your "struct winsize" is bad; you're stepping on
namespace which belongs to userspace. Since we can't use typedefs on
struct tags, I suggest:
struct __kstruct_winsize {
/* ... */
};
.. and userspace can do:
#define __kstruct_winsize winsize
> (e) These header files should be bounded with __USER_XXXXX_H conditionals:
>
> [include/user-i386/termios.h]
> #ifndef __USER_I386_TERMIOS_H
> #define __USER_I386_TERMIOS_H
> ...
> #endif /* __USER_I386_TERMIOS_H */
Good...
> (3) Remove all #if(n)def __KERNEL__ clauses.
>
> (4) Remove the -D__KERNEL__ from the master kernel Makefile.
Bad! There is code in the kernel which can compile in userspace for
testing. This is highly valuable and should be kept.
-hpa
Speaking of include/* cleanups, here's a work of about 10 minutes
(and yes, result does build):
* private stuff moved into fs/affs/affs.h
* include/linux/affs_fs_{i,sb}.h removed, since they do not contain
anything that might be used outside of fs/affs/*
* includes in fs/affs/* trimmed (as the matter of fact, only super.c
needed something not pulled by affs.h)
* duplicate MODULE_LICENSE removed from fs/affs/inode.c (fs/affs/super.c
already has exact same thing)
* bogus extern (affs_fs()) removed (used to be in affs_fs.h, function
in question simply doesn't exist)
That leaves affs_fs.h containing only AFFS_SUPER_MAGIC declaration and
amigaffs.h also seriously cleaned up; it still has a bunch of defines that
are almost certainly never used in userland (they refer to ->b_data and
in-core affs superblock), but these might in theory be useful (similar
stuff for ext2 *is* used by userland code).
If Roman is OK with their removal, AFFS_DATA and friends should also get
moved to fs/affs/affs.h.
Signed-off-by: Al Viro <[email protected]>
----
diff -urN RC10-rc2-bk12-base/fs/affs/affs.h RC10-rc2-bk12-current/fs/affs/affs.h
--- RC10-rc2-bk12-base/fs/affs/affs.h 1969-12-31 19:00:00.000000000 -0500
+++ RC10-rc2-bk12-current/fs/affs/affs.h 2004-11-29 17:20:17.536489880 -0500
@@ -0,0 +1,283 @@
+#include <linux/types.h>
+#include <linux/fs.h>
+#include <linux/buffer_head.h>
+#include <linux/affs_fs.h>
+#include <linux/amigaffs.h>
+
+#define AFFS_CACHE_SIZE PAGE_SIZE
+
+#define AFFS_MAX_PREALLOC 32
+#define AFFS_LC_SIZE (AFFS_CACHE_SIZE/sizeof(u32)/2)
+#define AFFS_AC_SIZE (AFFS_CACHE_SIZE/sizeof(struct affs_ext_key)/2)
+#define AFFS_AC_MASK (AFFS_AC_SIZE-1)
+
+struct affs_ext_key {
+ u32 ext; /* idx of the extended block */
+ u32 key; /* block number */
+};
+
+/*
+ * affs fs inode data in memory
+ */
+struct affs_inode_info {
+ u32 i_opencnt;
+ struct semaphore i_link_lock; /* Protects internal inode access. */
+ struct semaphore i_ext_lock; /* Protects internal inode access. */
+#define i_hash_lock i_ext_lock
+ u32 i_blkcnt; /* block count */
+ u32 i_extcnt; /* extended block count */
+ u32 *i_lc; /* linear cache of extended blocks */
+ u32 i_lc_size;
+ u32 i_lc_shift;
+ u32 i_lc_mask;
+ struct affs_ext_key *i_ac; /* associative cache of extended blocks */
+ u32 i_ext_last; /* last accessed extended block */
+ struct buffer_head *i_ext_bh; /* bh of last extended block */
+ loff_t mmu_private;
+ u32 i_protect; /* unused attribute bits */
+ u32 i_lastalloc; /* last allocated block */
+ int i_pa_cnt; /* number of preallocated blocks */
+#if 0
+ s32 i_original; /* if != 0, this is the key of the original */
+ u32 i_data[AFFS_MAX_PREALLOC]; /* preallocated blocks */
+ int i_cache_users; /* Cache cannot be freed while > 0 */
+ unsigned char i_hlink; /* This is a fake */
+ unsigned char i_pad;
+ s32 i_parent; /* parent ino */
+#endif
+ struct inode vfs_inode;
+};
+
+/* short cut to get to the affs specific inode data */
+static inline struct affs_inode_info *AFFS_I(struct inode *inode)
+{
+ return list_entry(inode, struct affs_inode_info, vfs_inode);
+}
+
+/*
+ * super-block data in memory
+ *
+ * Block numbers are adjusted for their actual size
+ *
+ */
+
+struct affs_bm_info {
+ u32 bm_key; /* Disk block number */
+ u32 bm_free; /* Free blocks in here */
+};
+
+struct affs_sb_info {
+ int s_partition_size; /* Partition size in blocks. */
+ int s_reserved; /* Number of reserved blocks. */
+ //u32 s_blksize; /* Initial device blksize */
+ u32 s_data_blksize; /* size of the data block w/o header */
+ u32 s_root_block; /* FFS root block number. */
+ int s_hashsize; /* Size of hash table. */
+ unsigned long s_flags; /* See below. */
+ uid_t s_uid; /* uid to override */
+ gid_t s_gid; /* gid to override */
+ umode_t s_mode; /* mode to override */
+ struct buffer_head *s_root_bh; /* Cached root block. */
+ struct semaphore s_bmlock; /* Protects bitmap access. */
+ struct affs_bm_info *s_bitmap; /* Bitmap infos. */
+ u32 s_bmap_count; /* # of bitmap blocks. */
+ u32 s_bmap_bits; /* # of bits in one bitmap blocks */
+ u32 s_last_bmap;
+ struct buffer_head *s_bmap_bh;
+ char *s_prefix; /* Prefix for volumes and assigns. */
+ int s_prefix_len; /* Length of prefix. */
+ char s_volume[32]; /* Volume prefix for absolute symlinks. */
+};
+
+#define SF_INTL 0x0001 /* International filesystem. */
+#define SF_BM_VALID 0x0002 /* Bitmap is valid. */
+#define SF_IMMUTABLE 0x0004 /* Protection bits cannot be changed */
+#define SF_QUIET 0x0008 /* chmod errors will be not reported */
+#define SF_SETUID 0x0010 /* Ignore Amiga uid */
+#define SF_SETGID 0x0020 /* Ignore Amiga gid */
+#define SF_SETMODE 0x0040 /* Ignore Amiga protection bits */
+#define SF_MUFS 0x0100 /* Use MUFS uid/gid mapping */
+#define SF_OFS 0x0200 /* Old filesystem */
+#define SF_PREFIX 0x0400 /* Buffer for prefix is allocated */
+#define SF_VERBOSE 0x0800 /* Talk about fs when mounting */
+
+/* short cut to get to the affs specific sb data */
+static inline struct affs_sb_info *AFFS_SB(struct super_block *sb)
+{
+ return sb->s_fs_info;
+}
+
+/* amigaffs.c */
+
+extern int affs_insert_hash(struct inode *inode, struct buffer_head *bh);
+extern int affs_remove_hash(struct inode *dir, struct buffer_head *rem_bh);
+extern int affs_remove_header(struct dentry *dentry);
+extern u32 affs_checksum_block(struct super_block *sb, struct buffer_head *bh);
+extern void affs_fix_checksum(struct super_block *sb, struct buffer_head *bh);
+extern void secs_to_datestamp(time_t secs, struct affs_date *ds);
+extern mode_t prot_to_mode(u32 prot);
+extern void mode_to_prot(struct inode *inode);
+extern void affs_error(struct super_block *sb, const char *function, const char *fmt, ...);
+extern void affs_warning(struct super_block *sb, const char *function, const char *fmt, ...);
+extern int affs_check_name(const unsigned char *name, int len);
+extern int affs_copy_name(unsigned char *bstr, struct dentry *dentry);
+
+/* bitmap. c */
+
+extern u32 affs_count_free_blocks(struct super_block *s);
+extern void affs_free_block(struct super_block *sb, u32 block);
+extern u32 affs_alloc_block(struct inode *inode, u32 goal);
+extern int affs_init_bitmap(struct super_block *sb, int *flags);
+extern void affs_free_bitmap(struct super_block *sb);
+
+/* namei.c */
+
+extern int affs_hash_name(struct super_block *sb, const u8 *name, unsigned int len);
+extern struct dentry *affs_lookup(struct inode *dir, struct dentry *dentry, struct nameidata *);
+extern int affs_unlink(struct inode *dir, struct dentry *dentry);
+extern int affs_create(struct inode *dir, struct dentry *dentry, int mode, struct nameidata *);
+extern int affs_mkdir(struct inode *dir, struct dentry *dentry, int mode);
+extern int affs_rmdir(struct inode *dir, struct dentry *dentry);
+extern int affs_link(struct dentry *olddentry, struct inode *dir,
+ struct dentry *dentry);
+extern int affs_symlink(struct inode *dir, struct dentry *dentry,
+ const char *symname);
+extern int affs_rename(struct inode *old_dir, struct dentry *old_dentry,
+ struct inode *new_dir, struct dentry *new_dentry);
+
+/* inode.c */
+
+extern unsigned long affs_parent_ino(struct inode *dir);
+extern struct inode *affs_new_inode(struct inode *dir);
+extern int affs_notify_change(struct dentry *dentry, struct iattr *attr);
+extern void affs_put_inode(struct inode *inode);
+extern void affs_delete_inode(struct inode *inode);
+extern void affs_clear_inode(struct inode *inode);
+extern void affs_read_inode(struct inode *inode);
+extern int affs_write_inode(struct inode *inode, int);
+extern int affs_add_entry(struct inode *dir, struct inode *inode, struct dentry *dentry, s32 type);
+
+/* file.c */
+
+void affs_free_prealloc(struct inode *inode);
+extern void affs_truncate(struct inode *);
+
+/* dir.c */
+
+extern void affs_dir_truncate(struct inode *);
+
+/* jump tables */
+
+extern struct inode_operations affs_file_inode_operations;
+extern struct inode_operations affs_dir_inode_operations;
+extern struct inode_operations affs_symlink_inode_operations;
+extern struct file_operations affs_file_operations;
+extern struct file_operations affs_file_operations_ofs;
+extern struct file_operations affs_dir_operations;
+extern struct address_space_operations affs_symlink_aops;
+extern struct address_space_operations affs_aops;
+extern struct address_space_operations affs_aops_ofs;
+
+extern struct dentry_operations affs_dentry_operations;
+extern struct dentry_operations affs_dentry_operations_intl;
+
+static inline void
+affs_set_blocksize(struct super_block *sb, int size)
+{
+ sb_set_blocksize(sb, size);
+}
+static inline struct buffer_head *
+affs_bread(struct super_block *sb, int block)
+{
+ pr_debug("affs_bread: %d\n", block);
+ if (block >= AFFS_SB(sb)->s_reserved && block < AFFS_SB(sb)->s_partition_size)
+ return sb_bread(sb, block);
+ return NULL;
+}
+static inline struct buffer_head *
+affs_getblk(struct super_block *sb, int block)
+{
+ pr_debug("affs_getblk: %d\n", block);
+ if (block >= AFFS_SB(sb)->s_reserved && block < AFFS_SB(sb)->s_partition_size)
+ return sb_getblk(sb, block);
+ return NULL;
+}
+static inline struct buffer_head *
+affs_getzeroblk(struct super_block *sb, int block)
+{
+ struct buffer_head *bh;
+ pr_debug("affs_getzeroblk: %d\n", block);
+ if (block >= AFFS_SB(sb)->s_reserved && block < AFFS_SB(sb)->s_partition_size) {
+ bh = sb_getblk(sb, block);
+ lock_buffer(bh);
+ memset(bh->b_data, 0 , sb->s_blocksize);
+ set_buffer_uptodate(bh);
+ unlock_buffer(bh);
+ return bh;
+ }
+ return NULL;
+}
+static inline struct buffer_head *
+affs_getemptyblk(struct super_block *sb, int block)
+{
+ struct buffer_head *bh;
+ pr_debug("affs_getemptyblk: %d\n", block);
+ if (block >= AFFS_SB(sb)->s_reserved && block < AFFS_SB(sb)->s_partition_size) {
+ bh = sb_getblk(sb, block);
+ wait_on_buffer(bh);
+ set_buffer_uptodate(bh);
+ return bh;
+ }
+ return NULL;
+}
+static inline void
+affs_brelse(struct buffer_head *bh)
+{
+ if (bh)
+ pr_debug("affs_brelse: %lld\n", (long long) bh->b_blocknr);
+ brelse(bh);
+}
+
+static inline void
+affs_adjust_checksum(struct buffer_head *bh, u32 val)
+{
+ u32 tmp = be32_to_cpu(((__be32 *)bh->b_data)[5]);
+ ((__be32 *)bh->b_data)[5] = cpu_to_be32(tmp - val);
+}
+static inline void
+affs_adjust_bitmapchecksum(struct buffer_head *bh, u32 val)
+{
+ u32 tmp = be32_to_cpu(((__be32 *)bh->b_data)[0]);
+ ((__be32 *)bh->b_data)[0] = cpu_to_be32(tmp - val);
+}
+
+static inline void
+affs_lock_link(struct inode *inode)
+{
+ down(&AFFS_I(inode)->i_link_lock);
+}
+static inline void
+affs_unlock_link(struct inode *inode)
+{
+ up(&AFFS_I(inode)->i_link_lock);
+}
+static inline void
+affs_lock_dir(struct inode *inode)
+{
+ down(&AFFS_I(inode)->i_hash_lock);
+}
+static inline void
+affs_unlock_dir(struct inode *inode)
+{
+ up(&AFFS_I(inode)->i_hash_lock);
+}
+static inline void
+affs_lock_ext(struct inode *inode)
+{
+ down(&AFFS_I(inode)->i_ext_lock);
+}
+static inline void
+affs_unlock_ext(struct inode *inode)
+{
+ up(&AFFS_I(inode)->i_ext_lock);
+}
diff -urN RC10-rc2-bk12-base/fs/affs/amigaffs.c RC10-rc2-bk12-current/fs/affs/amigaffs.c
--- RC10-rc2-bk12-base/fs/affs/amigaffs.c 2004-10-18 17:53:43.000000000 -0400
+++ RC10-rc2-bk12-current/fs/affs/amigaffs.c 2004-11-29 17:00:38.594716272 -0500
@@ -8,14 +8,7 @@
* Please send bug reports to: [email protected]
*/
-#include <stdarg.h>
-#include <linux/stat.h>
-#include <linux/time.h>
-#include <linux/affs_fs.h>
-#include <linux/string.h>
-#include <linux/mm.h>
-#include <linux/amigaffs.h>
-#include <linux/buffer_head.h>
+#include "affs.h"
extern struct timezone sys_tz;
diff -urN RC10-rc2-bk12-base/fs/affs/bitmap.c RC10-rc2-bk12-current/fs/affs/bitmap.c
--- RC10-rc2-bk12-base/fs/affs/bitmap.c 2004-10-18 17:55:28.000000000 -0400
+++ RC10-rc2-bk12-current/fs/affs/bitmap.c 2004-11-29 17:01:22.855987536 -0500
@@ -7,15 +7,7 @@
* block allocation, deallocation, calculation of free space.
*/
-#include <linux/time.h>
-#include <linux/affs_fs.h>
-#include <linux/stat.h>
-#include <linux/kernel.h>
-#include <linux/slab.h>
-#include <linux/string.h>
-#include <linux/bitops.h>
-#include <linux/amigaffs.h>
-#include <linux/buffer_head.h>
+#include "affs.h"
/* This is, of course, shamelessly stolen from fs/minix */
diff -urN RC10-rc2-bk12-base/fs/affs/dir.c RC10-rc2-bk12-current/fs/affs/dir.c
--- RC10-rc2-bk12-base/fs/affs/dir.c 2004-10-18 17:54:55.000000000 -0400
+++ RC10-rc2-bk12-current/fs/affs/dir.c 2004-11-29 17:02:15.392000848 -0500
@@ -13,17 +13,7 @@
*
*/
-#include <asm/uaccess.h>
-#include <linux/errno.h>
-#include <linux/fs.h>
-#include <linux/kernel.h>
-#include <linux/affs_fs.h>
-#include <linux/stat.h>
-#include <linux/string.h>
-#include <linux/mm.h>
-#include <linux/amigaffs.h>
-#include <linux/smp_lock.h>
-#include <linux/buffer_head.h>
+#include "affs.h"
static int affs_readdir(struct file *, void *, filldir_t);
diff -urN RC10-rc2-bk12-base/fs/affs/file.c RC10-rc2-bk12-current/fs/affs/file.c
--- RC10-rc2-bk12-base/fs/affs/file.c 2004-10-18 17:54:08.000000000 -0400
+++ RC10-rc2-bk12-current/fs/affs/file.c 2004-11-29 17:04:02.952649144 -0500
@@ -12,24 +12,7 @@
* affs regular file handling primitives
*/
-#include <asm/div64.h>
-#include <asm/uaccess.h>
-#include <asm/system.h>
-#include <linux/time.h>
-#include <linux/affs_fs.h>
-#include <linux/fcntl.h>
-#include <linux/kernel.h>
-#include <linux/errno.h>
-#include <linux/slab.h>
-#include <linux/stat.h>
-#include <linux/smp_lock.h>
-#include <linux/dirent.h>
-#include <linux/fs.h>
-#include <linux/amigaffs.h>
-#include <linux/mm.h>
-#include <linux/highmem.h>
-#include <linux/pagemap.h>
-#include <linux/buffer_head.h>
+#include "affs.h"
#if PAGE_SIZE < 4096
#error PAGE_SIZE must be at least 4096
diff -urN RC10-rc2-bk12-base/fs/affs/inode.c RC10-rc2-bk12-current/fs/affs/inode.c
--- RC10-rc2-bk12-base/fs/affs/inode.c 2004-10-18 17:54:07.000000000 -0400
+++ RC10-rc2-bk12-current/fs/affs/inode.c 2004-11-29 17:11:20.702101112 -0500
@@ -10,26 +10,7 @@
* (C) 1991 Linus Torvalds - minix filesystem
*/
-#include <asm/div64.h>
-#include <linux/errno.h>
-#include <linux/fs.h>
-#include <linux/slab.h>
-#include <linux/stat.h>
-#include <linux/time.h>
-#include <linux/affs_fs.h>
-#include <linux/kernel.h>
-#include <linux/mm.h>
-#include <linux/string.h>
-#include <linux/genhd.h>
-#include <linux/amigaffs.h>
-#include <linux/major.h>
-#include <linux/blkdev.h>
-#include <linux/init.h>
-#include <linux/smp_lock.h>
-#include <linux/buffer_head.h>
-#include <asm/system.h>
-#include <asm/uaccess.h>
-#include <linux/module.h>
+#include "affs.h"
extern struct inode_operations affs_symlink_inode_operations;
extern struct timezone sys_tz;
@@ -428,4 +409,3 @@
affs_unlock_link(inode);
goto done;
}
-MODULE_LICENSE("GPL");
diff -urN RC10-rc2-bk12-base/fs/affs/namei.c RC10-rc2-bk12-current/fs/affs/namei.c
--- RC10-rc2-bk12-base/fs/affs/namei.c 2004-10-18 17:53:46.000000000 -0400
+++ RC10-rc2-bk12-current/fs/affs/namei.c 2004-11-29 17:15:09.782275632 -0500
@@ -8,23 +8,10 @@
* (C) 1991 Linus Torvalds - minix filesystem
*/
-#include <linux/time.h>
-#include <linux/affs_fs.h>
-#include <linux/kernel.h>
-#include <linux/string.h>
-#include <linux/stat.h>
-#include <linux/fcntl.h>
-#include <linux/amigaffs.h>
-#include <linux/smp_lock.h>
-#include <linux/buffer_head.h>
-#include <asm/uaccess.h>
-
-#include <linux/errno.h>
+#include "affs.h"
typedef int (*toupper_t)(int);
-extern struct inode_operations affs_symlink_inode_operations;
-
static int affs_toupper(int ch);
static int affs_hash_dentry(struct dentry *, struct qstr *);
static int affs_compare_dentry(struct dentry *, struct qstr *, struct qstr *);
diff -urN RC10-rc2-bk12-base/fs/affs/super.c RC10-rc2-bk12-current/fs/affs/super.c
--- RC10-rc2-bk12-base/fs/affs/super.c 2004-10-18 17:54:39.000000000 -0400
+++ RC10-rc2-bk12-current/fs/affs/super.c 2004-11-29 17:17:59.947406600 -0500
@@ -11,26 +11,10 @@
*/
#include <linux/module.h>
-#include <linux/errno.h>
-#include <linux/fs.h>
-#include <linux/slab.h>
-#include <linux/stat.h>
-#include <linux/time.h>
-#include <linux/affs_fs.h>
-#include <linux/kernel.h>
-#include <linux/mm.h>
-#include <linux/string.h>
-#include <linux/genhd.h>
-#include <linux/amigaffs.h>
-#include <linux/major.h>
-#include <linux/blkdev.h>
#include <linux/init.h>
-#include <linux/smp_lock.h>
-#include <linux/buffer_head.h>
-#include <linux/vfs.h>
+#include <linux/statfs.h>
#include <linux/parser.h>
-#include <asm/system.h>
-#include <asm/uaccess.h>
+#include "affs.h"
extern struct timezone sys_tz;
diff -urN RC10-rc2-bk12-base/fs/affs/symlink.c RC10-rc2-bk12-current/fs/affs/symlink.c
--- RC10-rc2-bk12-base/fs/affs/symlink.c 2004-10-18 17:53:42.000000000 -0400
+++ RC10-rc2-bk12-current/fs/affs/symlink.c 2004-11-29 17:14:21.318643224 -0500
@@ -8,14 +8,7 @@
* affs symlink handling code
*/
-#include <linux/errno.h>
-#include <linux/fs.h>
-#include <linux/stat.h>
-#include <linux/affs_fs.h>
-#include <linux/amigaffs.h>
-#include <linux/pagemap.h>
-#include <linux/smp_lock.h>
-#include <linux/buffer_head.h>
+#include "affs.h"
static int affs_symlink_readpage(struct file *file, struct page *page)
{
diff -urN RC10-rc2-bk12-base/include/linux/affs_fs.h RC10-rc2-bk12-current/include/linux/affs_fs.h
--- RC10-rc2-bk12-base/include/linux/affs_fs.h 2004-10-18 17:53:06.000000000 -0400
+++ RC10-rc2-bk12-current/include/linux/affs_fs.h 2004-11-29 16:53:07.815245168 -0500
@@ -3,94 +3,5 @@
/*
* The affs filesystem constants/structures
*/
-
-#include <linux/types.h>
-
-#include <linux/affs_fs_i.h>
-#include <linux/affs_fs_sb.h>
-
#define AFFS_SUPER_MAGIC 0xadff
-
-struct affs_date;
-
-/* --- Prototypes ----------------------------------------------------------------------------- */
-
-/* amigaffs.c */
-
-extern int affs_insert_hash(struct inode *inode, struct buffer_head *bh);
-extern int affs_remove_hash(struct inode *dir, struct buffer_head *rem_bh);
-extern int affs_remove_header(struct dentry *dentry);
-extern u32 affs_checksum_block(struct super_block *sb, struct buffer_head *bh);
-extern void affs_fix_checksum(struct super_block *sb, struct buffer_head *bh);
-extern void secs_to_datestamp(time_t secs, struct affs_date *ds);
-extern mode_t prot_to_mode(u32 prot);
-extern void mode_to_prot(struct inode *inode);
-extern void affs_error(struct super_block *sb, const char *function, const char *fmt, ...);
-extern void affs_warning(struct super_block *sb, const char *function, const char *fmt, ...);
-extern int affs_check_name(const unsigned char *name, int len);
-extern int affs_copy_name(unsigned char *bstr, struct dentry *dentry);
-
-/* bitmap. c */
-
-extern u32 affs_count_free_blocks(struct super_block *s);
-extern void affs_free_block(struct super_block *sb, u32 block);
-extern u32 affs_alloc_block(struct inode *inode, u32 goal);
-extern int affs_init_bitmap(struct super_block *sb, int *flags);
-extern void affs_free_bitmap(struct super_block *sb);
-
-/* namei.c */
-
-extern int affs_hash_name(struct super_block *sb, const u8 *name, unsigned int len);
-extern struct dentry *affs_lookup(struct inode *dir, struct dentry *dentry, struct nameidata *);
-extern int affs_unlink(struct inode *dir, struct dentry *dentry);
-extern int affs_create(struct inode *dir, struct dentry *dentry, int mode, struct nameidata *);
-extern int affs_mkdir(struct inode *dir, struct dentry *dentry, int mode);
-extern int affs_rmdir(struct inode *dir, struct dentry *dentry);
-extern int affs_link(struct dentry *olddentry, struct inode *dir,
- struct dentry *dentry);
-extern int affs_symlink(struct inode *dir, struct dentry *dentry,
- const char *symname);
-extern int affs_rename(struct inode *old_dir, struct dentry *old_dentry,
- struct inode *new_dir, struct dentry *new_dentry);
-
-/* inode.c */
-
-extern unsigned long affs_parent_ino(struct inode *dir);
-extern struct inode *affs_new_inode(struct inode *dir);
-extern int affs_notify_change(struct dentry *dentry, struct iattr *attr);
-extern void affs_put_inode(struct inode *inode);
-extern void affs_delete_inode(struct inode *inode);
-extern void affs_clear_inode(struct inode *inode);
-extern void affs_read_inode(struct inode *inode);
-extern int affs_write_inode(struct inode *inode, int);
-extern int affs_add_entry(struct inode *dir, struct inode *inode, struct dentry *dentry, s32 type);
-
-/* super.c */
-
-extern int affs_fs(void);
-
-/* file.c */
-
-void affs_free_prealloc(struct inode *inode);
-extern void affs_truncate(struct inode *);
-
-/* dir.c */
-
-extern void affs_dir_truncate(struct inode *);
-
-/* jump tables */
-
-extern struct inode_operations affs_file_inode_operations;
-extern struct inode_operations affs_dir_inode_operations;
-extern struct inode_operations affs_symlink_inode_operations;
-extern struct file_operations affs_file_operations;
-extern struct file_operations affs_file_operations_ofs;
-extern struct file_operations affs_dir_operations;
-extern struct address_space_operations affs_symlink_aops;
-extern struct address_space_operations affs_aops;
-extern struct address_space_operations affs_aops_ofs;
-
-extern struct dentry_operations affs_dentry_operations;
-extern struct dentry_operations affs_dentry_operations_intl;
-
#endif
diff -urN RC10-rc2-bk12-base/include/linux/affs_fs_i.h RC10-rc2-bk12-current/include/linux/affs_fs_i.h
--- RC10-rc2-bk12-base/include/linux/affs_fs_i.h 2004-10-18 17:55:28.000000000 -0400
+++ RC10-rc2-bk12-current/include/linux/affs_fs_i.h 1969-12-31 19:00:00.000000000 -0500
@@ -1,59 +0,0 @@
-#ifndef _AFFS_FS_I
-#define _AFFS_FS_I
-
-#include <linux/a.out.h>
-#include <linux/fs.h>
-#include <asm/semaphore.h>
-
-#define AFFS_CACHE_SIZE PAGE_SIZE
-//#define AFFS_CACHE_SIZE (4*4)
-
-#define AFFS_MAX_PREALLOC 32
-#define AFFS_LC_SIZE (AFFS_CACHE_SIZE/sizeof(u32)/2)
-#define AFFS_AC_SIZE (AFFS_CACHE_SIZE/sizeof(struct affs_ext_key)/2)
-#define AFFS_AC_MASK (AFFS_AC_SIZE-1)
-
-struct affs_ext_key {
- u32 ext; /* idx of the extended block */
- u32 key; /* block number */
-};
-
-/*
- * affs fs inode data in memory
- */
-struct affs_inode_info {
- u32 i_opencnt;
- struct semaphore i_link_lock; /* Protects internal inode access. */
- struct semaphore i_ext_lock; /* Protects internal inode access. */
-#define i_hash_lock i_ext_lock
- u32 i_blkcnt; /* block count */
- u32 i_extcnt; /* extended block count */
- u32 *i_lc; /* linear cache of extended blocks */
- u32 i_lc_size;
- u32 i_lc_shift;
- u32 i_lc_mask;
- struct affs_ext_key *i_ac; /* associative cache of extended blocks */
- u32 i_ext_last; /* last accessed extended block */
- struct buffer_head *i_ext_bh; /* bh of last extended block */
- loff_t mmu_private;
- u32 i_protect; /* unused attribute bits */
- u32 i_lastalloc; /* last allocated block */
- int i_pa_cnt; /* number of preallocated blocks */
-#if 0
- s32 i_original; /* if != 0, this is the key of the original */
- u32 i_data[AFFS_MAX_PREALLOC]; /* preallocated blocks */
- int i_cache_users; /* Cache cannot be freed while > 0 */
- unsigned char i_hlink; /* This is a fake */
- unsigned char i_pad;
- s32 i_parent; /* parent ino */
-#endif
- struct inode vfs_inode;
-};
-
-/* short cut to get to the affs specific inode data */
-static inline struct affs_inode_info *AFFS_I(struct inode *inode)
-{
- return list_entry(inode, struct affs_inode_info, vfs_inode);
-}
-
-#endif
diff -urN RC10-rc2-bk12-base/include/linux/affs_fs_sb.h RC10-rc2-bk12-current/include/linux/affs_fs_sb.h
--- RC10-rc2-bk12-base/include/linux/affs_fs_sb.h 2004-10-18 17:53:22.000000000 -0400
+++ RC10-rc2-bk12-current/include/linux/affs_fs_sb.h 1969-12-31 19:00:00.000000000 -0500
@@ -1,57 +0,0 @@
-#ifndef _AFFS_FS_SB
-#define _AFFS_FS_SB
-
-/*
- * super-block data in memory
- *
- * Block numbers are adjusted for their actual size
- *
- */
-
-struct affs_bm_info {
- u32 bm_key; /* Disk block number */
- u32 bm_free; /* Free blocks in here */
-};
-
-struct affs_sb_info {
- int s_partition_size; /* Partition size in blocks. */
- int s_reserved; /* Number of reserved blocks. */
- //u32 s_blksize; /* Initial device blksize */
- u32 s_data_blksize; /* size of the data block w/o header */
- u32 s_root_block; /* FFS root block number. */
- int s_hashsize; /* Size of hash table. */
- unsigned long s_flags; /* See below. */
- uid_t s_uid; /* uid to override */
- gid_t s_gid; /* gid to override */
- umode_t s_mode; /* mode to override */
- struct buffer_head *s_root_bh; /* Cached root block. */
- struct semaphore s_bmlock; /* Protects bitmap access. */
- struct affs_bm_info *s_bitmap; /* Bitmap infos. */
- u32 s_bmap_count; /* # of bitmap blocks. */
- u32 s_bmap_bits; /* # of bits in one bitmap blocks */
- u32 s_last_bmap;
- struct buffer_head *s_bmap_bh;
- char *s_prefix; /* Prefix for volumes and assigns. */
- int s_prefix_len; /* Length of prefix. */
- char s_volume[32]; /* Volume prefix for absolute symlinks. */
-};
-
-#define SF_INTL 0x0001 /* International filesystem. */
-#define SF_BM_VALID 0x0002 /* Bitmap is valid. */
-#define SF_IMMUTABLE 0x0004 /* Protection bits cannot be changed */
-#define SF_QUIET 0x0008 /* chmod errors will be not reported */
-#define SF_SETUID 0x0010 /* Ignore Amiga uid */
-#define SF_SETGID 0x0020 /* Ignore Amiga gid */
-#define SF_SETMODE 0x0040 /* Ignore Amiga protection bits */
-#define SF_MUFS 0x0100 /* Use MUFS uid/gid mapping */
-#define SF_OFS 0x0200 /* Old filesystem */
-#define SF_PREFIX 0x0400 /* Buffer for prefix is allocated */
-#define SF_VERBOSE 0x0800 /* Talk about fs when mounting */
-
-/* short cut to get to the affs specific sb data */
-static inline struct affs_sb_info *AFFS_SB(struct super_block *sb)
-{
- return sb->s_fs_info;
-}
-
-#endif
diff -urN RC10-rc2-bk12-base/include/linux/amigaffs.h RC10-rc2-bk12-current/include/linux/amigaffs.h
--- RC10-rc2-bk12-base/include/linux/amigaffs.h 2004-10-18 17:53:23.000000000 -0400
+++ RC10-rc2-bk12-current/include/linux/amigaffs.h 2004-11-29 17:20:01.572916712 -0500
@@ -2,8 +2,6 @@
#define AMIGAFFS_H
#include <linux/types.h>
-#include <linux/buffer_head.h>
-#include <linux/string.h>
#include <asm/byteorder.h>
/* AmigaOS allows file names with up to 30 characters length.
@@ -20,107 +18,6 @@
#define AFFS_GET_HASHENTRY(data,hashkey) be32_to_cpu(((struct dir_front *)data)->hashtable[hashkey])
#define AFFS_BLOCK(sb, bh, blk) (AFFS_HEAD(bh)->table[AFFS_SB(sb)->s_hashsize-1-(blk)])
-static inline void
-affs_set_blocksize(struct super_block *sb, int size)
-{
- sb_set_blocksize(sb, size);
-}
-static inline struct buffer_head *
-affs_bread(struct super_block *sb, int block)
-{
- pr_debug("affs_bread: %d\n", block);
- if (block >= AFFS_SB(sb)->s_reserved && block < AFFS_SB(sb)->s_partition_size)
- return sb_bread(sb, block);
- return NULL;
-}
-static inline struct buffer_head *
-affs_getblk(struct super_block *sb, int block)
-{
- pr_debug("affs_getblk: %d\n", block);
- if (block >= AFFS_SB(sb)->s_reserved && block < AFFS_SB(sb)->s_partition_size)
- return sb_getblk(sb, block);
- return NULL;
-}
-static inline struct buffer_head *
-affs_getzeroblk(struct super_block *sb, int block)
-{
- struct buffer_head *bh;
- pr_debug("affs_getzeroblk: %d\n", block);
- if (block >= AFFS_SB(sb)->s_reserved && block < AFFS_SB(sb)->s_partition_size) {
- bh = sb_getblk(sb, block);
- lock_buffer(bh);
- memset(bh->b_data, 0 , sb->s_blocksize);
- set_buffer_uptodate(bh);
- unlock_buffer(bh);
- return bh;
- }
- return NULL;
-}
-static inline struct buffer_head *
-affs_getemptyblk(struct super_block *sb, int block)
-{
- struct buffer_head *bh;
- pr_debug("affs_getemptyblk: %d\n", block);
- if (block >= AFFS_SB(sb)->s_reserved && block < AFFS_SB(sb)->s_partition_size) {
- bh = sb_getblk(sb, block);
- wait_on_buffer(bh);
- set_buffer_uptodate(bh);
- return bh;
- }
- return NULL;
-}
-static inline void
-affs_brelse(struct buffer_head *bh)
-{
- if (bh)
- pr_debug("affs_brelse: %lld\n", (long long) bh->b_blocknr);
- brelse(bh);
-}
-
-static inline void
-affs_adjust_checksum(struct buffer_head *bh, u32 val)
-{
- u32 tmp = be32_to_cpu(((__be32 *)bh->b_data)[5]);
- ((__be32 *)bh->b_data)[5] = cpu_to_be32(tmp - val);
-}
-static inline void
-affs_adjust_bitmapchecksum(struct buffer_head *bh, u32 val)
-{
- u32 tmp = be32_to_cpu(((__be32 *)bh->b_data)[0]);
- ((__be32 *)bh->b_data)[0] = cpu_to_be32(tmp - val);
-}
-
-static inline void
-affs_lock_link(struct inode *inode)
-{
- down(&AFFS_I(inode)->i_link_lock);
-}
-static inline void
-affs_unlock_link(struct inode *inode)
-{
- up(&AFFS_I(inode)->i_link_lock);
-}
-static inline void
-affs_lock_dir(struct inode *inode)
-{
- down(&AFFS_I(inode)->i_hash_lock);
-}
-static inline void
-affs_unlock_dir(struct inode *inode)
-{
- up(&AFFS_I(inode)->i_hash_lock);
-}
-static inline void
-affs_lock_ext(struct inode *inode)
-{
- down(&AFFS_I(inode)->i_ext_lock);
-}
-static inline void
-affs_unlock_ext(struct inode *inode)
-{
- up(&AFFS_I(inode)->i_ext_lock);
-}
-
#ifdef __LITTLE_ENDIAN
#define BO_EXBITS 0x18UL
#elif defined(__BIG_ENDIAN)
On Mon, 29 Nov 2004, Alexandre Oliva wrote:
>
> I don't see it as obvious at all. The need for an agreement between
> two parties on an ABI doesn't imply that one party gets to define it
> and the other gets to follow it.
Sorry, but that's not how it works.
He who writes the code decides what it is. In this case, if the kernel
does a new extension, it's the kernel that gets to decide what it is.
Full stop.
If glibc wants to do something new, go wild. The kernel won't care.
And that's really the fundamental issue. The kernel does not care what
user land does. The kernel exports functionality, the kernel does _not_
ask user land to help.
That _does_ make it a one-way street. Sorry.
Linus
On Nov 29, 2004, at 15:01, H. Peter Anvin wrote:
> Most people seem to have suggested include/linux-abi for this; I would
> personally prefer include/linux-abi/arch for the second.
It seems like there are two ways to adjust the headers. We could move
the private headers to a new directory (include/kernel?), or we could
move
the public headers to a new directory (include/linux-abi?). I am
willing to
work on some simple and trivial patches to begin doing either one, if we
can reach an agreement as to which one is preferrable (and what to call
the new directory.)
>> (a) Transfer this stuff into a file of the same name in the new
>> directory. So, for example, the syscall number list from
>> include/asm-i386/unistd.h will be transferred to
>> include/user-i386/unistd.h.
> I'm not sure you can do such a 1:1 mapping. In fact, there are cases
> where you definitely don't want to, because the current structure
> doesn't make much sense.
I think that the initial trivial patches would probably preserve a 1:1
mapping, just for compatibility. On the other hand, future patches
might
rearrange things to make more logical sense.
>> (d) stdint types should be used where possible.
>>
>> [include/user-i386/termios.h]
>> struct winsize {
>> uint16_t ws_row;
>> uint16_t ws_col;
>> uint16_t ws_xpixel;
>> uint16_t ws_ypixel;
>> };
> Good, except your "struct winsize" is bad; you're stepping on
> namespace which belongs to userspace. Since we can't use typedefs on
> struct tags, I suggest:
>
> struct __kstruct_winsize {
> /* ... */
> };
>
> .. and userspace can do:
>
> #define __kstruct_winsize winsize
My initial suggestion would be to leave the types as-is, primarily for
the reasons in Linus' earlier email, but also for simplicity and to
prevent inadvertent breakage.
>> (3) Remove all #if(n)def __KERNEL__ clauses.
>>
>> (4) Remove the -D__KERNEL__ from the master kernel Makefile.
>
> Bad! There is code in the kernel which can compile in userspace for
> testing. This is highly valuable and should be kept.
I would propose that if we decide to move the public headers instead of
the internal kernel headers, we autogenerate headers for installation
that have a #warning wrapper if __KERNEL__ isn't defined.
Cheers,
Kyle Moffett
-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCM/CS/IT/U d- s++: a17 C++++>$ UB/L/X/*++++(+)>$ P+++(++++)>$
L++++(+++) E W++(+) N+++(++) o? K? w--- O? M++ V? PS+() PE+(-) Y+
PGP+++ t+(+++) 5 X R? tv-(--) b++++(++) DI+ D+ G e->++++$ h!*()>++$ r
!y?(-)
------END GEEK CODE BLOCK------
Followup to: <[email protected]>
By author: Linus Torvalds <[email protected]>
In newsgroup: linux.dev.kernel
>
> If glibc wants to do something new, go wild. The kernel won't care.
>
> And that's really the fundamental issue. The kernel does not care what
> user land does. The kernel exports functionality, the kernel does _not_
> ask user land to help.
>
> That _does_ make it a one-way street. Sorry.
>
And it SHOULD be a one-way street. A lot of the ugliness in the
current stuff comes from the fact that the kernel tries to export
libc4/5 internals, instead of the real kernel ABI.
-hpa
Followup to: <[email protected]>
By author: Kyle Moffett <[email protected]>
In newsgroup: linux.dev.kernel
>
> On Nov 29, 2004, at 15:01, H. Peter Anvin wrote:
> > Most people seem to have suggested include/linux-abi for this; I would
> > personally prefer include/linux-abi/arch for the second.
>
> It seems like there are two ways to adjust the headers. We could move
> the private headers to a new directory (include/kernel?), or we could
> move
> the public headers to a new directory (include/linux-abi?). I am
> willing to
> work on some simple and trivial patches to begin doing either one, if we
> can reach an agreement as to which one is preferrable (and what to call
> the new directory.)
>
If we can preserve compatibility, moving the private headers to a
separate place makes sense. However, preserving compatibility is not
a good thing, because a lot of the brokenness is in how things are
exported. Thus, you want libc to be able to have a linux/* wrapper
for backwards compatibility.
My first choice would be to put the ABI headers in linux/abi.
> > Good, except your "struct winsize" is bad; you're stepping on
> > namespace which belongs to userspace. Since we can't use typedefs on
> > struct tags, I suggest:
> >
> > struct __kstruct_winsize {
> > /* ... */
> > };
> >
> > .. and userspace can do:
> >
> > #define __kstruct_winsize winsize
>
> My initial suggestion would be to leave the types as-is, primarily for
> the reasons in Linus' earlier email, but also for simplicity and to
> prevent inadvertent breakage.
True; I hadn't considered the weirdness of the namespace pollution
issue. However, the issue with struct tags remain.
Note also that not all ABI issues can be well-described in C headers,
and certainly not the way it currently is. Syscall numbers and ioctl
APIs are examples there (I do some hideously ugly hacks in klibc to
try to auto-derive as much of the ABI as possible; this is part of
making klibc very easy to port to new architectures and keeps
maintenance in one place.) However, cleaning up the headers is a
major step forward.
> >> (3) Remove all #if(n)def __KERNEL__ clauses.
> >>
> >> (4) Remove the -D__KERNEL__ from the master kernel Makefile.
> >
> > Bad! There is code in the kernel which can compile in userspace for
> > testing. This is highly valuable and should be kept.
>
> I would propose that if we decide to move the public headers instead of
> the internal kernel headers, we autogenerate headers for installation
> that have a #warning wrapper if __KERNEL__ isn't defined.
-hpa
David Howells wrote :
> We've been discussing splitting the kernel headers into userspace API headers
> and kernel internal headers and deprecating the __KERNEL__ macro.
Linus Torvalds wrote :
> On Mon, 29 Nov 2004, Alexandre Oliva wrote:
> >
> > - Linux gets to define the ABI between kernel and userland, and
> > userland must duplicate the contents of headers in which the kernel
> > defines the kernel<->userland ABI, tracking changes in them in the
> > hope that nothing falls through the cracks
>
> This is unquestionably true. The kernel obviously _does_ define the ABI,
> and userland just lives with it. At some point you have to track things,
> just because new features etc just can't be sanely handled any other way.
>
> That said, I think we can make the tracking _easier_.
Nice discussion, but nobody is discussing my main concern,
even if Linus come close to it.
On my box, I do have multiple kernels. Some 2.4.X, some 2.6.X,
I may even find some 2.2.X on some of my boxes. Depending on what I
do, and the bug reports send to me, I boot one of these kernel or
another. Because Linus is so productive, I update my kernel
frequently, without changing the user space. I know that some
distributions are also offering various kernel versions, and let the
user choose.
So, which kernel ABI should be present on my system in
/usr/include/linux and /usr/include/asm ? Should I use the ABI from
2.6.X, 2.4.X or 2.2.X ? Should I take the time to create a sanitised
version of the kernel header every time I install a new kernel, even
though I may revert back to the previous kernel ? Should I recompile
applications using kernel headers at each kernel upgrade ?
Currently, it doesn't matter, as so few applications are using
those headers, and most changes are highly backward compatible. And
the application that are deeply intimate with the kernel probably do
run-time tracking of features already, so don't really need those
headers...
However, I believe we need to at least ask this question, even
if I don't think we will come up with a good answer...
Have fun...
Jean
On Mon, 29 Nov 2004, Jean Tourrilhes wrote:
>
> So, which kernel ABI should be present on my system in
> /usr/include/linux and /usr/include/asm ? Should I use the ABI from
> 2.6.X, 2.4.X or 2.2.X ?
I have always felt (pretty strongly) that the /usr/include/xxx contents
should not be kernel-dependent, but be linked to your glibc version.
That's why the symlink from /usr/include/xxx to /usr/src/linux/include/
has been deprecated for the last, oh about ten years now..
Yes, there are some _very_ specific things which might care about system
calls or ioctl's that have been added later, but let's face it, we don't
actually do that very often. The kernel may change at a rapid pace, but
user interfaces don't, and user interfaces that would bypass the C library
change even less frequently.
Linus
On Mon, Nov 29, 2004 at 03:00:17PM -0800, Linus Torvalds wrote:
>
>
> On Mon, 29 Nov 2004, Alexandre Oliva wrote:
> >
> > I don't see it as obvious at all. The need for an agreement between
> > two parties on an ABI doesn't imply that one party gets to define it
> > and the other gets to follow it.
>
> Sorry, but that's not how it works.
>
> He who writes the code decides what it is. In this case, if the kernel
> does a new extension, it's the kernel that gets to decide what it is.
> Full stop.
>
> If glibc wants to do something new, go wild. The kernel won't care.
>
> And that's really the fundamental issue. The kernel does not care what
> user land does. The kernel exports functionality, the kernel does _not_
> ask user land to help.
except for the user land kernel helpers ;)
via call_usermodehelper()
> That _does_ make it a one-way street. Sorry.
>
> Linus
sorry too,
Herbert
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
On Mon, Nov 29, 2004 at 03:00:17PM -0800, Linus Torvalds wrote:
> And that's really the fundamental issue. The kernel does not care what
> user land does. The kernel exports functionality, the kernel does _not_
> ask user land to help.
Reminds me of the Klingon programming language:
http://tinyurl.com/4rjco
7 : "Klingon function calls do not have 'parameters' -- they have
'arguments' -- and they ALWAYS WIN THEM."
8 : "What is this talk of 'release'? Klingons do not make software
'releases.' Our software 'escapes' leaving a bloody trail of designers and
quality assurance people in its wake."
9 : "Indentation?! - I will show you how to indent when I indent your
skull!"
--
http://www.PowerDNS.com Open source, database driven DNS Software
http://lartc.org Linux Advanced Routing & Traffic Control HOWTO
Linus Torvalds wrote:
>
> On Mon, 29 Nov 2004, Jean Tourrilhes wrote:
>
>> So, which kernel ABI should be present on my system in
>>/usr/include/linux and /usr/include/asm ? Should I use the ABI from
>>2.6.X, 2.4.X or 2.2.X ?
>
>
> I have always felt (pretty strongly) that the /usr/include/xxx contents
> should not be kernel-dependent, but be linked to your glibc version.
> That's why the symlink from /usr/include/xxx to /usr/src/linux/include/
> has been deprecated for the last, oh about ten years now..
>
> Yes, there are some _very_ specific things which might care about system
> calls or ioctl's that have been added later, but let's face it, we don't
> actually do that very often. The kernel may change at a rapid pace, but
> user interfaces don't, and user interfaces that would bypass the C library
> change even less frequently.
>
More to the point, though, is that they should be supersets of each
other, which means you should be able to upgrade them to the latest version.
Most of the ABI changes, after all, are new structures needed for
various things.
-hpa
On Mon, 29 Nov 2004 09:42:16 +0000, David Woodhouse <[email protected]> wrote:
> I've lost track of the number of times things have broken because of
> incorrect use of kernel headers from userspace. That's what we're trying
> to fix -- by putting only the bits which are _supposed_ to be visible
> into files which userspace sees, where we know they define part of the
> userspace API and hence we can be extremely careful when editing them.
how about generating the user-visible interfaces and structures from the headers
using something like sparse and defined annotations? I.e.:
struct iovec
{
void __user *iov_base;
__kernel_size_t iov_len;
} __user_api__;
I should have read the whole thread before posting. My apologies, Linus
Matthew Wilcox <[email protected]> said:
> On Fri, Nov 26, 2004 at 12:00:43PM +0000, David Woodhouse wrote:
> > On Fri, 2004-11-26 at 11:58 +0000, David Howells wrote:
> > > How about calling the interface headers "kapi*/" instead of "user*/".
> > > In case you haven't guessed, "kapi" would be short for "kernel-api".
> > I don't think that change really makes any difference. The nomenclature
> > really isn't _that_ important.
> Indeed. We could also make this transparent to userspace by using a script
> to copy the user-* headers to /usr/include. Something like this:
And get them promptly out of sync with glibc &c when the kernel changes.
--
Dr. Horst H. von Brand User #22616 counter.li.org
Departamento de Informatica Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria +56 32 654239
Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513
> But in general, I do kind of like the explicit marking. The same way we
> explicitly mark the functions inside the kernel that we expose to modules,
> we could try to mark the data structures and values that we expose to user
> space. That tends to "work".
That gets trickier when it comes to #defines, I think.
Do you really object that much to splitting the header files as I proposed?
This should obviate the requirement for such marking (because it's implicit in
the location). Getting rid of the #ifdefs would also make the "Thou shalt not
use #ifdef" camp happier.
Doing this really shouldn't break the syscall API/ABI at all; barring of
course the potential change of location within /usr/include/ of the installed
headers, but that is simple to address. All this does is to separate out the
userspace available structures and constants.
I can grant that you might not want to use stdint types because not all the
toolchains that might be used on the kernel support them, but that's not a
reason to throw out the whole proposal.
Personally, I'd prefer us to move to using standard C99 types in lieu of u32
and co at least for the interface to userspace because they are just that:
standard.
However, I can see that there may be objections due to the fact that types may
become different logical sizes, and thus cause warnings in printf-format
string checking and pointer casting.
Also, the comparison you made to modules isn't exactly valid. We annotate
variables and functions that modules are allowed to access, but we never
export these to userspace directly; and we don't annotate structures, inline
functions, constants and macros into which categories the user-visible stuff
falls for use by modules.
David
On Mon, 2004-11-29 at 09:41 -0800, Linus Torvalds wrote:
> In fact, in many ways I'd prefer to have source-level annotations like
> "this is exported to user space" over trying to gather things in one
> place.
I don't think you would; not once you really tried it.
That's what the littering of ifdef __KERNEL__ attempts to do, and
there's not really any better way of doing it than that. Whatever form
of annotation you came up with, it wouldn't be much different to using
__KERNEL__ and performing the 'export' step using unifdef. Changing the
sense of it so that we annotate the stuff which _is_ visible rather than
the stuff which _isn't_ doesn't really make it fundamentally different
either.
If we want to make stuff cleaner in userspace that means spreading the
headers much more liberally with such ifdefs or whatever trick we use to
replace them. It gets ugly quite fast -- and we already know that the
fix for having too many ifdefs is to split stuff up more sensibly.
By splitting the visible parts into separate files we actually help to
make people _think_ when they're exposing stuff to userspace, and it
would help to direct particularly close attention to reviewing
structures therein. For example, if the netfilter people tried to add
this to a header file which is explicitly for userspace consumption,
rather than hidden away in kernel private headers and just _happening_
to be part of the userspace ABI, they'd never get away with it:
struct ipt_rateinfo {
u_int32_t avg; /* Average secs between packets * scale */
u_int32_t burst; /* Period multiplier for upper limit. */
/* Used internally by the kernel */
unsigned long prev;
u_int32_t credit;
u_int32_t credit_cap, cost;
/* Ugly, ugly fucker. */
struct ipt_rateinfo *master;
};
Use of an 'unsigned long' which gratuitously changes the structure size
between 32-bit userspace and 64-bit kernel, without even any _reason_
for it being there. That kind of thing would be far more obvious if the
headers are split, and much less so if we continue as we are.
That and the ugliness of the __KERNEL__ or whatever other annotation
scheme we come up with are the main reasons why I think it wants to be
separate header files for sharing with userspace, rather than continuing
with the mess we have.
Let's go through the kernel headers and identify what actually needs to
be exported; yes. But let's not 'annotate' it and post-process the mess
of header files to sanitise it; let's just take the parts which need to
be shared, put them into different header files and include them from
there.
--
dwmw2
On Tue, 30 Nov 2004, David Howells wrote:
>
> > But in general, I do kind of like the explicit marking. The same way we
> > explicitly mark the functions inside the kernel that we expose to modules,
> > we could try to mark the data structures and values that we expose to user
> > space. That tends to "work".
>
> That gets trickier when it comes to #defines, I think.
>
> Do you really object that much to splitting the header files as I proposed?
I object sternuously to your "the header files". If you can't even say
_which_ header file, I'm not interested.
If you can say "these X header files have this specific problem Y, and if
we move part Z into a common area A, we'd solve it because B", that's a
different matter.
See what I'm saying? Whole-sale "move things around because we want to"
I'm not interested in. Specific problems, yes.
> Personally, I'd prefer us to move to using standard C99 types in lieu of u32
> and co at least for the interface to userspace because they are just that:
> standard.
No. I told you why it cannot and MUST NOT be done. Repeat after me:
WE MUST NOT POLLUTE THE NAMESPACE!
There are many cases where including a header file means you want the
structure, but it does _not_ mean that you can expose totally unrelated
types, even if those types happen to be used as part of the structure. See
what I'm saying?
In contrast, __u8/__u16/__xxx are explicitly defined by the relevant
standards to be "system name space", and that is _exactly_ why we don't
use the standard namespaces.
And the fact is, the rules for _when_ you export _which_ standard types
are just too damn hard, and more importantly, they are not something that
the kernel headers should even be deciding. They depend on glibc internal
things like __USE_BSD etc, and they depend on which standard you compile
for, and sometimes even on which _version_ of the standard you compile
for.
I will not have that mess in the kernel. Which means that we do _not_
export any of the standard names. Which means that we can't _use_ any of
them. End of story.
Linus
On Tue, 30 Nov 2004, David Woodhouse wrote:
>
> On Mon, 2004-11-29 at 09:41 -0800, Linus Torvalds wrote:
> > In fact, in many ways I'd prefer to have source-level annotations like
> > "this is exported to user space" over trying to gather things in one
> > place.
>
> I don't think you would; not once you really tried it.
>
> That's what the littering of ifdef __KERNEL__ attempts to do, and
> there's not really any better way of doing it than that.
The fact is, despite this stupid and way-too-long thread, #ifdef
__KERNEL__ has worked for over a decade, and works damn well, everything
considered.
Remember the second-system-syndrome? It comes from people wanting to fix
problems in the current implementation, without thinking about all the
things that it does _well_ (because the things that work are not on their
radar - they just work). And no, __KERNEL__ may not be pretty, but it's
damn simple to fix up, and one thing it does really well is allow
flexibility in a way that forcing structure does not.
I know people like "structure". But it's _waayyy_ overrated. The reason we
use C instead of some other programming environment is not that C is the
most highly structured language around, but it's the most _flexible_ one,
largely because it says "let's give people rope". It allows structure, but
if you want to cast things around and use gotos, it says "sure, master,
you're the boss".
Same thing here. The __KERNEL__ approach says "whatever you want, boss".
It doesn't get in the way. Maybe it doesn't actively _help_ you either,
but you never have to fight any structure it imposes on you.
Also, there _are_ better ways of annotating it than __KERNEL__. In
particular, if we had a "user annotation", I could make sparse sit up and
take notice, and _complain_ when you use a non-specific-sized type. That's
not just theory - Al Viro was talking about that at some point.
Linus
> No. I told you why it cannot and MUST NOT be done. Repeat after me:
>
> WE MUST NOT POLLUTE THE NAMESPACE!
Sorry, I misremembered that bit in your earlier email, but having reread it I
think I have to concur with it. I don't like it very much, but I don't think
there's a good way around it now.
David
On Tue, 2004-11-30 at 07:58 -0800, Linus Torvalds wrote:
> The fact is, despite this stupid and way-too-long thread, #ifdef
> __KERNEL__ has worked for over a decade, and works damn well, everything
> considered.
I don't agree with that. The current setup is a complete PITA and that's
why people are clamouring for us to clean the crap up.
But if you feel this strongly about it, then let's go ahead with the
existing mess. We can add more ifdef __KERNEL__ and add rules to the
makefiles to export it for userspace, generating something like the
existing glibc-kernheaders setup. And of course we can look at the
better alternative 'annotations' instead of using #ifdef __KERNEL__.
I think we'll all hate it, and I don't think it's going to really fix
the problem. But we can always reconsider the position later.
> Same thing here. The __KERNEL__ approach says "whatever you want, boss".
> It doesn't get in the way. Maybe it doesn't actively _help_ you either,
> but you never have to fight any structure it imposes on you.
Having to think before adding something that's user visible is a
_benefit_ not a disadvantage.
--
dwmw2
> The fact is, despite this stupid and way-too-long thread, #ifdef
> __KERNEL__ has worked for over a decade, and works damn well, everything
> considered.
It's a pain, which is why people want to do something about it.
> Remember the second-system-syndrome? It comes from people wanting to fix
> problems in the current implementation, without thinking about all the
> things that it does _well_ (because the things that work are not on their
> radar - they just work). And no, __KERNEL__ may not be pretty, but it's
> damn simple to fix up, and one thing it does really well is allow
> flexibility in a way that forcing structure does not.
And allows abuse and doesn't penalise laziness; both of which do happen in the
kernel header files.
> I know people like "structure". But it's _waayyy_ overrated.
> The reason we use C instead of some other programming environment is not
> that C is the most highly structured language around, but it's the most
> _flexible_ one, largely because it says "let's give people rope".
We use C & ASM for the kernel because you and others won't allow anything
else. Not that I'm necessarily endorsing the use of other languages - I like C
and ASM; I just look at stuff in the kernel occasionally and know that other
languages handle it better and more efficiently. That is, however, besides the
point.
> Same thing here. The __KERNEL__ approach says "whatever you want, boss".
> It doesn't get in the way. Maybe it doesn't actively _help_ you either,
> but you never have to fight any structure it imposes on you.
It can get in the way accidentally.
> Also, there _are_ better ways of annotating it than __KERNEL__. In
> particular, if we had a "user annotation", I could make sparse sit up and
> take notice, and _complain_ when you use a non-specific-sized type. That's
> not just theory - Al Viro was talking about that at some point.
If you did user annotations you'd have to solve the problem of applying it to
#defines and still allowing the constants to be used in assembly. Obviously
this is not impossible. This is trivial with __KERNEL__ or separation into
other files.
David
On Tue, 30 Nov 2004, David Woodhouse wrote:
>
> > Same thing here. The __KERNEL__ approach says "whatever you want, boss".
> > It doesn't get in the way. Maybe it doesn't actively _help_ you either,
> > but you never have to fight any structure it imposes on you.
>
> Having to think before adding something that's user visible is a
> _benefit_ not a disadvantage.
I've said this at least three times: if you can point to a _specific_
thing you want to move, go wild. I think the big waste in this discussion
has been that there have _not_ been specific suggestions, just total
sound-bites like "wouldn't it be great to move things to 'include/kapi'".
If you have a specific thing in mind, say instead something like
"Wouldn't it be great if we moved all the tty layer IOCTL numbers
into 'tty-ioctl-nr.h', and made the old header file just include
that header file, so that new libc users can get them from just
that header? And btw, here's the patch."
then I might listen. Notice how the only really constructive thing to come
out of this flame-fest has been a patch by Al that looked perfectly
reasonable, but that got totally drowned out by the arguing?
Note that even _if_ you have a specific thing in mind, I want to see that
somebody would say "yes, we'd use that organization". I would not be
surprised at all if glibc people said that they can't really use any
re-organization anyway, since they need to support old kernel setups too.
See? Changes that aren't specific enough, or don't actually help things is
what I'm against.
Linus
On Tue, 30 Nov 2004, David Howells wrote:
>
> If you did user annotations you'd have to solve the problem of applying it to
> #defines and still allowing the constants to be used in assembly. Obviously
> this is not impossible. This is trivial with __KERNEL__ or separation into
> other files.
Annotations don't have to be per-entry. It can be something as simple as
"this particular file is exposed to user space", or "things from here to
here are exposed to user space".
And while automation is good, sometimes automation isn't even worth it. I
suspect it might be good just to _comment_ the structures we expose to
user space, even if it's never used for anything but as a note to tell
people not to change it.
Linus
On Tue, 2004-11-30 at 08:53 -0800, Linus Torvalds wrote:
> I've said this at least three times: if you can point to a _specific_
> thing you want to move, go wild. I think the big waste in this discussion
> has been that there have _not_ been specific suggestions, just total
> sound-bites like "wouldn't it be great to move things to 'include/kapi'".
I've already done some -- see the contents of include/mtd/ for example,
which contain only the parts which userspace should see, while
include/linux/mtd/ is now entirely kernel-private stuff.
We were trying to reach a consensus on where new header files can be put
and indeed whether they should be split at all. We can all disappear for
a week or two and come back with a completely sanitised set of header
files with the userspace stuff in a separate directory and _then_ and be
told "no, we don't like that; do it differently".
A bunch of us have even made a start on such things to prove the
concept. Presenting it as a fait accompli just to have it rejected isn't
much fun though -- the idea was to get some form of agreement in
principle beforehand, then collaborate on actually doing it.
> Note that even _if_ you have a specific thing in mind, I want to see that
> somebody would say "yes, we'd use that organization". I would not be
> surprised at all if glibc people said that they can't really use any
> re-organization anyway, since they need to support old kernel setups too.
The idea in the proposal which David posted, which seemed perfectly
specific enough to me, was to move all the user-visible parts to
separate header files in a separate directory. Those same header files
would also be included by the kernel-private headers. This would be done
in such a way that it's a drop-in replacement for the existing sanitised
glibc-kernheaders package. Of _course_ it needs to be usable without
pain by the glibc people.
We can just go ahead and _do_ that if necessary, but it seemed like a
sane plan to agree on how to do it beforehand.
--
dwmw2
On Tue, Nov 30, 2004 at 08:53:34AM -0800, Linus Torvalds wrote:
> then I might listen. Notice how the only really constructive thing to come
> out of this flame-fest has been a patch by Al that looked perfectly
> reasonable, but that got totally drowned out by the arguing?
Frankly, I suspect that amount of effort that went into postings made in
that thread is approaching what it would take to clean the damn thing
up for good. I wasn't kidding about 10 minutes work to do that patch...
On Tue, 30 Nov 2004, David Woodhouse wrote:
>
> The idea in the proposal which David posted, which seemed perfectly
> specific enough to me, was to move all the user-visible parts to
> separate header files in a separate directory.
You call _that_ specific?
Hell no. You need to do it without breaking existing uses, as noted
earlier, and it's not specific at all. "all user visible parts" is a big
undertaking, if you can't make it smaller than that, then forget about it.
Basic rule in kernel engineering: you don't just rewrite the world. You do
it in incremental independent steps.
Any mtd-specific rewrite is obviously a go.
Linus
On wtorek 30 listopad 2004 19:21, Linus Torvalds wrote:
> You call _that_ specific?
>
> Hell no. You need to do it without breaking existing uses, as noted
> earlier, and it's not specific at all. "all user visible parts" is a big
> undertaking, if you can't make it smaller than that, then forget about it.
>
> Basic rule in kernel engineering: you don't just rewrite the world. You do
> it in incremental independent steps.
>
> Any mtd-specific rewrite is obviously a go.
Facts:
- current __KERNEL__ stuff is crap; just check any distro what they're using -
either some kind of userland headers or a patched version of kernel headers
- "You don't use kernel headers in userland!!"
small print: except this, this, and that dir, which are userland friendly
That's quite schizophrenic.
Specific problems:
- glibc has a copy of eg. networking definitions (mostly lots of numbers for
different protocols); if userland wants to use something that's not in glibc
already, it has to include linux' headers; which more often than not causes
conflicts with glibc
- the above problem is present in allmost all headers that have a glibc and
linux version
- glibc uses linux headers for getting ABI stuff in... dunno... four? five
cases? And *everything* else ends up getting duplicated.
People in this thread are trying to force you to agree to a specific location
where stuff like the above mentioned mtd can go to and to start accepting
patches (afaik there were a number of patches trying to introduce that
userland dir - all of them ignored). That's (mostly) all.
--
In the year eighty five ten
God is gonna shake his mighty head
He'll either say,
"I'm pleased where man has been"
Or tear it down, and start again
On Nov 30, 2004, Linus Torvalds <[email protected]> wrote:
> I object sternuously to your "the header files". If you can't even say
> _which_ header file, I'm not interested.
How about this formulation then:
- move anything that is not protected by #ifdef __KERNEL__ to the
ukabi header tree, adding an include in the beginning of the original
header that includes the ukabi header. Since ukabi stuff can't depend on
kernel-internal stuff, having this include as the first thing in the
existing header should work.
- figure out what should have been protected by #ifdef __KERNEL__ but
wasn't, and move it back
- wait for bug reports from the world and deal with them
- repeat until you feel you got it right
> See what I'm saying? Whole-sale "move things around because we want to"
> I'm not interested in. Specific problems, yes.
The specific problem we're trying to address is the creation of header
files that define the ABI between userland and kernel, that both
kernel and userland can use.
>> Personally, I'd prefer us to move to using standard C99 types in lieu of u32
>> and co at least for the interface to userspace because they are just that:
>> standard.
> No. I told you why it cannot and MUST NOT be done. Repeat after me:
> WE MUST NOT POLLUTE THE NAMESPACE!
David, Linus is right. Using standard types is not an option because
they're not built in type names; you have to include a header to bring
them in, and ideally you shouldn't gratuitously get the names defined
just because you include some unrelated header, although the C
Standard grants standard headers freedom to include other headers or
something along these lines; I don't recall the exact passage of the
Standard that says so, but we should strive to not pollute the user
namespace anyway.
--
Alexandre Oliva http://www.ic.unicamp.br/~oliva/
Red Hat Compiler Engineer aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist oliva@{lsd.ic.unicamp.br, gnu.org}
On Tue, 30 Nov 2004, Mariusz Mazur wrote:
>
> Facts:
> - current __KERNEL__ stuff is crap;
Fact: you're wrong. I _constantly_ get emails when something breaks. It
doesn't happen all that often, but it happens.
IOW, I _know_ from personal experience that people depend on it. No ifs,
buts and maybes about it.
> People in this thread are trying to force you to agree to a specific location
> where stuff like the above mentioned mtd can go to and to start accepting
> patches (afaik there were a number of patches trying to introduce that
> userland dir - all of them ignored). That's (mostly) all.
No. People in this thread have almost uniformly ignored my arguments.
If it's MTD-specific, nobody _cares_ where it goes. Somebody take a
fricking flying leap of faith here.
If that's all that people want, I hereby proclaim that
include/asm-xxx/user/xxxx.h
include/user/xxxx.h
is reserved for user-visible stuff. And now you can send me small and
localized patches that fix a _particular_ gripe.
But claiming that __KERNEL__ doesn't work is clearly a bunch of crapola.
As is the notion that you can somehow do it all. We do it in small pieces.
Happy now?
Linus
On Tue, 30 Nov 2004, Alexandre Oliva wrote:
>
> - move anything that is not protected by #ifdef __KERNEL__ to the
> ukabi header tree, adding an include in the beginning of the original
> header that includes the ukabi header.
No. I want stuff that goes into the ABI tree to be clearly _defined_ to be
user-visible. Not a "let's move it all there and then prune it".
Leave anything questionable in the current location. And never EVER move
anything that is kernel-internal to a new "clean" tree, because that would
be totally pointless. At that point, people would have to edit the "clean"
tree even for kernel internal stuff. No go.
Also, "ukabi" just isn't going to fly as a name. It's also not as simple
as you seem to think, since a lot of these ABI things are architecture-
dependent, which apparently all you guys have totally ignored.
I've suggested "include/user/" and "include/asm-xxx/user", which handles
architecture-specific parts too. I'm ok with doing it the other way
around, ie "include/user/" and "include/user/arch-xxxx".
And "user" might be "user-abi" or something like that, but it sure isn't
going to be some unreadable contraction.
And I _still_ want to see these patches only for things where somebody can
validly argue that
(a) it can't break anything (ie the old location still includes the new
one, exactly the same way)
(b) there are people who will actually take _advantage_ of that
particular file (ie "just because I think so" doesn't fly).
Moving files around is just too disruptive to be done without damn good
reason.
Linus
On Nov 29, 2004, Linus Torvalds <[email protected]> wrote:
> On Mon, 29 Nov 2004, Alexandre Oliva wrote:
>>
>> I don't see it as obvious at all. The need for an agreement between
>> two parties on an ABI doesn't imply that one party gets to define it
>> and the other gets to follow it.
> Sorry, but that's not how it works.
Then maybe this is the fundamental problem. As long as the kernel
doesn't recognize that an ABI is a contract, rather than an
imposition, kernel developers won't care.
But guess what? Kernel developers *do* care about maintaining the
userland ABI. They don't just go wild changing data types,
structures, renumbering system calls and all that sort of nasty stuff
because there's a contract with userland that they don't want to
break. Because breaking this contract would be a stupid thing, that
would require all affected userland to be modified to cope with the
change, even if that's just a recompile using the new kernel
headers... erhm... using the result of applying the kernel ABI
change in the copy/extract/whatever of kernel headers that userland is
entitled to use.
As long as we recognize that the notion that the kernel<->userland ABI
is a separate entity, that enables kernel and userland to exchange
information, rather than something that the kernel defines and
userland must follow, we give strength to the argument that userland
is not a derived work from the kernel. If this argument proves to be
false, then only works released under the GNU GPL or a compatible
license will be allowed to run atop of Linux, because libc would be a
derived work, its license would become full GPL, and any program
linked with it could only be distributed under the full GPL. Oh, and
just in case, IANAL :-)
Again, I wouldn't mind that at all, but I'm sure others feel
otherwise. Besides, as I wrote before, if this is the intent, I'd
rather have it exposed in clear, rather than seeing people being
misled to believe it's fair game to run non-GPLed software on
Linux-based operating systems.
> He who writes the code decides what it is.
I've been in projects in which I wrote ABI headers before the
corresponding kernel port had even started. Sometimes, quite
unfortunately, the kernel engineer went ahead and wrote headers with
slightly different assumptions, and then we had to get together to
define what the ABI was going to be.
In fact, the way you seem to want things to be, you want both kernel
and userland to write the code, and somehow hope they magically match
each other. That's not how things work. The way things work is that
either you get agreement first, or each end goes on its way and then
at some point they figure out where they diverged and fix things up,
or one end goes first and the other takes the definitions and uses
them. It's in no way a kernel->userland one-way street. It's a
language (for lack of a better word) that kernel and userland choose
to exchange information, not one that the kernel invents and then
requires others interested in talking to it to learn (you know, that
``Do you speak English/my language?´´ phrase so common in Hollywood
movies in which an American character needs to talk to someone abroad
:-)
Sometimes you'll even find formal ABI specifications, written down
before either userland or kernel start being developed. Claiming that
the kernel gets to set the ABI as it wishes, without worrying about
userland because it never calls userland is a very narrow view, and in
some cases it's even a lie. Think signal handling and process
start-up, for example: if you don't follow the userland ABI to call
the function pointer given to the signal function, or if you don't
follow the userland ABI to start the program's entry point, it's not
userland that is broken, it's the kernel.
> In this case, if the kernel
> does a new extension, it's the kernel that gets to decide what it is.
As in, say, NPTL? As if a number of userland considerations hadn't
driven most of the design decisions behind the then-defined ABI
extensions?
> If glibc wants to do something new, go wild. The kernel won't care.
This is not about glibc. That's not the only userland component.
There are several different alternatives one can use.
And then, the kernel will care, because it has to. A kernel without
userland is nothing. A kernel that kept introducing incompatible ABI
changes would die a very quick death. A kernel has to be very careful
about complying with the ABI it shares with userland, and you know
that, because you do care about it.
> And that's really the fundamental issue. The kernel does not care what
> user land does. The kernel exports functionality, the kernel does _not_
> ask user land to help.
I'm not so much talking about the hundreds of syscalls, the syscall
conventions and the number and type of arguments given to each
syscall. This is not even well defined in kernel headers. All that's
exported regarding syscalls is a mapping from syscall names to
numbers.
What I'm talking about is the data structures. Sure, the kernel can
define these however it likes. So can userland. The point is that,
if they don't match, when you get userland and kernel together, in the
hopes of getting something useful done, it doesn't work.
Both ends can come up with extensions, and do things the old way until
the other end picks up the extension. I'll give you that many
extensions make to the kernel before say glibc, but that's mostly just
like, when we create a new toolchain port, assembler and linker are
generally implemented before the compiler. I guess it just makes more
sense. It wastes less effort to make sure you got something right in
a lower layer before you build something atop of it. It's unlikely
that a change would make to glibc if it used a feature that isn't
available in the kernel, just because most often features undergo
(minor or sometimes major) changes when they're integrated, and it's
easier to fix things up if you haven't committed to the way that
didn't make it there yet.
> That _does_ make it a one-way street. Sorry.
Good, we're getting somewhere. This is surely better than a zero-way
street, i.e., one in which headers that specify the ABI don't flow in
any direction. If we're agreeing to make it one way, let's make it
so: have headers that are maintained as part of the kernel, and that
are imposed, if you like, on userland. Saying it's a one-way street
but denying others from taking what comes their way through this
one-way street hasn't been very effective :-)
--
Alexandre Oliva http://www.ic.unicamp.br/~oliva/
Red Hat Compiler Engineer aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist oliva@{lsd.ic.unicamp.br, gnu.org}
On Nov 30, 2004, Linus Torvalds <[email protected]> wrote:
> On Tue, 30 Nov 2004, Alexandre Oliva wrote:
>>
>> - move anything that is not protected by #ifdef __KERNEL__ to the
>> ukabi header tree, adding an include in the beginning of the original
>> header that includes the ukabi header.
> No. I want stuff that goes into the ABI tree to be clearly _defined_ to be
> user-visible. Not a "let's move it all there and then prune it".
Right. Which brings us to my second bullet. I *knew* I should have
made it the first :-/
> Also, "ukabi" just isn't going to fly as a name. It's also not as simple
> as you seem to think, since a lot of these ABI things are architecture-
> dependent, which apparently all you guys have totally ignored.
Looks like you missed the post that started this thread. It *did*
mention machine-independent and machine-specific user headers.
> I've suggested "include/user/" and "include/asm-xxx/user", which handles
> architecture-specific parts too. I'm ok with doing it the other way
> around, ie "include/user/" and "include/user/arch-xxxx".
As I pointed out, `user' is a very bad name. As you said yourself,
we're talking about the *kernel* ABI. So what's `user' supposed to
mean? Was I so successful in my arguments that you now see it as the
userland ABI? :-)
> (a) it can't break anything (ie the old location still includes the new
> one, exactly the same way)
You mean it can't break anything in a kernel build, or it can't break
anything except for userland apps that abused kernel headers and used
to get away with that?
> (b) there are people who will actually take _advantage_ of that
> particular file (ie "just because I think so" doesn't fly).
People who currently get to maintain duplicates of these header
contents will take immediate advantage of these changes, since they
will no longer have to maintain the duplicates. This is a major step
forward in terms of maintainability for everybody else, at no expense
to the kernel, that now gets to explicitly (as opposed to implicitly)
document what is the ABI it's willing to comply with. It's a win for
both ends.
--
Alexandre Oliva http://www.ic.unicamp.br/~oliva/
Red Hat Compiler Engineer aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist oliva@{lsd.ic.unicamp.br, gnu.org}
On Nov 30, 2004, Alexandre Oliva <[email protected]> wrote:
>> I've suggested "include/user/" and "include/asm-xxx/user", which handles
>> architecture-specific parts too. I'm ok with doing it the other way
>> around, ie "include/user/" and "include/user/arch-xxxx".
> As I pointed out, `user' is a very bad name. As you said yourself,
> we're talking about the *kernel* ABI. So what's `user' supposed to
> mean? Was I so successful in my arguments that you now see it as the
> userland ABI? :-)
I got carried away with joking and failed to repeat why I consider it
a bad name (assuming that, since you missed the beginning of the
thread, you probably missed the first reply I posted to the message
that started it): since we're going to install these headers in
/usr/include (where headers for userland live), /usr/include/user is
quite misleading. /usr/include/kernel would be far more
appropriate for this purpose.
Sure, we could take headers from linux-*/include/user and install them
in /usr/include/kernel, but then includes in there that reference
other headers in user/ or in asm-<arch>/ will cease to work.
So we should come up with a name that makes sense for both users of
these headers, which is why I suggested ukabi. linux/abi and
asm-<mach>/abi work just as well, and then we can soft-link `abi -> .'
in /usr/include/{linux,asm-<mach>} if needed. Ideally, we wouldn't
have to.
--
Alexandre Oliva http://www.ic.unicamp.br/~oliva/
Red Hat Compiler Engineer aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist oliva@{lsd.ic.unicamp.br, gnu.org}
On Tue, 30 Nov 2004, Alexandre Oliva wrote:
>
> Then maybe this is the fundamental problem. As long as the kernel
> doesn't recognize that an ABI is a contract, rather than an
> imposition, kernel developers won't care.
That's a silly analogy. Worse, it's a very flawed analogy.
If you want to use a legal analogy, the ABI is not a contract, it's a
public _license_.
Why? A contract is something that you cannot change unilaterally. Clearly
the kernel _can_ and will change the ABI without asking every existing
program for permission.
In a license, you can always just _expand_ the license eventually. Maybe
you originally licensed for "fly-fishing for trout", and later you can
expand that license to say "you can also catch crawfish" without impacting
your existing licensees.
And exactly as with an ABI, the only thing you can't do is _remove_ rights
without getting into trouble.
So get your analogies straight. The kernel ABI is _not_ a contract.
Linus
On Tue, 30 Nov 2004, Alexandre Oliva wrote:
> > (a) it can't break anything (ie the old location still includes the new
> > one, exactly the same way)
>
> You mean it can't break anything in a kernel build, or it can't break
> anything except for userland apps that abused kernel headers and used
> to get away with that?
It can't break userland either.
> > (b) there are people who will actually take _advantage_ of that
> > particular file (ie "just because I think so" doesn't fly).
>
> People who currently get to maintain duplicates of these header
> contents will take immediate advantage of these changes, since they
> will no longer have to maintain the duplicates.
Wrong. They'll _still_ have to maintain duplicates, since they can't rely
ont he end-user to have a recent enough kernel.
It's just that they can hopefully start _copying_ their dupliates more
easily. But if you think the duplication goes away, then I don't want you
to send me patches, because you don't understand the issues.
At RH you may see only the case where people do a whole-system upgrade.
Please realize that that is just _one_ schenario, and you should not
assume that it's the only valid one.
Linus
On Tue, Nov 30, 2004 at 12:47:50PM -0800, Linus Torvalds wrote:
If that's all that people want, I hereby proclaim that
>
> include/asm-xxx/user/xxxx.h
> include/user/xxxx.h
>
> is reserved for user-visible stuff. And now you can send me small and
> localized patches that fix a _particular_ gripe.
Please use:
include/$arch/user-asm/xxxx.h
include/user/xxxx.h
This allows us to
1) To include file foo.h we still distingush between user versus user-asm:
#include <user/foo.h> (include/user/foo.h)
#include <user-asm/foo.h> (include/$arch/user-asm/foo.h)
2) No symlinks needed - just proper options to gcc
Sam
This should do it for the top-level makefile:
===== Makefile 1.549 vs edited =====
--- 1.549/Makefile 2004-11-15 10:00:11 +01:00
+++ edited/Makefile 2004-11-30 23:31:09 +01:00
@@ -345,6 +345,10 @@
# Needed to be compatible with the O= option
LINUXINCLUDE := -Iinclude \
$(if $(KBUILD_SRC),-Iinclude2 -I$(srctree)/include)
+
+# Extend include path with user dir
+LINUXINCLUDE += -Iinclude/$(ARCH) \
+ $(if $(KBUILD_SRC), -I$(srctree)/include/$(ARCH))
CPPFLAGS := -D__KERNEL__ $(LINUXINCLUDE)
On Tue, 2004-11-30 at 14:25 -0800, Linus Torvalds wrote:
>
> On Tue, 30 Nov 2004, Alexandre Oliva wrote:
>
> > > (a) it can't break anything (ie the old location still includes the new
> > > one, exactly the same way)
> >
> > You mean it can't break anything in a kernel build, or it can't break
> > anything except for userland apps that abused kernel headers and used
> > to get away with that?
>
> It can't break userland either.
That depends on your definition of 'break'. It should prevent abuse.
To pick a specific example, since you like them: where userland programs
are including atomic.h, and hence writing programs which don't compile
on some architectures, and which compile on others but silently give
non-atomic results, it's perfectly acceptable and indeed advisable to
prevent compilation across the board.
Some people might call that breakage; I don't.
--
dwmw2
On wtorek 30 listopad 2004 23:33, Sam Ravnborg wrote:
> On Tue, Nov 30, 2004 at 12:47:50PM -0800, Linus Torvalds wrote:
> If that's all that people want, I hereby proclaim that
>
> > include/asm-xxx/user/xxxx.h
> > include/user/xxxx.h
> >
> > is reserved for user-visible stuff. And now you can send me small and
> > localized patches that fix a _particular_ gripe.
>
> Please use:
> include/$arch/user-asm/xxxx.h
> include/user/xxxx.h
Wrong. These dirs must be linked to /usr/include so they must have more
meaningfull names.
--
In the year eighty five ten
God is gonna shake his mighty head
He'll either say,
"I'm pleased where man has been"
Or tear it down, and start again
On Tue, Nov 30, 2004 at 10:21:52AM -0800, Linus Torvalds wrote:
>
>
> On Tue, 30 Nov 2004, David Woodhouse wrote:
> >
> > The idea in the proposal which David posted, which seemed perfectly
> > specific enough to me, was to move all the user-visible parts to
> > separate header files in a separate directory.
>
> You call _that_ specific?
>
> Hell no. You need to do it without breaking existing uses, as noted
> earlier, and it's not specific at all. "all user visible parts" is a big
> undertaking, if you can't make it smaller than that, then forget about it.
So we follow dhowell's plan with the following additions:
a) during the transition, include/linux/foo.h includes
include/user/foo.h if it exists and similarly for asm-*
b) when include/user is deemed sufficiently populated, a flag day is
declared and links from /usr/include are switched to them
c) #define __KERNEL__ is dropped from the compile
d) extraneous include/user/* includes are removed from include/linux
(though they'll often make perfect sense)
Step b) may be a long time in coming, but for interfaces where kernel
headers are frequently abused, we can start feeling benefits.
> Basic rule in kernel engineering: you don't just rewrite the world. You do
> it in incremental independent steps.
Even so, it's sometimes important for everyone to agree on a
destination before they set out..
--
Mathematics is the supreme nostalgia of our time.
On Tue, 30 Nov 2004, David Woodhouse wrote:
>
> That depends on your definition of 'break'. It should prevent abuse.
Not really.
It should prevent _future_ abuse.
The notion of "preventing existing xxx" is insane. You can't "prevent"
something that already happened unless you've come up with some new
interesting theory of causality.
> To pick a specific example, since you like them: where userland programs
> are including atomic.h, and hence writing programs which don't compile
> on some architectures, and which compile on others but silently give
> non-atomic results, it's perfectly acceptable and indeed advisable to
> prevent compilation across the board.
>
> Some people might call that breakage; I don't.
I do. The thing is, the people who _notice_ the breakage are often the
people who don't know what the hell to do about it.
The way to prevent _future_ abuse is by adding something like
#ifndef __KERNEL__
#warning "This really doesn't work"
#endif
which does that, and has the advantage of not breaking anything at all.
In other words: if you want to move things around just to break things,
THEN THAT IS INCREDIBLY STUPID. We don't do things to screw our users
over.
Feel free to send a patch.
Linus
> a) during the transition, include/linux/foo.h includes
> include/user/foo.h if it exists and similarly for asm-*
> b) when include/user is deemed sufficiently populated, a flag day is
> declared and links from /usr/include are switched to them
there are no such links, only copies (more or less modified)
On wtorek 30 listopad 2004 23:25, Linus Torvalds wrote:
> > You mean it can't break anything in a kernel build, or it can't break
> > anything except for userland apps that abused kernel headers and used
> > to get away with that?
>
> It can't break userland either.
Userland? And who's using vanilla kernel headers in userland? Duplicates
maintainers can always fix stuff independent of kernel (and I do).
> > > (b) there are people who will actually take _advantage_ of that
> > > particular file (ie "just because I think so" doesn't fly).
> >
> > People who currently get to maintain duplicates of these header
> > contents will take immediate advantage of these changes, since they
> > will no longer have to maintain the duplicates.
>
> Wrong. They'll _still_ have to maintain duplicates, since they can't rely
> ont he end-user to have a recent enough kernel.
The end user uses a distro that does these things for him. If not, he needs to
know his way around with finding working kernel headers, otherwise he won't
be able to build anything (he surely isn't able now - which btw you're trying
really hard to ignore).
I don't get your point.
> It's just that they can hopefully start _copying_ their dupliates more
> easily. But if you think the duplication goes away, then I don't want you
> to send me patches, because you don't understand the issues.
Yes. The more stuff gets separated, the less work duplicates maintainers have.
Once all userland stuff gets extracted, those maintainers won't have anything
else to do, maybe except releasing their final version and using it for
backward compatibility. And that's what we're aiming at.
--
In the year eighty five ten
God is gonna shake his mighty head
He'll either say,
"I'm pleased where man has been"
Or tear it down, and start again
On Tue, 30 Nov 2004, Matt Mackall wrote:
>
> So we follow dhowell's plan with the following additions:
No.
We do _not_ move stuff over that is questionable.
I thought that was clear by now. The rules are:
- we only move things that _have_ to move
- we don't break existing programs, and no "but they are broken already"
is not an excuse.
- we only move things where that _particular_ move can be shown to be
beneficial.
No whole-sale moves. No "let's break things that I think are broken". No
"let's change things because we can".
Well-defined moves. Both in content _and_ in reason.
Linus
On Nov 30, 2004, Linus Torvalds <[email protected]> wrote:
> On Tue, 30 Nov 2004, Alexandre Oliva wrote:
>> Then maybe this is the fundamental problem. As long as the kernel
>> doesn't recognize that an ABI is a contract, rather than an
>> imposition, kernel developers won't care.
> That's a silly analogy. Worse, it's a very flawed analogy.
> If you want to use a legal analogy, the ABI is not a contract, it's a
> public _license_.
I didn't mean to use a legal analogy. I meant contract in the
software engineering sense. Sorry if that wasn't clear.
--
Alexandre Oliva http://www.ic.unicamp.br/~oliva/
Red Hat Compiler Engineer aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist oliva@{lsd.ic.unicamp.br, gnu.org}
On Tue, Nov 30, 2004 at 11:44:21PM +0100, Mariusz Mazur wrote:
> On wtorek 30 listopad 2004 23:33, Sam Ravnborg wrote:
> > On Tue, Nov 30, 2004 at 12:47:50PM -0800, Linus Torvalds wrote:
> > If that's all that people want, I hereby proclaim that
> >
> > > include/asm-xxx/user/xxxx.h
> > > include/user/xxxx.h
> > >
> > > is reserved for user-visible stuff. And now you can send me small and
> > > localized patches that fix a _particular_ gripe.
> >
> > Please use:
> > include/$arch/user-asm/xxxx.h
> > include/user/xxxx.h
>
> Wrong. These dirs must be linked to /usr/include so they must have more
> meaningfull names.
WTF? I've got a dozen kernel trees hanging around, which one (and WTF any,
while we are at it) should be "linked to"?
On Nov 30, 2004, Linus Torvalds <[email protected]> wrote:
> Wrong. They'll _still_ have to maintain duplicates, since they can't rely
> ont he end-user to have a recent enough kernel.
No. They just won't have the headers for the newer kernel installed
on their system. They stick with the kernel ABI headers provided by a
baseline kernel, and that's probably from an older kernel tarball.
> It's just that they can hopefully start _copying_ their dupliates more
> easily.
It's still the headers from the kernel, copied or not. From a
different, baseline version, maybe.
> At RH you may see only the case where people do a whole-system upgrade.
I personally don't care about the case you perceive as the Red Hat
case, because I work on embedded systems development, in a different
organization within Red Hat than the people who develop Red Hat
Enterprise Linux and Fedora. So yes, in the case that actually
benefits me, it's even more of a whole-system thing, since for our
group it's *the* kernel, not *a* random kernel version. It's the
kernel we port and ship to the customer, and the corresponding
kernel<->userland ABI.
That said, I do understand the issues of kernel baseline ABI settings,
or at least part of them. I know you don't want to assume the
presence of a certain system call just because you're running a kernel
that implements it. In fact, I know you don't want to assume the
presence of a system call just because it's defined in linux/unistd.h.
As long as the userland software keeps this distinction in mind, I
don't see why taking the latest kernel<->userland ABI headers, that
describe the newest system calls and any corresponding data
structures, could hurt anything. In fact, if you don't have the
corresponding system call numbers and data structures, you can't
possibly take advantage of the new syscalls even if they happen to be
available, unless you duplicate the syscall numbers and data structure
definitions in your own code, which, from a software engineering
standpoint, is bad.
Of course, badly written software can assume the syscalls are present
just because their numbers are given, and then people who rely on such
bad software lose, but such is the nature of bad software.
--
Alexandre Oliva http://www.ic.unicamp.br/~oliva/
Red Hat Compiler Engineer aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist oliva@{lsd.ic.unicamp.br, gnu.org}
On Tue, 30 Nov 2004, Alexandre Oliva wrote:
> On Nov 30, 2004, Linus Torvalds <[email protected]> wrote:
>
> > On Tue, 30 Nov 2004, Alexandre Oliva wrote:
>
> >> Then maybe this is the fundamental problem. As long as the kernel
> >> doesn't recognize that an ABI is a contract, rather than an
> >> imposition, kernel developers won't care.
>
> > That's a silly analogy. Worse, it's a very flawed analogy.
>
> > If you want to use a legal analogy, the ABI is not a contract, it's a
> > public _license_.
>
> I didn't mean to use a legal analogy. I meant contract in the
> software engineering sense. Sorry if that wasn't clear.
Then your definition of a "contract" is flawed or your world-view has
nothing to do with reality.
An ABI is not a contract. It's somebody exposing a certain set of
features. It does _not_ mean that the party that gets exposed has any say
wrt the features. At most he can hope that the exported set doesn't
shrink or change semantically, but let's face it, even that does happen.
The Linux kernel has been very very good about this exposure, in the sense
that we try very very hard not to break it. I think that's a good thing.
But that in no way makes anything a "contract".
It's the same thing with CPU vendors. You as a programmer, didn't get to
dictate what the CPU vendor implemented. You take it or leave it, and at
most you can make suggestions about things that might make the CPU vendor
sell more chips. The ISA is not a "contract" - it's just an interface that
was handed down to you. You didn't get to negotiate, you got to say "yes"
or "no".
Linus
Christoph Hellwig wrote:
>>b) when include/user is deemed sufficiently populated, a flag day is
>>declared and links from /usr/include are switched to them
>
>
> there are no such links, only copies (more or less modified)
This may be somewhat heretical, but someone has to ask...
Once include/user/foo.h is sufficiently clean and sufficiently complete, is
there any reason to not allow such links?
Chris
On Nov 30, 2004, Al Viro <[email protected]> wrote:
> WTF? I've got a dozen kernel trees hanging around, which one (and WTF any,
> while we are at it) should be "linked to"?
Whichever you chose to install in your /usr/include, and use as the
kernel ABI definition as far as userland is concerned.
I don't think `make install' should touch /usr/include at all. It
should be a separate step, such that one can build a kernel abi
headers package out of the kernel source tree, ideally without even
having to configure it first, and use that as the kernel ABI
definition. You should only have to do this again *if* the ABI
changes (ideally no removals, only additions) *and* you want to take
advantage of them, *and* you're willing to rebuild userland pieces
that could take advantage of them.
On most systems, people won't do that, they'll just use whatever
kernel headers and glibc binaries their distro ships. If they want to
change any of these, they have to know what they're doing.
--
Alexandre Oliva http://www.ic.unicamp.br/~oliva/
Red Hat Compiler Engineer aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist oliva@{lsd.ic.unicamp.br, gnu.org}
On Tue, 30 Nov 2004, Mariusz Mazur wrote:
>
> Userland? And who's using vanilla kernel headers in userland? Duplicates
> maintainers can always fix stuff independent of kernel (and I do).
Dammit, how hard is it for people to admit that they aren't the only ones
around?
There are still people who use ancient setups, and upgrade the kernel.
They won't complain to _you_ or to redhat when their setup breaks, because
they aren't _using_ your setup. They complain to me.
This is what statisticians call "self-selection". It means that you guys
only see the part of the world that uses your stuff, so you think that
there is nothing else.
> I don't get your point.
Very obviously.
Linus
On Tue, Nov 30, 2004 at 02:55:00PM -0800, Linus Torvalds wrote:
>
>
> On Tue, 30 Nov 2004, Matt Mackall wrote:
> >
> > So we follow dhowell's plan with the following additions:
>
> No.
>
> We do _not_ move stuff over that is questionable.
Umm. I don't think you even read what I wrote.
> I thought that was clear by now. The rules are:
> - we only move things that _have_ to move
Yes. And when we move a definition from linux/foo.h to user/foo.h, we include
them from linux/foo.h..
> - we don't break existing programs, and no "but they are broken already"
> is not an excuse.
..which prevents userland breakage during the transition (and is generally what
you want in the kernel anyway).
> - we only move things where that _particular_ move can be shown to be
> beneficial.
It's my opinion that it will eventually be deemed beneficial to
separate out everything that has a legitimate reason to be used by
userspace. But I'm not suggesting getting there in one go, what I'm
suggesting is how to get there incrementally. To rehash:
1. create include/user and friends
2. when we run across a troublesome ABI definition:
create include/user/foo.h and move the definition there
make sure include/linux/foo.h includes it
userspace and kernel compile as before
send a patch
3. repeat step 2 as often as useful
4. add new user ABI always to include/user/
5. if at some point we find that all of userspace builds from
include/user/ without reference to include/linux/, declare
include/user the canonical ABI
6. drop #define __KERNEL__, etc..
7. drop any superfluous include/linux -> include/user includes (there
shouldn't be many)
--
Mathematics is the supreme nostalgia of our time.
On Tue, Nov 30, 2004 at 07:39:34PM -0200, Alexandre Oliva wrote:
> Sure, we could take headers from linux-*/include/user and install them
> in /usr/include/kernel, but then includes in there that reference
> other headers in user/ or in asm-<arch>/ will cease to work.
I covered this with an evil sed script earlier in the thread.
--
"Next the statesmen will invent cheap lies, putting the blame upon
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince
himself that the war is just, and will thank God for the better sleep
he enjoys after this process of grotesque self-deception." -- Mark Twain
On Tue, Nov 30, 2004 at 05:09:35PM -0600, Chris Friesen wrote:
> Christoph Hellwig wrote:
>
> >>b) when include/user is deemed sufficiently populated, a flag day is
> >>declared and links from /usr/include are switched to them
> >
> >
> >there are no such links, only copies (more or less modified)
Indeed.
> This may be somewhat heretical, but someone has to ask...
>
> Once include/user/foo.h is sufficiently clean and sufficiently complete, is
> there any reason to not allow such links?
Briefly it's preferable not to need the kernel source nor a particular
version of the kernel source around to compile things. Though a
complete include/user makes building a proper kernel-headers package
to install in /usr/include pretty trivial.
--
Mathematics is the supreme nostalgia of our time.
On ?roda 01 grudzie? 2004 00:03, Al Viro wrote:
> > Wrong. These dirs must be linked to /usr/include so they must have more
> > meaningfull names.
>
> WTF? I've got a dozen kernel trees hanging around, which one (and WTF any,
> while we are at it) should be "linked to"?
Linked, copied, mount --binded, whatever. Just not under the
name /usr/include/user, but something more meaningfull.
And to directly answer your question: you'll 'link' the newest kernel you've
got around (we all know linux doesn't break abi compatibility, so the newest
kernel will be fully compatible with all other kernel lying around, right? :)
--
In the year eighty five ten
God is gonna shake his mighty head
He'll either say,
"I'm pleased where man has been"
Or tear it down, and start again
Al Viro wrote:
> On Tue, Nov 30, 2004 at 11:44:21PM +0100, Mariusz Mazur wrote:
>>Wrong. These dirs must be linked to /usr/include so they must have more
>>meaningfull names.
>
>
> WTF? I've got a dozen kernel trees hanging around, which one (and WTF any,
> while we are at it) should be "linked to"?
Just my point of view, but I'd love to have /usr/include/linux and
/usr/include/asm linked to the headers for the currently running kernel.
For a while I was playing with adding a bunch of new syscalls, and it sure would
have been nice to be able to just include <asm/unistd.h> and get the up-to-date
syscalls for whatever kernel I happened to be running.
I've since switched to using a device node and doing ioctl() calls to get around
this, but the issue still stands as I need to either import the ioctl() command
numbers and data types, or else I have to duplicate the header in userspace and
keep the two versions up to date.
Chris
On Tue, 2004-11-30 at 14:51 -0800, Linus Torvalds wrote:
> The way to prevent _future_ abuse is by adding something like
>
> #ifndef __KERNEL__
> #warning "This really doesn't work"
> #endif
>
> which does that, and has the advantage of not breaking anything at all.
>
> In other words: if you want to move things around just to break things,
> THEN THAT IS INCREDIBLY STUPID. We don't do things to screw our users
> over.
Linus, you're arguing that it's better to let users use something which
is non-portable and silently does the wrong thing, as long as it
actually compiles. That this is preferable to making sure it doesn't
compile.
Now, there are cases (like _perhaps_ byteorder.h) where we should
probably allow this kind of 'abuse' to continue because it's fairly
harmless and it does actually _work_.
But atomic.h isn't an example of that.
If I wrote a userspace program which relies upon the md5sum of certain
kernel headers not changing, would you decree that the headers in
question should not change at _all_ because that would 'break existing
software'? Of course not; you have to draw the line somewhere. And I
would draw it somewhere between atomic.h and byteorder.h -- where would
_you_ draw it?
--
dwmw2
On Tue, 2004-11-30 at 14:55 -0800, Linus Torvalds wrote:
>
> On Tue, 30 Nov 2004, Matt Mackall wrote:
> >
> > So we follow dhowell's plan with the following additions:
>
> No.
>
> We do _not_ move stuff over that is questionable.
>
> I thought that was clear by now. The rules are:
> - we only move things that _have_ to move
> - we don't break existing programs, and no "but they are broken already"
> is not an excuse.
> - we only move things where that _particular_ move can be shown to be
> beneficial.
>
> No whole-sale moves. No "let's break things that I think are broken". No
> "let's change things because we can".
>
> Well-defined moves. Both in content _and_ in reason.
Fine. And yes, we do it piecemeal with each part being given individual
consideration.
I'd like to aim for a conclusion where everything which userspace needs
is in the new directories and nothing is required from the private
directories any more. But let's start with the really important parts
which really _are_ crying out to be fixed.
--
dwmw2
On Tue, 30 Nov 2004, David Woodhouse wrote:
>
> Linus, you're arguing that it's better to let users use something which
> is non-portable and silently does the wrong thing, as long as it
> actually compiles. That this is preferable to making sure it doesn't
> compile.
I'm saying that if it worked before, it should work after. And my
suggestion gives a nice warning.
> But atomic.h isn't an example of that.
Even atomic.h. I could well imagine that somebody includes atomic.h just
to get the thread-safe updates for some architectures. For example,
asm-alpha/atomic.h does it right, and I woul dnot be at all surprised if
somebody had noticed.
And your suggestion has the problem that the people who get bitten by a
non-compiling thing are not necessarily the same people who can fix it.
> software'? Of course not; you have to draw the line somewhere. And I
> would draw it somewhere between atomic.h and byteorder.h -- where would
> _you_ draw it?
I'll draw it at "somebody might validly use it", because the fact is, I
can't test it.
Which is why I want patches to this to be OBVIOUSLY CORRECT, dammit! How
hard is that to understand?
The fact is, the less benefit there is from a change, the more obviously
correct it has to be in all forms. Moving stuff around in header files
basically fixes zero bugs in the kernel, so the benefit for the kernel is
basically zero. Which means that the obviousness factor of the change had
better be pretty close to infinite.
Comprende?
Linus
In article <[email protected]>,
Chris Friesen <[email protected]> wrote:
>Christoph Hellwig wrote:
>
>>>b) when include/user is deemed sufficiently populated, a flag day is
>>>declared and links from /usr/include are switched to them
>>
>>
>> there are no such links, only copies (more or less modified)
>
>This may be somewhat heretical, but someone has to ask...
>
>Once include/user/foo.h is sufficiently clean and sufficiently complete, is
>there any reason to not allow such links?
It would be cleaner to use gcc -I /lib/modules/`uname -r`/build/include
so that you don't break regular compiles by accident.
Mike.
On Wed, 1 Dec 2004, David Woodhouse wrote:
>
> The concept isn't at all hard to understand. But no patch is 'obviously
> correct' if you want to protect against the _slightest_ possibility that
> people might be abusing something you're taking away.
I really disagree. That's kind of my point. We _can_ make sure that there
is abzolutely zero semantic content change.
People both inside and outside the kernel who use the old <linux/xxx.h>
headers will get exactly what they got before if we do it right.
> Some people might define __KERNEL__ on purpose when compiling something
> in userspace, to get something that would otherwise be hidden from them.
> Would you consider that sacrosanct too?
Why _do_ you want to break things? Do the cleanup. Don't do the breakage.
Linus
On Tue, 30 Nov 2004, Linus Torvalds wrote:
>
> Even atomic.h. I could well imagine that somebody includes atomic.h just
> to get the thread-safe updates for some architectures. For example,
> asm-alpha/atomic.h does it right, and I would not be at all surprised if
> somebody had noticed.
In fact, this is not entirely theoretical. I know people _have_ noticed,
because I've gotten queries from some projects that wanted to copy the
definitions for their alpha port. I only got those queries because my name
is on the file, and those people wanted to actually copy them, not just
include the header file. I don't know _how_ many people decided to just
do the #include.
There are probably more people familiar with the kernel source tree than
with GLIB which is probably the preferred way to do those things these
days. And a few years ago that choice wasn't even there.
Linus
On Tue, 2004-11-30 at 16:37 -0800, Linus Torvalds wrote:
>
> On Wed, 1 Dec 2004, David Woodhouse wrote:
> >
> > The concept isn't at all hard to understand. But no patch is 'obviously
> > correct' if you want to protect against the _slightest_ possibility that
> > people might be abusing something you're taking away.
>
> I really disagree. That's kind of my point. We _can_ make sure that there
> is abzolutely zero semantic content change.
We've _never_ made sure that there's absolutely zero semantic content
change in our private headers.
> People both inside and outside the kernel who use the old <linux/xxx.h>
> headers will get exactly what they got before if we do it right.
They'll get that if we change nothing at all.
> > Some people might define __KERNEL__ on purpose when compiling something
> > in userspace, to get something that would otherwise be hidden from them.
> > Would you consider that sacrosanct too?
>
> Why _do_ you want to break things? Do the cleanup. Don't do the breakage.
I'm trying to understand precisely where _you_ want to draw the line
between 'cleanup' and 'breakage'.
You obviously don't agree that including atomic.h was an entirely stupid
thing for userspace to be doing in the first place, and that it's
reasonable for that not to compile in future; you believe that it should
continue to compile and silently do the wrong thing. We've covered that
much.
So where _do_ you draw the line? Trying to use spinlock.h? Defining
__KERNEL__ before including kernel headers?
I can try to apply common sense here, having discussed it with a bunch
of other people. But you don't seem to agree with that; hence further
guidance would be appreciated. But we can just submit patches to do the
cleanups and see what you take and what you don't, if you really want to
make us jump through hoops for the sake of it without indicating what
you'd accept and what you'd not beforehand.
--
dwmw2
On Wed, 1 Dec 2004, David Woodhouse wrote:
>
> > I really disagree. That's kind of my point. We _can_ make sure that there
> > is abzolutely zero semantic content change.
>
> We've _never_ made sure that there's absolutely zero semantic content
> change in our private headers.
Not true. For example, I don't take spelling fixes that do semantic
content changes.
See? The more trivial the fix, the more obviously correct it has to be.
This isn't even a "fix". It's a cleanup. It goes under the same rules a
spelling fix does.
Linus
On Tue, 2004-11-30 at 16:57 -0800, Linus Torvalds wrote:
> This isn't even a "fix". It's a cleanup. It goes under the same rules
> a spelling fix does.
So you don't see a long-term technical benefit in cleaning up the
API/ABI we export to userspace so that userspace stops depending on
stuff which just isn't supposed to be there? It's all just cosmetic
masturbation as far as you're concerned? There's no point in trying to
get to the point where we don't need to separately maintain a
glibc-kernheaders package because it can be taken directly from the
kernel?
--
dwmw2
On Tue, 2004-11-30 at 16:10 -0800, Linus Torvalds wrote:
> I'll draw it at "somebody might validly use it", because the fact is, I
> can't test it.
>
> Which is why I want patches to this to be OBVIOUSLY CORRECT, dammit! How
> hard is that to understand?
The concept isn't at all hard to understand. But no patch is 'obviously
correct' if you want to protect against the _slightest_ possibility that
people might be abusing something you're taking away.
Some people might define __KERNEL__ on purpose when compiling something
in userspace, to get something that would otherwise be hidden from them.
Would you consider that sacrosanct too?
--
dwmw2
On Wed, 1 Dec 2004, David Woodhouse wrote:
>
> On Tue, 2004-11-30 at 16:57 -0800, Linus Torvalds wrote:
> > This isn't even a "fix". It's a cleanup. It goes under the same rules
> > a spelling fix does.
>
> So you don't see a long-term technical benefit in cleaning up the
> API/ABI we export to userspace so that userspace stops depending on
> stuff which just isn't supposed to be there?
I see _benefits_, but exactly because they aren't immediate, I think we
should be careful.
Also, exactly because moving things around is a total _nightmare_ from a
diff and source maintenance perspective, we should be _extra_ careful.
It's almost impossible to see something that changed when that something
is _also_ being moved around. You don't see it in the diff, and you don't
see it in any of the tools - a small semantic change that would have been
absolutely OBVIOUS if it was in a regular diff gets totally drowned out by
the "move" diff.
Put another way: when people do things like re-indenting a driver (which
has _exactly_ the same issues: cleanup for the future, but totally
destroys all chance of seeing what the changes actually were), I want the
patch that does the re-indentation to be totally independent of the actual
changes.
Ie we do one patch that _only_ does the whitespace changes, and try to
make sure that it doesn't break anything even by mistake. People even
compile the thing and verify that it is 100% the same thing as an object
file. Then that gets applied. Only _after_ that should you make changes.
Do we follow this religiously? For small things, clearly no. We sometimes
mix whitespace and real changes. But we sure as hell shouldn't. At least
not for anything bigger.
The EXACT same thing is true of any header file movement. DO NOT CHANGE
SEMANTICS WHEN YOU MOVE STUFF!
It's not just my personal hangup. It's a damn good idea.
I will hereby just ignore this thread. Please remove me from the Cc, I've
wasted way too much time on this total idiocy already. I've made my
opinions extremely clear, and quite frankly, I don't want to hear anything
on this any more for at least a week. It's just not worth it.
Linus
On Tue, Nov 30, 2004 at 11:50:56PM +0000, David Woodhouse wrote:
> Now, there are cases (like _perhaps_ byteorder.h) where we should
> probably allow this kind of 'abuse' to continue because it's fairly
> harmless and it does actually _work_.
OK, time for my pet peeve: linux/cdrom.h.
#include <asm/byteorder.h>
[...]
/* for CDROM_PACKET_COMMAND ioctl */
struct cdrom_generic_command
{
[...]
struct request_sense __user *sense;
[...]
}
struct request_sense {
#if defined(__BIG_ENDIAN_BITFIELD)
__u8 valid : 1;
__u8 error_code : 7;
#elif defined(__LITTLE_ENDIAN_BITFIELD)
__u8 error_code : 7;
__u8 valid : 1;
#endif
__u8 segment_number;
[...]
}
At least one architecture's asm/byteorder.h has been C++-incompatible
in the past (I forget what; it was 2 years ago that I cared about it).
So to fix this, we need a asm-*/user/byteorder.h that only describes
the endianness. Except ... what about mips/mipsel?
--
"Next the statesmen will invent cheap lies, putting the blame upon
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince
himself that the war is just, and will thank God for the better sleep
he enjoys after this process of grotesque self-deception." -- Mark Twain