M?ns Rullg?rd wrote:
> Excuse my ignorance, but why can't the headers from the kernel still
> be used. They seem to be working fine here.
For current kernels, the "official" method is to have cleaned up copies
of the kernel headers shipped with glibc and placed in
/usr/include/linux and /usr/include/asm. The "real" headers will often
work, but not always,
To complicate things, if you add new stuff to the kernel (new ioctl
commands, etc.) then your app needs to either link against the "real"
headers, or else duplicate the definitions.
Its kind of a mess.
In an ideal world there would be clean "userspace" headers shipped with
the kernel, and the kernel would then use those headers plus the
kernel-only stuff.
Chris
--
Chris Friesen | MailStop: 043/33/F10
Nortel Networks | work: (613) 765-0557
3500 Carling Avenue | fax: (613) 765-2986
Nepean, ON K2H 8E9 Canada | email: [email protected]
Chris Friesen <[email protected]> writes:
> For current kernels, the "official" method is to have cleaned up
> copies of the kernel headers shipped with glibc and placed in
> /usr/include/linux and /usr/include/asm.
Not sure about it being "official".
It may make sense if it's a distribution - many users don't install
kernel sources. Still, from a technical point of view, it should be
a straight copy of kernel includes - we don't want to maintain two
separate sets, do we?
> The "real" headers will
> often work, but not always,
Then they should be fixed.
I.e. parts for internal kernel use should be wrapped with #ifdef __KERNEL__.
Personally I consider every kernel header which prevents successful
user space compilation buggy.
> To complicate things, if you add new stuff to the kernel (new ioctl
> commands, etc.) then your app needs to either link against the "real"
> headers, or else duplicate the definitions.
>
> Its kind of a mess.
Precisely. This is why we need just one header set.
> In an ideal world there would be clean "userspace" headers shipped
> with the kernel, and the kernel would then use those headers plus the
> kernel-only stuff.
Not sure about it. How is it different from clean "kernel" headers
shipped (obviously) with the kernel?
The "non-problem" here is, IMHO, that the cleaning of kernel headers
is quite trivial, and thus nobody is interested :-)
--
Krzysztof Halasa, B*FH
Krzysztof Halasa <[email protected]> writes:
> The "non-problem" here is, IMHO, that the cleaning of kernel headers
> is quite trivial, and thus nobody is interested :-)
It usually is. I guess I'll start sending patches whenever I need to
change one to get something compiled. Usually it's so trivial that I
think I must have made a mistake somewhere else.
--
M?ns Rullg?rd
[email protected]
[Also sent to [email protected]]
On Mon, Mar 01, 2004 at 07:10:31PM +0100, Krzysztof Halasa wrote:
>
> The "non-problem" here is, IMHO, that the cleaning of kernel headers
> is quite trivial, and thus nobody is interested :-)
The problem is that there is no infrastructure for abi only .h files (mainly).
Matthew Wilcox + others IIRC has already posted a few patches, but I do not see
this happening until 2.7 opens up.
When the proper infrastructure is agreed upon on lot's of people will put some effort
in this janitorial type of work. But imagine all the small mistakes, something we do not
want in 2.6.
IIRC the current agreed scheme is something along the lines of this:
abi/abi-linux/* Userspace relevant parts of include/linux
abi/abi-asm/ symlink to abi/abi-$(ARCH)
abi/abi-i386 i386 specific userland abi
abi/abi-ppc ppc ....
So a header file in include/linux with a counterpart in abi could look like this:
include/linux/wait.h:
#include <abi-linux/wait.h>
#include <linux/config.h>
typedef struct __wait_queue wait_queue_t;
...
abi/abi-linux/wait.h:
#define WNOHANG 0x00000001
#define WUNTRACED 0x00000002
This proposal meets some resistence related to internal issues such as
renaming of internal types etc.
But in the end the gain from a scheme like this outweights the drawbacks - IMHO.
And the backward compatible stuff can be located in abi where it may belong -
if really needed.
Sam
On Monday 01 of March 2004 19:10, Krzysztof Halasa wrote:
> Not sure about it being "official".
> It may make sense if it's a distribution - many users don't install
> kernel sources. Still, from a technical point of view, it should be
> a straight copy of kernel includes - we don't want to maintain two
> separate sets, do we?
We do. Abi changes are rather rare so keeping them as a separate tree wouldn't
add too much work for subsystem maintainers, and it would be a Good Thing
(tm) to have one place where the whole current abi can be seen. One thing to
note is that linux headers duplicate many structures and definitions that
should be (and are) provided by glibc. This causes collisions. My current
practice is to clear offending linux headers of their content, and simply
include appropriate libc headers (with a nice warning) from them.
I can say that about 60-80% of current linux headers do not have proper
separation of kernel only code (counting in headers that are kernel only, but
have no visible signs of that fact) and adding that separation would take a
while. And even if successfull, it would add significant maintainer burden to
keep the whole thing working (it would probably look like crap too).
And we have to remember 2.4 compatibilities (which linux-libc-headers have) -
is 2.6 kernel a place for them?
(keep in mind I know squat about kernel development, so don't rely on my
opinions too much)
My current objective is to remove *all* remaining kernel code from
linux-libc-headers and add as much 2.4 compatibilities as possible. And this
will take a loooong time since the majority of headers are kernel only, but
are not marked as such, or are marked as such on one arch only (ppc guys
rock :), and assumption that removal of those headers from other archs is
safe is... well... just an assumption.
And as to leaving this as a separate project, or trying to push it into the
kernel - well, I see pros and cons of both options (and it's not my
decision :).
--
In the year eighty five ten
God is gonna shake his mighty head
He'll either say,
"I'm pleased where man has been"
Or tear it down, and start again
On Wed, 3 Mar 2004, Mariusz Mazur wrote:
> On Monday 01 of March 2004 19:10, Krzysztof Halasa wrote:
> > Not sure about it being "official".
> > It may make sense if it's a distribution - many users don't install
> > kernel sources. Still, from a technical point of view, it should be
> > a straight copy of kernel includes - we don't want to maintain two
> > separate sets, do we?
>
> We do. Abi changes are rather rare so keeping them as a separate tree wouldn't
> add too much work for subsystem maintainers, and it would be a Good Thing
> (tm) to have one place where the whole current abi can be seen. One thing to
> note is that linux headers duplicate many structures and definitions that
> should be (and are) provided by glibc. This causes collisions. My current
> practice is to clear offending linux headers of their content, and simply
> include appropriate libc headers (with a nice warning) from them.
> I can say that about 60-80% of current linux headers do not have proper
> separation of kernel only code (counting in headers that are kernel only, but
> have no visible signs of that fact) and adding that separation would take a
> while. And even if successfull, it would add significant maintainer burden to
> keep the whole thing working (it would probably look like crap too).
> And we have to remember 2.4 compatibilities (which linux-libc-headers have) -
> is 2.6 kernel a place for them?
> (keep in mind I know squat about kernel development, so don't rely on my
> opinions too much)
>
> My current objective is to remove *all* remaining kernel code from
> linux-libc-headers and add as much 2.4 compatibilities as possible. And this
> will take a loooong time since the majority of headers are kernel only, but
> are not marked as such, or are marked as such on one arch only (ppc guys
> rock :), and assumption that removal of those headers from other archs is
> safe is... well... just an assumption.
>
> And as to leaving this as a separate project, or trying to push it into the
> kernel - well, I see pros and cons of both options (and it's not my
> decision :).
Then maybe we need 3 types of includes (each in separate directory I
think):
1. kernel only - cannot be seen from user space, for current kernel, no
comatibility
2. user only - cannot be seen from kernel (mainly for compatibility), will
not change very often
3. shared - really portions of includes with definitions that are shared
between current kernel headers and user headers, are included by both 1.
and 2. internally (= both kernel and user space does not include them
directly), extra care should be taken when changing it
This way there are no _KERNEL_ ifdefs needed any more and there is no code
or data duplications (= no inconsistencies between 1. and 2.).
And if some definitions in kernel 6.6.6 will be obsolete but should be
kept for compatibility with user space then it will be moved from 3. to 2.
and apropriate #include commands will be removed from 1. (and from 2.
too).
What do You think about this? (I know I am completly wrong :))
Grzegorz Kulewski
Sam Ravnborg wrote:
>
> IIRC the current agreed scheme is something along the lines of this:
>
> abi/abi-linux/* Userspace relevant parts of include/linux
> abi/abi-asm/ symlink to abi/abi-$(ARCH)
> abi/abi-i386 i386 specific userland abi
> abi/abi-ppc ppc ....
>
> So a header file in include/linux with a counterpart in abi could look like this:
>
> include/linux/wait.h:
> #include <abi-linux/wait.h>
>
> #include <linux/config.h>
> typedef struct __wait_queue wait_queue_t;
> ...
>
>
> abi/abi-linux/wait.h:
> #define WNOHANG 0x00000001
> #define WUNTRACED 0x00000002
>
>
> This proposal meets some resistence related to internal issues such as
> renaming of internal types etc.
> But in the end the gain from a scheme like this outweights the drawbacks - IMHO.
>
> And the backward compatible stuff can be located in abi where it may belong -
> if really needed.
>
I think the main issue is that it's going to take a fair amount of work,
and hence needs to wait until 2.7.
The other thing to consider if whether or not there should be some way
to export things that aren't easily expressible as #defines...
-hpa
Sam Ravnborg <[email protected]> writes:
> IIRC the current agreed scheme is something along the lines of this:
>
> abi/abi-linux/* Userspace relevant parts of include/linux
> abi/abi-asm/ symlink to abi/abi-$(ARCH)
> abi/abi-i386 i386 specific userland abi
> abi/abi-ppc ppc ....
More efforts, no real effects.
I don't think we need such an infrastructure.
The normal headers should just be usable for user-space inclusion.
However I realize that the difference isn't that important, as long
as we don't duplicate the definitions etc.
> So a header file in include/linux with a counterpart in abi could look
> like this:
>
> include/linux/wait.h:
> #include <abi-linux/wait.h>
>
> #include <linux/config.h>
> typedef struct __wait_queue wait_queue_t;
> ...
>
>
> abi/abi-linux/wait.h:
> #define WNOHANG 0x00000001
> #define WUNTRACED 0x00000002
why not:
include/linux/wait.h:
#define WNOHANG 0x00000001
#define WUNTRACED 0x00000002
#ifdef __KERNEL__
#include <linux/config.h>
...
#endif /* __KERNEL__ */
> But in the end the gain from a scheme like this outweights the drawbacks
> - IMHO.
Such as? In comparison to a (fixed) present situation?
--
Krzysztof Halasa, B*FH
Mariusz Mazur <[email protected]> writes:
> We do. Abi changes are rather rare so keeping them as a separate tree
> wouldn't
> add too much work for subsystem maintainers, and it would be a Good Thing
> (tm) to have one place where the whole current abi can be seen.
Sure. One place, yes. But separate header sets are IMHO too much crazy.
> One thing to
> note is that linux headers duplicate many structures and definitions that
> should be (and are) provided by glibc. This causes collisions. My current
> practice is to clear offending linux headers of their content, and simply
> include appropriate libc headers (with a nice warning) from them.
Is it the kernel which is based on glibc, or is it glibc (and the rest of
the userland as glibc isn't that special) using the kernel interface?
The kernel doesn't need glibc at all, I don't know why do you want it
to require some external headers to compile.
Should the kernel behave differently when compiled with different glibc
header sets? :-)
IMHO all the defines should be in the kernel tree. Glibc can and should
use them, as it uses the ABI.
> I can say that about 60-80% of current linux headers do not have proper
> separation of kernel only code (counting in headers that are kernel only,
> but
> have no visible signs of that fact)
No doubt.
> and adding that separation would take a
> while. And even if successfull, it would add significant maintainer burden
> to
> keep the whole thing working (it would probably look like crap too).
Don't think so. Anyway, there is no other acceptable option. The kernel
must compile without glibc headers (without any external headers, to
be precise) - as it doesn't need any external software to work.
The open question (of much less importance) is if we want to keep
the existing include/ layout or to move public parts to include/linux-abi
etc. It still has to reside in the kernel tree, though. I'd go with the
former for now as it requires less work. OTOH the latter might be
cleaner.
> And we have to remember 2.4 compatibilities (which linux-libc-headers
> have) -
> is 2.6 kernel a place for them?
Examples?
If they are part of kernel API/ABI, then of course they are still used
by 2.6 kernel and they need to be there. If they aren't used by the
kernel (old #define names for instance) they should go to glibc headers
(#ifndef xxx #define xxx etc.).
--
Krzysztof Halasa, B*FH
Krzysztof Halasa wrote:
> Sam Ravnborg <[email protected]> writes:
>
>
>>IIRC the current agreed scheme is something along the lines of this:
>>
>>abi/abi-linux/* Userspace relevant parts of include/linux
>>abi/abi-asm/ symlink to abi/abi-$(ARCH)
>>abi/abi-i386 i386 specific userland abi
>>abi/abi-ppc ppc ....
>
>
> More efforts, no real effects.
> I don't think we need such an infrastructure.
> The normal headers should just be usable for user-space inclusion.
No, this is a big pain :)
The ABI headers shared with userspace need to be split from definitions
that are only used in kernel space. #ifdef __KERNEL__ is a source of
frustration :)
Jeff
Jeff Garzik wrote:
> Krzysztof Halasa wrote:
>
>> Sam Ravnborg <[email protected]> writes:
>>
>>
>>> IIRC the current agreed scheme is something along the lines of this:
>>>
>>> abi/abi-linux/* Userspace relevant parts of include/linux
>>> abi/abi-asm/ symlink to abi/abi-$(ARCH)
>>> abi/abi-i386 i386 specific userland abi
>>> abi/abi-ppc ppc ....
>>
>>
>>
>> More efforts, no real effects.
>> I don't think we need such an infrastructure.
>> The normal headers should just be usable for user-space inclusion.
>
>
> No, this is a big pain :)
>
> The ABI headers shared with userspace need to be split from definitions
> that are only used in kernel space. #ifdef __KERNEL__ is a source of
> frustration :)
>
The biggest problem is that what is kernel ABI hasn't been very well
thought about. The result is that the split between the what the kernel
exports and what libc is supposed to provide itself is very haphazard.
And yes, #ifdef __KERNEL__ is a reflection of that.
-hpa
On Thursday 04 of March 2004 15:13, Krzysztof Halasa wrote:
> Is it the kernel which is based on glibc, or is it glibc (and the rest of
> the userland as glibc isn't that special) using the kernel interface?
>
> The kernel doesn't need glibc at all, I don't know why do you want it
> to require some external headers to compile.
> Should the kernel behave differently when compiled with different glibc
> header sets? :-)
I never said kernel should require glibc - it shouldn't (mind you I don't do
kernelland headers). But kernel does duplicate each and every structure
provided by glibc. It has to. The Bad Thing (tm) is that all (well... allmost
all - lots of linux headers don't parse correctly in userspace) of those
structures get exported to userland. And programmers use them. They don't
include <sys/resource.h>, but <linux/resource.h>. And that causes conflicts
(and is bad practice).
> IMHO all the defines should be in the kernel tree. Glibc can and should
> use them, as it uses the ABI.
Parts of abi that are standardized
(http://www.opengroup.org/onlinepubs/007904975/ - this thing; check the
headers section), should be imho provided by C libs. These things do not
change (they can't or everything would blow up) and I see no reason why glibc
should rely on having additional headers available, just to do what it's
supposed to.
> The open question (of much less importance) is if we want to keep
> the existing include/ layout or to move public parts to include/linux-abi
> etc. It still has to reside in the kernel tree, though. I'd go with the
> former for now as it requires less work. OTOH the latter might be
> cleaner.
Userland headers should be kept in /usr/include/{asm,linux}. I see no reason
to change that (kernel headers have no business being in /usr/include btw).
As to linux-common linux-kernelonly and linux-userland headers (linux-common
used by both) - I just find it weird for userland to require kernel sources.
Linux is supposed to have stable abi.
> > And we have to remember 2.4 compatibilities (which linux-libc-headers
> > have) -
> > is 2.6 kernel a place for them?
>
> Examples?
> If they are part of kernel API/ABI, then of course they are still used
> by 2.6 kernel and they need to be there. If they aren't used by the
> kernel (old #define names for instance) they should go to glibc headers
> (#ifndef xxx #define xxx etc.).
Additionall defines mostly. Probably some extra structures.
--
In the year eighty five ten
God is gonna shake his mighty head
He'll either say,
"I'm pleased where man has been"
Or tear it down, and start again
Mariusz Mazur <[email protected]> writes:
> I never said kernel should require glibc - it shouldn't (mind you I don't do
> kernelland headers). But kernel does duplicate each and every structure
> provided by glibc. It has to.
No, there is no such requirement. Glibc doesn't need to duplicate kernel
defs either.
Do you really want duplicated code (definitions etc)?
> The Bad Thing (tm) is that all (well... allmost
> all - lots of linux headers don't parse correctly in userspace)
That's bad, for sure.
> of those
> structures get exported to userland. And programmers use them. They don't
> include <sys/resource.h>, but <linux/resource.h>. And that causes conflicts
> (and is bad practice).
So the kernel headers are buggy (and should be fixed) and the glibc
headers contain unneeded code (and they should be fixed as well).
>> IMHO all the defines should be in the kernel tree. Glibc can and should
>> use them, as it uses the ABI.
>
> Parts of abi that are standardized
> (http://www.opengroup.org/onlinepubs/007904975/ - this thing; check the
> headers section), should be imho provided by C libs.
The C library alone obviously provide some functionality such as string.h
functions and such API should be defined in glibc headers.
However, a part of the glibc (or any libc) functionality is provided by
the kernel itself. It is still correct to say that those are provided
by libc, and that this is libc providing the headers. It doesn't mean
the syscalls can't go straight to the kernel and that the libc
headers can't just include/be the kernel ones.
> These things do not
> change (they can't or everything would blow up) and I see no reason why glibc
> should rely on having additional headers available, just to do what it's
> supposed to.
Because glibc has to rely on underlying kernel services while the user
space programs are executed.
You know, Linux can be run on different architectures. Do you want to
keep different sets of header files for each arch? Yes, the APIs differs.
BTW: the user space programs don't necessarily need glibc. They may need
the kernel API headers, of course.
You may even want to use another C library. Would you copy the headers
in such case again?
> Userland headers should be kept in /usr/include/{asm,linux}.
Because?
> As to linux-common linux-kernelonly and linux-userland headers (linux-common
> used by both) - I just find it weird for userland to require kernel sources.
> Linux is supposed to have stable abi.
No, it isn't. It's supposed to have a backward-compatible ABI.
You don't want to block progress, do you?
However, this isn't only about progress. A functionality can be removed,
too.
>> Examples?
>> If they are part of kernel API/ABI, then of course they are still used
>> by 2.6 kernel and they need to be there. If they aren't used by the
>> kernel (old #define names for instance) they should go to glibc headers
>> (#ifndef xxx #define xxx etc.).
>
> Additionall defines mostly. Probably some extra structures.
I'd go with the above, then.
--
Krzysztof Halasa, B*FH
On Fri, 5 Mar 2004, Krzysztof Halasa wrote:
> BTW: the user space programs don't necessarily need glibc. They may need
> the kernel API headers, of course.
> You may even want to use another C library. Would you copy the headers
> in such case again?
I agree.
> > As to linux-common linux-kernelonly and linux-userland headers (linux-common
> > used by both) - I just find it weird for userland to require kernel sources.
But some programs (hdparm?) and/or libs must (for example most C libs like
glibc)...
> >> If they are part of kernel API/ABI, then of course they are still used
> >> by 2.6 kernel and they need to be there. If they aren't used by the
> >> kernel (old #define names for instance) they should go to glibc headers
> >> (#ifndef xxx #define xxx etc.).
> >
> > Additionall defines mostly. Probably some extra structures.
>
> I'd go with the above, then.
But how synchronize kernel development with glibcs (+ all other C libs +
all other programs that must interface directly with the kernel)?
When kernel developers want to stop using something, how they should tell
that to glibc developers and others. And how these non-kernel developers
can change their projects in very short time to start using new kernel
headers (= define things removed from the kernel in their own headers)?
Most admins change kernels more often than all other programs and libs
and these programs can potenitially not be updated yet.
My proposal is to move these things from linux-common to linux-userland
instead of removng them from the kernel immendiatelly.
So no compatibility with user program and libs will be broken (and after a
few releases of kernel, when all programs and libs will be updated, these
things can be removed completly from kernel headers of all three types).
Grzegorz Kulewski
ex inx user :)
Grzegorz Kulewski <[email protected]> writes:
> But how synchronize kernel development with glibcs (+ all other C libs +
> all other programs that must interface directly with the kernel)?
The kernel API should be stable enough, at least within stable trees.
Outside stable trees we should still maintain backward compatibility,
except when it comes to things which wouldn't work anyway (non-libc API
such as ipfw/ipchains/iptables etc).
> When kernel developers want to stop using something, how they should tell
> that to glibc developers and others.
They should not stop :-)
Adding things is another story.
> Most admins change kernels more often than all other programs and libs
> and these programs can potenitially not be updated yet.
I don't think so.
Applying a kernel patch usually doesn't change the API. Changing kernel
config should never change the API.
> My proposal is to move these things from linux-common to linux-userland
> instead of removng them from the kernel immendiatelly.
> So no compatibility with user program and libs will be broken (and after a
> few releases of kernel, when all programs and libs will be updated, these
> things can be removed completly from kernel headers of all three types).
Things which are IMHO not options:
- removing kernel headers (i.e. headers which are required by kernel
files to compile) from the kernel tree,
- duplicating kernel header definitions in external packages (a straight
copy for distro users is ok, things which need maintenance aren't).
Existing duplicates should be dealt with.
--
Krzysztof Halasa, B*FH
> Changing kernel config should never change the API.
Linux would be in pretty sad shape if that held.
Incompatible API changes should be rare, but they are an essential part
of our continuing healthy evolution. In particular, there's a _long_
list of hardware devices that have at some time or other worked on
Linux, but don't anymore - usually because no one is still maintaining
the driver needed. But sometimes other API's have to change or
disappear as well.
Would you say that a city or county should _never_ raze a building or
remove a road as part of development?
Change happens. Deal with it.
--
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.650.933.1373
Paul Jackson <[email protected]> writes:
>> Changing kernel config should never change the API.
>
> Linux would be in pretty sad shape if that held.
>
> Incompatible API changes should be rare, but they are an essential part
> of our continuing healthy evolution. In particular, there's a _long_
> list of hardware devices that have at some time or other worked on
> Linux, but don't anymore - usually because no one is still maintaining
> the driver needed. But sometimes other API's have to change or
> disappear as well.
You're talking about the kernel development while I meant changing
kernel configuration. They're very distant things.
And note that it's all about C API and not about some API between
different parts of the kernel.
--
Krzysztof Halasa, B*FH
> You're talking about the kernel development ...
No. I mean that even the C API that the kernel presents to
user code will sometimes change or have parts disappear.
To be avoided, yes. Worth going to substantial lengths
to avoid, yes. But not absolutely prohibited.
--
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.650.933.1373
Paul Jackson <[email protected]> writes:
>> You're talking about the kernel development ...
>
> No. I mean that even the C API that the kernel presents to
> user code will sometimes change or have parts disappear.
And it can be caused by changing kernel config?
Care to show an example?
--
Krzysztof Halasa, B*FH