Hi
Some recent changes to include/linux/sysctl.h breaks glibc.
Problem is that __sysctl_args have been modified to use '__user',
but that is only defined if __KERNEL__ is defined, because that
is the only time compiler.h is included.
------------------------------------------------
--- linux-2.5.69-bk2/include/linux/sysctl.h 2003-05-05
01:53:31.000000000 +0200
+++ linux-2.5.69-bk12/include/linux/sysctl.h 2003-05-18
22:12:39.000000000 +0200
@@ -36,11 +36,11 @@
member of a struct __sysctl_args to have? */
struct __sysctl_args {
- int *name;
+ int __user *name;
int nlen;
- void *oldval;
- size_t *oldlenp;
- void *newval;
+ void __user *oldval;
+ size_t __user *oldlenp;
+ void __user *newval;
size_t newlen;
unsigned long __unused[4];
};
--------------------------------------------------
Question: Is this expected behaviour ?
Thanks,
--
Martin Schlemmer
On Sun, May 18, 2003 at 10:21:56PM +0200, Martin Schlemmer wrote:
> Some recent changes to include/linux/sysctl.h breaks glibc.
> Problem is that __sysctl_args have been modified to use '__user',
> but that is only defined if __KERNEL__ is defined, because that
> is the only time compiler.h is included.
Don't include the kernel headers in userspace.
-- wli
On Sun, 2003-05-18 at 22:49, William Lee Irwin III wrote:
> On Sun, May 18, 2003 at 10:21:56PM +0200, Martin Schlemmer wrote:
> > Some recent changes to include/linux/sysctl.h breaks glibc.
> > Problem is that __sysctl_args have been modified to use '__user',
> > but that is only defined if __KERNEL__ is defined, because that
> > is the only time compiler.h is included.
>
> Don't include the kernel headers in userspace.
>
Yes, the standard answer. So what kernel headers should glibc
be compiled against then ?
--
Martin Schlemmer
On Sun, May 18, 2003 at 11:12:19PM +0200, Martin Schlemmer wrote:
> Yes, the standard answer. So what kernel headers should glibc
> be compiled against then ?
None. But as glibc still hasn't been fixed use kernel headers from linux 2.4.
On Mon, May 19, 2003 at 12:43:44PM +0200, Martin Schlemmer wrote:
> Right, so who are going to tell the glibc guys that ?
> -----------------------------------------------------------------
> configure: error: GNU libc requires kernel header files from
> Linux 2.0.10 or later to be installed before configuring.
> The kernel header files are found usually in /usr/include/asm and
> /usr/include/linux; make sure these directories use files from
> Linux 2.0.10 or later. This check uses <linux/version.h>, so
> make sure that file was built correctly when installing the kernel
> header
> files. To use kernel headers not from /usr/include/linux, use the
> configure option --with-headers.
> -----------------------------------------------------------------
IIRC you're supposed to use some sort of sanitized copy, not the things
directly. IMHO the current state of affairs sucks as there is no
standard set of ABI headers, but grabbing them right out of the kernel
is definitely not the way to go.
-- wli
On Mon, 2003-05-19 at 07:38, Christoph Hellwig wrote:
> On Sun, May 18, 2003 at 11:12:19PM +0200, Martin Schlemmer wrote:
> > Yes, the standard answer. So what kernel headers should glibc
> > be compiled against then ?
>
> None. But as glibc still hasn't been fixed use kernel headers from linux 2.4.
>
Right, so who are going to tell the glibc guys that ?
-----------------------------------------------------------------
configure: error: GNU libc requires kernel header files from
Linux 2.0.10 or later to be installed before configuring.
The kernel header files are found usually in /usr/include/asm and
/usr/include/linux; make sure these directories use files from
Linux 2.0.10 or later. This check uses <linux/version.h>, so
make sure that file was built correctly when installing the kernel
header
files. To use kernel headers not from /usr/include/linux, use the
configure option --with-headers.
-----------------------------------------------------------------
I do not mind if thing break due to compatibility if needed to
enhance an API, etc ... gcc does that enough.
The big problem however is that every time somebody screws up
user space due to changes in kernel headers, the reply is always
the same: "do not use kernel headers in user land"
Ok, lets say we stop doing that. How do anything user side find
out specifics at compile time related to the kernel it should run
on ? If we may not include kernel headers, what then?
Please understand, this is not me trolling, but everybody always
have the same answer for issues like this, but no solutions.
Thanks,
--
Martin Schlemmer
On Mon, 2003-05-19 at 12:51, William Lee Irwin III wrote:
> IIRC you're supposed to use some sort of sanitized copy, not the things
> directly. IMHO the current state of affairs sucks as there is no
> standard set of ABI headers, but grabbing them right out of the kernel
> is definitely not the way to go.
>
Ok, anybody know of an effort to get this done ?
Also, what about odd things that are more kernel dependant
like imon support in fam for example ? The imon.h will not
be in the 'sanitized copy' ....
Regards,
--
Martin Schlemmer
On Mon, 2003-05-19 at 12:51, William Lee Irwin III wrote:
>> IIRC you're supposed to use some sort of sanitized copy, not the things
>> directly. IMHO the current state of affairs sucks as there is no
>> standard set of ABI headers, but grabbing them right out of the kernel
>> is definitely not the way to go.
On Mon, May 19, 2003 at 01:14:02PM +0200, Martin Schlemmer wrote:
> Ok, anybody know of an effort to get this done ?
> Also, what about odd things that are more kernel dependant
> like imon support in fam for example ? The imon.h will not
> be in the 'sanitized copy' ....
It probably has to be hand-crafted for cases like the above.
I'm unaware of anyone actually doing anything about this.
-- wli
On Mon, 2003-05-19 at 13:14, Martin Schlemmer wrote:
> On Mon, 2003-05-19 at 12:51, William Lee Irwin III wrote:
>
> > IIRC you're supposed to use some sort of sanitized copy, not the things
> > directly. IMHO the current state of affairs sucks as there is no
> > standard set of ABI headers, but grabbing them right out of the kernel
> > is definitely not the way to go.
> >
>
> Ok, anybody know of an effort to get this done ?
Red Hat Linux ships with a mostly sanitized header set for this.
>
> Also, what about odd things that are more kernel dependant
> like imon support in fam for example ? The imon.h will not
> be in the 'sanitized copy' ....
apps that have such deep knowledge about internals are supposed to
provide their own copy of the headers in RHL at least. But there are few
of those.
On Mon, May 19, 2003 at 01:14:02PM +0200, Martin Schlemmer wrote:
> like imon support in fam for example ? The imon.h will not
> be in the 'sanitized copy' ....
Imon is a braindead IRIX feature and SGI stopped doing Linux patches
for it when dnotify became more common. So WTF are you talking about?
On Mon, May 19, 2003 at 12:43:44PM +0200, Martin Schlemmer wrote:
> Ok, lets say we stop doing that. How do anything user side find
> out specifics at compile time related to the kernel it should run
> on ?
They don't. You can run the same userspace on a wide range of kernels.
I'd just leave the job of selcting your headers to the distro vendor -
if they are too stupid to get their headers sanitized I'd
just use a different distro.
On Mon, 2003-05-19 at 13:43, Christoph Hellwig wrote:
> On Mon, May 19, 2003 at 01:14:02PM +0200, Martin Schlemmer wrote:
> > like imon support in fam for example ? The imon.h will not
> > be in the 'sanitized copy' ....
>
> Imon is a braindead IRIX feature and SGI stopped doing Linux patches
> for it when dnotify became more common. So WTF are you talking about?
>
It was just an example ffs.
--
Martin Schlemmer
On Mon, 2003-05-19 at 13:45, Christoph Hellwig wrote:
> On Mon, May 19, 2003 at 12:43:44PM +0200, Martin Schlemmer wrote:
> > Ok, lets say we stop doing that. How do anything user side find
> > out specifics at compile time related to the kernel it should run
> > on ?
>
> They don't. You can run the same userspace on a wide range of kernels.
> I'd just leave the job of selcting your headers to the distro vendor -
> if they are too stupid to get their headers sanitized I'd
> just use a different distro.
>
Ok, so say they use LFS ?
Point is just that people like you keep on bitching about not
using sanitized kernel headers, but do nothing about it, or
until today have said nothing about 'sanitized headers'.
--
Martin Schlemmer
On Mon, May 19, 2003 at 02:56:25PM +0200, Martin Schlemmer wrote:
> Point is just that people like you keep on bitching about not
> using sanitized kernel headers, but do nothing about it, or
> until today have said nothing about 'sanitized headers'.
Why don't you just get the glibc-kernheaders package from rawhide
(or the equivalent from your prefered distribution) and stop
complaining?
On Mon, May 19, 2003 at 02:06:18PM +0100, Christoph Hellwig wrote:
> On Mon, May 19, 2003 at 02:56:25PM +0200, Martin Schlemmer wrote:
> > Point is just that people like you keep on bitching about not
> > using sanitized kernel headers, but do nothing about it, or
> > until today have said nothing about 'sanitized headers'.
>
> Why don't you just get the glibc-kernheaders package from rawhide
> (or the equivalent from your prefered distribution) and stop
> complaining?
Since this is brought up againg.
Today we have:
include/asm-<arch>
include/linux
include/<subsystem>
Within include/linux grep told me that "#ifdef __KERNEL__" was present
in 219 places. So a lot of header files are kept in a shape that allows
them to be included from user-land.
I see two possible solutions here:
1) Automate the process of creating sanitized user-land headers.
- It should be a trivial task to do - but will require some effort
to be done right. Does any tool exist to do it today?
Automation allows glibc to user kernel headers for a recent
kernel without looking up a distribution specific set of header
files somewhere.
Dave M brougt up one issue the other day, when he added a new define
to a header file. No need to wait for a distribution to pick up.
2) Create a user level hirachy under include/ [obviously 2.7 material]
include/linux/user
include/<subsystem>/user
include/asm-<arch>/user
In reality we will see a lot of files in include/linux that looks like:
sched.h:
part 1 - all relevant includes:
#include <linux/.....>
...
part 2 - user level interface
#include <linux/user/sched.h>
part 3 - kernel specific definitions.
extern rwlock_t tasklist_lock;
extern spinlock_t mmlist_lock;
I am well aware that by default glibc shall NOT use kernel headers.
But IMHO it is too difficult to upgrate to a new version of the kernel
headers.
Care to explain what is wrong with the above approaches - I may have
missed something obvious.
Sam
On Mon, 19 May 2003, Sam Ravnborg wrote:
>
> Within include/linux grep told me that "#ifdef __KERNEL__" was present
> in 219 places. So a lot of header files are kept in a shape that allows
> them to be included from user-land.
No. They're definitely _not_ "kept in shape".
A number of headers have historical baggage, mainly to support the
old libc5 habits, and because removing the ifdef's is something that
nobody has felt was worth the pain.
I think the only header file that should be considered truly exported is
something like "asm/posix_types.h". For the others, we'll add __KERNEL__
protection on demand if the glibc guys can give good arguments that it
helps them do the "copy-and-cleanup" phase.
Linus
The question still remains unanswered. How do you get the sanitized
headers? "Use your vendor" is NOT the answer. People need to get off
their high horses and stop passing the buck to the ambiguous "vendor".
Even the vendor (gentoo in this case) needs the answer on how to make
sanitized headers.
We are nearly 70 versions into 2.5 and approaching 2.6 quickly. There
are numerous userland programs that are now using kernel headers because
2.4 headers simply don't suffice anymore -- they don't support the
hardware or interfaces of today's equipment. It used to be "use the
headers that you compiled the system with [glibc]" but that doesn't even
work anymore.
We don't need the bullsh*t answer "don't use kernel headers" because
there are oodles of packages now that require the information that is in
kernel headers for one reason or another, some fishy, some legit. In a
perfect world we'd have a nice set of APIs for everybody that was always
kept freshly up to date from all the maintainers. If it isn't
centralized and synchronized in the kernel headers there every package
out there is going to be maintaining patches against the kernel source
so their module builds correctly and this leads to patch co-existence
nightmares.
Someone please step up to the plate and explain how to convert kernel
headers into sanitized headers for /usr/include.
Thank you and flames > /dev/null
David
>They don't. You can run the same userspace on a wide range of kernels.
>I'd just leave the job of selcting your headers to the distro vendor -
>if they are too stupid to get their headers sanitized I'd
>just use a different distro.
>
How about not passing the buck off to "the vendor" and helping "this
vendor" make sanitized headers.
Not everybody copies RH and changes the text files from "RH" to "my
distribution name".
Gentoo is my very strongly preferred distribution and I have a lot of
respect and admiration for Martin for stepping in this bull*shit pool
and trying to get something accomplished.
David
Christoph Hellwig wrote:
>On Mon, May 19, 2003 at 02:56:25PM +0200, Martin Schlemmer wrote:
>
>
>>Point is just that people like you keep on bitching about not
>>using sanitized kernel headers, but do nothing about it, or
>>until today have said nothing about 'sanitized headers'.
>>
>>
>
>Why don't you just get the glibc-kernheaders package from rawhide
>(or the equivalent from your prefered distribution) and stop
>complaining?
>
On Mon, 2003-05-19 at 19:59, David Ford wrote:
> Someone please step up to the plate and explain how to convert kernel
> headers into sanitized headers for /usr/include.
It seems you totally miseed the entire point.
It shouldn't be an automatic conversion. It should be a well thought
subset cleaned from kernel private stuff.
I maintain such a subset for my employer and it's free for all to use
(it's GPL after all).
I didn't miss the point. I don't use RH, and I'm not in a mood to
switch to RH just because RH has an LK headers maintainer.
The question is how to make these headers. Who decides what should and
shouldn't be available to userland? What of the myriad of tools which
make modules, or use deep kernel headers? What of the packages that try
hard to keep subset headers synchronized but get frustrated because
update patches get dropped? What am I supposed to do when I want to use
several of these packages and their updates conflict with each other?
AFAIK, you don't have a package that contains all the -current- headers
for all the current versions of all these various projects applied to
the kernel headers and then sanitized. I want to use my hardware that
is supported by version X of the package's software but the headers only
have version M supported. Wireless extensions for example.
With everybody maintaining separate headers things get messy.
The question is how to make these headers. Nobody wants to say how and
when someone needs an answer, even a distro maintainer, the answer is a
flippant "don't use kernel headers / use your vendor". I haven't seen
otherwise and believe me, I would latch on to the answer because I'm
always having to tailer headers to make things work for a variety of
distributions.
David
Arjan van de Ven wrote:
>On Mon, 2003-05-19 at 19:59, David Ford wrote:
>
>
>
>>Someone please step up to the plate and explain how to convert kernel
>>headers into sanitized headers for /usr/include.
>>
>>
>
>It seems you totally miseed the entire point.
>It shouldn't be an automatic conversion. It should be a well thought
>subset cleaned from kernel private stuff.
>
>I maintain such a subset for my employer and it's free for all to use
>(it's GPL after all).
>
>
On Mon, May 19, 2003 at 02:05:38PM -0400, David Ford wrote:
> How about not passing the buck off to "the vendor" and helping "this
> vendor" make sanitized headers.
Wjhy? Someone already did the work so why should I help to duplicate
the effort. Something doesn't have to be bad just because it's from
redhat.
> Not everybody copies RH and changes the text files from "RH" to "my
> distribution name".
It would certainly help gentoo..
On Mon, May 19, 2003 at 02:23:40PM -0400, David Ford wrote:
> I didn't miss the point.
The rest of your mail suggests otherwise
> I don't use RH, and I'm not in a mood to switch to RH just because
> RH has an LK headers maintainer.
It;s not about using RH. At all. You obviously didn't read
my mail. First you say "nobody is doing the work" to which I respond
"I am, and you even don't have to use RH to benifit from it".
> The question is how to make these headers. Who decides what should and
> shouldn't be available to userland?
For the header set I maintain, I did. Based on the criterium "does
this describe a kernel<->userland ABI".
> AFAIK, you don't have a package that contains all the -current- headers
> for all the current versions of all these various projects applied to
> the kernel headers and then sanitized.
You really don't get it.
If a userspace app needs something REALLY special from headers, it
should bloody well come with that header.
> I want to use my hardware that
> is supported by version X of the package's software but the headers only
> have version M supported. Wireless extensions for example.
Ok to take your example: the wireless extension using apps
should include THEIR header for the extensions THEIR released userland
is for, unless they want to use the sanitized headers (which should have
latest). The kernel<->userspace ABI is stable, and compatible between
kernel versions.
> The question is how to make these headers. Nobody wants to say how and
> when someone needs an answer, even a distro maintainer, the answer is a
> flippant "don't use kernel headers / use your vendor". I haven't seen
> otherwise
I tried several times to tell you, you just don't want to hear the answer
it seems.
On Llu, 2003-05-19 at 19:05, David Ford wrote:
> How about not passing the buck off to "the vendor" and helping "this
> vendor" make sanitized headers.
Nothing stops gentoo contributing - or you
On Mon, 2003-05-19 at 20:34, Arjan van de Ven wrote:
> > I don't use RH, and I'm not in a mood to switch to RH just because
> > RH has an LK headers maintainer.
>
> It;s not about using RH. At all. You obviously didn't read
> my mail. First you say "nobody is doing the work" to which I respond
> "I am, and you even don't have to use RH to benifit from it".
>
Much appreciated for the work already done.
> > AFAIK, you don't have a package that contains all the -current- headers
> > for all the current versions of all these various projects applied to
> > the kernel headers and then sanitized.
>
> You really don't get it.
> If a userspace app needs something REALLY special from headers, it
> should bloody well come with that header.
>
> > I want to use my hardware that
> > is supported by version X of the package's software but the headers only
> > have version M supported. Wireless extensions for example.
>
> Ok to take your example: the wireless extension using apps
> should include THEIR header for the extensions THEIR released userland
> is for, unless they want to use the sanitized headers (which should have
> latest). The kernel<->userspace ABI is stable, and compatible between
> kernel versions.
>
Another question that comes up ... with the many API changes from 2.4
to 2.5 ... how many issues is there with new sys calls missing, etc
(sorry, only got home now.).
>
> > The question is how to make these headers. Nobody wants to say how and
> > when someone needs an answer, even a distro maintainer, the answer is a
> > flippant "don't use kernel headers / use your vendor". I haven't seen
> > otherwise
>
> I tried several times to tell you, you just don't want to hear the answer
> it seems.
I think on the one hand the question is also ... how far will a
developer of one distro go to help another. I cannot say that I
have had much success in the past to get a response from one of the
'big guys' to help me/us (the 'small guys') =)
Might it be a good thing to start an official project for this ?
Thanks,
--
Martin Schlemmer
On Monday, May 19, 2003 1:44 PM, Martin Schlemmer wrote:
> I think on the one hand the question is also ... how far
> will a developer of one distro go to help another. I
> cannot say that I have had much success in the past to
> get a response from one of the 'big guys' to help me/us
> (the 'small guys') =)
AFAIK, it doesn't matter if a distro helps another or not. As per Arjan van
de Ven's comment, I would think any code they release in terms of header
files based on original GPL source is itself GPL, and therefore
includable/usable/modifyable/redistributable by any distro.
Red Hat (or insert other large distro vendor here) might not want to
explicitly "help" their little competitors, but they have appeared to solve
this problem (according to other posts) and there's no reason you can't base
your own work off of that...
--eric
On Mon, 2003-05-19 at 21:54, Mudama, Eric wrote:
> On Monday, May 19, 2003 1:44 PM, Martin Schlemmer wrote:
> > I think on the one hand the question is also ... how far
> > will a developer of one distro go to help another. I
> > cannot say that I have had much success in the past to
> > get a response from one of the 'big guys' to help me/us
> > (the 'small guys') =)
>
> AFAIK, it doesn't matter if a distro helps another or not. As per Arjan van
> de Ven's comment, I would think any code they release in terms of header
> files based on original GPL source is itself GPL, and therefore
> includable/usable/modifyable/redistributable by any distro.
>
> Red Hat (or insert other large distro vendor here) might not want to
> explicitly "help" their little competitors, but they have appeared to solve
> this problem (according to other posts) and there's no reason you can't base
> your own work off of that...
>
Was actually more referring to accepting patches to fix issues, or even
just replying in general on the patch.
--
Martin Schlemmer
Gentoo Linux Developer, Desktop/System Team Developer
Cape Town, South Africa
Followup to: <[email protected]>
By author: William Lee Irwin III <[email protected]>
In newsgroup: linux.dev.kernel
>
> IIRC you're supposed to use some sort of sanitized copy, not the things
> directly. IMHO the current state of affairs sucks as there is no
> standard set of ABI headers, but grabbing them right out of the kernel
> is definitely not the way to go.
>
This "cure" sucks worse than the disease. Now you're putting it onto
everyone who maintains userspace to do the same repetitive task of
"sanitizing" this. Especially for things this trivial, this is a
ridiculous concept.
For 2.7, getting real exportable ABI headers is so bloody necessary
it's not even funny. However, for 2.5, breaking things randomly is
not the way to go.
-hpa
--
<[email protected]> at work, <[email protected]> in private!
"Unix gives you enough rope to shoot yourself in the foot."
Architectures needed: ia64 m68k mips64 ppc ppc64 s390 s390x sh v850 x86-64
Followup to: <[email protected]>
By author: Linus Torvalds <[email protected]>
In newsgroup: linux.dev.kernel
>
> A number of headers have historical baggage, mainly to support the
> old libc5 habits, and because removing the ifdef's is something that
> nobody has felt was worth the pain.
>
> I think the only header file that should be considered truly exported is
> something like "asm/posix_types.h". For the others, we'll add __KERNEL__
> protection on demand if the glibc guys can give good arguments that it
> helps them do the "copy-and-cleanup" phase.
>
Copy and cleanup isn't realistic either, though, because it doesn't
track ABI changes. ABI headers is the only realistic solution. We
can't realistically get real ABI headers for 2.5, so please don't just
break things randomly until then.
-hpa
--
<[email protected]> at work, <[email protected]> in private!
"Unix gives you enough rope to shoot yourself in the foot."
Architectures needed: ia64 m68k mips64 ppc ppc64 s390 s390x sh v850 x86-64
Followup to: <[email protected]>
By author: Martin Schlemmer <[email protected]>
In newsgroup: linux.dev.kernel
>
> I think on the one hand the question is also ... how far will a
> developer of one distro go to help another. I cannot say that I
> have had much success in the past to get a response from one of the
> 'big guys' to help me/us (the 'small guys') =3D)
>
> Might it be a good thing to start an official project for this ?
>
Yes. It's sometimes referenced as the "ABI header" project, and is
the only way out of this mess.
"Copy and cleanup" is not maintainable, so then you end up having to
make changes in multiple places every time. Not a good idea. Thus,
what we need is ABI headers, which can be included from both kernel
and userspace. This is different from the current situation, in which
userspace includes random kernel headers and hopes someone has put
#ifdef __KERNEL__ in all the right places. Not sustainable.
Constructing those ABI headers is going to take a lot of effort. For
example <linux/abi/types.h> shouldn't export dev_t, it should export
something like __kernel_dev64_t, and the structures that uses these
things should be adjusted accordingly. The user-space library might
include this from <sys/types.h> and have typedef __kernel_dev64_t
dev_t, but that's user-space policy.
-hpa
--
<[email protected]> at work, <[email protected]> in private!
"Unix gives you enough rope to shoot yourself in the foot."
Architectures needed: ia64 m68k mips64 ppc ppc64 s390 s390x sh v850 x86-64
On Mon, May 19, 2003 at 08:06:32PM +0200, Arjan van de Ven wrote:
>
> I maintain such a subset for my employer and it's free for all to use
> (it's GPL after all).
Do you have other tools, except good judment to help you in the process?
Some perl scripts or other goodies maybe.
Sam
On Mon, May 19, 2003 at 11:35:30PM +0200, Sam Ravnborg wrote:
> On Mon, May 19, 2003 at 08:06:32PM +0200, Arjan van de Ven wrote:
> >
> > I maintain such a subset for my employer and it's free for all to use
> > (it's GPL after all).
>
>
> Do you have other tools, except good judment to help you in the process?
> Some perl scripts or other goodies maybe.
there is unifdef floating around that I'm going to use from now on ;)
(I just recently found it somewhere, google for it)
"H. Peter Anvin" <[email protected]> writes:
> Followup to: <[email protected]>
> By author: Linus Torvalds <[email protected]>
> In newsgroup: linux.dev.kernel
> >
> > A number of headers have historical baggage, mainly to support the
> > old libc5 habits, and because removing the ifdef's is something that
> > nobody has felt was worth the pain.
> >
> > I think the only header file that should be considered truly exported is
> > something like "asm/posix_types.h". For the others, we'll add __KERNEL__
> > protection on demand if the glibc guys can give good arguments that it
> > helps them do the "copy-and-cleanup" phase.
> >
>
> Copy and cleanup isn't realistic either, though, because it doesn't
> track ABI changes.
ABI changes or ABI additions?
If the ABI is not fixed that is a bug. Admittedly some interfaces
in the development kernel are still under development and so have not
stabilized on an ABI but that is a different issue.
> ABI headers is the only realistic solution. We
> can't realistically get real ABI headers for 2.5, so please don't just
> break things randomly until then.
As the ABI remains fixed I remain unconvinced. Multiple implementations
against the same ABI should be possible. The real question which is the
more scalable way to do the code.
What I find truly puzzling is that after years glibc still needs
kernel headers at all.
Eric
Eric W. Biederman wrote:
>
> ABI changes or ABI additions?
>
> If the ABI is not fixed that is a bug. Admittedly some interfaces
> in the development kernel are still under development and so have not
> stabilized on an ABI but that is a different issue.
>
ABI fixes and ABI additions, as well as outright ABI changes (yes they
suck, but they happen.)
>
>>ABI headers is the only realistic solution. We
>>can't realistically get real ABI headers for 2.5, so please don't just
>>break things randomly until then.
>
> As the ABI remains fixed I remain unconvinced. Multiple implementations
> against the same ABI should be possible. The real question which is the
> more scalable way to do the code.
The ABI doesn't remain fixed. Like everything else it evolves.
> What I find truly puzzling is that after years glibc still needs
> kernel headers at all.
What I find truly puzzling is that obviously intelligent people like
yourself still seem to think that ABIs remain fixed.
-hpa
Followup to: <[email protected]>
By author: William Lee Irwin III <[email protected]>
In newsgroup: linux.dev.kernel
>> IIRC you're supposed to use some sort of sanitized copy, not the things
>> directly. IMHO the current state of affairs sucks as there is no
>> standard set of ABI headers, but grabbing them right out of the kernel
>> is definitely not the way to go.
On Mon, May 19, 2003 at 02:14:00PM -0700, H. Peter Anvin wrote:
> This "cure" sucks worse than the disease. Now you're putting it onto
> everyone who maintains userspace to do the same repetitive task of
> "sanitizing" this. Especially for things this trivial, this is a
> ridiculous concept.
> For 2.7, getting real exportable ABI headers is so bloody necessary
> it's not even funny. However, for 2.5, breaking things randomly is
> not the way to go.
I would rather have real exportable ABI headers, yes. We don't have
them and AFAIK sanitized copies are the current policy.
-- wli
William Lee Irwin III wrote:
>
> On Mon, May 19, 2003 at 02:14:00PM -0700, H. Peter Anvin wrote:
>
>>This "cure" sucks worse than the disease. Now you're putting it onto
>>everyone who maintains userspace to do the same repetitive task of
>>"sanitizing" this. Especially for things this trivial, this is a
>>ridiculous concept.
>>For 2.7, getting real exportable ABI headers is so bloody necessary
>>it's not even funny. However, for 2.5, breaking things randomly is
>>not the way to go.
>
> I would rather have real exportable ABI headers, yes. We don't have
> them and AFAIK sanitized copies are the current policy.
>
Unfortunately "the current policy" is unrealistic, and repeating it
doesn't make it any less so.
-hpa
On Mon, May 19, 2003 at 02:14:00PM -0700, H. Peter Anvin wrote:
>>> This "cure" sucks worse than the disease. Now you're putting it onto
>>> everyone who maintains userspace to do the same repetitive task of
>>> "sanitizing" this. Especially for things this trivial, this is a
>>> ridiculous concept.
>>> For 2.7, getting real exportable ABI headers is so bloody necessary
>>> it's not even funny. However, for 2.5, breaking things randomly is
>>> not the way to go.
William Lee Irwin III wrote:
>> I would rather have real exportable ABI headers, yes. We don't have
>> them and AFAIK sanitized copies are the current policy.
On Mon, May 19, 2003 at 03:46:01PM -0700, H. Peter Anvin wrote:
> Unfortunately "the current policy" is unrealistic, and repeating it
> doesn't make it any less so.
No contest there; unfortunately unrealistic amounts of work seem to
be required to get around the general state of affairs at times. =(
Does it really have to be 2.7? It seems most of this would be header
reorganization with no runtime impact on the kernel.
-- wli
William Lee Irwin III wrote:
>
> On Mon, May 19, 2003 at 03:46:01PM -0700, H. Peter Anvin wrote:
>
>>Unfortunately "the current policy" is unrealistic, and repeating it
>>doesn't make it any less so.
>
> No contest there; unfortunately unrealistic amounts of work seem to
> be required to get around the general state of affairs at times. =(
> Does it really have to be 2.7? It seems most of this would be header
> reorganization with no runtime impact on the kernel.
>
It is; it doesn't have to be in 2.7 if someone would be willing to step
up and do it in time for 2.6. Personally I wouldn't have a prayer; I'm
already way to deep into too many projects...
-=hpa
"H. Peter Anvin" <[email protected]> writes:
> Eric W. Biederman wrote:
> >
> > ABI changes or ABI additions?
> >
> > If the ABI is not fixed that is a bug. Admittedly some interfaces
> > in the development kernel are still under development and so have not
> > stabilized on an ABI but that is a different issue.
> >
>
> ABI fixes and ABI additions, as well as outright ABI changes (yes they
> suck, but they happen.)
And ABI changes from the ABI exposed by a stable kernel are _B_U_G_S.
Yes _b_u_g_s happen but they are still _b_u_g_s.
It is legal to stop supporting an interface but changing the interface
in such a way that you must know the version of the kernel you are
running on is a _b_u_g.
> >>ABI headers is the only realistic solution. We
> >>can't realistically get real ABI headers for 2.5, so please don't just
> >>break things randomly until then.
> >
> > As the ABI remains fixed I remain unconvinced. Multiple implementations
> > against the same ABI should be possible. The real question which is the
> > more scalable way to do the code.
>
> The ABI doesn't remain fixed. Like everything else it evolves.
I meant to say the existing subset of the ABI, remains fixed.
Evolution by accretion is fine.
> > What I find truly puzzling is that after years glibc still needs
> > kernel headers at all.
>
> What I find truly puzzling is that obviously intelligent people like
> yourself still seem to think that ABIs remain fixed.
Simply because if the existing subset of an ABI does not remain fixed
it is a _b_u_g. Only once in a blue moon is there a chance for
an incompatible change, like when switching to x86-64.
And given my experiences running old a.out binaries Linux does a pretty good
job at this. And I am not willing to admit that changes that
break backwards compatibility (except for applications that depended
on implementation bugs) are anything except bugs. That would only
encourage changes that should not happen.
I do agree however that the ABI should be better documented. And that
being able to automatically generate headers from the official
documentation would be advantageous. With good documentation it
should actually be harder to change an ABI because what the old ABI
was will be clearer.
And I do agree that if the kernel builds with headers automatically
generated from the official documentation. Then it will be easy
to guarantee they are in sync.
But there is no reason not to write documentation today about what the
kernel interfaces are and convert glibc and the kernel later when
it is convenient to their development cycles.
What I do not is see the necessity of using automation to follow the
documentation.
Eric
On Llu, 2003-05-19 at 23:31, H. Peter Anvin wrote:
> What I find truly puzzling is that obviously intelligent people like
> yourself still seem to think that ABIs remain fixed.
Because with a few deep system stuff exceptions they do, although they
certainly extend. Rogue for 0.98.5 still runs on 2.4.21 (although you
may have fun finding libc2.2.2)
Eric W. Biederman wrote:
>>
>>ABI fixes and ABI additions, as well as outright ABI changes (yes they
>>suck, but they happen.)
>
> And ABI changes from the ABI exposed by a stable kernel are _B_U_G_S.
> Yes _b_u_g_s happen but they are still _b_u_g_s.
>
> It is legal to stop supporting an interface but changing the interface
> in such a way that you must know the version of the kernel you are
> running on is a _b_u_g.
>
Yes. And guess what? Bugs happen. Sometimes you can't fix them either
because the "new" usage has gotten established. However, that's not the
point.
Your message assumes that the ABI remains fixed. This is totally and
utterly and undeniably WRONG. There are rules for how it may evolve,
but it very much does evolve. No amount of handwaving or putting
underscores in weird places will change that.
>
> But there is no reason not to write documentation today about what the
> kernel interfaces are and convert glibc and the kernel later when
> it is convenient to their development cycles.
>
> What I do not is see the necessity of using automation to follow the
> documentation.
>
Otherwise you have three places to manually make your changes (usually
more) -- the documentation, the kernel, and glibc... and really you also
have klibc, uclibc, dietlibc, and God knows what else.
Automation is the way to maintain these together and in concert, to
avoid your "B_ U_ G_ S_." This isn't just a Good Thing, this is the
only sane possibility.
Does that mean C source code is the only possible format? It most
certainly *doesn't* -- in fact one could argue it's not even a very good
format -- as long as C source code is one of the possible productions.
-hpa
Alan Cox wrote:
> On Llu, 2003-05-19 at 23:31, H. Peter Anvin wrote:
>
>>What I find truly puzzling is that obviously intelligent people like
>>yourself still seem to think that ABIs remain fixed.
>
> Because with a few deep system stuff exceptions they do, although they
> certainly extend. Rogue for 0.98.5 still runs on 2.4.21 (although you
> may have fun finding libc2.2.2)
That's my point, though: the ABI *as a whole* does not remain fixed,
even though it evolves according to rules, one of which is try to
maintain backwards compatibility.
The "copy-and-modify" mantra doesn't take that into account, nevermind
the GPL issues.
-hpa
On Mon, May 19, 2003 at 11:20:16PM +0100, Alan Cox wrote:
> > seem to think that ABIs remain fixed.
>
> Because with a few deep system stuff exceptions they do, although they
> certainly extend. Rogue for 0.98.5 still runs on 2.4.21 (although you
> may have fun finding libc2.2.2)
ftp://ftp.win.tue.nl/pub/linux-local/libcbin/libc-2.2.2/ ?
"H. Peter Anvin" <[email protected]> writes:
> Yes. And guess what? Bugs happen. Sometimes you can't fix them either
> because the "new" usage has gotten established. However, that's not the
> point.
>
> Your message assumes that the ABI remains fixed. This is totally and
> utterly and undeniably WRONG.
It is correct by definition, the existing part of an ABI may not change.
> There are rules for how it may evolve,
> but it very much does evolve. No amount of handwaving or putting
> underscores in weird places will change that.
Sure you can add new functionality in a defined way. You can add to
the ABI. You may not modify an existing definition. But that has
no implications for existing working code.
> > But there is no reason not to write documentation today about what the
> > kernel interfaces are and convert glibc and the kernel later when
> > it is convenient to their development cycles.
> >
> > What I do not is see the necessity of using automation to follow the
> > documentation.
> >
>
> Otherwise you have three places to manually make your changes (usually
> more) -- the documentation, the kernel, and glibc... and really you also
> have klibc, uclibc, dietlibc, and God knows what else.
But since the older interface still remains you don't have to change
klibc, uclibc, dietlibc, and whatever else does not care. Making
it easy to change an existing ABI is simply wrong.
If you were to increment the syscall numbers by one no amount of automation
in the world would make that a kernel that would be usable on a production
box. The only way it could even work is if you were to declare that
kernel uses a new ABI.
> Automation is the way to maintain these together and in concert, to
> avoid your "B_ U_ G_ S_." This isn't just a Good Thing, this is the
> only sane possibility.
If things must be maintained in concert it is a bug.
With a fixed ABI people take advantage of new features as they
care for them. And in general to use new features requires new code.
If people are adding and changing ioctls/sysctls/prctls left and right,
and that is what is causing the maintenance problem, then that is the
problem. And that is where the problem needs to be reigned in at.
> Does that mean C source code is the only possible format? It most
> certainly *doesn't* -- in fact one could argue it's not even a very good
> format -- as long as C source code is one of the possible productions.
Agreed.
Calling it documentation simply makes it clear what the headers are,
and suggest that the machine readable for does not need to be C source
code.
Eric
On Mon, 19 May 2003 17:55:06 MDT, Eric W. Biederman said:
> If things must be maintained in concert it is a bug.
>
> With a fixed ABI people take advantage of new features as they
> care for them. And in general to use new features requires new code.
And if the kernel headers aren't maintained in concert with the kernel,
new userspace code can't reach the new features.
Therefor, by your definition, the current situation is a bug.
Try compiling code that uses futexes on a system that has a kernel that
supports them, but kernel-headers that haven't been upgraded to mention them.
The kernel has the new code, the userspace has the new code, but gcc will
turn around and whinge about the new code because there's a piece missing in
between. So people *CANT* take advantage of the new features (unless they
do something silly like drag their own foo.h file around where it can get
out of sync with reality).
Eric W. Biederman wrote:
> "H. Peter Anvin" <[email protected]> writes:
>
>>Yes. And guess what? Bugs happen. Sometimes you can't fix them either
>>because the "new" usage has gotten established. However, that's not the
>>point.
>>
>>Your message assumes that the ABI remains fixed. This is totally and
>>utterly and undeniably WRONG.
>
> It is correct by definition, the existing part of an ABI may not change.
>
"THE ABI" does not consitute "THE EXISTING PART OF AN ABI". They are
two different things.
>>>But there is no reason not to write documentation today about what the
>>>kernel interfaces are and convert glibc and the kernel later when
>>>it is convenient to their development cycles.
>>>
>>>What I do not is see the necessity of using automation to follow the
>>>documentation.
>>>
>>Otherwise you have three places to manually make your changes (usually
>>more) -- the documentation, the kernel, and glibc... and really you also
>>have klibc, uclibc, dietlibc, and God knows what else.
>
> But since the older interface still remains you don't have to change
> klibc, uclibc, dietlibc, and whatever else does not care. Making
> it easy to change an existing ABI is simply wrong.
No, it's not. It provides a chance to fix bugs quickly and easily.
Furthermore, it provides a clean, uniform way to distribute ABI
extensions.
> If you were to increment the syscall numbers by one no amount of automation
> in the world would make that a kernel that would be usable on a production
> box. The only way it could even work is if you were to declare that
> kernel uses a new ABI.
Syscall numbers are an extremely bad example (ioctl structures is more
like it), but since you brought it up: how would you deal with the
hypotetical case that two syscalls were assigned the same number?
You're screwed no matter what, and making it hard to fix doesn't improve
the situation in any way.
>>Automation is the way to maintain these together and in concert, to
>>avoid your "B_ U_ G_ S_." This isn't just a Good Thing, this is the
>>only sane possibility.
>
> If things must be maintained in concert it is a bug.
Things *DO* have to be maintained in concert if you want to get things
done. I'm not talking atomic changes as I'm talking getting changes in
within a reasonable time frame.
> With a fixed ABI people take advantage of new features as they
> care for them. And in general to use new features requires new code.
>
> If people are adding and changing ioctls/sysctls/prctls left and right,
> and that is what is causing the maintenance problem, then that is the
> problem. And that is where the problem needs to be reigned in at.
And your proposal for this is...?
Let's face it, you're going to need these kinds of things. You're also
going to need a way to push these features into use. You're also going
to want to have an automated way to make things like strace (my personal
favourite debugging tool) produce useful output.
>>Does that mean C source code is the only possible format? It most
>>certainly *doesn't* -- in fact one could argue it's not even a very good
>>format -- as long as C source code is one of the possible productions.
>
> Agreed.
>
> Calling it documentation simply makes it clear what the headers are,
> and suggest that the machine readable for does not need to be C source
> code.
What it really is is an ABI specification, not documentation. The
difference is that the specification *is* the ABI, by definition,
whereas documentation is a *description* of what the ABI is. The latter
may or may not be up to date and correct.
Automation prevents bugs. This is a good thing.
-hpa
[email protected] writes:
> On Mon, 19 May 2003 17:55:06 MDT, Eric W. Biederman said:
> > If things must be maintained in concert it is a bug.
> >
> > With a fixed ABI people take advantage of new features as they
> > care for them. And in general to use new features requires new code.
>
> And if the kernel headers aren't maintained in concert with the kernel,
> new userspace code can't reach the new features.
>
> Therefor, by your definition, the current situation is a bug.
Yes, glibc uses kernel headers.
> Try compiling code that uses futexes on a system that has a kernel that
> supports them, but kernel-headers that haven't been upgraded to mention them.
> The kernel has the new code, the userspace has the new code, but gcc will
> turn around and whinge about the new code because there's a piece missing in
> between. So people *CANT* take advantage of the new features (unless they
> do something silly like drag their own foo.h file around where it can get
> out of sync with reality).
Or the build against a library that does that. There are not that
many libraries.
For a lot of system calls it is actively dangerous to assume dev_t ==
__kernel_dev_t. As glibc does some cute things in there.
Eric
On Tue, 2003-05-20 at 00:17, H. Peter Anvin wrote:
> Your message assumes that the ABI remains fixed. This is totally and
> utterly and undeniably WRONG. There are rules for how it may evolve,
> but it very much does evolve. No amount of handwaving or putting
> underscores in weird places will change that.
To a large extent, however, it merely grows. And in a lot of cases when
it grows due to new syscalls, new interfaces, etc., you have to add
matching code to glibc to use them _anyway_, so it's no problem for
glibc's version of the headers to lag behind until the appropriate
support is added.
You are, however, correct that the correct fix is to have completely
separate headers which define the ABI. Then the real kernel headers in
include/linux and include/asm can include them, and C libraries can also
use them without contamination.
This requires that someone sit down and cut'n'paste large amounts of
structures and definitions from include/linux/*.h into the new header
files. I've been tempted to do that on occasion but what's held me up
has been the fact that there isn't yet a consensus on how it should be
laid out.
For compatibility with older libc, one approach would be to add a new
directory to the include path which matches the existing layout
(linux/usrinclude/linux, linux/usrinclude/asm-*), and use #include_next
from the actual kernel headers to pull in those files.
Alternatively, we could go further and take the opportunity to rearrange
stuff further; I'm not sure what we really gain from that though other
than extra pain.
If Linus would approve a strategy for rearranging the headers such that
people can work on it without suspecting that they're just wasting their
time, I think it could get done for 2.6.
It's not the kind of thing you do in private and present as a fait
accomplis -- if it isn't quite right, you end up having to do the whole
thing from scratch, afaict.
--
dwmw2
? but admittedly not all.
David Woodhouse wrote:
>
> To a large extent, however, it merely grows. And in a lot of cases when
> it grows due to new syscalls, new interfaces, etc., you have to add
> matching code to glibc to use them _anyway_, so it's no problem for
> glibc's version of the headers to lag behind until the appropriate
> support is added.
>
... unless you need the new feature, may it be an ioctl to support your
device driver, or whatnot.
Most ABI changes do not require
> You are, however, correct that the correct fix is to have completely
> separate headers which define the ABI. Then the real kernel headers in
> include/linux and include/asm can include them, and C libraries can also
> use them without contamination.
>
> This requires that someone sit down and cut'n'paste large amounts of
> structures and definitions from include/linux/*.h into the new header
> files. I've been tempted to do that on occasion but what's held me up
> has been the fact that there isn't yet a consensus on how it should be
> laid out.
I maintain the proposal I have given before:
<linux/abi/*.h> as the header file namespace;
<linux/*.h> <asm/*.h> namespaces reserved for compatibility (once the
migration is complete these are owned by the libc)
Types use the __kernel_* namespace *only*; structures use struct __kernel_*.
Some form of export of the expected syscall ABI as well as syscall
numbering.
> For compatibility with older libc, one approach would be to add a new
> directory to the include path which matches the existing layout
> (linux/usrinclude/linux, linux/usrinclude/asm-*), and use #include_next
> from the actual kernel headers to pull in those files.
>
> Alternatively, we could go further and take the opportunity to rearrange
> stuff further; I'm not sure what we really gain from that though other
> than extra pain.
I don't think the <linux/*.h> namespace as its currently laid out is
very functional for exporting ABIs, so I'd like to
A bigger issue is if this really should be done in C. A worthwhile
thought: if this is done correctly then most or all of the 64/32 compat
code (or any other arch1-on-arch2 compatibility) should be able to be
automatically generated. If not, it almost certainly isn't done
correctly...
> If Linus would approve a strategy for rearranging the headers such that
> people can work on it without suspecting that they're just wasting their
> time, I think it could get done for 2.6.
>
> It's not the kind of thing you do in private and present as a fait
> accomplis -- if it isn't quite right, you end up having to do the whole
> thing from scratch, afaict.
Agreed.
-hpa
On Mon, 19 May 2003 18:40:18 MDT, Eric W. Biederman said:
> Or the build against a library that does that. There are not that
> many libraries.
So I get around the lack of foo-gram support in glibc by
writing/compiling/installing a -lfoo-gram, which of course gets its
header definitions from the same header file that kernheaders isn't
supplying, or by dragging around the same local foo-gram.h that I'd
otherwise be dragging around with my app.
Somehow, this doesn't seem like progress to me. And of course, it just
sets me up for *odd* debugging problems when my foo-gram.h is at version
0.17 and the kernel and glibc are up to 0.25, and features mysteriously
misbehave depending on the exact -I include order on the system I'm
doing the build on, and whether the particular version of gcc has the
automagic auto-ambush handling of /usr/local/include, and the phase of
the moon...
Yes, I know I can build against a library. Point is that saying I can
do it doesn't change the fact it's a hack and an ongoing maintenance problem.
> For a lot of system calls it is actively dangerous to assume dev_t ==
> __kernel_dev_t. As glibc does some cute things in there.
I thought that sort of fun and games was *WHY* userspace can't use the
kernel headers in the first place?
"H. Peter Anvin" <[email protected]> writes:
> Types use the __kernel_* namespace *only*; structures use struct __kernel_*.
What, do you mean _every_ kernel internal type would use such names?
That's completely horrid...
-Miles
--
Come now, if we were really planning to harm you, would we be waiting here,
beside the path, in the very darkest part of the forest?
Or for anyone to explain or assist in the matter.
Alan Cox wrote:
>On Llu, 2003-05-19 at 19:05, David Ford wrote:
>
>
>>How about not passing the buck off to "the vendor" and helping "this
>>vendor" make sanitized headers.
>>
>>
>
>Nothing stops gentoo contributing - or you
>
Miles Bader wrote:
> "H. Peter Anvin" <[email protected]> writes:
>
>>Types use the __kernel_* namespace *only*; structures use struct __kernel_*.
>
>
> What, do you mean _every_ kernel internal type would use such names?
> That's completely horrid...
>
No, I don't mean that.
The kernel-internal headers would typedef these to different names, e.g.
/* linux/types.h */
#include <linux/abi/types.h>
/* Kernel internal types */
typedef __kernel_dev64_t dev_t;
typedef __kernel_ino_t ino_t;
... etc ...
-hpa
"H. Peter Anvin" <[email protected]> writes:
> The kernel-internal headers would typedef these to different names, e.g.
>
> /* linux/types.h */
> #include <linux/abi/types.h>
> /* Kernel internal types */
> typedef __kernel_dev64_t dev_t;
> typedef __kernel_ino_t ino_t;
I see... I guess that seems reasonable.
-Miles
--
`Cars give people wonderful freedom and increase their opportunities.
But they also destroy the environment, to an extent so drastic that
they kill all social life' (from _A Pattern Language_)
> On Mon, May 19, 2003 at 03:46:01PM -0700, H. Peter Anvin wrote:
> > Unfortunately "the current policy" is unrealistic, and repeating it
> > doesn't make it any less so.
>
> No contest there; unfortunately unrealistic amounts of work seem to
> be required to get around the general state of affairs at times. =(
> Does it really have to be 2.7? It seems most of this would be header
> reorganization with no runtime impact on the kernel.
>
Well, this is a tender issue - both for users and non kernel developers
working on a project that may (according to the author at least *g*)
need kernel headers.
Getting something done sooner than later would thus be appreciated from
many venues, and will also get rid of your head aces =)
Trying to get back to my point ... I am sure there are many people
that will help getting this done, if only somebody with enough knowledge
of how the kernel devs want the ABI headers to look like will take some
kind of lead in the effort - i2c and sensor support in 2.5 is a good
example.
Regards,
--
Martin Schlemmer
"H. Peter Anvin" <[email protected]> writes:
> David Woodhouse wrote:
> >
> > To a large extent, however, it merely grows. And in a lot of cases when
> > it grows due to new syscalls, new interfaces, etc., you have to add
> > matching code to glibc to use them _anyway_, so it's no problem for
> > glibc's version of the headers to lag behind until the appropriate
> > support is added.
> >
>
> ... unless you need the new feature, may it be an ioctl to support your
> device driver, or whatnot.
>
> Most ABI changes do not require
Ugh. I now see both sides of this.
ABI changes always require writing new code to take advantage
of it but that code does not need to live in libc. And libc is expected
to have the definitions necessary to compile the new code.
> > You are, however, correct that the correct fix is to have completely
> > separate headers which define the ABI. Then the real kernel headers in
> > include/linux and include/asm can include them, and C libraries can also
> > use them without contamination.
> >
> > This requires that someone sit down and cut'n'paste large amounts of
> > structures and definitions from include/linux/*.h into the new header
> > files. I've been tempted to do that on occasion but what's held me up
> > has been the fact that there isn't yet a consensus on how it should be
> > laid out.
>
> I maintain the proposal I have given before:
>
> <linux/abi/*.h> as the header file namespace;
What about <linux-abi/*.h>
> <linux/*.h> <asm/*.h> namespaces reserved for compatibility (once the
> migration is complete these are owned by the libc)
I think removing the abi namespace totally from the legacy directories
helps this to a small extent.
> Types use the __kernel_* namespace *only*; structures use struct __kernel_*.
>
> Some form of export of the expected syscall ABI as well as syscall
> numbering.
Prototypes for everything including ioctls.
> > For compatibility with older libc, one approach would be to add a new
> > directory to the include path which matches the existing layout
> > (linux/usrinclude/linux, linux/usrinclude/asm-*), and use #include_next
> > from the actual kernel headers to pull in those files.
> >
> > Alternatively, we could go further and take the opportunity to rearrange
> > stuff further; I'm not sure what we really gain from that though other
> > than extra pain.
>
> I don't think the <linux/*.h> namespace as its currently laid out is
> very functional for exporting ABIs, so I'd like to
>
> A bigger issue is if this really should be done in C. A worthwhile
> thought: if this is done correctly then most or all of the 64/32 compat
> code (or any other arch1-on-arch2 compatibility) should be able to be
> automatically generated. If not, it almost certainly isn't done
> correctly...
Potentially. Certain things like type conversions may be a non-trivial
exercise. It is certainly true that it should be possible to build
the entire decoding logic for strace. Even in the mixed architecture
cases.
For cases that are hard to automate see: sys_old_mmap vs. sys_mmap2,
on x86. Potentially it can be done but it requires a powerful stub
generator.
> > If Linus would approve a strategy for rearranging the headers such that
> > people can work on it without suspecting that they're just wasting their
> > time, I think it could get done for 2.6.
> >
> > It's not the kind of thing you do in private and present as a fait
> > accomplis -- if it isn't quite right, you end up having to do the whole
> > thing from scratch, afaict.
>
> Agreed.
If it is done purely as headers certainly. For a header generator it
might be something you can get away with. Because the core repository
would not need to change just the world useable form. This may
be the one good argument for doing this in something other than C.
Eric
Eric W. Biederman wrote:
>
> Ugh. I now see both sides of this.
>
> ABI changes always require writing new code to take advantage
> of it but that code does not need to live in libc. And libc is expected
> to have the definitions necessary to compile the new code.
>
Bingo.
>>
>>I maintain the proposal I have given before:
>>
>><linux/abi/*.h> as the header file namespace;
>
> What about <linux-abi/*.h>
>
>><linux/*.h> <asm/*.h> namespaces reserved for compatibility (once the
>>migration is complete these are owned by the libc)
>
> I think removing the abi namespace totally from the legacy directories
> helps this to a small extent.
>
It might, but the <linux/abi/*.h> namespace hasn't been used, so it
isn't really legacy in the sense that you can have a possible conflict.
Having said that, <linuxabi/*.h> is the least bleacherous nonlegacy
alternative I've seen (I'd like to skip the dash, though.) <abi/*.h> is
already taken.
>
>>Types use the __kernel_* namespace *only*; structures use struct __kernel_*.
>>
>>Some form of export of the expected syscall ABI as well as syscall
>>numbering.
>
>
> Prototypes for everything including ioctls.
>
Absolutely. As I said in another post, if this is done correctly, a
whole bunch of the ABI compatbility stuff like the 32-on-64 things
should be possible to generate automatically. If that isn't true, it's
done wrong.
>>
>>A bigger issue is if this really should be done in C. A worthwhile
>>thought: if this is done correctly then most or all of the 64/32 compat
>>code (or any other arch1-on-arch2 compatibility) should be able to be
>>automatically generated. If not, it almost certainly isn't done
>>correctly...
>
>
> Potentially. Certain things like type conversions may be a non-trivial
> exercise. It is certainly true that it should be possible to build
> the entire decoding logic for strace. Even in the mixed architecture
> cases.
>
> For cases that are hard to automate see: sys_old_mmap vs. sys_mmap2,
> on x86. Potentially it can be done but it requires a powerful stub
> generator.
>
That one is somewhat tricky, and probably needs some ad hoc code.
However, you get a long way toward the goal if you can express the
following:
void * sys_mmap(void *, size_t, int, struct { int, int, off32_t });
void * sys_mmap2(void *, size_t, int, int, int, off32_t);
void * sys_mmap64(void *, size_t, int, int, int, struct { off64_t });
(The latter doesn't exist at the moment AFAIK, but at some point it'll
be necessary.)
You can get even further if you can express the fact that the final
structure in sys_old_mmap are "memory arguments" -- ordinary arguments
spilled to memory because we ran out of registers ("prematurely" in the
case of sys_mmap, due to backwards compatibility constraints.)
>
>>>If Linus would approve a strategy for rearranging the headers such that
>>>people can work on it without suspecting that they're just wasting their
>>>time, I think it could get done for 2.6.
>>>
>>>It's not the kind of thing you do in private and present as a fait
>>>accomplis -- if it isn't quite right, you end up having to do the whole
>>>thing from scratch, afaict.
>>
>>Agreed.
>
> If it is done purely as headers certainly. For a header generator it
> might be something you can get away with. Because the core repository
> would not need to change just the world useable form. This may
> be the one good argument for doing this in something other than C.
>
On the other hand, Linus just released "sparse", which is a pretty nice
C parser that might be useful for this, especially with some kind of
custom annotation capability. I talked to him about this a while ago,
actually.
-hpa
Hi David.
>> Your message assumes that the ABI remains fixed. This is
>> totally and utterly and undeniably WRONG. There are rules
>> for how it may evolve, but it very much does evolve. No
>> amount of hand-waving or putting underscores in weird
>> places will change that.
> To a large extent, however, it merely grows. And in a lot
> of cases when it grows due to new syscalls, new interfaces,
> etc., you have to add matching code to glibc to use them
> _anyway_, so it's no problem for glibc's version of the
> headers to lag behind until the appropriate support is
> added.
>
> You are, however, correct that the correct fix is to have
> completely separate headers which define the ABI. Then
> the real kernel headers in include/linux and include/asm
> can include them, and C libraries can also use them without
> contamination.
First, is there anything in include/asm that is actually needed
outside the kernel itself? There shouldn't be, and if there is,
it needs to be moved to a header under include/linux that is
included in the relevant include/asm file.
Assuming that has been done, the first step in making this
change would be to create the directory all this is to go into.
Any suggestions ???
My suggestion would be to move the current headers to a new
directory and put the public headers in their place. This
would produce the following include directory layout:
* include/linux/ The "sanitised" header files that are
safe for use by external software.
* include/kernel/ Versions of those header files that are
used by the kernel, each including the
relevant sanitised header files.
* include/asm/ Header files used only by the kernel,
all including the relevant sanitised
header files.
The basic rule would be that external software can make free
use of headers under include/linux but should never make any
use whatsoever of headers under include/kernel or include/asm
for any reason.
If anything external to the kernel needs to refer to something
only defined under include/kernel/ then they should report that
here for consideration as something needing a sanitised version
and we could thus ensure that any oversights were dealt with.
> This requires that someone sit down and cut'n'paste large
> amounts of structures and definitions from include/linux/*.h
> into the new header files. I've been tempted to do that on
> occasion but what's held me up has been the fact that there
> isn't yet a consensus on how it should be laid out.
Personally, I'm of the opinion that this will only be resolved
when somebody sits down and just does it as they see fit.
> For compatibility with older libc, one approach would be to
> add a new directory to the include path which matches the
> existing layout (linux/usrinclude/linux, linux/usrinclude/asm-*),
> and use #include_next from the actual kernel headers to pull in
> those files.
> Alternatively, we could go further and take the opportunity
> to rearrange stuff further; I'm not sure what we really gain
> from that though other than extra pain.
I personally can't see the point of that.
> If Linus would approve a strategy for rearranging the headers
> such that people can work on it without suspecting that they're
> just wasting their time, I think it could get done for 2.6.
>
> It's not the kind of thing you do in private and present as a
> fait accompli -- if it isn't quite right, you end up having
> to do the whole thing from scratch, afaict.
The only way I can see this ever being done is one header at a time.
I'm willing to have a go at doing this along the lines of the above
scheme, and the first patch would be one to create the directory
with a set of empty files therein.
Best wishes from Riley.
---
* Nothing as pretty as a smile, nothing as ugly as a frown.
---
Outgoing mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.481 / Virus Database: 277 - Release Date: 13-May-2003
On Tue, May 20, 2003 at 01:28:56AM +0200, Andries Brouwer wrote:
> On Mon, May 19, 2003 at 11:20:16PM +0100, Alan Cox wrote:
>> (although you may have fun finding libc2.2.2)
> ftp://ftp.win.tue.nl/pub/linux-local/libcbin/libc-2.2.2/ ?
Well, no. This URL doesn't exist. I guess you meant
ftp://ftp.win.tue.nl/pub/linux-local/libc.archive/libcbin/libc-2.2.2/
Best regards,
--
Lionel Elie Mamane
Riley Williams wrote:
> First, is there anything in include/asm that is actually needed
> outside the kernel itself? There shouldn't be, and if there is,
> it needs to be moved to a header under include/linux that is
> included in the relevant include/asm file.
It's handy for stuff like getting high res timestamps without having to ifdef
the case of building on different architechtures. Also, there are files under
include/linux which end up including asm stuff in turn.
> The basic rule would be that external software can make free
> use of headers under include/linux but should never make any
> use whatsoever of headers under include/kernel or include/asm
> for any reason.
What if the include/linux files themselves make use of the asm files?
Chris
--
Chris Friesen | MailStop: 043/33/F10
Nortel Networks | work: (613) 765-0557
3500 Carling Avenue | fax: (613) 765-2986
Nepean, ON K2H 8E9 Canada | email: [email protected]
Chris Friesen wrote:
> Riley Williams wrote:
>
>> First, is there anything in include/asm that is actually needed
>> outside the kernel itself? There shouldn't be, and if there is,
>> it needs to be moved to a header under include/linux that is
>> included in the relevant include/asm file.
>
> It's handy for stuff like getting high res timestamps without having to
> ifdef the case of building on different architechtures. Also, there are
> files under include/linux which end up including asm stuff in turn.
>
>> The basic rule would be that external software can make free
>> use of headers under include/linux but should never make any
>> use whatsoever of headers under include/kernel or include/asm
>> for any reason.
>
>
> What if the include/linux files themselves make use of the asm files?
>
No, not acceptable.
The thing is, trying to redefine the old namespaces is hopeless at this
point. Hence the proposed new namespace <linux/abi/*.h> ...
<linux/abi/arch/*.h> would be my preference for an arch-specific
subnamespace.
Thus the rule is:
a) <linux/abi/*> files MUST NOT include files outside <linux/abi/*>
b) <linux/*.h> and <asm/*.h> are legacy namespaces. They should be
considered to be completely different in kernel and userspace -- in
effect, glibc will eventually ship with its own set of these headers.
c) <linux/abi/*> files should be clean for inclusion from either kernel
or userspace.
-hpa
"H. Peter Anvin" <[email protected]> writes:
> Eric W. Biederman wrote:
> > Ugh. I now see both sides of this.
> > ABI changes always require writing new code to take advantage
> > of it but that code does not need to live in libc. And libc is expected
> > to have the definitions necessary to compile the new code.
> >
>
> Bingo.
Several more little more points.
In general this only affects applications that are tied tightly to
the kernel.
What the applications want is a kernel abi specification repository,
which glibc is not, at least in a timely fashion, and it is not the
emphasis of any libc so it is quite likely to be neglected.
So for applications today that cannot wait for the code to be
incorporated into glibc the only reasonable advice is to include
their own header files (that do not conflict with libc header files),
for the ioctls and definitions they need. This is a maintainable if
not beautiful strategy.
> >>I maintain the proposal I have given before:
> >>
> >><linux/abi/*.h> as the header file namespace;
> > What about <linux-abi/*.h>
> >
> >><linux/*.h> <asm/*.h> namespaces reserved for compatibility (once the
> >>migration is complete these are owned by the libc)
> > I think removing the abi namespace totally from the legacy directories
> > helps this to a small extent.
> >
>
> It might, but the <linux/abi/*.h> namespace hasn't been used, so it isn't really
>
> legacy in the sense that you can have a possible conflict. Having said that,
> <linuxabi/*.h> is the least bleacherous nonlegacy alternative I've seen (I'd
> like to skip the dash, though.) <abi/*.h> is already taken.
No problem with skipping the dash. In general I would like to be able
to: ``rm -rf /usr/include/linux/ /usr/include/asm/'' and then compile
applications. This is not something I would recommend in a
production system any time soon but it would help in cruft detection.
> >>Types use the __kernel_* namespace *only*; structures use struct __kernel_*.
> >>
> >>Some form of export of the expected syscall ABI as well as syscall
> >>numbering.
> > Prototypes for everything including ioctls.
> >
>
> Absolutely. As I said in another post, if this is done correctly, a whole bunch
> of the ABI compatbility stuff like the 32-on-64 things should be possible to
> generate automatically. If that isn't true, it's done wrong.
>
> >>
> >>A bigger issue is if this really should be done in C. A worthwhile
> >>thought: if this is done correctly then most or all of the 64/32 compat
> >>code (or any other arch1-on-arch2 compatibility) should be able to be
> >>automatically generated. If not, it almost certainly isn't done
> >>correctly...
> > Potentially. Certain things like type conversions may be a non-trivial
> > exercise. It is certainly true that it should be possible to build
> > the entire decoding logic for strace. Even in the mixed architecture
> > cases.
> > For cases that are hard to automate see: sys_old_mmap vs. sys_mmap2,
> > on x86. Potentially it can be done but it requires a powerful stub
> > generator.
> >
>
> That one is somewhat tricky, and probably needs some ad hoc code. However, you
> get a long way toward the goal if you can express the following:
My basic point with respect to arch-on-arch compatibility is that
sys_mmap -> sys_mmap2 (where they both take an off32_t argument) is
classic case of the arch on arch compatibility.
The case is actually more interesting because sys_mmap2 uses
a 4K aligned offset. Instead of a byte aligned offset. Which
makes sys_mmap64 unnecessary for the time being.
But I am certainly for it where the work can be done easily.
A more interesting possibility to me at least is network transparency.
> >>>If Linus would approve a strategy for rearranging the headers such that
> >>>people can work on it without suspecting that they're just wasting their
> >>>time, I think it could get done for 2.6.
> >>>
> >>>It's not the kind of thing you do in private and present as a fait
> >>>accomplis -- if it isn't quite right, you end up having to do the whole
> >>>thing from scratch, afaict.
> >>
> >>Agreed.
> > If it is done purely as headers certainly. For a header generator it
> > might be something you can get away with. Because the core repository
> > would not need to change just the world useable form. This may
> > be the one good argument for doing this in something other than C.
> >
>
> On the other hand, Linus just released "sparse", which is a pretty nice C parser
> that might be useful for this, especially with some kind of custom annotation
> capability. I talked to him about this a while ago, actually.
Against doing this in C there are 3 arguments.
1) C may not be precise enough.
2) C may not contain enough information.
3) Reorganization by an automated tool may not be possible.
I don't know for certain that the ability to reorganize the generated
interfaces from original source is important. A good design may
make this a non-issue. But it may help prototyping so I mention it.
I suspect a good implementation of this would resemble to the CORBA
IDL. A relative of C but with different restrictions.
Another very quite piece of the puzzle is how to include in the in
kernel DSO.
Eric
Followup to: <[email protected]>
By author: [email protected]
In newsgroup: linux.dev.kernel
>
> > For a lot of system calls it is actively dangerous to assume dev_t ==
> > __kernel_dev_t. As glibc does some cute things in there.
>
> I thought that sort of fun and games was *WHY* userspace can't use the
> kernel headers in the first place?
>
Indeed. Because they try to export dev_t, not __kernel_dev_t (unless
you know exactly what you're doing, which most people don't.)
dev_t should be defined by the library ABI, not by the kernel ABI.
-hpa
--
<[email protected]> at work, <[email protected]> in private!
"Unix gives you enough rope to shoot yourself in the foot."
Architectures needed: ia64 m68k mips64 ppc ppc64 s390 s390x sh v850 x86-64
Martin Schlemmer wrote:
>
> The only issue that we might have, is that <linux/abi/*> will once
> again break many things. Sure, if we have to fix them once to get
> this fixed for good, why not.
>
> On the other hand, why not leave it at <linux/*.h> and <asm/*.h>
> as the location of the ABI, and then move all kernel only
> related stuff to <kernel/*.h> (or whatever, just the concept which
> count ...) which can then include whatever it needs form the other
> places (linux/asm)?
>
Because then there will be a push to be backward compatible, and the
current structure is hideously poor for this purpose.
-hpa
"H. Peter Anvin" <[email protected]> writes:
> Martin Schlemmer wrote:
> > The only issue that we might have, is that <linux/abi/*> will once
> > again break many things. Sure, if we have to fix them once to get
> > this fixed for good, why not.
> > On the other hand, why not leave it at <linux/*.h> and <asm/*.h>
> > as the location of the ABI, and then move all kernel only
> > related stuff to <kernel/*.h> (or whatever, just the concept which
> > count ...) which can then include whatever it needs form the other
> > places (linux/asm)?
> >
>
> Because then there will be a push to be backward compatible, and the current
> structure is hideously poor for this purpose.
In fact all but the most kernel dependent programs should still
not be using these headers directly.
Given that glibc plays interesting tricks with dev_t and friends.
Only an application that knows what it is doing should bypass
glibcs abstraction layer.
Mostly these headers will be there for glibc to include, and rexport
in sys/*
Eric
On Tue, 2003-05-20 at 19:06, H. Peter Anvin wrote:
> > What if the include/linux files themselves make use of the asm files?
> >
>
> No, not acceptable.
>
> The thing is, trying to redefine the old namespaces is hopeless at this
> point. Hence the proposed new namespace <linux/abi/*.h> ...
> <linux/abi/arch/*.h> would be my preference for an arch-specific
> subnamespace.
>
> Thus the rule is:
>
> a) <linux/abi/*> files MUST NOT include files outside <linux/abi/*>
>
> b) <linux/*.h> and <asm/*.h> are legacy namespaces. They should be
> considered to be completely different in kernel and userspace -- in
> effect, glibc will eventually ship with its own set of these headers.
>
> c) <linux/abi/*> files should be clean for inclusion from either kernel
> or userspace.
>
The only issue that we might have, is that <linux/abi/*> will once
again break many things. Sure, if we have to fix them once to get
this fixed for good, why not.
On the other hand, why not leave it at <linux/*.h> and <asm/*.h>
as the location of the ABI, and then move all kernel only
related stuff to <kernel/*.h> (or whatever, just the concept which
count ...) which can then include whatever it needs form the other
places (linux/asm)?
Regards,
--
Martin Schlemmer