We suddenly started seeing freezing problems on a number of machines
in the past couple of days. There is no pattern as far as I can tell.
It has happened while running OpenGL apps, netscape, or even when not
doing anything. The machine will usually just hang and while it's
still pingable, it's totally unresponsive and you can't remotely log in.
The 2.4.3 kernel usually hangs forever; the machines with 2.4.9
kernels usually come back within 10-15 secs.
Every time this happens we see the following message in the system log:
Dec 6 21:29:00 stranger kernel: ldt allocation failed
The machines are:
Hardware:
- IBM Intellistation M Pro
- dual 2 GHz P4's
- 2 GB RAM
- NVIDIA Quadro DCC card
Software:
- Red Hat 7.1
- kernel 2.4.3smp or 2.4.9smp
- XFree 4.1
- Ximian Gnome 1.4
- NVIDIA drivers 1.0-1541
Once this problem occurs, even if the hang is temporary, the machine
is extremely flakey. Almost any app will start causing ldt allocation
failure messages in the system log. Only a reboot really helps.
Questions:
- Does this ring a bell to anybody? What is ldt anyway and what would
cause an allocation to fail?
- I still keep one of these messages in this state. Is there anything
I can probe to get useful debugging info?
Any help would greatly be appreciated.
Thanks,
KV
--
___________________________________________________________________
Studio Tools [email protected]
Pixar Animation Studios http://www.pixar.com/
> We suddenly started seeing freezing problems on a number of machines
> in the past couple of days. There is no pattern as far as I can tell.
> It has happened while running OpenGL apps, netscape, or even when not
> doing anything.
> Software:
> - NVIDIA drivers 1.0-1541
Sorry, we do not have the source to NVIDIA's driver, so we cannot help you
debug this problem. Please contact NVIDIA support.
Regards,
Dan
On Fri, 7 Dec 2001, Dan Maas wrote:
> > We suddenly started seeing freezing problems on a number of machines
> > in the past couple of days. There is no pattern as far as I can tell.
> > It has happened while running OpenGL apps, netscape, or even when not
> > doing anything.
>
> > Software:
> > - NVIDIA drivers 1.0-1541
>
> Sorry, we do not have the source to NVIDIA's driver, so we cannot help you
> debug this problem. Please contact NVIDIA support.
I don't see how one can magically tell that this is an NVIDIA problem.
I am contacting NVIDIA at the same time and it's possible that the
problem is with their drivers. However, there is something going on
in the kernel and I imagine that even if the NVIDIA drivers are
triggering the problem, there are other modules/apps that can bring
about the same behavior.
KV
--
___________________________________________________________________
Studio Tools [email protected]
Pixar Animation Studios http://www.pixar.com/
> I don't see how one can magically tell that this is an NVIDIA problem.
We don't know. But since we don't have their source and they have our
source only they can tell you.
> in the kernel and I imagine that even if the NVIDIA drivers are
> triggering the problem, there are other modules/apps that can bring
> about the same behavior.
Possibly, but you'll have to ask Nvidia to debug it for you. If you can
reproduce a bug by
- removing the nvidia modules so they wont be loaded
- hard booting the machine
- triggering the bug without loading the nvidia drivers
then please report it. If not, its between you and nvidia.
On Fri, 7 Dec 2001, Alan Cox wrote:
> > I don't see how one can magically tell that this is an NVIDIA problem.
>
> We don't know. But since we don't have their source and they have our
> source only they can tell you.
>
> > in the kernel and I imagine that even if the NVIDIA drivers are
> > triggering the problem, there are other modules/apps that can bring
> > about the same behavior.
>
> Possibly, but you'll have to ask Nvidia to debug it for you. If you can
> reproduce a bug by
> - removing the nvidia modules so they wont be loaded
> - hard booting the machine
> - triggering the bug without loading the nvidia drivers
>
> then please report it. If not, its between you and nvidia.
Without turning this into a religous debate, is there any way to
find out what those messages really mean? When does one run into
an ldt allocation failure? What is ldt? I am not just trying to
solve this problem; I also want to understand what is going on
in the kernel.
Thanks for any explanations.
KV
--
___________________________________________________________________
Studio Tools [email protected]
Pixar Animation Studios http://www.pixar.com/
> Without turning this into a religous debate, is there any way to
> find out what those messages really mean? When does one run into
> an ldt allocation failure? What is ldt? I am not just trying to
> solve this problem; I also want to understand what is going on
> in the kernel.
Have a look in arch/i386/kernel/*.c. LDT is a descriptor table of segment
registers. Its generally used by emulators to run things like win16 binaries
and sometimes by threaded apps to do thread specific address spaces by
giving each thread a different %fs or %gs segment register
The nVidia kernel drivers are open source. You can get them from
ftp://ftp.nvidia.com/pub/drivers/english/XFree86_40/1.0-2313/NVIDIA_kernel-1.0-2313.tar.gz
the 0.9 serious of drivers were buggy and crashed a lot, earning them a
bad rep. But the 1.0 series are faster and more stable than their
windows counterparts- I havn't had one crash, even with a faulty card
that kills windows constantly.
On Fri, 2001-12-07 at 19:15, Alan Cox wrote:
> > I don't see how one can magically tell that this is an NVIDIA problem.
>
> We don't know. But since we don't have their source and they have our
> source only they can tell you.
>
> > in the kernel and I imagine that even if the NVIDIA drivers are
> > triggering the problem, there are other modules/apps that can bring
> > about the same behavior.
>
> Possibly, but you'll have to ask Nvidia to debug it for you. If you can
> reproduce a bug by
> - removing the nvidia modules so they wont be loaded
> - hard booting the machine
> - triggering the bug without loading the nvidia drivers
>
> then please report it. If not, its between you and nvidia.
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com
> The nVidia kernel drivers are open source. You can get them from
> ftp://ftp.nvidia.com/pub/drivers/english/XFree86_40/1.0-2313/NVIDIA_kernel-1.0-2313.tar.gz
The Nvidia drivers are not open source. Please stop pedalling this myth. All
you are doing is risking confusing other unfortunates into buying nvidia
hardware and finding there is no open 3D support.
> bad rep. But the 1.0 series are faster and more stable than their
> windows counterparts- I havn't had one crash, even with a faulty card
> that kills windows constantly.
So tell me why I keep getting people who remove the Nvidia drivers and have
their machine work. I'm sure some of them are hardware related issues, but
all of them ?
Alan
At 09:15 07/12/01, Alan Cox wrote:
> > I don't see how one can magically tell that this is an NVIDIA problem.
>
>We don't know. But since we don't have their source and they have our
>source only they can tell you.
Not doing any debugging is fine.
However, looking at 2.4.16/arch/i386/kernel/process.c::copy_segments()
which generates this message it seems odd: It returns void, yet it can fail
because it is doing a vmalloc().
When the vmalloc() fails, the new_mm->context.segments is set to NULL and
the function returns.
That seems wrong, no? Shouldn't there be a panic() when the allocation
fails at least? Or even better the function should perhaps return an error
code?
Considering there is only one caller (kernel/fork.c::copy_mm()) it would be
easy to modify copy_mm() to handle a returned error code gracefully and
goto fail_nomem, which would in turn result in kernel/fork.c::do_fork(),
the only caller of copy_mm(), cleaning up properly and returning an error code.
Or have I missed something and the situation where the ldt is missing can
be recovered from? - I would think (without looking into this in the kernel
code) that loosing the local descriptor table would be rather detrimental
on the first context switch to the new process created by fork... And
considering all kinds of errors are being handled in this code path, except
for the vmalloc() failure, it seems like a good idea to add the appropriate
fail mechanism.
If nvidia is causing this to get triggered they will likely run into
problems elsewhere anyway and we don't care but we should get the kernel
working. As it is AFAICS we have a potential DOS, just get the box close to
OOM and start calling man 2 fork and/or man 3 clone and you could trigger
this with a finite probability.
Best regards,
Anton
--
"I've not lost my mind. It's backed up on tape somewhere." - Unknown
--
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Linux NTFS Maintainer / WWW: http://linux-ntfs.sf.net/
ICQ: 8561279 / WWW: http://www-stu.christs.cam.ac.uk/~aia21/
At 09:45 07/12/01, Anton Altaparmakov wrote:
>At 09:15 07/12/01, Alan Cox wrote:
>> > I don't see how one can magically tell that this is an NVIDIA problem.
>>
>>We don't know. But since we don't have their source and they have our
>>source only they can tell you.
>
>Not doing any debugging is fine.
>
>However, looking at 2.4.16/arch/i386/kernel/process.c::copy_segments()
>which generates this message it seems odd: It returns void, yet it can
>fail because it is doing a vmalloc().
>
>When the vmalloc() fails, the new_mm->context.segments is set to NULL and
>the function returns.
>
>That seems wrong, no? Shouldn't there be a panic() when the allocation
>fails at least? Or even better the function should perhaps return an error
>code?
>
>Considering there is only one caller (kernel/fork.c::copy_mm()) it would
>be easy to modify copy_mm() to handle a returned error code gracefully and
>goto fail_nomem, which would in turn result in kernel/fork.c::do_fork(),
>the only caller of copy_mm(), cleaning up properly and returning an error code.
>
>Or have I missed something and the situation where the ldt is missing can
>be recovered from? - I would think (without looking into this in the
>kernel code) that loosing the local descriptor table would be rather
>detrimental on the first context switch to the new process created by
>fork... And considering all kinds of errors are being handled in this code
>path, except for the vmalloc() failure, it seems like a good idea to add
>the appropriate fail mechanism.
>
>If nvidia is causing this to get triggered they will likely run into
>problems elsewhere anyway and we don't care but we should get the kernel
>working. As it is AFAICS we have a potential DOS, just get the box close
>to OOM and start calling man 2 fork and/or man 3 clone and you could trigger
s/man 3 clone/man 2 clone/
> this with a finite probability.
>
>Best regards,
>
> Anton
>
>
>--
> "I've not lost my mind. It's backed up on tape somewhere." - Unknown
>--
>Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
>Linux NTFS Maintainer / WWW: http://linux-ntfs.sf.net/
>ICQ: 8561279 / WWW: http://www-stu.christs.cam.ac.uk/~aia21/
>
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to [email protected]
>More majordomo info at http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at http://www.tux.org/lkml/
--
"I've not lost my mind. It's backed up on tape somewhere." - Unknown
--
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Linux NTFS Maintainer / WWW: http://linux-ntfs.sf.net/
ICQ: 8561279 / WWW: http://www-stu.christs.cam.ac.uk/~aia21/
> When the vmalloc() fails, the new_mm->context.segments is set to NULL and
> the function returns.
>
> That seems wrong, no? Shouldn't there be a panic() when the allocation
> fails at least? Or even better the function should perhaps return an error
> code?
>
> Considering there is only one caller (kernel/fork.c::copy_mm()) it would be
> easy to modify copy_mm() to handle a returned error code gracefully and
That sounds like an appropriate change.
On Fri, 2001-12-07 at 19:43, Alan Cox wrote:
> > The nVidia kernel drivers are open source. You can get them from
> > ftp://ftp.nvidia.com/pub/drivers/english/XFree86_40/1.0-2313/NVIDIA_kernel-1.0-2313.tar.gz
>
> The Nvidia drivers are not open source. Please stop pedalling this myth. All
> you are doing is risking confusing other unfortunates into buying nvidia
> hardware and finding there is no open 3D support.
I wan't trying to state that it is GPL or similar, but the kernel driver
source code IS available for free under a restrictive license. the GLX
drivers are closed.
> > bad rep. But the 1.0 series are faster and more stable than their
> > windows counterparts- I havn't had one crash, even with a faulty card
> > that kills windows constantly.
>
> So tell me why I keep getting people who remove the Nvidia drivers and have
> their machine work. I'm sure some of them are hardware related issues, but
> all of them ?
I can't comment on the stability in all configurations, but I've found
them to work quite well with my box since 1.0 came out. the 0.9 serious
were notorious for crashing though.
> Alan
_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com
On 7 Dec 2001, James Davies wrote:
> source code IS available for free under a restrictive license. the GLX
> drivers are closed.
Bzzzt...
>From the tarball you posted a link to..
-rwxr-xr-x 1 davej users 889615 Nov 27 20:39 Module-nvkernel*
(davej@noodles:NVIDIA_kernel-1.0-2313)$ file Module-nvkernel
Module-nvkernel: ELF 32-bit LSB relocatable, Intel 80386, version 1, not
stripped
No-one but nvidia knows whats in this.
regards,
Dave.
--
| Dave Jones. http://www.codemonkey.org.uk
| SuSE Labs
> > The Nvidia drivers are not open source. Please stop pedalling this myth. All
> > you are doing is risking confusing other unfortunates into buying nvidia
> > hardware and finding there is no open 3D support.
>
> I wan't trying to state that it is GPL or similar, but the kernel driver
> source code IS available for free under a restrictive license. the GLX
> drivers are closed.
No. The kernel driver source code is not available. Stop trying to mislead
people.
Alan
Ok I see my mistake now. While the kernel driver contains source code,
it is only enough to link with the current kernel correctly, and the
rest binary. Sorry.
On Fri, 2001-12-07 at 23:41, Dave Jones wrote:
> On 7 Dec 2001, James Davies wrote:
>
> > source code IS available for free under a restrictive license. the GLX
> > drivers are closed.
>
> Bzzzt...
>
> >From the tarball you posted a link to..
>
> -rwxr-xr-x 1 davej users 889615 Nov 27 20:39 Module-nvkernel*
>
> (davej@noodles:NVIDIA_kernel-1.0-2313)$ file Module-nvkernel
> Module-nvkernel: ELF 32-bit LSB relocatable, Intel 80386, version 1, not
> stripped
>
> No-one but nvidia knows whats in this.
>
> regards,
> Dave.
>
> --
> | Dave Jones. http://www.codemonkey.org.uk
> | SuSE Labs
_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com
This problem is very familiar. It was fixed in an -ac patch against
2.4.9 (although I don't recall the exact one), and is included in
subsequent Linus kernels. Without this fix, I experienced similar
problems with X on multiprocessor workstations.
Jeffrey H. Ingber (jhingber _at_ ix.netcom.com)
On Fri, 2001-12-07 at 00:40, Kiril Vidimce wrote:
>
> We suddenly started seeing freezing problems on a number of machines
> in the past couple of days. There is no pattern as far as I can tell.
> It has happened while running OpenGL apps, netscape, or even when not
> doing anything. The machine will usually just hang and while it's
> still pingable, it's totally unresponsive and you can't remotely log in.
> The 2.4.3 kernel usually hangs forever; the machines with 2.4.9
> kernels usually come back within 10-15 secs.
>
> Every time this happens we see the following message in the system log:
>
> Dec 6 21:29:00 stranger kernel: ldt allocation failed
>
> The machines are:
>
> Hardware:
> - IBM Intellistation M Pro
> - dual 2 GHz P4's
> - 2 GB RAM
> - NVIDIA Quadro DCC card
>
> Software:
> - Red Hat 7.1
> - kernel 2.4.3smp or 2.4.9smp
> - XFree 4.1
> - Ximian Gnome 1.4
> - NVIDIA drivers 1.0-1541
>
> Once this problem occurs, even if the hang is temporary, the machine
> is extremely flakey. Almost any app will start causing ldt allocation
> failure messages in the system log. Only a reboot really helps.
>
> Questions:
> - Does this ring a bell to anybody? What is ldt anyway and what would
> cause an allocation to fail?
>
> - I still keep one of these messages in this state. Is there anything
> I can probe to get useful debugging info?
>
> Any help would greatly be appreciated.
>
> Thanks,
> KV
> --
> ___________________________________________________________________
> Studio Tools [email protected]
> Pixar Animation Studios http://www.pixar.com/
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
On 7 December 2001 07:45, Anton Altaparmakov wrote:
> However, looking at 2.4.16/arch/i386/kernel/process.c::copy_segments()
> which generates this message it seems odd: It returns void, yet it can fail
> because it is doing a vmalloc().
>
> When the vmalloc() fails, the new_mm->context.segments is set to NULL and
> the function returns.
>
> That seems wrong, no? Shouldn't there be a panic() when the allocation
> fails at least? Or even better the function should perhaps return an error
> code?
>
> Considering there is only one caller (kernel/fork.c::copy_mm()) it would be
> easy to modify copy_mm() to handle a returned error code gracefully and
> goto fail_nomem, which would in turn result in kernel/fork.c::do_fork(),
> the only caller of copy_mm(), cleaning up properly and returning an error
> code.
I am ignorant on the subject, but why LDT is used in Linux at all?
LDT register can be set to 0, this can speed up task switch time and save
some memory used for LDT.
I see a i386 specific syscall (kernel/ldt.c:sys_modify_ldt()) and
/*
* default LDT is a single-entry callgate to lcall7 for iBCS
* and a callgate to lcall27 for Solaris/x86 binaries
*/
set_call_gate(&default_ldt[0],lcall7);
set_call_gate(&default_ldt[4],lcall27);
in kernel/traps.c:trap_init().
Is it used elsewhere?
--
vda
Denis Vlasenko <[email protected]> writes:
>
> I am ignorant on the subject, but why LDT is used in Linux at all?
> LDT register can be set to 0, this can speed up task switch time and save
> some memory used for LDT.
glibc thread local data uses an LDT for the segment register.
glibc 2.3 seems to plan to use segment register based thread local data for
even non threaded programs, so it would be a good idea to optimize LDT
allocation a bit (= not allocate 64K of vmalloc space every time
sys_modify_ldt is called - there is only 8MB of it)
-Andi
> I am ignorant on the subject, but why LDT is used in Linux at all?
> LDT register can be set to 0, this can speed up task switch time and save
> some memory used for LDT.
Wine and certain threaded applications
On Wed, Feb 06, 2002 at 02:19:07PM +0100, Andi Kleen wrote:
> glibc 2.3 seems to plan to use segment register based thread local data for
> even non threaded programs, so it would be a good idea to optimize LDT
> allocation a bit (= not allocate 64K of vmalloc space every time
> sys_modify_ldt is called - there is only 8MB of it)
Manfred Spraul had a patch that did this.
--
| Dave Jones. http://www.codemonkey.org.uk
| SuSE Labs
On Wed, Feb 06, 2002 at 02:13:45PM +0000, Alan Cox wrote:
> > glibc 2.3 seems to plan to use segment register based thread local data for
> > even non threaded programs, so it would be a good idea to optimize LDT
> > allocation a bit (= not allocate 64K of vmalloc space every time
> > sys_modify_ldt is called - there is only 8MB of it)
>
> I think it would be a good idea to modify the glibc authors in that case.
> The ldt costs real performance on task switches. It would be very dumb of
> glibc to use it except when justified in the bigger picture - ie threaded
> apps
Are you sure it does? LGDT with non zero argument shouldn't be that costly.
The %fs switching adds some locked cycles for reloading the segment cache,
but because Windows uses that I would it expect to be reasonably optimized
on CPUs.
I actually tried to complain because on x86-64 it is more costly, but to
no avail.
-Andi
> Are you sure it does? LGDT with non zero argument shouldn't be that costly.
> The %fs switching adds some locked cycles for reloading the segment cache,
> but because Windows uses that I would it expect to be reasonably optimized
> on CPUs.
Its measurable, even on an Athlon.
> I actually tried to complain because on x86-64 it is more costly, but to
> no avail.
The more I see from glibc the more I realise that Linus is right - it needs
replacing
> glibc 2.3 seems to plan to use segment register based thread local data for
> even non threaded programs, so it would be a good idea to optimize LDT
> allocation a bit (= not allocate 64K of vmalloc space every time
> sys_modify_ldt is called - there is only 8MB of it)
I think it would be a good idea to modify the glibc authors in that case.
The ldt costs real performance on task switches. It would be very dumb of
glibc to use it except when justified in the bigger picture - ie threaded
apps
On 6 February 2002 11:19, Andi Kleen wrote:
> Denis Vlasenko <[email protected]> writes:
> > I am ignorant on the subject, but why LDT is used in Linux at all?
> > LDT register can be set to 0, this can speed up task switch time and save
> > some memory used for LDT.
>
> glibc thread local data uses an LDT for the segment register.
>
> glibc 2.3 seems to plan to use segment register based thread local data for
> even non threaded programs, so it would be a good idea to optimize LDT
> allocation a bit (= not allocate 64K of vmalloc space every time
> sys_modify_ldt is called - there is only 8MB of it)
What do they use on arches without LDT or equivalent?
--
vda
On Wed, Feb 06, 2002 at 04:02:25PM -0200, Denis Vlasenko wrote:
> On 6 February 2002 11:19, Andi Kleen wrote:
> > Denis Vlasenko <[email protected]> writes:
> > > I am ignorant on the subject, but why LDT is used in Linux at all?
> > > LDT register can be set to 0, this can speed up task switch time and save
> > > some memory used for LDT.
> >
> > glibc thread local data uses an LDT for the segment register.
> >
> > glibc 2.3 seems to plan to use segment register based thread local data for
> > even non threaded programs, so it would be a good idea to optimize LDT
> > allocation a bit (= not allocate 64K of vmalloc space every time
> > sys_modify_ldt is called - there is only 8MB of it)
>
> What do they use on arches without LDT or equivalent?
Most sane architectures reserve a thread pointer register (%g6 resp. %g7 on
sparc, tp on ia64, ppc will use %r2, alpha uses a fast pall call as thread
"register", s390 uses user access register 0 (and s390x uar 0 and 1), etc.).
On register starved ia32 there aren't too many spare registers, so %gs is
used instead.
Jakub
> > allocation a bit (= not allocate 64K of vmalloc space every time
> > sys_modify_ldt is called - there is only 8MB of it)
>
> What do they use on arches without LDT or equivalent?
Generally on such platforms you have enough registers to use a register
for your thread specific storage. In fact even the kernel does that - you'll
find 'current' on some platforms is a global register variable
On Wed, 6 Feb 2002, Alan Cox wrote:
> I think it would be a good idea to modify the glibc authors in that case.
Did you mean "notify," or are you REALLY serious about this ;-)
--
bill davidsen <[email protected]>
CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.
> On Wed, 6 Feb 2002, Alan Cox wrote:
> > I think it would be a good idea to modify the glibc authors in that case.
> Did you mean "notify," or are you REALLY serious about this ;-)
Can we have a kernel FAQ entry about non-US humour 8)
Actually Jakub has already replied to my email with a glibc folks paper
on the thread stuff (confusingly called tls just to confuse everyone who
reads internet standards) and useful discussion can I hope go from that
point.
Alan
On Wed, Feb 06, 2002 at 04:08:45PM +0000, Alan Cox wrote:
> Actually Jakub has already replied to my email with a glibc folks paper
> on the thread stuff (confusingly called tls just to confuse everyone who
where can we find this ?
thanks
john
--
"Mathemeticians stand on each other's shoulders while computer scientists
stand on each other's toes."
- Richard Hamming
On Wed, Feb 06, 2002 at 02:34:06PM +0000, Alan Cox wrote:
> The more I see from glibc the more I realise that Linus is right - it needs
> replacing
Felix Leitner, the diet-libc man, loves to point out the difference between
his popen.c and the one in glibc. The dietlibc version is obviously correct
and 37 lines. The glibc version is 337 lines and full of goobledygook.
Regards,
bert
--
http://www.PowerDNS.com Versatile DNS Software & Services
http://www.tk the dot in .tk
Netherlabs BV / Rent-a-Nerd.nl - Nerd Available -
Linux Advanced Routing & Traffic Control: http://ds9a.nl/lartc
> where can we find this ?
http://people.redhat.com/drepper/tls.pdf
On Wed, 6 Feb 2002, Jakub Jelinek wrote:
> Most sane architectures reserve a thread pointer register (%g6 resp. %g7 on
> sparc, tp on ia64, ppc will use %r2, alpha uses a fast pall call as thread
> "register", s390 uses user access register 0 (and s390x uar 0 and 1), etc.).
> On register starved ia32 there aren't too many spare registers, so %gs is
> used instead.
Actually really sane architectures, such as MIPS, have no unused
registers floating around just in case someone needs one in the next ten
years or so. They require an ABI change which can only be justified if
the benefit is large. So far I failed to see the benefit, but hopefully
it's only a fault of mine.
--
+ Maciej W. Rozycki, Technical University of Gdansk, Poland +
+--------------------------------------------------------------+
+ e-mail: [email protected], PGP key available +
In article <[email protected]> you wrote:
>> I am ignorant on the subject, but why LDT is used in Linux at all?
>> LDT register can be set to 0, this can speed up task switch time and save
>> some memory used for LDT.
>
> Wine and certain threaded applications
Xenix/286 binary emulation.
Followup to: <[email protected]>
By author: Jakub Jelinek <[email protected]>
In newsgroup: linux.dev.kernel
>
> Most sane architectures reserve a thread pointer register (%g6 resp. %g7 on
> sparc, tp on ia64, ppc will use %r2, alpha uses a fast pall call as thread
> "register", s390 uses user access register 0 (and s390x uar 0 and 1), etc.).
> On register starved ia32 there aren't too many spare registers, so %gs is
> used instead.
>
x86-64, interestingly, retains vestigial meaning of the %fs and %gs
registers (but no others) to use as a base pointer for this reason
alone.
-hpa
--
<[email protected]> at work, <[email protected]> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt <[email protected]>
On Wed, Feb 06, 2002 at 10:12:31AM -0500, Jakub Jelinek wrote:
> Most sane architectures reserve a thread pointer register (%g6 resp. %g7 on
> sparc, tp on ia64, ppc will use %r2, alpha uses a fast pall call as thread
> "register", s390 uses user access register 0 (and s390x uar 0 and 1), etc.).
> On register starved ia32 there aren't too many spare registers, so %gs is
> used instead.
So the x86 designers have provided all sorts of shadow registers and extensive
high speed caches and the glibc developers deliberately choose to defeat all that
expensive optimization?
Followup to: <[email protected]>
By author: [email protected]
In newsgroup: linux.dev.kernel
>
> On Wed, Feb 06, 2002 at 10:12:31AM -0500, Jakub Jelinek wrote:
> > Most sane architectures reserve a thread pointer register (%g6 resp. %g7 on
> > sparc, tp on ia64, ppc will use %r2, alpha uses a fast pall call as thread
> > "register", s390 uses user access register 0 (and s390x uar 0 and 1), etc.).
> > On register starved ia32 there aren't too many spare registers, so %gs is
> > used instead.
>
> So the x86 designers have provided all sorts of shadow registers and extensive
> high speed caches and the glibc developers deliberately choose to defeat all that
> expensive optimization?
>
Uhm... none of that "expensive optimization" will help you here,
because you need *architectural storage*. Archtectural storage is
always a fundamentally limited resource, regardless of how many shadow
registers you add.
However, an x86 has a whole additional register file which is rarely
used these days -- the segment register file. The segment register
file is mainly useful when the main usage is address offsetting, since
it provides an extra input into the address adder. For this
particular purpose, using it is very much the sane thing to do.
As far as %gs switching is concerned, using %gs doesn't automatically
mean using the LDT. There are two ways you can avoid setting up LDTs
in single-threaded apps, and still allow an ABI compatible with the
threaded apps:
a) For single-threaded apps, define %gs == %ds. Less than ideal for
several reasons, but no kernel mods needed.
b) Have the kernel provide another GDT value which can be used by the
single-threaded apps.
-hpa
--
<[email protected]> at work, <[email protected]> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt <[email protected]>
On Wed, Feb 06, 2002 at 12:19:37PM -0800, H. Peter Anvin wrote:
> Followup to: <[email protected]>
> By author: Jakub Jelinek <[email protected]>
> In newsgroup: linux.dev.kernel
> >
> > Most sane architectures reserve a thread pointer register (%g6 resp. %g7 on
> > sparc, tp on ia64, ppc will use %r2, alpha uses a fast pall call as thread
> > "register", s390 uses user access register 0 (and s390x uar 0 and 1), etc.).
> > On register starved ia32 there aren't too many spare registers, so %gs is
> > used instead.
> >
>
> x86-64, interestingly, retains vestigial meaning of the %fs and %gs
> registers (but no others) to use as a base pointer for this reason
> alone.
Well, on x86-64 this is purely x86-64 ABI designers decision, they could
pick one of %r8 - %r15 and use that as thread pointer instead (and were
recommended to do so).
Jakub
Jakub Jelinek wrote:
>>>
>>x86-64, interestingly, retains vestigial meaning of the %fs and %gs
>>registers (but no others) to use as a base pointer for this reason
>>alone.
>
> Well, on x86-64 this is purely x86-64 ABI designers decision, they could
> pick one of %r8 - %r15 and use that as thread pointer instead (and were
> recommended to do so).
>
Wasting an architectural register is still bloody expensive, even if 1
among 16 is less than 1 among 8. They'll be using %gs as thread pointer
in userspace and CPU pointer in kernel space.
-hpa
On Wed, Feb 06, 2002 at 01:00:12PM -0800, H. Peter Anvin wrote:
> Uhm... none of that "expensive optimization" will help you here,
> because you need *architectural storage*. Archtectural storage is
> always a fundamentally limited resource, regardless of how many shadow
> registers you add.
>
> However, an x86 has a whole additional register file which is rarely
> used these days -- the segment register file. The segment register
> file is mainly useful when the main usage is address offsetting, since
> it provides an extra input into the address adder. For this
> particular purpose, using it is very much the sane thing to do.
>
> As far as %gs switching is concerned, using %gs doesn't automatically
> mean using the LDT. There are two ways you can avoid setting up LDTs
> in single-threaded apps, and still allow an ABI compatible with the
> threaded apps:
>
> a) For single-threaded apps, define %gs == %ds. Less than ideal for
> several reasons, but no kernel mods needed.
This is not possible, since then %gs:0 (which is TLS base) cannot be read.
We would have to change the TLS ABI (thus become incompatible e.g. with Sun)
- note that these sequences are emitted by compiler or linker (the latter in
code transformations) and pick some hardcoded constant, say %gs:0x1000.
This would mean though that every single non-threaded app would need to
mmap a page at 4KB. If this offset was not constant, it would slow down all
TLS accesses.
> b) Have the kernel provide another GDT value which can be used by the
> single-threaded apps.
Like above, a fixed address for mmap would have to be chosen, but the
advantage would be that the TLS ABI would need no changing.
Simply kernel would add 0x33 to GDT as 4KB -> 0xc0000000 user data segment
and apps could put that value into %gs if not using threads.
But I think there is c) and d).
c) is just minor modification of current ldt handling in kernel, which would
mean a single entry LDT (residing in task_struct) could be used instead
of vmalloced one - this has the disadvantage of lldt on almost every
context switch
d) default to a single-entry per-cpu LDT, which only non-linux personality
apps and apps needing more than 1 LDT entry (threaded apps, wine/dosemu/...)
would not use. Non-linux apps would use current default_ldt and those
needing more than one LDT would use the current vmalloced mm private area.
If a task would be using this per-cpu LDT (common case), context switch
would do lldt only if previous task was not using the per-cpu LDT
(unlikely) and just store task_struct->thread.ldt_word_0 and ldt_word_1
into the per-cpu LDT (dunno how expensive is that, but IMHO it should be
cheaper than full load_LDT).
Jakub
Jakub Jelinek wrote:
>
>>b) Have the kernel provide another GDT value which can be used by the
>> single-threaded apps.
>>
>
> Like above, a fixed address for mmap would have to be chosen, but the
> advantage would be that the TLS ABI would need no changing.
> Simply kernel would add 0x33 to GDT as 4KB -> 0xc0000000 user data segment
> and apps could put that value into %gs if not using threads.
>
> But I think there is c) and d).
> c) is just minor modification of current ldt handling in kernel, which would
> mean a single entry LDT (residing in task_struct) could be used instead
> of vmalloced one - this has the disadvantage of lldt on almost every
> context switch
> d) default to a single-entry per-cpu LDT, which only non-linux personality
> apps and apps needing more than 1 LDT entry (threaded apps, wine/dosemu/...)
> would not use. Non-linux apps would use current default_ldt and those
> needing more than one LDT would use the current vmalloced mm private area.
> If a task would be using this per-cpu LDT (common case), context switch
> would do lldt only if previous task was not using the per-cpu LDT
> (unlikely) and just store task_struct->thread.ldt_word_0 and ldt_word_1
> into the per-cpu LDT (dunno how expensive is that, but IMHO it should be
> cheaper than full load_LDT).
>
Actually d) is probably better done by allowing CPUs to put *one* entry in
the GDT instead of requesting an LDT. Since the LDT takes up a GDT entry
anyway, this should be simple enough.
-hpa
On Wed, Feb 06, 2002 at 02:05:52PM -0800, H. Peter Anvin wrote:
> >>b) Have the kernel provide another GDT value which can be used by the
> >> single-threaded apps.
> >>
> >
> > Like above, a fixed address for mmap would have to be chosen, but the
> > advantage would be that the TLS ABI would need no changing.
> > Simply kernel would add 0x33 to GDT as 4KB -> 0xc0000000 user data segment
> > and apps could put that value into %gs if not using threads.
> >
> > But I think there is c) and d).
> > c) is just minor modification of current ldt handling in kernel, which would
> > mean a single entry LDT (residing in task_struct) could be used instead
> > of vmalloced one - this has the disadvantage of lldt on almost every
> > context switch
> > d) default to a single-entry per-cpu LDT, which only non-linux personality
> > apps and apps needing more than 1 LDT entry (threaded apps, wine/dosemu/...)
> > would not use. Non-linux apps would use current default_ldt and those
> > needing more than one LDT would use the current vmalloced mm private area.
> > If a task would be using this per-cpu LDT (common case), context switch
> > would do lldt only if previous task was not using the per-cpu LDT
> > (unlikely) and just store task_struct->thread.ldt_word_0 and ldt_word_1
> > into the per-cpu LDT (dunno how expensive is that, but IMHO it should be
> > cheaper than full load_LDT).
> >
>
>
> Actually d) is probably better done by allowing CPUs to put *one* entry in
> the GDT instead of requesting an LDT. Since the LDT takes up a GDT entry
> anyway, this should be simple enough.
On UP sure, on SMP you'd need to allocate as many GDT entries as there are
CPUs and special case this in __switch_to too (something like
loadsegment(fs, next->fs);
if ((next->gs & 7) == 3 && next->gs >= 0x30 && next->gs < 0x30 + NR_CPUS * 8)
next->gs = smp_processor_id() * 8 + 0x33;
loadsegment(gs, next->gs);
) which is kind of ugly (and userland must not save/restore %gs itself then).
Unlike d) with LDT, where unmodified glibc could work with older kernels
too, thus would mean strict kernel minimum version requirement (with LDT d)
it would be just an optimization).
Jakub
Followup to: <[email protected]>
By author: Jakub Jelinek <[email protected]>
In newsgroup: linux.dev.kernel
>
> Unlike d) with LDT, where unmodified glibc could work with older kernels
> too, thus would mean strict kernel minimum version requirement (with LDT d)
> it would be just an optimization).
>
Just make it a fallback option.
-hpa
--
<[email protected]> at work, <[email protected]> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt <[email protected]>
> This is not possible, since then %gs:0 (which is TLS base) cannot be read.
> We would have to change the TLS ABI (thus become incompatible e.g. with Sun)
Sun who have canned their x86 product it seems. I don't feel "the standard
requires we suck" is an appropriate justification for anything. If there
is not a sane way to follow the standard - break it. If there is a sane way
then all fair and good, find it and use it
Alan
On Thu, Feb 07, 2002 at 12:21:22AM +0000, Alan Cox wrote:
> > This is not possible, since then %gs:0 (which is TLS base) cannot be read.
> > We would have to change the TLS ABI (thus become incompatible e.g. with Sun)
>
> Sun who have canned their x86 product it seems. I don't feel "the standard
> requires we suck" is an appropriate justification for anything. If there
> is not a sane way to follow the standard - break it. If there is a sane way
> then all fair and good, find it and use it
We have changed it already (e.g,. for regparm(1), fewer relocs, shorter insn
sequences, etc.), but with exception of 2 non-dynamic relocs (which get
different numbers) we are still compatible.
But as written later, just using a different GDT descriptor could avoid
having to change the ABI, but would still have the undesirable property of
requiring every app to mmap a new page at fixed location.
Jakub
Alan Cox wrote:
>>This is not possible, since then %gs:0 (which is TLS base) cannot be read.
>>We would have to change the TLS ABI (thus become incompatible e.g. with Sun)
>>
>
> Sun who have canned their x86 product it seems. I don't feel "the standard
> requires we suck" is an appropriate justification for anything. If there
> is not a sane way to follow the standard - break it. If there is a sane way
> then all fair and good, find it and use it
>
I do have to agree that zero-basing the TLS pointer seems saner than not
doing so.
-hpa
Jakub Jelinek wrote:
>
> We have changed it already (e.g,. for regparm(1), fewer relocs, shorter insn
> sequences, etc.), but with exception of 2 non-dynamic relocs (which get
> different numbers) we are still compatible.
> But as written later, just using a different GDT descriptor could avoid
> having to change the ABI, but would still have the undesirable property of
> requiring every app to mmap a new page at fixed location.
>
It's not that bad, though, especially if your fixed location is just
beneath the standard mmap base address (e.g. 0x3f000000 -- leave some room
for future expansion -- or thereabouts on the standard 3:1 split space.)
Do you support tlsalloc() or whatever it is called?
-hpa