Frankly, I'm amazed the kernel worked for long without this.
Every linker script thinks the section is called .data.percpu.
Without this patch, every CPU ends up sharing the same "per-cpu"
variable.
This might explain the wierd per-cpu problem reports from Andrew and
Dave, and also that nagging feeling that I'm an idiot...
Rusty.
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.
--- working-2.5.33-hotcpu-cpudown-i386/include/asm-generic/percpu.h.~1~ Wed Aug 28 09:29:50 2002
+++ working-2.5.33-hotcpu-cpudown-i386/include/asm-generic/percpu.h Wed Sep 4 12:32:34 2002
@@ -10,7 +10,7 @@
/* Separate out the type, so (int[3], foo) works. */
#ifndef MODULE
#define DEFINE_PER_CPU(type, name) \
- __attribute__((__section__(".percpu"))) __typeof__(type) name##__per_cpu
+ __attribute__((__section__(".data.percpu"))) __typeof__(type) name##__per_cpu
#endif
/* var is in discarded region: offset to particular copy we want */
Rusty Russell wrote:
>
> Frankly, I'm amazed the kernel worked for long without this.
>
> Every linker script thinks the section is called .data.percpu.
> Without this patch, every CPU ends up sharing the same "per-cpu"
> variable.
>
wow. Either this was working by fluke, or Bill's softirq problem
just got solved.
Rusty Russell wrote:
>> Frankly, I'm amazed the kernel worked for long without this.
>> Every linker script thinks the section is called .data.percpu.
>> Without this patch, every CPU ends up sharing the same "per-cpu"
>> variable.
On Tue, Sep 03, 2002 at 07:56:47PM -0700, Andrew Morton wrote:
> wow. Either this was working by fluke, or Bill's softirq problem
> just got solved.
It's pretty easy to find out, my machines don't get past
execve("/sbin/init", argv_init, envp_init); without some kind of softirq
fix. The boot cycle is not swift though... in progress.
Cheers,
Bill
From: Rusty Russell <[email protected]>
Date: Wed, 04 Sep 2002 12:35:41 +1000
This might explain the wierd per-cpu problem reports from Andrew and
Dave, and also that nagging feeling that I'm an idiot...
Verifying... no without the explicit initializers the per-cpu stuff
still ends up in the BSS with egcs-2.9X, even with your fix applied.
On Wed, Sep 04, 2002 at 12:35:41PM +1000, Rusty Russell wrote:
> Frankly, I'm amazed the kernel worked for long without this.
> Every linker script thinks the section is called .data.percpu.
> Without this patch, every CPU ends up sharing the same "per-cpu"
> variable.
> This might explain the wierd per-cpu problem reports from Andrew and
> Dave, and also that nagging feeling that I'm an idiot...
Hmm, 2.5.33 is doing some *really* weird crap. OTOH it doesn't appear
to be tripping the BUG() in softirq.c, and disks seem to be doing okay.
It survived 4 parallel mkfs's. I'll follow up with some kind of
bugreport on the PCI and/or starfire.c front in a separate post.
Cheers,
Bill
In message <[email protected]> you write:
> From: Rusty Russell <[email protected]>
> Date: Wed, 04 Sep 2002 12:35:41 +1000
>
> This might explain the wierd per-cpu problem reports from Andrew and
> Dave, and also that nagging feeling that I'm an idiot...
>
> Verifying... no without the explicit initializers the per-cpu stuff
> still ends up in the BSS with egcs-2.9X, even with your fix applied.
OK. I really hate working around wierd toolchain bugs (I use 2.95.4
here and it's fine), and adding an initializer to the macro is ugly.
If you're not going to upgrade your compiler, will you accept a gcc
patch to fix this? If so, where can I get the source to your exact
version?
Thanks,
Rusty.
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.
Followup to: <[email protected]>
By author: Rusty Russell <[email protected]>
In newsgroup: linux.dev.kernel
>
> In message <[email protected]> you write:
> > From: Rusty Russell <[email protected]>
> > Date: Wed, 04 Sep 2002 12:35:41 +1000
> >
> > This might explain the wierd per-cpu problem reports from Andrew and
> > Dave, and also that nagging feeling that I'm an idiot...
> >
> > Verifying... no without the explicit initializers the per-cpu stuff
> > still ends up in the BSS with egcs-2.9X, even with your fix applied.
>
> OK. I really hate working around wierd toolchain bugs (I use 2.95.4
> here and it's fine), and adding an initializer to the macro is ugly.
>
> If you're not going to upgrade your compiler, will you accept a gcc
> patch to fix this? If so, where can I get the source to your exact
> version?
>
gcc puts all uninitialized variables in .bss, and it apparently can't
be overridden. This seems to be a side effect of the way gcc handles
common variables.
-hpa
--
<[email protected]> at work, <[email protected]> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt <[email protected]>
From: Rusty Russell <[email protected]>
Date: Wed, 04 Sep 2002 14:04:22 +1000
OK. I really hate working around wierd toolchain bugs (I use 2.95.4
here and it's fine), and adding an initializer to the macro is ugly.
Mine is 2.92.11
If you're not going to upgrade your compiler, will you accept a gcc
patch to fix this? If so, where can I get the source to your exact
version?
Oh, "I'm" willing to upgrade "my" compiler, it's my users
that are the problem. If you impose 3.1 or whatever, I get less
people testing on sparc64 as a result.
From: Linus Torvalds <[email protected]>
Date: Tue, 3 Sep 2002 22:17:39 -0700 (PDT)
Well, I don't think you have to go to 3.1.x.
Yes I do, anything after our standard sparc64 (ie. egcs-2.92.11)
compiler up until 3.1.x is seriously unusable for kernel builds.
On Tue, 3 Sep 2002, David S. Miller wrote:
> Oh, "I'm" willing to upgrade "my" compiler, it's my users
> that are the problem. If you impose 3.1 or whatever, I get less
> people testing on sparc64 as a result.
Well, I don't think you have to go to 3.1.x.
gcc-2.96 at least seems to do all right. Apparently 2.95 does too. It's
only the truly ancient compilers that don't work.
Linus
In message <[email protected]> you write:
> If you're not going to upgrade your compiler, will you accept a gcc
> patch to fix this? If so, where can I get the source to your exact
> version?
>
> Oh, "I'm" willing to upgrade "my" compiler, it's my users
> that are the problem. If you impose 3.1 or whatever, I get less
> people testing on sparc64 as a result.
I understand. It's OK to piss off one Sparc64 Linux user, but if you
piss off both of them, you're in trouble 8)
I was actually proposing to provide a patch to an egcs version of your
choice which you could provide as kgcc to your users. But RH seem to
have taken down the SRPM for 2.92.11 (egcs64-19980921-4.src.rpm).
Rusty.
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.
From: Rusty Russell <[email protected]>
Date: Wed, 04 Sep 2002 16:00:35 +1000
I was actually proposing to provide a patch to an egcs version of your
choice which you could provide as kgcc to your users. But RH seem to
have taken down the SRPM for 2.92.11 (egcs64-19980921-4.src.rpm).
The last sparc64 Red Hat distribution was 6.2 And it still works
unchanged with current kernels. :-)
Actually Rusty what's the big deal, add an "initializer"
arg to the macro. It doesn't hurt anyone, it doesn't lose
any space in the kernel image, and the macro arg reminds
people to do it.
I think it's a small price to pay to keep a longer range
of compilers supported :-)
In message <[email protected]> you write:
>
> Actually Rusty what's the big deal, add an "initializer"
> arg to the macro. It doesn't hurt anyone, it doesn't lose
> any space in the kernel image, and the macro arg reminds
> people to do it.
>
> I think it's a small price to pay to keep a longer range
> of compilers supported :-)
I disagree. They might not have a convenient (static) initializer, in
which case it's simply cruel and unusual, to work around an obscure
compiler bug.
Rusty.
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.
Hi,
On Tue, 3 Sep 2002, David S. Miller wrote:
> Oh, "I'm" willing to upgrade "my" compiler, it's my users that are the
> problem. If you impose 3.1 or whatever, I get less people testing on
> sparc64 as a result.
Oh, well. I couldn't use gcc 3.* on sparc64 for some weird reason, gcc
2.95* is unable to handle all that crap, and the egcs hackup can't handle
the current kernel code. So what?
Thunder
--
--./../...-/. -.--/---/..-/.-./..././.-../..-. .---/..-/.../- .-
--/../-./..-/-/./--..-- ../.----./.-../.-.. --./../...-/. -.--/---/..-
.- -/---/--/---/.-./.-./---/.--/.-.-.-
--./.-/-.../.-./.././.-../.-.-.-
From: Thunder from the hill <[email protected]>
Date: Wed, 4 Sep 2002 09:44:51 -0600 (MDT)
I couldn't use gcc 3.* on sparc64 for some weird reason
3.0 should be broken yes, but 3.1 definitely works as I use
it here for occaisionaly test builds all the time.
In article <[email protected]> Rusty Russell wrote:
> Frankly, I'm amazed the kernel worked for long without this.
> Every linker script thinks the section is called .data.percpu.
> Without this patch, every CPU ends up sharing the same "per-cpu"
> variable.
> This might explain the wierd per-cpu problem reports from Andrew and
> Dave, and also that nagging feeling that I'm an idiot...
Not only does this fix the tasklet BUG with 2.5.32 but it also fixes a serial
console hang with my 2.5.32 version of Ingo/Davem/Alexey's scalable timers
code that I have been debugging for the last two days. I use
a per-cpu tasklet to run the timers, so it was probably killing me
there.
Thanks
--
Dipankar Sarma <[email protected]> http://lse.sourceforge.net
Linux Technology Center, IBM Software Lab, Bangalore, India.
> In article <[email protected]> Rusty Russell wrote:
>> This might explain the wierd per-cpu problem reports from Andrew and
>> Dave, and also that nagging feeling that I'm an idiot...
On Thu, Sep 05, 2002 at 02:40:23PM +0530, Dipankar Sarma wrote:
> Not only does this fix the tasklet BUG with 2.5.32 but it also fixes a serial
> console hang with my 2.5.32 version of Ingo/Davem/Alexey's scalable timers
> code that I have been debugging for the last two days. I use
> a per-cpu tasklet to run the timers, so it was probably killing me
> there.
Are you running on NUMA-Q? Do you also see the tty deadlock?
Thanks,
Bill
On Thu, Sep 05, 2002 at 02:09:59AM -0700, William Lee Irwin III wrote:
> Are you running on NUMA-Q? Do you also see the tty deadlock?
>
No, a 4CPU PIII box. What tty deadlock ?
Thanks
--
Dipankar Sarma <[email protected]> http://lse.sourceforge.net
Linux Technology Center, IBM Software Lab, Bangalore, India.
On Thu, Sep 05, 2002 at 02:09:59AM -0700, William Lee Irwin III wrote:
>> Are you running on NUMA-Q? Do you also see the tty deadlock?
On Fri, Sep 06, 2002 at 12:38:53AM +0530, Dipankar Sarma wrote:
> No, a 4CPU PIII box. What tty deadlock ?
> Thanks
The one reported in the "mysterious tty deadlock" thread.
Basically, some kind of badness happens where it has to be sent
a signal-generating character before generating output a good
chunk of the time, where extra carriage returns sometimes help.
Cheers,
Bill
Hi!
> > Actually Rusty what's the big deal, add an "initializer"
> > arg to the macro. It doesn't hurt anyone, it doesn't lose
> > any space in the kernel image, and the macro arg reminds
> > people to do it.
> >
> > I think it's a small price to pay to keep a longer range
> > of compilers supported :-)
>
> I disagree. They might not have a convenient (static) initializer, in
> which case it's simply cruel and unusual, to work around an obscure
> compiler bug.
Ugh? of course it will always have convient initializer, namely zero.
Pavel
--
Philips Velo 1: 1"x4"x8", 300gram, 60, 12MB, 40bogomips, linux, mutt,
details at http://atrey.karlin.mff.cuni.cz/~pavel/velo/index.html.
In message <[email protected]> you write:
> Hi!
>
> > > Actually Rusty what's the big deal, add an "initializer"
> > > arg to the macro. It doesn't hurt anyone, it doesn't lose
> > > any space in the kernel image, and the macro arg reminds
> > > people to do it.
> > >
> > > I think it's a small price to pay to keep a longer range
> > > of compilers supported :-)
> >
> > I disagree. They might not have a convenient (static) initializer, in
> > which case it's simply cruel and unusual, to work around an obscure
> > compiler bug.
>
> Ugh? of course it will always have convient initializer, namely zero.
What if you really need to initialize them at runtime? You're putting
a static initializer there simply to mask an obscure toolchain bug.
I'd really prefer dotting:
/* FIXME: Initializer required so gcc 2.96 doesn't put in BSS */
DEFINE_PER_CPU(int, xxx) = 0;
everywhere, which can be deleted later, to enforcing it for everyone
in the infrastructure when it doesn't always make sense.
Rusty.
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.
From: Rusty Russell <[email protected]>
Date: Mon, 09 Sep 2002 13:45:02 +1000
/* FIXME: Initializer required so gcc 2.96 doesn't put in BSS */
DEFINE_PER_CPU(int, xxx) = 0;
everywhere, which can be deleted later
Why everywhere? If you do it in the macro, then when you want
to delete the initializer remove the macro arg. Then you cover
both in-kernel and cases in external sources because the build
will break for them so they'll know to fixup their macro invocation.
I hate when things change silently, and if you don't put it in
the macro, someone will get it wrong and we'll find out only
when someone with an older compiler uses that feature on an SMP
machine.
Read as: This is bad programming practice Rusty, regardless of
your personal views about this particular GCC bug or
what you think is a legitimate initializer.
See spinlock_t's declaration for older GCC versions, that is how we
traditionally deal with compiler bugs like this, explicitly so that it
is impossible to "miss" something and get it wrong.
Everytime I asked Linus to rearrange some piece of code that used
spinlock_t initializers to workaround the "empty initializer at
end of struct" GCC bug he always told me "someone will get it
wrong if we do it that way" And I think he's right.
On 3 Sep 2002 21:52:45 -0700
"H. Peter Anvin" <[email protected]> wrote:
> Followup to: <[email protected]>
> By author: Rusty Russell <[email protected]>
> In newsgroup: linux.dev.kernel
> >
> > In message <[email protected]> you write:
> > > From: Rusty Russell <[email protected]>
> > > Date: Wed, 04 Sep 2002 12:35:41 +1000
> > >
> > > This might explain the wierd per-cpu problem reports from Andrew and
> > > Dave, and also that nagging feeling that I'm an idiot...
> > >
> > > Verifying... no without the explicit initializers the per-cpu stuff
> > > still ends up in the BSS with egcs-2.9X, even with your fix applied.
> >
> > OK. I really hate working around wierd toolchain bugs (I use 2.95.4
> > here and it's fine), and adding an initializer to the macro is ugly.
>
> gcc puts all uninitialized variables in .bss, and it apparently can't
> be overridden. This seems to be a side effect of the way gcc handles
> common variables.
Err... no, as I said, it doesn't happen with 2.95.4 or 3.0.4.
Rusty.
--
there are those who do and those who hang on and you don't see too
many doers quoting their contemporaries. -- Larry McVoy
In message <[email protected]> you write:
> From: Rusty Russell <[email protected]>
> Date: Mon, 09 Sep 2002 13:45:02 +1000
>
> /* FIXME: Initializer required so gcc 2.96 doesn't put in BSS */
> DEFINE_PER_CPU(int, xxx) = 0;
>
> everywhere, which can be deleted later
>
> Why everywhere? If you do it in the macro, then when you want
> to delete the initializer remove the macro arg. Then you cover
> both in-kernel and cases in external sources because the build
> will break for them so they'll know to fixup their macro invocation.
So, you break everyone's compile later *and* you have to figure out
which initializers were required, and which ones were bogus.
*Planning* to break an interface is just stupid.
> See spinlock_t's declaration for older GCC versions, that is how we
> traditionally deal with compiler bugs like this, explicitly so that it
> is impossible to "miss" something and get it wrong.
Yeah, but you can still leave a spinlock uninitialized, and it'll
work.
I'm not making an interface harder to use because of one of 17
architectures refuses to fix a bug in their toolchain. Just not
happening, sorry.
I want *you* to feel the pain, not spread it around by leaving turds
throughout the code long after the bug is forgotten:
diff -urpN --exclude TAGS -X /home/rusty/devel/kernel/kernel-patches/current-dontdiff --minimal working-2.5.33-cset-1.621/Makefile working-2.5.33-cset-1.621-catch-percpu/Makefile
--- working-2.5.33-cset-1.621/Makefile 2002-09-09 13:31:44.000000000 +1000
+++ working-2.5.33-cset-1.621-catch-percpu/Makefile 2002-09-09 18:06:51.000000000 +1000
@@ -282,6 +282,8 @@ cmd_link_vmlinux = $(LD) $(LDFLAGS) $(LD
-o vmlinux
# set -e makes the rule exit immediately on error
+# Final awk script makes sure per-cpu vars are in per-cpu section, as
+# old gcc (eg egcs 2.92.11) ignores section attribute if uninitialized.
define rule_link_vmlinux
set -e
@@ -292,6 +294,7 @@ define rule_link_vmlinux
$(call cmd,link_vmlinux)
echo 'cmd_$@ := $(cmd_link_vmlinux)' > $(@D)/.$(@F).cmd
$(NM) vmlinux | grep -v '\(compiled\)\|\(\.o$$\)\|\( [aUw] \)\|\(\.\.ng$$\)\|\(LASH[RL]DI\)' | sort > System.map
+ awk '/ __per_cpu_start$/ { IN_PER_CPU=1 } / __per_cpu_end$/ { IN_PER_CPU=0 } /__per_cpu$/ { if (!IN_PER_CPU) { print $3 " not in per-cpu section" > "/dev/stderr"; FOUND=1; } } END { exit FOUND; }' < System.map
endef
vmlinux: $(vmlinux-objs) FORCE
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.
From: Rusty Russell <[email protected]>
Date: Mon, 09 Sep 2002 18:17:22 +1000
In message <[email protected]> you write:
I'm not making an interface harder to use because of one of 17
architectures refuses to fix a bug in their toolchain.
It's not a sparc64 specific problem, as has been noted.
We are extended the life of an older toolchain on more than
one platform.
Note that Andrew Morton found the problem on one of his older
x86 EGCS's about the same time I found it on sparc64.
I want *you* to feel the pain, not spread it around by leaving turds
throughout the code long after the bug is forgotten:
Aha, but it is you putting the turd comments all over. I'm
suggesting to put the turd in one place, the header file.
And how difficult is it to discern which initializers were
needed? Hmmm let me see, if it was all zero --> removing it
is harmless.
Wow, that was hard. :-)
Both of us are advocating adding shit to the tree, the only argument
is which stinks less from a maintainence perspective.
From: Rusty Russell <[email protected]>
Date: Mon, 9 Sep 2002 18:06:19 +1000
On 3 Sep 2002 21:52:45 -0700
"H. Peter Anvin" <[email protected]> wrote:
> gcc puts all uninitialized variables in .bss, and it apparently can't
> be overridden. This seems to be a side effect of the way gcc handles
> common variables.
Err... no, as I said, it doesn't happen with 2.95.4 or 3.0.4.
But note that older GCCs do have the problem and it isn't platform
specific at all.
Rusty wrote:
> Yeah, but you can still leave a spinlock uninitialized, and it'll
> work.
If your architecture has load-and-zero as its only atomic primitive,
leaving spinlocks uninitialised will _not_ work ;-)
--
Revolutions do not require corporate support.
On Mon, 2002-09-09 at 09:15, David S. Miller wrote:
> Note that Andrew Morton found the problem on one of his older
> x86 EGCS's about the same time I found it on sparc64.
egcs gets so many long long things wrong on x86 that its only valid use
IMHO for 2.5 is as a syntax checker. Is it really worth an ugly hack for
a compiler one major developer has a personal affliction for and a port
that has a tiny user base and now has a working compiler.
Maybe if we put less gunk in the kernel they'd fix gcc more often 8)
Alan Cox wrote:
>
> On Mon, 2002-09-09 at 09:15, David S. Miller wrote:
> > Note that Andrew Morton found the problem on one of his older
> > x86 EGCS's about the same time I found it on sparc64.
>
> egcs gets so many long long things wrong on x86 that its only valid use
> IMHO for 2.5 is as a syntax checker. Is it really worth an ugly hack for
> a compiler one major developer has a personal affliction for and a port
> that has a tiny user base and now has a working compiler.
>
> Maybe if we put less gunk in the kernel they'd fix gcc more often 8)
Please. I'm not a reason for hanging onto egcs-1.1.2; and I'll
downgrade to 2.95.2 when egcs-1.1.2 is retired (as I did a while back).
But as long as we need to support 1.1.2, I use it. To detect
breakage, and because it compiles kernels 30% faster.
In message <[email protected]> you write:
> Rusty wrote:
> > Yeah, but you can still leave a spinlock uninitialized, and it'll
> > work.
>
> If your architecture has load-and-zero as its only atomic primitive,
> leaving spinlocks uninitialised will _not_ work ;-)
<sigh>
Context: static initializers. ie. you can use dynamic initialization
on your spinlocks.
Rusty.
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.
In message <[email protected]> you write:
> I want *you* to feel the pain, not spread it around by leaving turds
> throughout the code long after the bug is forgotten:
>
> Aha, but it is you putting the turd comments all over. I'm
> suggesting to put the turd in one place, the header file.
.... and every user of it...
> And how difficult is it to discern which initializers were
> needed? Hmmm let me see, if it was all zero --> removing it
> is harmless.
Let's not get onto initializer wars: I initialize all my variables
exactly once, so there's serious semantic difference between "static
int x;" and "static int x = 0;" in my code.
> Both of us are advocating adding shit to the tree, the only argument
> is which stinks less from a maintainence perspective.
Hey, I'm not stopping you sending a patch to Linus, but given how we
deprecated compilers on x86, I don't think he'll have sympathy for
you in 2.6.
Rusty.
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.