Greeting,
FYI, we noticed the following commit (built with gcc-4.9):
commit: c3a340f7e7eadac7662ab104ceb16432e5a4c6b2 ("sched: Have sched_class_highest define by vmlinux.lds.h")
https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git sched/core
in testcase: trinity
with following parameters:
runtime: 300s
test-description: Trinity is a linux system call fuzz tester.
test-url: http://codemonkey.org.uk/projects/trinity/
on test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 16G
caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace):
+------------------------------------------+------------+------------+
| | 590d697963 | c3a340f7e7 |
+------------------------------------------+------------+------------+
| boot_successes | 4 | 0 |
| boot_failures | 0 | 4 |
| invalid_opcode:#[##] | 0 | 4 |
| EIP:sched_init | 0 | 4 |
| Kernel_panic-not_syncing:Fatal_exception | 0 | 4 |
| kernel_BUG_at_kernel/sched/core.c | 0 | 4 |
+------------------------------------------+------------+------------+
If you fix the issue, kindly add following tag
Reported-by: kernel test robot <[email protected]>
[ 1.813939] ** If you see this message and you are not debugging **
[ 1.814781] ** the kernel, report this immediately to your vendor! **
[ 1.815661] ** **
[ 1.816509] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE **
[ 1.817362] **********************************************************
[ 1.820914] invalid opcode: 0000 [#1] SMP DEBUG_PAGEALLOC
[ 1.821634] CPU: 0 PID: 0 Comm: swapper Not tainted 5.8.0-rc1-00024-gc3a340f7e7ead #2
[ 1.822630] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-1 04/01/2014
[ 1.823710] EIP: sched_init+0x3d/0x2fb
[ 1.824191] Code: 20 a6 e5 c1 3d 40 a6 e5 c1 0f 95 c0 08 c2 75 1a b8 a0 a6 e5 c1 3d c0 a6 e5 c1 75 0e b8 20 a7 e5 c1 3d 40 a7 e5 c1 74 04 0f 0b <0f> 0b 55 89 e5 57 56 53 83 ec 08 e8 67 04 00 00 8b 3d 80 ad ec c1
[ 1.826607] EAX: c1e5a601 EBX: c1dac484 ECX: 00000000 EDX: 00000001
[ 1.827434] ESI: 00000000 EDI: 000003f8 EBP: c1e93f9c ESP: c1e93f78
[ 1.828241] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 EFLAGS: 00210002
[ 1.829112] CR0: 80050033 CR2: ffc70000 CR3: 021ce000 CR4: 00000690
[ 1.829919] Call Trace:
[ 1.830240] ? start_kernel+0x43f/0x677
[ 1.830738] i386_start_kernel+0x48/0x4a
[ 1.831261] startup_32_smp+0x15f/0x164
[ 1.831757] Modules linked in:
[ 1.832165] ---[ end trace db9d28bea02c9e7f ]---
[ 1.832761] EIP: sched_init+0x3d/0x2fb
[ 1.833243] Code: 20 a6 e5 c1 3d 40 a6 e5 c1 0f 95 c0 08 c2 75 1a b8 a0 a6 e5 c1 3d c0 a6 e5 c1 75 0e b8 20 a7 e5 c1 3d 40 a7 e5 c1 74 04 0f 0b <0f> 0b 55 89 e5 57 56 53 83 ec 08 e8 67 04 00 00 8b 3d 80 ad ec c1
[ 1.835656] EAX: c1e5a601 EBX: c1dac484 ECX: 00000000 EDX: 00000001
[ 1.836463] ESI: 00000000 EDI: 000003f8 EBP: c1e93f9c ESP: c1e93f78
[ 1.838603] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 EFLAGS: 00210002
[ 1.839496] CR0: 80050033 CR2: ffc70000 CR3: 021ce000 CR4: 00000690
[ 1.840300] Kernel panic - not syncing: Fatal exception
[ 1.840969] ------------[ cut here ]------------
[ 1.840970] kernel BUG at kernel/sched/core.c:6652!
Elapsed time: 60
qemu-img create -f qcow2 disk-vm-snb-94-0 256G
qemu-img create -f qcow2 disk-vm-snb-94-1 256G
To reproduce:
# build kernel
cd linux
cp config-5.8.0-rc1-00024-gc3a340f7e7ead .config
make HOSTCC=gcc-4.9 CC=gcc-4.9 ARCH=i386 olddefconfig prepare modules_prepare bzImage
git clone https://github.com/intel/lkp-tests.git
cd lkp-tests
bin/lkp qemu -k <bzImage> job-script # job-script is attached in this email
Thanks,
lkp
On Mon, Jun 29, 2020 at 08:31:27AM +0800, kernel test robot wrote:
> Greeting,
>
> FYI, we noticed the following commit (built with gcc-4.9):
>
> commit: c3a340f7e7eadac7662ab104ceb16432e5a4c6b2 ("sched: Have sched_class_highest define by vmlinux.lds.h")
> [ 1.840970] kernel BUG at kernel/sched/core.c:6652!
W T H
$ readelf -Wa defconfig-build/vmlinux | grep sched_class
62931: c1e62d20 0 NOTYPE GLOBAL DEFAULT 2 __begin_sched_classes
65736: c1e62f40 96 OBJECT GLOBAL DEFAULT 2 stop_sched_class
71813: c1e62dc0 96 OBJECT GLOBAL DEFAULT 2 fair_sched_class
78689: c1e62d40 96 OBJECT GLOBAL DEFAULT 2 idle_sched_class
78953: c1e62fa0 0 NOTYPE GLOBAL DEFAULT 2 __end_sched_classes
79090: c1e62e40 96 OBJECT GLOBAL DEFAULT 2 rt_sched_class
79431: c1e62ec0 96 OBJECT GLOBAL DEFAULT 2 dl_sched_class
$ printf "%d\n" $((0xc1e62dc0 - 0xc1e62d40))
128
So even though the object is 96 bytes in size, has an explicit 32 byte
alignment, the array ends up with a stride of 128 bytes !?!?!
Consistently so with GCC-4.9. Any other GCC I tried does the sane thing.
Full patch included below.
Anybody any clue wth 4.9 is doing crazy things like this?
---
commit c3a340f7e7eadac7662ab104ceb16432e5a4c6b2
Author: Steven Rostedt (VMware) <[email protected]>
Date: Thu Dec 19 16:44:53 2019 -0500
sched: Have sched_class_highest define by vmlinux.lds.h
Now that the sched_class descriptors are defined by the linker script, and
this needs to be aware of the existance of stop_sched_class when SMP is
enabled or not, as it is used as the "highest" priority when defined. Move
the declaration of sched_class_highest to the same location in the linker
script that inserts stop_sched_class, and this will also make it easier to
see what should be defined as the highest class, as this linker script
location defines the priorities as well.
Signed-off-by: Steven Rostedt (VMware) <[email protected]>
Signed-off-by: Peter Zijlstra (Intel) <[email protected]>
Link: https://lkml.kernel.org/r/[email protected]
diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
index 2186d7b01af6..66fb84c3dc7e 100644
--- a/include/asm-generic/vmlinux.lds.h
+++ b/include/asm-generic/vmlinux.lds.h
@@ -114,11 +114,14 @@
* relation to each other.
*/
#define SCHED_DATA \
+ STRUCT_ALIGN(); \
+ __begin_sched_classes = .; \
*(__idle_sched_class) \
*(__fair_sched_class) \
*(__rt_sched_class) \
*(__dl_sched_class) \
- *(__stop_sched_class)
+ *(__stop_sched_class) \
+ __end_sched_classes = .;
/*
* Align to a 32 byte boundary equal to the
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 0208b71bef80..81640fe0eae8 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -6646,6 +6646,14 @@ void __init sched_init(void)
unsigned long ptr = 0;
int i;
+ /* Make sure the linker didn't screw up */
+ BUG_ON(&idle_sched_class + 1 != &fair_sched_class ||
+ &fair_sched_class + 1 != &rt_sched_class ||
+ &rt_sched_class + 1 != &dl_sched_class);
+#ifdef CONFIG_SMP
+ BUG_ON(&dl_sched_class + 1 != &stop_sched_class);
+#endif
+
wait_bit_init();
#ifdef CONFIG_FAIR_GROUP_SCHED
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 336887607b3d..4165c06d1d7b 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -1811,7 +1811,7 @@ struct sched_class {
#ifdef CONFIG_FAIR_GROUP_SCHED
void (*task_change_group)(struct task_struct *p, int type);
#endif
-};
+} __aligned(32); /* STRUCT_ALIGN(), vmlinux.lds.h */
static inline void put_prev_task(struct rq *rq, struct task_struct *prev)
{
@@ -1825,17 +1825,18 @@ static inline void set_next_task(struct rq *rq, struct task_struct *next)
next->sched_class->set_next_task(rq, next, false);
}
-#ifdef CONFIG_SMP
-#define sched_class_highest (&stop_sched_class)
-#else
-#define sched_class_highest (&dl_sched_class)
-#endif
+/* Defined in include/asm-generic/vmlinux.lds.h */
+extern struct sched_class __begin_sched_classes[];
+extern struct sched_class __end_sched_classes[];
+
+#define sched_class_highest (__end_sched_classes - 1)
+#define sched_class_lowest (__begin_sched_classes - 1)
#define for_class_range(class, _from, _to) \
- for (class = (_from); class != (_to); class = class->next)
+ for (class = (_from); class != (_to); class--)
#define for_each_class(class) \
- for_class_range(class, sched_class_highest, NULL)
+ for_class_range(class, sched_class_highest, sched_class_lowest)
extern const struct sched_class stop_sched_class;
extern const struct sched_class dl_sched_class;
On 30/06/2020 14.46, Peter Zijlstra wrote:
> On Mon, Jun 29, 2020 at 08:31:27AM +0800, kernel test robot wrote:
>> Greeting,
>>
>> FYI, we noticed the following commit (built with gcc-4.9):
>>
>> commit: c3a340f7e7eadac7662ab104ceb16432e5a4c6b2 ("sched: Have sched_class_highest define by vmlinux.lds.h")
>
>> [ 1.840970] kernel BUG at kernel/sched/core.c:6652!
>
> W T H
>
> $ readelf -Wa defconfig-build/vmlinux | grep sched_class
> 62931: c1e62d20 0 NOTYPE GLOBAL DEFAULT 2 __begin_sched_classes
> 65736: c1e62f40 96 OBJECT GLOBAL DEFAULT 2 stop_sched_class
> 71813: c1e62dc0 96 OBJECT GLOBAL DEFAULT 2 fair_sched_class
> 78689: c1e62d40 96 OBJECT GLOBAL DEFAULT 2 idle_sched_class
> 78953: c1e62fa0 0 NOTYPE GLOBAL DEFAULT 2 __end_sched_classes
> 79090: c1e62e40 96 OBJECT GLOBAL DEFAULT 2 rt_sched_class
> 79431: c1e62ec0 96 OBJECT GLOBAL DEFAULT 2 dl_sched_class
>
> $ printf "%d\n" $((0xc1e62dc0 - 0xc1e62d40))
> 128
>
> So even though the object is 96 bytes in size, has an explicit 32 byte
> alignment, the array ends up with a stride of 128 bytes !?!?!
>
> Consistently so with GCC-4.9. Any other GCC I tried does the sane thing.
Does that include gcc 4.8, or is it only "anything newer than 4.9"?
>
> Full patch included below.
>
> Anybody any clue wth 4.9 is doing crazy things like this?
Perhaps
https://gcc.gnu.org/onlinedocs/gcc-4.9.4/gcc/Variable-Attributes.html#Variable-Attributes:
When used on a struct, or struct member, the aligned attribute can
only increase the alignment; in order to decrease it, the packed
attribute must be specified as well. When used as part of a typedef, the
aligned attribute can both increase and decrease alignment, and
specifying the packed attribute generates a warning.
is part of the explanation. But this is seriously weird. I don't know
which .config you or the buildbot used, but I took an i386_defconfig
with SMP=n to get a small enough struct sched_class (and disable
retpoline stuff), added
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 81640fe0eae8..53c0d3ba62ba 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -71,6 +71,13 @@ unsigned int sysctl_sched_rt_period = 1000000;
__read_mostly int scheduler_running;
+void foo(void)
+{
+ extern void bar(int);
+ bar(sizeof(struct sched_class));
+ bar(_Alignof(struct sched_class));
+}
+
/*
* part of the period that we allow rt tasks to run in us.
* default: 0.95s
and apparently _Alignof is only 16:
00002c90 <foo>:
2c90: 55 push %ebp
2c91: b8 60 00 00 00 mov $0x60,%eax
2c96: 89 e5 mov %esp,%ebp
2c98: e8 fc ff ff ff call 2c99 <foo+0x9>
2c99: R_386_PC32 bar
2c9d: b8 10 00 00 00 mov $0x10,%eax
2ca2: e8 fc ff ff ff call 2ca3 <foo+0x13>
2ca3: R_386_PC32 bar
Neverthess, readelf -S --wide kernel/sched/fair.o:
Section Headers:
[Nr] Name Type Addr Off Size ES Flg
Lk Inf Al
[35] __fair_sched_class PROGBITS 00000000 002980 000060 00 A
0 0 64
so the section it was put in has an alignment of 64. The generated
assembly is indeed
.globl fair_sched_class
.section __fair_sched_class,"a",@progbits
.align 64
/me goes brew coffee
On Tue, Jun 30, 2020 at 03:55:05PM +0200, Rasmus Villemoes wrote:
> > Consistently so with GCC-4.9. Any other GCC I tried does the sane thing.
>
> Does that include gcc 4.8, or is it only "anything newer than 4.9"?
It includes 4.8 :-)
> so the section it was put in has an alignment of 64. The generated
> assembly is indeed
>
> .globl fair_sched_class
> .section __fair_sched_class,"a",@progbits
> .align 64
>
> /me goes brew coffee
Right.. so I now have the below patch, and with that I get:
62931: c1e62c20 0 NOTYPE GLOBAL DEFAULT 2 __begin_sched_classes
65736: c1e62e40 128 OBJECT GLOBAL DEFAULT 2 stop_sched_class
71813: c1e62cc0 128 OBJECT GLOBAL DEFAULT 2 fair_sched_class
78689: c1e62c40 128 OBJECT GLOBAL DEFAULT 2 idle_sched_class
78953: c1e62ec0 0 NOTYPE GLOBAL DEFAULT 2 __end_sched_classes
79090: c1e62d40 128 OBJECT GLOBAL DEFAULT 2 rt_sched_class
79431: c1e62dc0 128 OBJECT GLOBAL DEFAULT 2 dl_sched_class
Which has me stumped on __begin_sched_classes being on a 32byte edge
(and crashes differently due to that).
Argh!!
diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
index 66fb84c3dc7ee..b4704fb12b2dd 100644
--- a/include/asm-generic/vmlinux.lds.h
+++ b/include/asm-generic/vmlinux.lds.h
@@ -108,6 +108,17 @@
#define SBSS_MAIN .sbss
#endif
+/*
+ * Align to a 32 byte boundary equal to the
+ * alignment gcc 4.5 uses for a struct
+ */
+#if GCC_VERSION >= 40900 && GCC_VERSION < 50000
+#define STRUCT_ALIGNMENT 64
+#else
+#define STRUCT_ALIGNMENT 32
+#endif
+#define STRUCT_ALIGN() . = ALIGN(STRUCT_ALIGNMENT)
+
/*
* The order of the sched class addresses are important, as they are
* used to determine the order of the priority of each sched class in
@@ -123,13 +134,6 @@
*(__stop_sched_class) \
__end_sched_classes = .;
-/*
- * Align to a 32 byte boundary equal to the
- * alignment gcc 4.5 uses for a struct
- */
-#define STRUCT_ALIGNMENT 32
-#define STRUCT_ALIGN() . = ALIGN(STRUCT_ALIGNMENT)
-
/* The actual configuration determine if the init/exit sections
* are handled as text/data or they can be discarded (which
* often happens at runtime)
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 4165c06d1d7bd..33251d0ab62e7 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -67,6 +67,7 @@
#include <linux/tsacct_kern.h>
#include <asm/tlb.h>
+#include <asm-generic/vmlinux.lds.h>
#ifdef CONFIG_PARAVIRT
# include <asm/paravirt.h>
@@ -1811,7 +1812,7 @@ struct sched_class {
#ifdef CONFIG_FAIR_GROUP_SCHED
void (*task_change_group)(struct task_struct *p, int type);
#endif
-} __aligned(32); /* STRUCT_ALIGN(), vmlinux.lds.h */
+} __aligned(STRUCT_ALIGNMENT); /* STRUCT_ALIGN(), vmlinux.lds.h */
static inline void put_prev_task(struct rq *rq, struct task_struct *prev)
{
On Tue, Jun 30, 2020 at 04:02:31PM +0200, Peter Zijlstra wrote:
> On Tue, Jun 30, 2020 at 03:55:05PM +0200, Rasmus Villemoes wrote:
>
> > > Consistently so with GCC-4.9. Any other GCC I tried does the sane thing.
> >
> > Does that include gcc 4.8, or is it only "anything newer than 4.9"?
>
> It includes 4.8 :-)
>
> > so the section it was put in has an alignment of 64. The generated
> > assembly is indeed
> >
> > .globl fair_sched_class
> > .section __fair_sched_class,"a",@progbits
> > .align 64
> >
> > /me goes brew coffee
>
> Right.. so I now have the below patch, and with that I get:
>
> 62931: c1e62c20 0 NOTYPE GLOBAL DEFAULT 2 __begin_sched_classes
> 65736: c1e62e40 128 OBJECT GLOBAL DEFAULT 2 stop_sched_class
> 71813: c1e62cc0 128 OBJECT GLOBAL DEFAULT 2 fair_sched_class
> 78689: c1e62c40 128 OBJECT GLOBAL DEFAULT 2 idle_sched_class
> 78953: c1e62ec0 0 NOTYPE GLOBAL DEFAULT 2 __end_sched_classes
> 79090: c1e62d40 128 OBJECT GLOBAL DEFAULT 2 rt_sched_class
> 79431: c1e62dc0 128 OBJECT GLOBAL DEFAULT 2 dl_sched_class
>
>
> Which has me stumped on __begin_sched_classes being on a 32byte edge
> (and crashes differently due to that).
>
> Argh!!
Steve suggested adding a dummy variable before the lot and this actually
works... But this just cannot be right :-(
---
diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
index 66fb84c3dc7ee..9c0ee5cf73a50 100644
--- a/include/asm-generic/vmlinux.lds.h
+++ b/include/asm-generic/vmlinux.lds.h
@@ -108,6 +108,17 @@
#define SBSS_MAIN .sbss
#endif
+/*
+ * Align to a 32 byte boundary equal to the
+ * alignment gcc 4.5 uses for a struct
+ */
+#if GCC_VERSION >= 40900 && GCC_VERSION < 50000
+#define STRUCT_ALIGNMENT 64
+#else
+#define STRUCT_ALIGNMENT 32
+#endif
+#define STRUCT_ALIGN() . = ALIGN(STRUCT_ALIGNMENT)
+
/*
* The order of the sched class addresses are important, as they are
* used to determine the order of the priority of each sched class in
@@ -115,6 +126,7 @@
*/
#define SCHED_DATA \
STRUCT_ALIGN(); \
+ *(__dummy_sched_class) \
__begin_sched_classes = .; \
*(__idle_sched_class) \
*(__fair_sched_class) \
@@ -123,13 +135,6 @@
*(__stop_sched_class) \
__end_sched_classes = .;
-/*
- * Align to a 32 byte boundary equal to the
- * alignment gcc 4.5 uses for a struct
- */
-#define STRUCT_ALIGNMENT 32
-#define STRUCT_ALIGN() . = ALIGN(STRUCT_ALIGNMENT)
-
/* The actual configuration determine if the init/exit sections
* are handled as text/data or they can be discarded (which
* often happens at runtime)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 81640fe0eae8f..f8535a3438819 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -6641,6 +6641,9 @@ static struct kmem_cache *task_group_cache __read_mostly;
DECLARE_PER_CPU(cpumask_var_t, load_balance_mask);
DECLARE_PER_CPU(cpumask_var_t, select_idle_mask);
+const struct sched_class dummy_sched_class
+ __attribute__((section("__dummy_sched_class")));
+
void __init sched_init(void)
{
unsigned long ptr = 0;
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 4165c06d1d7bd..33251d0ab62e7 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -67,6 +67,7 @@
#include <linux/tsacct_kern.h>
#include <asm/tlb.h>
+#include <asm-generic/vmlinux.lds.h>
#ifdef CONFIG_PARAVIRT
# include <asm/paravirt.h>
@@ -1811,7 +1812,7 @@ struct sched_class {
#ifdef CONFIG_FAIR_GROUP_SCHED
void (*task_change_group)(struct task_struct *p, int type);
#endif
-} __aligned(32); /* STRUCT_ALIGN(), vmlinux.lds.h */
+} __aligned(STRUCT_ALIGNMENT); /* STRUCT_ALIGN(), vmlinux.lds.h */
static inline void put_prev_task(struct rq *rq, struct task_struct *prev)
{
On Tue, Jun 30, 2020 at 04:02:31PM +0200, Peter Zijlstra wrote:
> On Tue, Jun 30, 2020 at 03:55:05PM +0200, Rasmus Villemoes wrote:
>
> > > Consistently so with GCC-4.9. Any other GCC I tried does the sane thing.
> >
> > Does that include gcc 4.8, or is it only "anything newer than 4.9"?
>
> It includes 4.8 :-)
>
> > so the section it was put in has an alignment of 64. The generated
> > assembly is indeed
> >
> > .globl fair_sched_class
> > .section __fair_sched_class,"a",@progbits
> > .align 64
> >
> > /me goes brew coffee
>
> Right.. so I now have the below patch, and with that I get:
>
> 62931: c1e62c20 0 NOTYPE GLOBAL DEFAULT 2 __begin_sched_classes
> 65736: c1e62e40 128 OBJECT GLOBAL DEFAULT 2 stop_sched_class
> 71813: c1e62cc0 128 OBJECT GLOBAL DEFAULT 2 fair_sched_class
> 78689: c1e62c40 128 OBJECT GLOBAL DEFAULT 2 idle_sched_class
> 78953: c1e62ec0 0 NOTYPE GLOBAL DEFAULT 2 __end_sched_classes
> 79090: c1e62d40 128 OBJECT GLOBAL DEFAULT 2 rt_sched_class
> 79431: c1e62dc0 128 OBJECT GLOBAL DEFAULT 2 dl_sched_class
>
>
> Which has me stumped on __begin_sched_classes being on a 32byte edge
> (and crashes differently due to that).
OK, when I look at defconfig-build/arch/x86/kernel/vmlinux.lds I get:
. = ALIGN(32); *(__dummy_sched_class) __begin_sched_classes = .;
So I'm thinking the GCC_VERSION thing works for sched.h but not for
arch//x86/kernel/vmlinux.lds.S, lovely. Let me try and figure out why.
On Tue, Jun 30, 2020 at 02:46:28PM +0200, Peter Zijlstra wrote:
> On Mon, Jun 29, 2020 at 08:31:27AM +0800, kernel test robot wrote:
> > Greeting,
> >
> > FYI, we noticed the following commit (built with gcc-4.9):
> >
> > commit: c3a340f7e7eadac7662ab104ceb16432e5a4c6b2 ("sched: Have sched_class_highest define by vmlinux.lds.h")
>
> > [ 1.840970] kernel BUG at kernel/sched/core.c:6652!
>
> W T H
>
> $ readelf -Wa defconfig-build/vmlinux | grep sched_class
> 62931: c1e62d20 0 NOTYPE GLOBAL DEFAULT 2 __begin_sched_classes
> 65736: c1e62f40 96 OBJECT GLOBAL DEFAULT 2 stop_sched_class
> 71813: c1e62dc0 96 OBJECT GLOBAL DEFAULT 2 fair_sched_class
> 78689: c1e62d40 96 OBJECT GLOBAL DEFAULT 2 idle_sched_class
> 78953: c1e62fa0 0 NOTYPE GLOBAL DEFAULT 2 __end_sched_classes
> 79090: c1e62e40 96 OBJECT GLOBAL DEFAULT 2 rt_sched_class
> 79431: c1e62ec0 96 OBJECT GLOBAL DEFAULT 2 dl_sched_class
>
> $ printf "%d\n" $((0xc1e62dc0 - 0xc1e62d40))
> 128
>
> So even though the object is 96 bytes in size, has an explicit 32 byte
> alignment, the array ends up with a stride of 128 bytes !?!?!
>
> Consistently so with GCC-4.9. Any other GCC I tried does the sane thing.
>
> Full patch included below.
>
> Anybody any clue wth 4.9 is doing crazy things like this?
>
> ---
This seems to make everything work, it builds and boots for 4.9 and
builds x86_64-defconfig with clang11 (just to check a !GCC compiler).
diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
index 66fb84c3dc7ee..49a9aaa1e2424 100644
--- a/include/asm-generic/vmlinux.lds.h
+++ b/include/asm-generic/vmlinux.lds.h
@@ -108,6 +108,17 @@
#define SBSS_MAIN .sbss
#endif
+/*
+ * Align to a 32 byte boundary equal to the
+ * alignment gcc 4.5 uses for a struct
+ */
+#if __GNUC__ == 4 && __GNUC_MINOR__ == 9
+#define STRUCT_ALIGNMENT 64
+#else
+#define STRUCT_ALIGNMENT 32
+#endif
+#define STRUCT_ALIGN() . = ALIGN(STRUCT_ALIGNMENT)
+
/*
* The order of the sched class addresses are important, as they are
* used to determine the order of the priority of each sched class in
@@ -123,13 +134,6 @@
*(__stop_sched_class) \
__end_sched_classes = .;
-/*
- * Align to a 32 byte boundary equal to the
- * alignment gcc 4.5 uses for a struct
- */
-#define STRUCT_ALIGNMENT 32
-#define STRUCT_ALIGN() . = ALIGN(STRUCT_ALIGNMENT)
-
/* The actual configuration determine if the init/exit sections
* are handled as text/data or they can be discarded (which
* often happens at runtime)
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 4165c06d1d7bd..33251d0ab62e7 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -67,6 +67,7 @@
#include <linux/tsacct_kern.h>
#include <asm/tlb.h>
+#include <asm-generic/vmlinux.lds.h>
#ifdef CONFIG_PARAVIRT
# include <asm/paravirt.h>
@@ -1811,7 +1812,7 @@ struct sched_class {
#ifdef CONFIG_FAIR_GROUP_SCHED
void (*task_change_group)(struct task_struct *p, int type);
#endif
-} __aligned(32); /* STRUCT_ALIGN(), vmlinux.lds.h */
+} __aligned(STRUCT_ALIGNMENT); /* STRUCT_ALIGN(), vmlinux.lds.h */
static inline void put_prev_task(struct rq *rq, struct task_struct *prev)
{
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 85c2ce9104eb93517db2037699471c517e81f9b4
Gitweb: https://git.kernel.org/tip/85c2ce9104eb93517db2037699471c517e81f9b4
Author: Peter Zijlstra <[email protected]>
AuthorDate: Tue, 30 Jun 2020 16:49:05 +02:00
Committer: Peter Zijlstra <[email protected]>
CommitterDate: Wed, 08 Jul 2020 11:39:00 +02:00
sched, vmlinux.lds: Increase STRUCT_ALIGNMENT to 64 bytes for GCC-4.9
For some mysterious reason GCC-4.9 has a 64 byte section alignment for
structures, all other GCC versions (and Clang) tested (including 4.8
and 5.0) are fine with the 32 bytes alignment.
Getting this right is important for the new SCHED_DATA macro that
creates an explicitly ordered array of 'struct sched_class' in the
linker script and expect pointer arithmetic to work.
Fixes: c3a340f7e7ea ("sched: Have sched_class_highest define by vmlinux.lds.h")
Reported-by: kernel test robot <[email protected]>
Signed-off-by: Peter Zijlstra (Intel) <[email protected]>
Link: https://lkml.kernel.org/r/[email protected]
---
include/asm-generic/vmlinux.lds.h | 18 +++++++++++-------
kernel/sched/sched.h | 3 ++-
2 files changed, 13 insertions(+), 8 deletions(-)
diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
index 66fb84c..3ceb4b7 100644
--- a/include/asm-generic/vmlinux.lds.h
+++ b/include/asm-generic/vmlinux.lds.h
@@ -109,6 +109,17 @@
#endif
/*
+ * GCC 4.5 and later have a 32 bytes section alignment for structures.
+ * Except GCC 4.9, that feels the need to align on 64 bytes.
+ */
+#if __GNUC__ == 4 && __GNUC_MINOR__ == 9
+#define STRUCT_ALIGNMENT 64
+#else
+#define STRUCT_ALIGNMENT 32
+#endif
+#define STRUCT_ALIGN() . = ALIGN(STRUCT_ALIGNMENT)
+
+/*
* The order of the sched class addresses are important, as they are
* used to determine the order of the priority of each sched class in
* relation to each other.
@@ -123,13 +134,6 @@
*(__stop_sched_class) \
__end_sched_classes = .;
-/*
- * Align to a 32 byte boundary equal to the
- * alignment gcc 4.5 uses for a struct
- */
-#define STRUCT_ALIGNMENT 32
-#define STRUCT_ALIGN() . = ALIGN(STRUCT_ALIGNMENT)
-
/* The actual configuration determine if the init/exit sections
* are handled as text/data or they can be discarded (which
* often happens at runtime)
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 5aa6661..9bef2dd 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -67,6 +67,7 @@
#include <linux/tsacct_kern.h>
#include <asm/tlb.h>
+#include <asm-generic/vmlinux.lds.h>
#ifdef CONFIG_PARAVIRT
# include <asm/paravirt.h>
@@ -1810,7 +1811,7 @@ struct sched_class {
#ifdef CONFIG_FAIR_GROUP_SCHED
void (*task_change_group)(struct task_struct *p, int type);
#endif
-} __aligned(32); /* STRUCT_ALIGN(), vmlinux.lds.h */
+} __aligned(STRUCT_ALIGNMENT); /* STRUCT_ALIGN(), vmlinux.lds.h */
static inline void put_prev_task(struct rq *rq, struct task_struct *prev)
{
On 6/30/20 7:49 AM, Peter Zijlstra wrote:
> On Tue, Jun 30, 2020 at 02:46:28PM +0200, Peter Zijlstra wrote:
>> On Mon, Jun 29, 2020 at 08:31:27AM +0800, kernel test robot wrote:
>>> Greeting,
>>>
>>> FYI, we noticed the following commit (built with gcc-4.9):
>>>
>>> commit: c3a340f7e7eadac7662ab104ceb16432e5a4c6b2 ("sched: Have sched_class_highest define by vmlinux.lds.h")
>>
>>> [ 1.840970] kernel BUG at kernel/sched/core.c:6652!
>>
>> W T H
>>
>> $ readelf -Wa defconfig-build/vmlinux | grep sched_class
>> 62931: c1e62d20 0 NOTYPE GLOBAL DEFAULT 2 __begin_sched_classes
>> 65736: c1e62f40 96 OBJECT GLOBAL DEFAULT 2 stop_sched_class
>> 71813: c1e62dc0 96 OBJECT GLOBAL DEFAULT 2 fair_sched_class
>> 78689: c1e62d40 96 OBJECT GLOBAL DEFAULT 2 idle_sched_class
>> 78953: c1e62fa0 0 NOTYPE GLOBAL DEFAULT 2 __end_sched_classes
>> 79090: c1e62e40 96 OBJECT GLOBAL DEFAULT 2 rt_sched_class
>> 79431: c1e62ec0 96 OBJECT GLOBAL DEFAULT 2 dl_sched_class
>>
>> $ printf "%d\n" $((0xc1e62dc0 - 0xc1e62d40))
>> 128
>>
>> So even though the object is 96 bytes in size, has an explicit 32 byte
>> alignment, the array ends up with a stride of 128 bytes !?!?!
>>
>> Consistently so with GCC-4.9. Any other GCC I tried does the sane thing.
>>
>> Full patch included below.
>>
>> Anybody any clue wth 4.9 is doing crazy things like this?
>>
>> ---
>
> This seems to make everything work, it builds and boots for 4.9 and
> builds x86_64-defconfig with clang11 (just to check a !GCC compiler).
Hi Peter,
This patch causes all files under kernel/sched/* that include sched.h to
be rebuilt whenever the value of CONFIG_BLK_DEV_INITRD. There are at
least two build systems (buildroot and OpenWrt) that toggle this
configuration value in order to produce a kernel image without an
initramfs, and one with.
On ARM we get all of these to be needlessly rebuilt:
CC kernel/sched/core.o
CC kernel/sched/loadavg.o
CC kernel/sched/clock.o
CC kernel/sched/cputime.o
CC kernel/sched/idle.o
CC kernel/sched/fair.o
CC kernel/sched/rt.o
CC kernel/sched/deadline.o
CC kernel/sched/wait.o
CC kernel/sched/wait_bit.o
CC kernel/sched/swait.o
CC kernel/sched/completion.o
CC kernel/sched/cpupri.o
CC kernel/sched/cpudeadline.o
CC kernel/sched/topology.o
CC kernel/sched/stop_task.o
CC kernel/sched/pelt.o
CC kernel/sched/debug.o
CC kernel/sched/cpufreq.o
CC kernel/sched/membarrier.o
Short of moving the STRUCT_ALIGNMENT to a separate header that would not
be subject to any configuration key change, can you think of a good way
to avoid these rebuilds, including for architectures like ARM that ship
their own vmlinux.lds.h? I would not say this is a bug, but it is
definitively an inconvenience.
Thanks!
--
Florian