Hi all,
Changes since 20081210:
New tree:
nommu
Undropped tree:
sound
Dropped trees (temporarily):
v4l-dvb (build problem)
mtd (difficult conflicts)
drm (build problem)
semaphore-removal (due to unfixed conflicts against Linus' tree)
cpu_alloc (build problem)
perfmon3 (concerns from the x86 team)
audit (difficult conflicts)
nommu (build problem)
staging (build failure)
The driver-core tree gained a build failure that needed a commit reverted.
The ftrace tree gained a conflict against Linus' tree.
The pci tree gained a conflict against the driver-core tree.
The mtd tree gained 3 conflicts against the arm tree which I could not
easily resolve, so it was dropped.
The ttydev tree gained a conflict against the async_tx tree requiring a
commit from the async_tx tree to be reverted.
The nommu tree gained conflicts against the slab and kmemcheck trees and
also a build failure so it was dropped.
----------------------------------------------------------------------------
I have created today's linux-next tree at
git://git.kernel.org/pub/scm/linux/kernel/git/sfr/linux-next.git
(patches at
http://www.kernel.org/pub/linux/kernel/people/sfr/linux-next/). If you
are tracking the linux-next tree using git, you should not use "git pull"
to do so as that will try to merge the new linux-next release with the
old one. You should use "git fetch" as mentioned in the FAQ on the wiki
(see below).
You can see which trees have been included by looking in the Next/Trees
file in the source. There are also quilt-import.log and merge.log files
in the Next directory. Between each merge, the tree was built with
a ppc64_defconfig for powerpc and an allmodconfig for x86_64. After the
final fixups (if any), it is also built with powerpc allnoconfig,
44x_defconfig and allyesconfig (minus CONFIG_PROFILE_ALL_BRANCHES) and
i386, sparc and sparc64 defconfig.
Below is a summary of the state of the merge.
We are up to 132 trees (counting Linus' and 15 trees of patches pending for
Linus' tree), more are welcome (even if they are currently empty).
Thanks to those who have contributed, and to those who haven't, please do.
Status of my local build tests will be at
http://kisskb.ellerman.id.au/linux-next . If maintainers want to give
advice about cross compilers/configs that work, we are always open to add
more builds.
Thanks to Jan Dittmer for adding the linux-next tree to his build tests
at http://l4x.org/k/ , the guys at http://test.kernel.org/ and Randy
Dunlap for doing many randconfig builds.
There is a wiki covering stuff to do with linux-next at
http://linux.f-seidel.de/linux-next/pmwiki/ . Thanks to Frank Seidel.
--
Cheers,
Stephen Rothwell [email protected]
$ git checkout master
$ git reset --hard stable
Merging origin/master
Merging arm-current/master
Merging m68k-current/for-linus
Merging powerpc-merge/merge
Merging sparc-current/master
Merging scsi-rc-fixes/master
Merging net-current/master
Merging sound-current/for-linus
Merging pci-current/for-linus
Merging wireless-current/master
Merging kbuild-current/master
Merging quilt/driver-core.current
Merging quilt/usb.current
Merging cpufreq-current/fixes
Merging input-current/for-linus
Merging md-current/for-2.6.26
Merging audit-current/for-linus
Merging dwmw2/master
Merging arm/devel
Merging avr32/avr32-arch
Merging blackfin/for-linus
Merging cris/for-next
Merging ia64/test
Merging m68k/for-next
Merging m68knommu/for-next
Merging mips/mips-for-linux-next
Merging parisc/master
Merging powerpc/next
Merging 4xx/next
Merging galak/next
Merging pxa/for-next
Merging s390/features
CONFLICT (content): Merge conflict in drivers/char/Makefile
Merging sh/master
Merging sparc/master
Merging x86/auto-x86-next
Merging xtensa/master
Merging quilt/driver-core
CONFLICT (content): Merge conflict in arch/powerpc/platforms/85xx/mpc85xx_mds.c
Created commit 1a20fa9: Revert "mtd: struct device - replace bus_id with dev_name(), dev_set_name()"
Merging quilt/usb
Merging tip-core/auto-core-next
Merging cpus4096/auto-cpus4096-next
Merging ftrace/auto-ftrace-next
CONFLICT (content): Merge conflict in include/linux/ftrace.h
CONFLICT (content): Merge conflict in scripts/recordmcount.pl
Merging genirq/auto-genirq-next
Merging safe-poison-pointers/auto-safe-poison-pointers-next
Merging sched/auto-sched-next
Merging stackprotector/auto-stackprotector-next
Merging timers/auto-timers-next
Merging pci/linux-next
CONFLICT (content): Merge conflict in drivers/pci/pci-driver.c
Merging quilt/device-mapper
Merging hid/for-next
Merging quilt/i2c
Merging quilt/jdelvare-hwmon
Merging quilt/kernel-doc
Merging jfs/next
Merging kbuild/master
Merging quilt/ide
Merging libata/NEXT
Merging nfs/linux-next
Merging xfs/master
Merging infiniband/for-next
Merging acpi/test
Merging nfsd/nfsd-next
Merging ieee1394/for-next
Merging ubi/linux-next
Merging kvm/master
CONFLICT (content): Merge conflict in arch/x86/kernel/reboot.c
Merging dlm/next
Merging scsi/master
Merging ocfs2/linux-next
Merging ext4/next
Merging async_tx/next
Merging udf/for_next
Merging net/master
CONFLICT (delete/modify): arch/sparc64/kernel/idprom.c deleted in HEAD and modified in net/master. Version net/master of arch/sparc64/kernel/idprom.c left in tree.
CONFLICT (content): Merge conflict in fs/lockd/host.c
CONFLICT (content): Merge conflict in fs/lockd/mon.c
$ git rm -f arch/sparc64/kernel/idprom.c
Applying: net: async_tx merge fix
Merging mtd/master
CONFLICT (content): Merge conflict in arch/arm/mach-pxa/corgi.c
CONFLICT (content): Merge conflict in arch/arm/mach-pxa/poodle.c
CONFLICT (content): Merge conflict in arch/arm/mach-pxa/spitz.c
$ git reset --hard
Merging wireless/master
Merging crypto/master
Merging vfs/for-next
Merging sound/for-next
Merging cpufreq/next
Merging v9fs/for-next
Merging rr_cpumask/master
Merging quilt/rr
CONFLICT (content): Merge conflict in arch/x86/kernel/setup.c
CONFLICT (content): Merge conflict in kernel/cpu.c
Merging cifs/master
CONFLICT (content): Merge conflict in fs/cifs/connect.c
Applying: cifs: update for new IP4/6 address printing
Merging mmc/next
Merging gfs2/master
Merging input/next
Merging semaphore/semaphore
Merging bkl-removal/bkl-removal
CONFLICT (content): Merge conflict in fs/ioctl.c
Merging ubifs/linux-next
Merging lsm/for-next
Merging block/for-next
Merging embedded/master
Merging firmware/master
CONFLICT (content): Merge conflict in drivers/net/tg3.c
CONFLICT (content): Merge conflict in firmware/Makefile
CONFLICT (content): Merge conflict in firmware/WHENCE
Merging pcmcia/master
Merging battery/master
Merging leds/for-mm
Merging backlight/for-mm
Merging kgdb/kgdb-next
Merging slab/for-next
Merging uclinux/for-next
Merging md/for-next
Merging kmemcheck/auto-kmemcheck-next
CONFLICT (content): Merge conflict in MAINTAINERS
CONFLICT (content): Merge conflict in arch/x86/mm/Makefile
CONFLICT (content): Merge conflict in mm/slab.c
CONFLICT (content): Merge conflict in mm/slub.c
Merging generic-ipi/auto-generic-ipi-next
Merging mfd/for-next
Merging hdlc/hdlc-next
Merging voltage/for-next
Merging security-testing/next
CONFLICT (content): Merge conflict in drivers/char/tty_audit.c
CONFLICT (content): Merge conflict in fs/cifs/cifs_fs_sb.h
CONFLICT (content): Merge conflict in fs/ocfs2/namei.c
CONFLICT (content): Merge conflict in fs/xfs/linux-2.6/xfs_cred.h
CONFLICT (content): Merge conflict in fs/xfs/linux-2.6/xfs_globals.h
CONFLICT (content): Merge conflict in fs/xfs/linux-2.6/xfs_ioctl.c
CONFLICT (content): Merge conflict in fs/xfs/xfs_vnodeops.h
Merging lblnet/master
CONFLICT (content): Merge conflict in Documentation/feature-removal-schedule.txt
Merging quilt/ttydev
CONFLICT (content): Merge conflict in drivers/serial/8250.c
CONFLICT (content): Merge conflict in fs/devpts/inode.c
$ git reset --hard
Created commit a59876f: Revert "iop13xx: workaround errata that causes uart interrupts to be missed"
CONFLICT (content): Merge conflict in fs/devpts/inode.c
Merging agp/agp-next
Merging oprofile/auto-oprofile-next
Merging fastboot/auto-fastboot-next
Merging sparseirq/auto-sparseirq-next
Merging iommu/auto-iommu-next
Merging uwb/for-upstream
Merging watchdog/master
CONFLICT (content): Merge conflict in drivers/watchdog/iTCO_wdt.c
Merging proc/proc
Merging bdev/master
Merging dwmw2-iommu/master
Merging cputime/cputime
CONFLICT (content): Merge conflict in kernel/sched.c
Merging osd/linux-next
Merging fatfs/master
Merging fuse/for-next
Merging jc_docs/docs-next
Merging nommu/master
CONFLICT (content): Merge conflict in Documentation/sysctl/vm.txt
CONFLICT (content): Merge conflict in kernel/fork.c
Merging trivial/for-next
Merging quilt/staging
CONFLICT (delete/modify): drivers/staging/winbond/linux/wbusb.c deleted in quilt/staging and modified in HEAD. Version HEAD of drivers/staging/winbond/linux/wbusb.c left in tree.
CONFLICT (content): Merge conflict in drivers/staging/wlan-ng/p80211netdev.c
CONFLICT (content): Merge conflict in drivers/staging/wlan-ng/p80211wext.c
$ git rm -f drivers/staging/winbond/linux/wbusb.c
$ git reset --hard HEAD^
Merging scsi-post-merge/master
$ git reset --hard HEAD^^
Merging trivial/for-next
On Thu, Dec 11, 2008 at 12:04, Stephen Rothwell <[email protected]> wrote:
> Hi all,
>
> Changes since 20081210:
>
> New tree:
> nommu
>
> Undropped tree:
> sound
>
> Dropped trees (temporarily):
> v4l-dvb (build problem)
> mtd (difficult conflicts)
> drm (build problem)
> semaphore-removal (due to unfixed conflicts against Linus' tree)
> cpu_alloc (build problem)
> perfmon3 (concerns from the x86 team)
> audit (difficult conflicts)
> nommu (build problem)
> staging (build failure)
>
> The driver-core tree gained a build failure that needed a commit reverted.
>
> The ftrace tree gained a conflict against Linus' tree.
>
> The pci tree gained a conflict against the driver-core tree.
>
> The mtd tree gained 3 conflicts against the arm tree which I could not
> easily resolve, so it was dropped.
>
> The ttydev tree gained a conflict against the async_tx tree requiring a
> commit from the async_tx tree to be reverted.
>
> The nommu tree gained conflicts against the slab and kmemcheck trees and
> also a build failure so it was dropped.
>
> ----------------------------------------------------------------------------
Hi.
I'm seeing this warning early in boot logs. It does not appear on 2.6.28-rc7.
Not sure how long it's been around. Haven't built -next for some time.
[ 0.004000] Intel machine check reporting enabled on CPU#0.
[ 0.004000] using mwait in idle threads.
[ 0.004000] Checking 'hlt' instruction... <4>------------[ cut here
]------------
[ 0.004167] WARNING: at kernel/sched.c:4364 sub_preempt_count+0xae/0xc0()
[ 0.004266] Hardware name: HP Compaq nx7300 (GB848ES#ACB)
[ 0.004361] Modules linked in:
[ 0.004497] Pid: 0, comm: swapper Not tainted 2.6.28-rc8-next-20081211 #117
[ 0.004595] Call Trace:
[ 0.004689] [<c01324d6>] warn_slowpath+0x86/0xa0
[ 0.004789] [<c0155d00>] ? check_usage_forwards+0x10/0xb0
[ 0.004886] [<c015394a>] ? save_trace+0x3a/0xa0
[ 0.004981] [<c015742d>] ? mark_lock+0x37d/0xe00
[ 0.005076] [<c01580f9>] ? __lock_acquire+0x249/0x610
[ 0.005175] [<c04a3a02>] ? _spin_unlock_irq+0x22/0x50
[ 0.005272] [<c0158e50>] ? trace_hardirqs_on_caller+0x70/0x1a0
[ 0.005369] [<c04a3a0d>] ? _spin_unlock_irq+0x2d/0x50
[ 0.005465] [<c0124bfe>] sub_preempt_count+0xae/0xc0
[ 0.005564] [<c0137012>] _local_bh_enable+0x52/0xc0
[ 0.005661] [<c013725f>] __do_softirq+0x11f/0x170
[ 0.005756] [<c0137140>] ? __do_softirq+0x0/0x170
[ 0.005851] <IRQ> [<c0137729>] ? irq_exit+0x89/0xa0
[ 0.005993] [<c01059ed>] ? do_IRQ+0xad/0x120
[ 0.006088] [<c0103aac>] ? common_interrupt+0x2c/0x34
[ 0.006184] [<c013007b>] ? mmput+0x2b/0xc0
[ 0.006281] [<c06735a8>] ? check_bugs+0xb8/0xe0
[ 0.006379] [<c066b7ea>] ? start_kernel+0x26a/0x310
[ 0.006475] [<c066b270>] ? unknown_bootoption+0x0/0x210
[ 0.006572] [<c066b077>] ? __init_begin+0x77/0xb0
[ 0.006674] ---[ end trace 4eaa2a86a8e2da22 ]---
[ 0.016004] OK.
[ 0.016560] ACPI: Core revision 20081031
[ 0.044495] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
Config and full dmesg attached.
On Thu, Dec 11, 2008 at 15:40, Alexey Zaytsev <[email protected]> wrote:
> On Thu, Dec 11, 2008 at 12:04, Stephen Rothwell <[email protected]> wrote:
>> Hi all,
>>
>> Changes since 20081210:
>>
>> New tree:
>> nommu
>>
>> Undropped tree:
>> sound
>>
>> Dropped trees (temporarily):
>> v4l-dvb (build problem)
>> mtd (difficult conflicts)
>> drm (build problem)
>> semaphore-removal (due to unfixed conflicts against Linus' tree)
>> cpu_alloc (build problem)
>> perfmon3 (concerns from the x86 team)
>> audit (difficult conflicts)
>> nommu (build problem)
>> staging (build failure)
>>
>> The driver-core tree gained a build failure that needed a commit reverted.
>>
>> The ftrace tree gained a conflict against Linus' tree.
>>
>> The pci tree gained a conflict against the driver-core tree.
>>
>> The mtd tree gained 3 conflicts against the arm tree which I could not
>> easily resolve, so it was dropped.
>>
>> The ttydev tree gained a conflict against the async_tx tree requiring a
>> commit from the async_tx tree to be reverted.
>>
>> The nommu tree gained conflicts against the slab and kmemcheck trees and
>> also a build failure so it was dropped.
>>
>> ----------------------------------------------------------------------------
>
> Hi.
>
> I'm seeing this warning early in boot logs. It does not appear on 2.6.28-rc7.
> Not sure how long it's been around. Haven't built -next for some time.
>
> [ 0.004000] Intel machine check reporting enabled on CPU#0.
> [ 0.004000] using mwait in idle threads.
> [ 0.004000] Checking 'hlt' instruction... <4>------------[ cut here
> ]------------
> [ 0.004167] WARNING: at kernel/sched.c:4364 sub_preempt_count+0xae/0xc0()
> [ 0.004266] Hardware name: HP Compaq nx7300 (GB848ES#ACB)
> [ 0.004361] Modules linked in:
> [ 0.004497] Pid: 0, comm: swapper Not tainted 2.6.28-rc8-next-20081211 #117
> [ 0.004595] Call Trace:
> [ 0.004689] [<c01324d6>] warn_slowpath+0x86/0xa0
> [ 0.004789] [<c0155d00>] ? check_usage_forwards+0x10/0xb0
> [ 0.004886] [<c015394a>] ? save_trace+0x3a/0xa0
> [ 0.004981] [<c015742d>] ? mark_lock+0x37d/0xe00
> [ 0.005076] [<c01580f9>] ? __lock_acquire+0x249/0x610
> [ 0.005175] [<c04a3a02>] ? _spin_unlock_irq+0x22/0x50
> [ 0.005272] [<c0158e50>] ? trace_hardirqs_on_caller+0x70/0x1a0
> [ 0.005369] [<c04a3a0d>] ? _spin_unlock_irq+0x2d/0x50
> [ 0.005465] [<c0124bfe>] sub_preempt_count+0xae/0xc0
> [ 0.005564] [<c0137012>] _local_bh_enable+0x52/0xc0
> [ 0.005661] [<c013725f>] __do_softirq+0x11f/0x170
> [ 0.005756] [<c0137140>] ? __do_softirq+0x0/0x170
> [ 0.005851] <IRQ> [<c0137729>] ? irq_exit+0x89/0xa0
> [ 0.005993] [<c01059ed>] ? do_IRQ+0xad/0x120
> [ 0.006088] [<c0103aac>] ? common_interrupt+0x2c/0x34
> [ 0.006184] [<c013007b>] ? mmput+0x2b/0xc0
> [ 0.006281] [<c06735a8>] ? check_bugs+0xb8/0xe0
> [ 0.006379] [<c066b7ea>] ? start_kernel+0x26a/0x310
> [ 0.006475] [<c066b270>] ? unknown_bootoption+0x0/0x210
> [ 0.006572] [<c066b077>] ? __init_begin+0x77/0xb0
> [ 0.006674] ---[ end trace 4eaa2a86a8e2da22 ]---
> [ 0.016004] OK.
> [ 0.016560] ACPI: Core revision 20081031
> [ 0.044495] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
>
> Config and full dmesg attached.
>
The warning can also be reproduced in qemu, so it was easy to bisect.
commit 7317d7b87edb41a9135e30be1ec3f7ef817c53dd
Author: Nick Piggin <[email protected]>
Date: Tue Sep 30 20:50:27 2008 +1000
sched: improve preempt debugging
This patch helped me out with a problem I recently had....
Basically, when the kernel lock is held, then preempt_count
underflow does not
get detected until it is released which may be a long time (and arbitrarily,
eg at different points it may be rescheduled). If the bkl is released at
schedule, the resulting output is actually fairly cryptic...
With any other lock that elevates preempt_count, it is illegal to schedule
under it (which would get found pretty quickly). bkl allows scheduling with
preempt_count elevated, which makes underflows hard to debug.
Signed-off-by: Ingo Molnar <[email protected]>
I understand that not this particular commit is buggy, but at least
I've got someone to add to the CC. ;)
Also the author's e-mail looks suspicious.
* Alexey Zaytsev <[email protected]> wrote:
> On Thu, Dec 11, 2008 at 15:40, Alexey Zaytsev <[email protected]> wrote:
> > On Thu, Dec 11, 2008 at 12:04, Stephen Rothwell <[email protected]> wrote:
> >> Hi all,
> >>
> >> Changes since 20081210:
> >>
> >> New tree:
> >> nommu
> >>
> >> Undropped tree:
> >> sound
> >>
> >> Dropped trees (temporarily):
> >> v4l-dvb (build problem)
> >> mtd (difficult conflicts)
> >> drm (build problem)
> >> semaphore-removal (due to unfixed conflicts against Linus' tree)
> >> cpu_alloc (build problem)
> >> perfmon3 (concerns from the x86 team)
> >> audit (difficult conflicts)
> >> nommu (build problem)
> >> staging (build failure)
> >>
> >> The driver-core tree gained a build failure that needed a commit reverted.
> >>
> >> The ftrace tree gained a conflict against Linus' tree.
> >>
> >> The pci tree gained a conflict against the driver-core tree.
> >>
> >> The mtd tree gained 3 conflicts against the arm tree which I could not
> >> easily resolve, so it was dropped.
> >>
> >> The ttydev tree gained a conflict against the async_tx tree requiring a
> >> commit from the async_tx tree to be reverted.
> >>
> >> The nommu tree gained conflicts against the slab and kmemcheck trees and
> >> also a build failure so it was dropped.
> >>
> >> ----------------------------------------------------------------------------
> >
> > Hi.
> >
> > I'm seeing this warning early in boot logs. It does not appear on 2.6.28-rc7.
> > Not sure how long it's been around. Haven't built -next for some time.
> >
> > [ 0.004000] Intel machine check reporting enabled on CPU#0.
> > [ 0.004000] using mwait in idle threads.
> > [ 0.004000] Checking 'hlt' instruction... <4>------------[ cut here
> > ]------------
> > [ 0.004167] WARNING: at kernel/sched.c:4364 sub_preempt_count+0xae/0xc0()
> > [ 0.004266] Hardware name: HP Compaq nx7300 (GB848ES#ACB)
> > [ 0.004361] Modules linked in:
> > [ 0.004497] Pid: 0, comm: swapper Not tainted 2.6.28-rc8-next-20081211 #117
> > [ 0.004595] Call Trace:
> > [ 0.004689] [<c01324d6>] warn_slowpath+0x86/0xa0
> > [ 0.004789] [<c0155d00>] ? check_usage_forwards+0x10/0xb0
> > [ 0.004886] [<c015394a>] ? save_trace+0x3a/0xa0
> > [ 0.004981] [<c015742d>] ? mark_lock+0x37d/0xe00
> > [ 0.005076] [<c01580f9>] ? __lock_acquire+0x249/0x610
> > [ 0.005175] [<c04a3a02>] ? _spin_unlock_irq+0x22/0x50
> > [ 0.005272] [<c0158e50>] ? trace_hardirqs_on_caller+0x70/0x1a0
> > [ 0.005369] [<c04a3a0d>] ? _spin_unlock_irq+0x2d/0x50
> > [ 0.005465] [<c0124bfe>] sub_preempt_count+0xae/0xc0
> > [ 0.005564] [<c0137012>] _local_bh_enable+0x52/0xc0
> > [ 0.005661] [<c013725f>] __do_softirq+0x11f/0x170
> > [ 0.005756] [<c0137140>] ? __do_softirq+0x0/0x170
> > [ 0.005851] <IRQ> [<c0137729>] ? irq_exit+0x89/0xa0
> > [ 0.005993] [<c01059ed>] ? do_IRQ+0xad/0x120
> > [ 0.006088] [<c0103aac>] ? common_interrupt+0x2c/0x34
> > [ 0.006184] [<c013007b>] ? mmput+0x2b/0xc0
> > [ 0.006281] [<c06735a8>] ? check_bugs+0xb8/0xe0
> > [ 0.006379] [<c066b7ea>] ? start_kernel+0x26a/0x310
> > [ 0.006475] [<c066b270>] ? unknown_bootoption+0x0/0x210
> > [ 0.006572] [<c066b077>] ? __init_begin+0x77/0xb0
> > [ 0.006674] ---[ end trace 4eaa2a86a8e2da22 ]---
> > [ 0.016004] OK.
> > [ 0.016560] ACPI: Core revision 20081031
> > [ 0.044495] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
> >
> > Config and full dmesg attached.
> >
>
> The warning can also be reproduced in qemu, so it was easy to bisect.
>
> commit 7317d7b87edb41a9135e30be1ec3f7ef817c53dd
> Author: Nick Piggin <[email protected]>
> Date: Tue Sep 30 20:50:27 2008 +1000
>
> sched: improve preempt debugging
>
> This patch helped me out with a problem I recently had....
>
> Basically, when the kernel lock is held, then preempt_count
> underflow does not
> get detected until it is released which may be a long time (and arbitrarily,
> eg at different points it may be rescheduled). If the bkl is released at
> schedule, the resulting output is actually fairly cryptic...
>
> With any other lock that elevates preempt_count, it is illegal to schedule
> under it (which would get found pretty quickly). bkl allows scheduling with
> preempt_count elevated, which makes underflows hard to debug.
>
> Signed-off-by: Ingo Molnar <[email protected]>
>
> I understand that not this particular commit is buggy, but at least
> I've got someone to add to the CC. ;)
>
> Also the author's e-mail looks suspicious.
Suspicious in what way?
Ingo
On Wed, Dec 17, 2008 at 00:08, Ingo Molnar <[email protected]> wrote:
>>
>> The warning can also be reproduced in qemu, so it was easy to bisect.
>>
>> commit 7317d7b87edb41a9135e30be1ec3f7ef817c53dd
>> Author: Nick Piggin <[email protected]>
>> Date: Tue Sep 30 20:50:27 2008 +1000
>>
>> sched: improve preempt debugging
>>
>> This patch helped me out with a problem I recently had....
>>
>> Basically, when the kernel lock is held, then preempt_count
>> underflow does not
>> get detected until it is released which may be a long time (and arbitrarily,
>> eg at different points it may be rescheduled). If the bkl is released at
>> schedule, the resulting output is actually fairly cryptic...
>>
>> With any other lock that elevates preempt_count, it is illegal to schedule
>> under it (which would get found pretty quickly). bkl allows scheduling with
>> preempt_count elevated, which makes underflows hard to debug.
>>
>> Signed-off-by: Ingo Molnar <[email protected]>
>>
>> I understand that not this particular commit is buggy, but at least
>> I've got someone to add to the CC. ;)
>>
>> Also the author's e-mail looks suspicious.
>
> Suspicious in what way?
Does not appear in signed-off-by.
>
> Ingo
>
* Alexey Zaytsev <[email protected]> wrote:
> On Wed, Dec 17, 2008 at 00:08, Ingo Molnar <[email protected]> wrote:
>
> >>
> >> The warning can also be reproduced in qemu, so it was easy to bisect.
> >>
> >> commit 7317d7b87edb41a9135e30be1ec3f7ef817c53dd
> >> Author: Nick Piggin <[email protected]>
> >> Date: Tue Sep 30 20:50:27 2008 +1000
> >>
> >> sched: improve preempt debugging
> >>
> >> This patch helped me out with a problem I recently had....
> >>
> >> Basically, when the kernel lock is held, then preempt_count
> >> underflow does not
> >> get detected until it is released which may be a long time (and arbitrarily,
> >> eg at different points it may be rescheduled). If the bkl is released at
> >> schedule, the resulting output is actually fairly cryptic...
> >>
> >> With any other lock that elevates preempt_count, it is illegal to schedule
> >> under it (which would get found pretty quickly). bkl allows scheduling with
> >> preempt_count elevated, which makes underflows hard to debug.
> >>
> >> Signed-off-by: Ingo Molnar <[email protected]>
> >>
> >> I understand that not this particular commit is buggy, but at least
> >> I've got someone to add to the CC. ;)
> >>
> >> Also the author's e-mail looks suspicious.
> >
> > Suspicious in what way?
>
> Does not appear in signed-off-by.
yes, because i applied and tested it through before Nick was comfortable
with signing off on it.
Ingo
On Sun, Dec 14, 2008 at 17:34, Alexey Zaytsev <[email protected]> wrote:
> On Thu, Dec 11, 2008 at 15:40, Alexey Zaytsev <[email protected]> wrote:
>> On Thu, Dec 11, 2008 at 12:04, Stephen Rothwell <[email protected]> wrote:
>>> Hi all,
>>>
>>>
>>> ----------------------------------------------------------------------------
>>
>> Hi.
Almost a month later, the warning is still there, and now also in Linus' git.
Am I the only one who sees it?
[ 0.004000] Intel machine check reporting enabled on CPU#0.
[ 0.004000] using mwait in idle threads.
[ 0.004000] Checking 'hlt' instruction... <4>------------[ cut here
]------------
[ 0.004150] WARNING: at kernel/sched.c:4435 sub_preempt_count+0xae/0xc0()
[ 0.004247] Hardware name: HP Compaq nx7300 (GB848ES#ACB)
[ 0.004342] Modules linked in:
[ 0.004477] Pid: 0, comm: swapper Not tainted 2.6.28-06859-gede6f5a #179
[ 0.004575] Call Trace:
[ 0.004672] [<c012fbe6>] warn_slowpath+0x86/0xa0
[ 0.004770] [<c014e99b>] ? tick_check_oneshot_change+0x4b/0x100
[ 0.004868] [<c046dc00>] ? _spin_unlock_irq+0x10/0x30
[ 0.004963] [<c047045e>] sub_preempt_count+0xae/0xc0
[ 0.005060] [<c0134787>] _local_bh_enable+0x27/0xa0
[ 0.005155] [<c0134ac7>] __do_softirq+0xf7/0x150
[ 0.005250] [<c01349d0>] ? __do_softirq+0x0/0x150
[ 0.005345] <IRQ> [<c014ec2e>] ? tick_nohz_update_jiffies+0xe/0x50
[ 0.005488] [<c013494f>] ? irq_exit+0x7f/0x90
[ 0.005584] [<c0104f23>] ? do_IRQ+0xa3/0x120
[ 0.005678] [<c01038a7>] ? common_interrupt+0x27/0x2c
[ 0.005773] [<c013007b>] ? try_acquire_console_sem+0x1b/0x30
[ 0.005872] [<c05f7378>] ? check_bugs+0xb8/0xe0
[ 0.005967] [<c05ef98a>] ? start_kernel+0x25a/0x2f0
[ 0.006062] [<c05ef270>] ? unknown_bootoption+0x0/0x210
[ 0.006159] [<c05ef07c>] ? __init_begin+0x7c/0xb0
[ 0.006260] ---[ end trace 4eaa2a86a8e2da22 ]---
[ 0.016003] OK.
[ 0.016538] ACPI: Core revision 20080926
>>
>> I'm seeing this warning early in boot logs. It does not appear on 2.6.28-rc7.
>> Not sure how long it's been around. Haven't built -next for some time.
>>
>> [ 0.004000] Intel machine check reporting enabled on CPU#0.
>> [ 0.004000] using mwait in idle threads.
>> [ 0.004000] Checking 'hlt' instruction... <4>------------[ cut here
>> ]------------
>> [ 0.004167] WARNING: at kernel/sched.c:4364 sub_preempt_count+0xae/0xc0()
>> [ 0.004266] Hardware name: HP Compaq nx7300 (GB848ES#ACB)
>> [ 0.004361] Modules linked in:
>> [ 0.004497] Pid: 0, comm: swapper Not tainted 2.6.28-rc8-next-20081211 #117
>> [ 0.004595] Call Trace:
>> [ 0.004689] [<c01324d6>] warn_slowpath+0x86/0xa0
>> [ 0.004789] [<c0155d00>] ? check_usage_forwards+0x10/0xb0
>> [ 0.004886] [<c015394a>] ? save_trace+0x3a/0xa0
>> [ 0.004981] [<c015742d>] ? mark_lock+0x37d/0xe00
>> [ 0.005076] [<c01580f9>] ? __lock_acquire+0x249/0x610
>> [ 0.005175] [<c04a3a02>] ? _spin_unlock_irq+0x22/0x50
>> [ 0.005272] [<c0158e50>] ? trace_hardirqs_on_caller+0x70/0x1a0
>> [ 0.005369] [<c04a3a0d>] ? _spin_unlock_irq+0x2d/0x50
>> [ 0.005465] [<c0124bfe>] sub_preempt_count+0xae/0xc0
>> [ 0.005564] [<c0137012>] _local_bh_enable+0x52/0xc0
>> [ 0.005661] [<c013725f>] __do_softirq+0x11f/0x170
>> [ 0.005756] [<c0137140>] ? __do_softirq+0x0/0x170
>> [ 0.005851] <IRQ> [<c0137729>] ? irq_exit+0x89/0xa0
>> [ 0.005993] [<c01059ed>] ? do_IRQ+0xad/0x120
>> [ 0.006088] [<c0103aac>] ? common_interrupt+0x2c/0x34
>> [ 0.006184] [<c013007b>] ? mmput+0x2b/0xc0
>> [ 0.006281] [<c06735a8>] ? check_bugs+0xb8/0xe0
>> [ 0.006379] [<c066b7ea>] ? start_kernel+0x26a/0x310
>> [ 0.006475] [<c066b270>] ? unknown_bootoption+0x0/0x210
>> [ 0.006572] [<c066b077>] ? __init_begin+0x77/0xb0
>> [ 0.006674] ---[ end trace 4eaa2a86a8e2da22 ]---
>> [ 0.016004] OK.
>> [ 0.016560] ACPI: Core revision 20081031
>> [ 0.044495] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
>>
>> Config and full dmesg attached.
>>
>
> The warning can also be reproduced in qemu, so it was easy to bisect.
>
> commit 7317d7b87edb41a9135e30be1ec3f7ef817c53dd
> Author: Nick Piggin <[email protected]>
> Date: Tue Sep 30 20:50:27 2008 +1000
>
> sched: improve preempt debugging
>
> This patch helped me out with a problem I recently had....
>
> Basically, when the kernel lock is held, then preempt_count
> underflow does not
> get detected until it is released which may be a long time (and arbitrarily,
> eg at different points it may be rescheduled). If the bkl is released at
> schedule, the resulting output is actually fairly cryptic...
>
> With any other lock that elevates preempt_count, it is illegal to schedule
> under it (which would get found pretty quickly). bkl allows scheduling with
> preempt_count elevated, which makes underflows hard to debug.
>
> Signed-off-by: Ingo Molnar <[email protected]>
>
> I understand that not this particular commit is buggy, but at least
> I've got someone to add to the CC. ;)
On Wed, 7 Jan 2009, Alexey Zaytsev wrote:
>
> Almost a month later, the warning is still there, and now also in Linus' git.
> Am I the only one who sees it?
Possibly. But that may be because most people don't have DEBUG_PREEMPT.
> [ 0.004150] WARNING: at kernel/sched.c:4435 sub_preempt_count+0xae/0xc0()
> [ 0.004247] Hardware name: HP Compaq nx7300 (GB848ES#ACB)
> [ 0.004342] Modules linked in:
> [ 0.004477] Pid: 0, comm: swapper Not tainted 2.6.28-06859-gede6f5a #179
> [ 0.004575] Call Trace:
> [ 0.004672] [<c012fbe6>] warn_slowpath+0x86/0xa0
> [ 0.004770] [<c014e99b>] ? tick_check_oneshot_change+0x4b/0x100
> [ 0.004868] [<c046dc00>] ? _spin_unlock_irq+0x10/0x30
> [ 0.004963] [<c047045e>] sub_preempt_count+0xae/0xc0
> [ 0.005060] [<c0134787>] _local_bh_enable+0x27/0xa0
Hmm. _local_bh_enable() would make the preempt_count go negative or fall
below 1 (with kernel lock held).
> [ 0.005155] [<c0134ac7>] __do_softirq+0xf7/0x150
> [ 0.005250] [<c01349d0>] ? __do_softirq+0x0/0x150
> [ 0.005345] <IRQ> [<c014ec2e>] ? tick_nohz_update_jiffies+0xe/0x50
> [ 0.005488] [<c013494f>] ? irq_exit+0x7f/0x90
> [ 0.005584] [<c0104f23>] ? do_IRQ+0xa3/0x120
> [ 0.005678] [<c01038a7>] ? common_interrupt+0x27/0x2c
> [ 0.005773] [<c013007b>] ? try_acquire_console_sem+0x1b/0x30
> [ 0.005872] [<c05f7378>] ? check_bugs+0xb8/0xe0
> [ 0.005967] [<c05ef98a>] ? start_kernel+0x25a/0x2f0
.. and it happens early on, when we take an interrupt in check_bugs.
Are we ready to enable interrupts there? Maybe the page fault we took (on
purpose) enabled interrupts and we now take the irq much too early.
Or maybe the initial kernel lock didn't set preempt_count to 1.
Ingo, any ideas?
Linus
* Linus Torvalds <[email protected]> wrote:
> On Wed, 7 Jan 2009, Alexey Zaytsev wrote:
> >
> > Almost a month later, the warning is still there, and now also in
> > Linus' git. Am I the only one who sees it?
>
> Possibly. But that may be because most people don't have DEBUG_PREEMPT.
hm, i never saw this warning and i run tons of different kernels (on
different hw with different build environments) so if this was a more
generic BKL init problem i'd expect to have seen it one way or another.
But i think the bug that Alexey is seeing is genuine:
> > [ 0.004150] WARNING: at kernel/sched.c:4435 sub_preempt_count+0xae/0xc0()
> > [ 0.004247] Hardware name: HP Compaq nx7300 (GB848ES#ACB)
> > [ 0.004342] Modules linked in:
> > [ 0.004477] Pid: 0, comm: swapper Not tainted 2.6.28-06859-gede6f5a #179
> > [ 0.004575] Call Trace:
> > [ 0.004672] [<c012fbe6>] warn_slowpath+0x86/0xa0
> > [ 0.004770] [<c014e99b>] ? tick_check_oneshot_change+0x4b/0x100
> > [ 0.004868] [<c046dc00>] ? _spin_unlock_irq+0x10/0x30
> > [ 0.004963] [<c047045e>] sub_preempt_count+0xae/0xc0
> > [ 0.005060] [<c0134787>] _local_bh_enable+0x27/0xa0
>
> Hmm. _local_bh_enable() would make the preempt_count go negative or fall
> below 1 (with kernel lock held).
>
> > [ 0.005155] [<c0134ac7>] __do_softirq+0xf7/0x150
> > [ 0.005250] [<c01349d0>] ? __do_softirq+0x0/0x150
> > [ 0.005345] <IRQ> [<c014ec2e>] ? tick_nohz_update_jiffies+0xe/0x50
> > [ 0.005488] [<c013494f>] ? irq_exit+0x7f/0x90
> > [ 0.005584] [<c0104f23>] ? do_IRQ+0xa3/0x120
> > [ 0.005678] [<c01038a7>] ? common_interrupt+0x27/0x2c
> > [ 0.005773] [<c013007b>] ? try_acquire_console_sem+0x1b/0x30
> > [ 0.005872] [<c05f7378>] ? check_bugs+0xb8/0xe0
> > [ 0.005967] [<c05ef98a>] ? start_kernel+0x25a/0x2f0
>
> .. and it happens early on, when we take an interrupt in check_bugs.
>
> Are we ready to enable interrupts there? Maybe the page fault we took
> (on purpose) enabled interrupts and we now take the irq much too early.
>
> Or maybe the initial kernel lock didn't set preempt_count to 1.
>
> Ingo, any ideas?
i dont think it's a BKL problem - i think something in linux-next which
went upstream recently leaked a preempt count.
What is weird is that we have detection for softirq preemption leaks:
if (unlikely(prev_count != preempt_count())) {
printk(KERN_ERR "huh, entered softirq %td %p"
"with preempt_count %08x,"
" exited with %08x?\n", h - softirq_vec,
h->action, prev_count, preempt_count());
preempt_count() = prev_count;
And this did not appear to have triggered for Alexey. So this probably
cannot be a softirq handler leak, nor an initcall handler leak.
The only possibility seems to be for this to be a preempt leak in hardirq
handler - i.e. locking inbalance by some driver. It did not get detected
there (because the preempt count was kept elevated by the BKL and by the
softirq handler), but when the softirq got released. The stack has some
hardirq signs, but no specific blame for which driver might be
responsible.
This would be a candidate for bisection i suspect, barring any better,
more specific ideas (like a specific timeframe for when this hit upstream,
or a better stacktrace of the hardirq context that triggered this softirq
instance).
Ingo
On Wed, Jan 7, 2009 at 20:17, Ingo Molnar <[email protected]> wrote:
>
> * Linus Torvalds <[email protected]> wrote:
>
>> On Wed, 7 Jan 2009, Alexey Zaytsev wrote:
>> >
>> > Almost a month later, the warning is still there, and now also in
>> > Linus' git. Am I the only one who sees it?
>>
>> Possibly. But that may be because most people don't have DEBUG_PREEMPT.
>
> hm, i never saw this warning and i run tons of different kernels (on
> different hw with different build environments) so if this was a more
> generic BKL init problem i'd expect to have seen it one way or another.
>
> But i think the bug that Alexey is seeing is genuine:
Not really. As I mentioned before, It is reproducible in qemu, but
happens a bit earlier:
[ 0.004000] Memory: 117972k/131008k available (3686k kernel code,
12484k reserved, 1567k data, 328k init, 0k highmem)
[ 0.004000] virtual kernel memory layout:
[ 0.004000] fixmap : 0xfff85000 - 0xfffff000 ( 488 kB)
[ 0.004000] pkmap : 0xff800000 - 0xffc00000 (4096 kB)
[ 0.004000] vmalloc : 0xc87f0000 - 0xff7fe000 ( 880 MB)
[ 0.004000] lowmem : 0xc0000000 - 0xc7ff0000 ( 127 MB)
[ 0.004000] .init : 0xc0627000 - 0xc0679000 ( 328 kB)
[ 0.004000] .data : 0xc04998e1 - 0xc0621678 (1567 kB)
[ 0.004000] .text : 0xc0100000 - 0xc04998e1 (3686 kB)
[ 0.004000] Checking if this processor honours the WP bit even in
supervisor mode...Ok.
[ 0.004000] SLUB: Genslabs=12, HWalign=32, Order=0-3, MinObjects=0,
CPUs=1, Nodes=1
[ 0.004768] ------------[ cut here ]------------
[ 0.007851] WARNING: at kernel/sched.c:4435 sub_preempt_count+0xaf/0xc0()
[ 0.008000] Hardware name:
[ 0.008000] Modules linked in:
[ 0.008000] Pid: 0, comm: swapper Not tainted 2.6.28-next-20090107 #181
[ 0.008000] Call Trace:
[ 0.008000] [<c0133526>] warn_slowpath+0x86/0xa0
[ 0.008000] [<c015887b>] ? mark_lock+0x38b/0xe00
[ 0.008000] [<c015b2b5>] ? __lock_acquire+0x475/0xa60
[ 0.008000] [<c0494d72>] ? _spin_unlock_irq+0x22/0x50
[ 0.008000] [<c013cf53>] ? run_timer_softirq+0x193/0x1c0
[ 0.008000] [<c0494d7d>] ? _spin_unlock_irq+0x2d/0x50
[ 0.008000] [<c049776f>] sub_preempt_count+0xaf/0xc0
[ 0.008000] [<c01387a2>] _local_bh_enable+0x52/0xc0
[ 0.008000] [<c0138bdf>] __do_softirq+0x11f/0x170
[ 0.008000] [<c0138ac0>] ? __do_softirq+0x0/0x170
[ 0.008000] <IRQ> [<c0138a44>] ? irq_exit+0x84/0x90
[ 0.008000] [<c01053e3>] ? do_IRQ+0xa3/0x120
[ 0.008000] [<c01039ec>] ? common_interrupt+0x2c/0x34
[ 0.008000] [<c0494d37>] ? _spin_unlock_irqrestore+0x57/0x70
[ 0.008000] [<c01686b3>] ? do_irq_select_affinity+0x183/0x2a0
[ 0.008000] [<c01687f1>] ? setup_irq+0x21/0x30
[ 0.008000] [<c063a8bb>] ? time_init_hook+0x2b/0x30
[ 0.008000] [<c062b116>] ? hpet_time_init+0x16/0x20
[ 0.008000] [<c06279f5>] ? start_kernel+0x245/0x360
[ 0.008000] [<c0627270>] ? unknown_bootoption+0x0/0x210
[ 0.008000] [<c0627106>] ? reserve_ebda_region+0x56/0x70
[ 0.008000] [<c062707d>] ? __init_begin+0x7d/0xb0
[ 0.008000] ---[ end trace 4eaa2a86a8e2da22 ]---
[ 0.008125] Calibrating delay loop (skipped), value calculated
using timer frequency.. 3724.22 BogoMIPS (lpj=7448448)
[ 0.016700] Mount-cache hash table entries: 512
[ 0.027953] CPU: L1 I cache: 32K, L1 D cache: 32K
[ 0.029292] CPU: L2 cache: 2048K
[ 0.032826] Checking 'hlt' instruction... OK.
[ 0.054785] SMP alternatives: switching to UP code
And last time I bisected, it pointed to:
commit 7317d7b87edb41a9135e30be1ec3f7ef817c53dd
Author: Nick Piggin <[email protected]>
Date: Tue Sep 30 20:50:27 2008 +1000
sched: improve preempt debugging
This patch helped me out with a problem I recently had....
Basically, when the kernel lock is held, then preempt_count
underflow does not
get detected until it is released which may be a long time (and arbitrarily,
eg at different points it may be rescheduled). If the bkl is released at
schedule, the resulting output is actually fairly cryptic...
With any other lock that elevates preempt_count, it is illegal to schedule
under it (which would get found pretty quickly). bkl allows scheduling with
preempt_count elevated, which makes underflows hard to debug.
Signed-off-by: Ingo Molnar <[email protected]>
so at least a dumb bisection won't do here.
.config attached for those who were not on cc last time I sent it.
* Alexey Zaytsev <[email protected]> wrote:
> And last time I bisected, it pointed to:
>
> commit 7317d7b87edb41a9135e30be1ec3f7ef817c53dd
> Author: Nick Piggin <[email protected]>
> Date: Tue Sep 30 20:50:27 2008 +1000
>
> sched: improve preempt debugging
>
>
> This patch helped me out with a problem I recently had....
>
> Basically, when the kernel lock is held, then preempt_count
> underflow does not
> get detected until it is released which may be a long time (and arbitrarily,
> eg at different points it may be rescheduled). If the bkl is released at
> schedule, the resulting output is actually fairly cryptic...
>
> With any other lock that elevates preempt_count, it is illegal to schedule
> under it (which would get found pretty quickly). bkl allows scheduling with
> preempt_count elevated, which makes underflows hard to debug.
>
> Signed-off-by: Ingo Molnar <[email protected]>
>
> so at least a dumb bisection won't do here.
ah, sorry for being a slow starter, i missed that bit - merge window
attention span troubles ...
I think the kernel_locked() check added here is plain buggy against IRQ
contexts: we drop the BKL spinlock and reduce current->kernel_depth
non-atomically.
So kernel_locked() can become detached from the preempt_count().
Nick, can you think of any better way of still saving this debug check, or
should we revert it?
Although it seems a bit weird how consistently you seem to be able to
trigger it - as this seems to be a narrow race. Is there an IRQ storm
there perhaps, or something widens things up for Qemu to inject an IRQ
right there?
Ingo
On Wed, Jan 7, 2009 at 21:47, Ingo Molnar <[email protected]> wrote:
>
> * Alexey Zaytsev <[email protected]> wrote:
>
>> And last time I bisected, it pointed to:
>>
>> commit 7317d7b87edb41a9135e30be1ec3f7ef817c53dd
>> Author: Nick Piggin <[email protected]>
>> Date: Tue Sep 30 20:50:27 2008 +1000
>>
>> sched: improve preempt debugging
>>
>>
>> This patch helped me out with a problem I recently had....
>>
>> Basically, when the kernel lock is held, then preempt_count
>> underflow does not
>> get detected until it is released which may be a long time (and arbitrarily,
>> eg at different points it may be rescheduled). If the bkl is released at
>> schedule, the resulting output is actually fairly cryptic...
>>
>> With any other lock that elevates preempt_count, it is illegal to schedule
>> under it (which would get found pretty quickly). bkl allows scheduling with
>> preempt_count elevated, which makes underflows hard to debug.
>>
>> Signed-off-by: Ingo Molnar <[email protected]>
>>
>> so at least a dumb bisection won't do here.
>
> ah, sorry for being a slow starter, i missed that bit - merge window
> attention span troubles ...
>
> I think the kernel_locked() check added here is plain buggy against IRQ
> contexts: we drop the BKL spinlock and reduce current->kernel_depth
> non-atomically.
>
> So kernel_locked() can become detached from the preempt_count().
>
> Nick, can you think of any better way of still saving this debug check, or
> should we revert it?
>
> Although it seems a bit weird how consistently you seem to be able to
> trigger it - as this seems to be a narrow race. Is there an IRQ storm
> there perhaps, or something widens things up for Qemu to inject an IRQ
> right there?
I'm not sure about the qemu case, but at least on my laptop it happens
somewhere along
arch/x86/kernel/cpu/bugs.c:
92 printk(KERN_INFO "Checking 'hlt' instruction... ");
93 if (!boot_cpu_data.hlt_works_ok) {
94 printk("disabled\n");
95 return;
96 }
97 halt();
98 halt();
99 halt();
100 halt();
101 printk("OK.\n");
where an interrupt has to come in order to get the cpu from hlt, so
there is no surprise that I'm seeing this on every single boot. ;)