2009-10-11 22:10:57

by Rafael J. Wysocki

[permalink] [raw]
Subject: 2.6.32-rc4: Reported regressions from 2.6.31

[Note:
We started to get more regression reports for 2.6.32-rc (100%+ jump in the
last 10 days. Nevertheless, we're still receiving new reports of regressions
from 2.6.30.]

This message contains a list of some regressions from 2.6.31, for which there
are no fixes in the mainline I know of. If any of them have been fixed already,
please let me know.

If you know of any other unresolved regressions from 2.6.31, please let me know
either and I'll add them to the list. Also, please let me know if any of the
entries below are invalid.

Each entry from the list will be sent additionally in an automatic reply to
this message with CCs to the people involved in reporting and handling the
issue.


Listed regressions statistics:

Date Total Pending Unresolved
----------------------------------------
2009-10-12 48 31 27
2009-10-02 22 15 9


Unresolved regressions
----------------------

Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14392
Subject : Touchpad "paste" stops working after suspend to RAM
Submitter : Carlos R. Mafra <[email protected]>
Date : 2009-10-11 16:21 (1 days old)
References : http://marc.info/?l=linux-kernel&m=125527987316493&w=4
Handled-By : Dmitry Torokhov <[email protected]>


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14390
Subject : "bind" a device to a driver doesn't not work anymore
Submitter : Éric Piel <[email protected]>
Date : 2009-10-11 0:04 (1 days old)
References : http://marc.info/?l=linux-kernel&m=125521979921241&w=4


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14389
Subject : Build system issue
Submitter : Peter Zijlstra <[email protected]>
Date : 2009-10-09 8:58 (3 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=575543347b5baed0ca927cb90ba8807396fe9cc9
References : http://marc.info/?l=linux-kernel&m=125507914909152&w=4


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14387
Subject : deadlock with fallocate
Submitter : Thomas Neumann <[email protected]>
Date : 2009-10-07 3:00 (5 days old)
References : http://marc.info/?l=linux-kernel&m=125488495526471&w=4
Handled-By : Christoph Hellwig <[email protected]>


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14386
Subject : GPF in snd_hda_intel
Submitter : Luca Tettamanti <[email protected]>
Date : 2009-10-10 13:01 (2 days old)
References : http://marc.info/?l=linux-kernel&m=125517999408019&w=4


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14384
Subject : tbench regression with 2.6.32-rc1
Submitter : Zhang, Yanmin <[email protected]>
Date : 2009-10-09 9:51 (3 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=59abf02644c45f1591e1374ee7bb45dc757fcb88
References : http://marc.info/?l=linux-kernel&m=125508216713138&w=4
Handled-By : Peter Zijlstra <[email protected]>


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14383
Subject : hackbench regression with kernel 2.6.32-rc1
Submitter : Zhang, Yanmin <[email protected]>
Date : 2009-10-09 9:19 (3 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=29cd8bae396583a2ee9a3340db8c5102acf9f6fd
References : http://marc.info/?l=linux-kernel&m=125508007510274&w=4
Handled-By : Peter Zijlstra <[email protected]>


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14381
Subject : iwlagn lost connection after s2ram (with warnings)
Submitter : Carlos R. Mafra <[email protected]>
Date : 2009-10-07 14:20 (5 days old)
References : http://marc.info/?l=linux-kernel&m=125492569119947&w=4


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14380
Subject : Video tearing/glitching with T400 laptops
Submitter : Theodore Ts'o <[email protected]>
Date : 2009-10-02 22:40 (10 days old)
References : http://marc.info/?l=linux-kernel&m=125452324520623&w=4
Handled-By : Jesse Barnes <[email protected]>


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14379
Subject : ACPI Warning for _SB_.BAT0._BIF: Converted Buffer to expected String
Submitter : Justin Mattock <[email protected]>
Date : 2009-10-08 21:46 (4 days old)
References : http://marc.info/?l=linux-kernel&m=125504031328941&w=4


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14378
Subject : Problems with net/core/skbuff.c
Submitter : Massimo Cetra <[email protected]>
Date : 2009-10-08 14:51 (4 days old)
References : http://marc.info/?l=linux-kernel&m=125501488220358&w=4


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14376
Subject : Kernel NULL pointer dereference/ kvm subsystem
Submitter : Don Dupuis <[email protected]>
Date : 2009-10-06 14:38 (6 days old)
References : http://marc.info/?l=linux-kernel&m=125484025021737&w=4


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14374
Subject : MCEs caused by commit db8be50c4307dac2b37305fc59c8dc0f978d09ea
Submitter : Nick Piggin <[email protected]>
Date : 2009-10-02 7:34 (10 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=db8be50c4307dac2b37305fc59c8dc0f978d09ea
References : http://marc.info/?l=linux-kernel&m=125446885705223&w=4


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14373
Subject : Task blocked for more than 120 seconds
Submitter : Zeno Davatz <[email protected]>
Date : 2009-10-02 10:16 (10 days old)
References : http://marc.info/?l=linux-kernel&m=125447858618412&w=4


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14372
Subject : Wireless not working after suspend-resume
Submitter : Fabio Comolli <[email protected]>
Date : 2009-10-03 15:36 (9 days old)
References : http://lkml.org/lkml/2009/10/3/91


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14370
Subject : ext4 corruptions
Submitter : Alexey Fisher <[email protected]>
Date : 2009-10-09 19:20 (3 days old)
References : http://marc.info/?l=linux-kernel&m=125511643504864&w=4


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14355
Subject : USB serial regression after 2.6.31.1 with Huawei E169 GSM modem
Submitter : Benjamin Herrenschmidt <[email protected]>
Date : 2009-10-10 03:07 (2 days old)
References : http://marc.info/?l=linux-kernel&m=125513456327542&w=4


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14354
Subject : Bad corruption with 2.6.32-rc1 and upwards
Submitter : Holger Freyther <[email protected]>
Date : 2009-10-09 15:42 (3 days old)


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14353
Subject : BUG: sleeping function called from invalid context at kernel/mutex.c:280
Submitter : Miles Lane <[email protected]>
Date : 2009-10-05 3:39 (7 days old)
References : http://marc.info/?l=linux-kernel&m=125471432208671&w=4


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14352
Subject : WARNING: at net/mac80211/scan.c:267
Submitter : Maciej Rutecki <[email protected]>
Date : 2009-10-08 00:30 (4 days old)
References : http://bugzilla.intellinuxwireless.org/show_bug.cgi?id=2089#c7


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14334
Subject : pcmcia suspend regression from 2.6.31.1 to 2.6.31.2 - Dell Inspiron 600m
Submitter : Jose Marino <[email protected]>
Date : 2009-10-06 15:44 (6 days old)


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14299
Subject : oops in wireless, iwl3945 related?
Submitter : Pavel Machek <[email protected]>
Date : 2009-09-29 17:12 (13 days old)
References : http://marc.info/?l=linux-kernel&m=125424439725743&w=4


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14298
Subject : warning at manage.c:361 (set_irq_wake), matrix-keypad related?
Submitter : Pavel Machek <[email protected]>
Date : 2009-09-30 20:07 (12 days old)
References : http://marc.info/?l=linux-kernel&m=125434130703538&w=4


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14297
Subject : console resume broken since ba15ab0e8d
Submitter : Sascha Hauer <[email protected]>
Date : 2009-09-30 15:11 (12 days old)
References : http://marc.info/?l=linux-kernel&m=125432349404060&w=4


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14296
Subject : spitz boots but suspend/resume is broken
Submitter : Pavel Machek <[email protected]>
Date : 2009-09-30 12:06 (12 days old)
References : http://marc.info/?l=linux-kernel&m=125431244516449&w=4


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14279
Subject : Suspend to RAM freeze totally since 2.6.32-rc1 - Acer Aspire 1511Lmi laptop
Submitter : Christian Casteyde <[email protected]>
Date : 2009-09-30 18:14 (12 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=5f68563996e812f9ca35b3939ad2a42e5d254d66


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14277
Subject : Caught 8-bit read from freed memory in b43 driver at association
Submitter : Christian Casteyde <[email protected]>
Date : 2009-09-30 18:06 (12 days old)


Regressions with patches
------------------------

Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14382
Subject : Transmit failure in et131x.
Submitter : Nick Bowler <[email protected]>
Date : 2009-10-08 14:08 (4 days old)
References : http://marc.info/?l=linux-kernel&m=125501117713744&w=4
Handled-By : Alan Cox <[email protected]>
Patch : http://patchwork.kernel.org/patch/52698/


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14375
Subject : Intel(R) I/OAT DMA Engine init failed
Submitter : Alexander Beregalov <[email protected]>
Date : 2009-10-02 9:46 (10 days old)
References : http://marc.info/?l=linux-kernel&m=125447680016160&w=4
Handled-By : Dan Williams <[email protected]>
Patch : http://patchwork.kernel.org/patch/51808/


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14302
Subject : Kernel panic on i386 machine when booting with profile=2
Submitter : Shi, Alex <[email protected]>
Date : 2009-10-01 3:23 (11 days old)
References : http://marc.info/?l=linux-kernel&m=125436749607199&w=4
Handled-By : Alex Shi <[email protected]>
Patch : http://patchwork.kernel.org/patch/50813/


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14278
Subject : New message "NOHZ: local_softirq_pending 08" at each ping request
Submitter : Christian Casteyde <[email protected]>
Date : 2009-09-30 18:12 (12 days old)
Handled-By : Michael Buesch <[email protected]>
Patch : http://bugzilla.kernel.org/attachment.cgi?id=23220


For details, please visit the bug entries and follow the links given in
references.

As you can see, there is a Bugzilla entry for each of the listed regressions.
There also is a Bugzilla entry used for tracking the regressions from 2.6.31,
unresolved as well as resolved, at:

http://bugzilla.kernel.org/show_bug.cgi?id=14230

Please let me know if there are any Bugzilla entries that should be added to
the list in there.

Thanks,
Rafael


2009-10-11 22:18:24

by Rafael J. Wysocki

[permalink] [raw]
Subject: [Bug #14277] Caught 8-bit read from freed memory in b43 driver at association

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31. Please verify if it still should be listed and let me know
(either way).


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14277
Subject : Caught 8-bit read from freed memory in b43 driver at association
Submitter : Christian Casteyde <[email protected]>
Date : 2009-09-30 18:06 (12 days old)

2009-10-11 22:25:16

by Rafael J. Wysocki

[permalink] [raw]
Subject: [Bug #14278] New message "NOHZ: local_softirq_pending 08" at each ping request

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31. Please verify if it still should be listed and let me know
(either way).


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14278
Subject : New message "NOHZ: local_softirq_pending 08" at each ping request
Submitter : Christian Casteyde <[email protected]>
Date : 2009-09-30 18:12 (12 days old)
Handled-By : Michael Buesch <[email protected]>
Patch : http://bugzilla.kernel.org/attachment.cgi?id=23220

2009-10-11 22:25:18

by Rafael J. Wysocki

[permalink] [raw]
Subject: [Bug #14279] Suspend to RAM freeze totally since 2.6.32-rc1 - Acer Aspire 1511Lmi laptop

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31. Please verify if it still should be listed and let me know
(either way).


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14279
Subject : Suspend to RAM freeze totally since 2.6.32-rc1 - Acer Aspire 1511Lmi laptop
Submitter : Christian Casteyde <[email protected]>
Date : 2009-09-30 18:14 (12 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=5f68563996e812f9ca35b3939ad2a42e5d254d66

2009-10-11 22:25:51

by Rafael J. Wysocki

[permalink] [raw]
Subject: [Bug #14296] spitz boots but suspend/resume is broken

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31. Please verify if it still should be listed and let me know
(either way).


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14296
Subject : spitz boots but suspend/resume is broken
Submitter : Pavel Machek <[email protected]>
Date : 2009-09-30 12:06 (12 days old)
References : http://marc.info/?l=linux-kernel&m=125431244516449&w=4

2009-10-11 22:26:03

by Rafael J. Wysocki

[permalink] [raw]
Subject: [Bug #14334] pcmcia suspend regression from 2.6.31.1 to 2.6.31.2 - Dell Inspiron 600m

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31. Please verify if it still should be listed and let me know
(either way).


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14334
Subject : pcmcia suspend regression from 2.6.31.1 to 2.6.31.2 - Dell Inspiron 600m
Submitter : Jose Marino <[email protected]>
Date : 2009-10-06 15:44 (6 days old)

2009-10-11 22:25:25

by Rafael J. Wysocki

[permalink] [raw]
Subject: [Bug #14297] console resume broken since ba15ab0e8d

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31. Please verify if it still should be listed and let me know
(either way).


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14297
Subject : console resume broken since ba15ab0e8d
Submitter : Sascha Hauer <[email protected]>
Date : 2009-09-30 15:11 (12 days old)
References : http://marc.info/?l=linux-kernel&m=125432349404060&w=4

2009-10-11 22:29:01

by Rafael J. Wysocki

[permalink] [raw]
Subject: [Bug #14302] Kernel panic on i386 machine when booting with profile=2

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31. Please verify if it still should be listed and let me know
(either way).


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14302
Subject : Kernel panic on i386 machine when booting with profile=2
Submitter : Shi, Alex <[email protected]>
Date : 2009-10-01 3:23 (11 days old)
References : http://marc.info/?l=linux-kernel&m=125436749607199&w=4
Handled-By : Alex Shi <[email protected]>
Patch : http://patchwork.kernel.org/patch/50813/

2009-10-11 22:26:00

by Rafael J. Wysocki

[permalink] [raw]
Subject: [Bug #14352] WARNING: at net/mac80211/scan.c:267

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31. Please verify if it still should be listed and let me know
(either way).


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14352
Subject : WARNING: at net/mac80211/scan.c:267
Submitter : Maciej Rutecki <[email protected]>
Date : 2009-10-08 00:30 (4 days old)
References : http://bugzilla.intellinuxwireless.org/show_bug.cgi?id=2089#c7

2009-10-11 22:25:26

by Rafael J. Wysocki

[permalink] [raw]
Subject: [Bug #14298] warning at manage.c:361 (set_irq_wake), matrix-keypad related?

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31. Please verify if it still should be listed and let me know
(either way).


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14298
Subject : warning at manage.c:361 (set_irq_wake), matrix-keypad related?
Submitter : Pavel Machek <[email protected]>
Date : 2009-09-30 20:07 (12 days old)
References : http://marc.info/?l=linux-kernel&m=125434130703538&w=4

2009-10-11 22:29:48

by Rafael J. Wysocki

[permalink] [raw]
Subject: [Bug #14299] oops in wireless, iwl3945 related?

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31. Please verify if it still should be listed and let me know
(either way).


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14299
Subject : oops in wireless, iwl3945 related?
Submitter : Pavel Machek <[email protected]>
Date : 2009-09-29 17:12 (13 days old)
References : http://marc.info/?l=linux-kernel&m=125424439725743&w=4

2009-10-11 22:26:05

by Rafael J. Wysocki

[permalink] [raw]
Subject: [Bug #14353] BUG: sleeping function called from invalid context at kernel/mutex.c:280

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31. Please verify if it still should be listed and let me know
(either way).


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14353
Subject : BUG: sleeping function called from invalid context at kernel/mutex.c:280
Submitter : Miles Lane <[email protected]>
Date : 2009-10-05 3:39 (7 days old)
References : http://marc.info/?l=linux-kernel&m=125471432208671&w=4

2009-10-11 22:26:09

by Rafael J. Wysocki

[permalink] [raw]
Subject: [Bug #14355] USB serial regression after 2.6.31.1 with Huawei E169 GSM modem

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31. Please verify if it still should be listed and let me know
(either way).


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14355
Subject : USB serial regression after 2.6.31.1 with Huawei E169 GSM modem
Submitter : Benjamin Herrenschmidt <[email protected]>
Date : 2009-10-10 03:07 (2 days old)
References : http://marc.info/?l=linux-kernel&m=125513456327542&w=4

2009-10-11 22:26:11

by Rafael J. Wysocki

[permalink] [raw]
Subject: [Bug #14370] ext4 corruptions

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31. Please verify if it still should be listed and let me know
(either way).


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14370
Subject : ext4 corruptions
Submitter : Alexey Fisher <[email protected]>
Date : 2009-10-09 19:20 (3 days old)
References : http://marc.info/?l=linux-kernel&m=125511643504864&w=4

2009-10-11 22:29:54

by Rafael J. Wysocki

[permalink] [raw]
Subject: [Bug #14354] Bad corruption with 2.6.32-rc1 and upwards

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31. Please verify if it still should be listed and let me know
(either way).


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14354
Subject : Bad corruption with 2.6.32-rc1 and upwards
Submitter : Holger Freyther <[email protected]>
Date : 2009-10-09 15:42 (3 days old)

2009-10-11 22:29:18

by Rafael J. Wysocki

[permalink] [raw]
Subject: [Bug #14372] Wireless not working after suspend-resume

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31. Please verify if it still should be listed and let me know
(either way).


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14372
Subject : Wireless not working after suspend-resume
Submitter : Fabio Comolli <[email protected]>
Date : 2009-10-03 15:36 (9 days old)
References : http://lkml.org/lkml/2009/10/3/91

2009-10-11 22:26:49

by Rafael J. Wysocki

[permalink] [raw]
Subject: [Bug #14374] MCEs caused by commit db8be50c4307dac2b37305fc59c8dc0f978d09ea

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31. Please verify if it still should be listed and let me know
(either way).


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14374
Subject : MCEs caused by commit db8be50c4307dac2b37305fc59c8dc0f978d09ea
Submitter : Nick Piggin <[email protected]>
Date : 2009-10-02 7:34 (10 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=db8be50c4307dac2b37305fc59c8dc0f978d09ea
References : http://marc.info/?l=linux-kernel&m=125446885705223&w=4

2009-10-11 22:29:20

by Rafael J. Wysocki

[permalink] [raw]
Subject: [Bug #14373] Task blocked for more than 120 seconds

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31. Please verify if it still should be listed and let me know
(either way).


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14373
Subject : Task blocked for more than 120 seconds
Submitter : Zeno Davatz <[email protected]>
Date : 2009-10-02 10:16 (10 days old)
References : http://marc.info/?l=linux-kernel&m=125447858618412&w=4

2009-10-11 22:27:35

by Rafael J. Wysocki

[permalink] [raw]
Subject: [Bug #14378] Problems with net/core/skbuff.c

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31. Please verify if it still should be listed and let me know
(either way).


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14378
Subject : Problems with net/core/skbuff.c
Submitter : Massimo Cetra <[email protected]>
Date : 2009-10-08 14:51 (4 days old)
References : http://marc.info/?l=linux-kernel&m=125501488220358&w=4

2009-10-11 22:26:44

by Rafael J. Wysocki

[permalink] [raw]
Subject: [Bug #14375] Intel(R) I/OAT DMA Engine init failed

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31. Please verify if it still should be listed and let me know
(either way).


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14375
Subject : Intel(R) I/OAT DMA Engine init failed
Submitter : Alexander Beregalov <[email protected]>
Date : 2009-10-02 9:46 (10 days old)
References : http://marc.info/?l=linux-kernel&m=125447680016160&w=4
Handled-By : Dan Williams <[email protected]>
Patch : http://patchwork.kernel.org/patch/51808/

2009-10-11 22:26:17

by Rafael J. Wysocki

[permalink] [raw]
Subject: [Bug #14376] Kernel NULL pointer dereference/ kvm subsystem

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31. Please verify if it still should be listed and let me know
(either way).


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14376
Subject : Kernel NULL pointer dereference/ kvm subsystem
Submitter : Don Dupuis <[email protected]>
Date : 2009-10-06 14:38 (6 days old)
References : http://marc.info/?l=linux-kernel&m=125484025021737&w=4

2009-10-11 22:28:32

by Rafael J. Wysocki

[permalink] [raw]
Subject: [Bug #14380] Video tearing/glitching with T400 laptops

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31. Please verify if it still should be listed and let me know
(either way).


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14380
Subject : Video tearing/glitching with T400 laptops
Submitter : Theodore Ts'o <[email protected]>
Date : 2009-10-02 22:40 (10 days old)
References : http://marc.info/?l=linux-kernel&m=125452324520623&w=4
Handled-By : Jesse Barnes <[email protected]>

2009-10-11 22:26:48

by Rafael J. Wysocki

[permalink] [raw]
Subject: [Bug #14382] Transmit failure in et131x.

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31. Please verify if it still should be listed and let me know
(either way).


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14382
Subject : Transmit failure in et131x.
Submitter : Nick Bowler <[email protected]>
Date : 2009-10-08 14:08 (4 days old)
References : http://marc.info/?l=linux-kernel&m=125501117713744&w=4
Handled-By : Alan Cox <[email protected]>
Patch : http://patchwork.kernel.org/patch/52698/

2009-10-11 22:28:28

by Rafael J. Wysocki

[permalink] [raw]
Subject: [Bug #14379] ACPI Warning for _SB_.BAT0._BIF: Converted Buffer to expected String

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31. Please verify if it still should be listed and let me know
(either way).


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14379
Subject : ACPI Warning for _SB_.BAT0._BIF: Converted Buffer to expected String
Submitter : Justin Mattock <[email protected]>
Date : 2009-10-08 21:46 (4 days old)
References : http://marc.info/?l=linux-kernel&m=125504031328941&w=4

2009-10-11 22:26:19

by Rafael J. Wysocki

[permalink] [raw]
Subject: [Bug #14381] iwlagn lost connection after s2ram (with warnings)

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31. Please verify if it still should be listed and let me know
(either way).


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14381
Subject : iwlagn lost connection after s2ram (with warnings)
Submitter : Carlos R. Mafra <[email protected]>
Date : 2009-10-07 14:20 (5 days old)
References : http://marc.info/?l=linux-kernel&m=125492569119947&w=4

2009-10-11 22:28:07

by Rafael J. Wysocki

[permalink] [raw]
Subject: [Bug #14383] hackbench regression with kernel 2.6.32-rc1

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31. Please verify if it still should be listed and let me know
(either way).


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14383
Subject : hackbench regression with kernel 2.6.32-rc1
Submitter : Zhang, Yanmin <[email protected]>
Date : 2009-10-09 9:19 (3 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=29cd8bae396583a2ee9a3340db8c5102acf9f6fd
References : http://marc.info/?l=linux-kernel&m=125508007510274&w=4
Handled-By : Peter Zijlstra <[email protected]>

2009-10-11 22:26:45

by Rafael J. Wysocki

[permalink] [raw]
Subject: [Bug #14384] tbench regression with 2.6.32-rc1

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31. Please verify if it still should be listed and let me know
(either way).


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14384
Subject : tbench regression with 2.6.32-rc1
Submitter : Zhang, Yanmin <[email protected]>
Date : 2009-10-09 9:51 (3 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=59abf02644c45f1591e1374ee7bb45dc757fcb88
References : http://marc.info/?l=linux-kernel&m=125508216713138&w=4
Handled-By : Peter Zijlstra <[email protected]>

2009-10-11 22:26:50

by Rafael J. Wysocki

[permalink] [raw]
Subject: [Bug #14386] GPF in snd_hda_intel

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31. Please verify if it still should be listed and let me know
(either way).


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14386
Subject : GPF in snd_hda_intel
Submitter : Luca Tettamanti <[email protected]>
Date : 2009-10-10 13:01 (2 days old)
References : http://marc.info/?l=linux-kernel&m=125517999408019&w=4

2009-10-11 22:27:50

by Rafael J. Wysocki

[permalink] [raw]
Subject: [Bug #14387] deadlock with fallocate

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31. Please verify if it still should be listed and let me know
(either way).


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14387
Subject : deadlock with fallocate
Submitter : Thomas Neumann <[email protected]>
Date : 2009-10-07 3:00 (5 days old)
References : http://marc.info/?l=linux-kernel&m=125488495526471&w=4
Handled-By : Christoph Hellwig <[email protected]>

2009-10-11 22:26:24

by Rafael J. Wysocki

[permalink] [raw]
Subject: [Bug #14389] Build system issue

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31. Please verify if it still should be listed and let me know
(either way).


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14389
Subject : Build system issue
Submitter : Peter Zijlstra <[email protected]>
Date : 2009-10-09 8:58 (3 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=575543347b5baed0ca927cb90ba8807396fe9cc9
References : http://marc.info/?l=linux-kernel&m=125507914909152&w=4

2009-10-11 22:26:52

by Rafael J. Wysocki

[permalink] [raw]
Subject: [Bug #14392] Touchpad "paste" stops working after suspend to RAM

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31. Please verify if it still should be listed and let me know
(either way).


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14392
Subject : Touchpad "paste" stops working after suspend to RAM
Submitter : Carlos R. Mafra <[email protected]>
Date : 2009-10-11 16:21 (1 days old)
References : http://marc.info/?l=linux-kernel&m=125527987316493&w=4
Handled-By : Dmitry Torokhov <[email protected]>

2009-10-11 22:26:25

by Rafael J. Wysocki

[permalink] [raw]
Subject: [Bug #14390] "bind" a device to a driver doesn't not work anymore

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31. Please verify if it still should be listed and let me know
(either way).


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14390
Subject : "bind" a device to a driver doesn't not work anymore
Submitter : Éric Piel <[email protected]>
Date : 2009-10-11 0:04 (1 days old)
References : http://marc.info/?l=linux-kernel&m=125521979921241&w=4

2009-10-11 22:29:53

by Alan

[permalink] [raw]
Subject: Re: [Bug #14382] Transmit failure in et131x.

On Mon, 12 Oct 2009 00:22:05 +0200 (CEST)
"Rafael J. Wysocki" <[email protected]> wrote:

> This message has been generated automatically as a part of a report
> of recent regressions.
>
> The following bug entry is on the current list of known regressions
> from 2.6.31. Please verify if it still should be listed and let me know
> (either way).

Patch sent for testing, nothing heard back. I think I've fixed it - I
certainly found a bug 8)

2009-10-11 23:12:09

by Linus Torvalds

[permalink] [raw]
Subject: Re: 2.6.32-rc4: Reported regressions from 2.6.31



On Mon, 12 Oct 2009, Rafael J. Wysocki wrote:
>
> Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14374
> Subject : MCEs caused by commit db8be50c4307dac2b37305fc59c8dc0f978d09ea
> Submitter : Nick Piggin <[email protected]>
> Date : 2009-10-02 7:34 (10 days old)
> First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=db8be50c4307dac2b37305fc59c8dc0f978d09ea
> References : http://marc.info/?l=linux-kernel&m=125446885705223&w=4

I think we're going to have to just revert that commit. It's wrong.

It would be nice to do USB handoff earlier, but

- the correct fix is to make the damn IOMMU code to do it's thing later

- the USB handoff cannot be done with just PCI config cycles, and will
need to do ioremap's etc, which in turn will need the resources to be
allocated, which in turn is not possible at FIXUP_HEADER time.

so however much we migth _wish_ to do what that commit does, the commit is
obvious crap in retrospect.

The IOMMU should not be reprogrammed until _after_ we have handled and
re-initialized all the devices, so I really think the problem is the IOMMU
code, not the USB code.

(Of course, the underlying problem is that USB legacy mode handling by
BIOS with SMM is some seriously insane crap, no question about that. At
the same time, the IOMMU code is just clearly wrong in removing the legacy
mappings before we have initialized all devices, so the revert is simply
the right thing to do)

Linus

2009-10-11 23:15:19

by Linus Torvalds

[permalink] [raw]
Subject: Re: 2.6.32-rc4: Reported regressions from 2.6.31


On Mon, 12 Oct 2009, Rafael J. Wysocki wrote:
>
> Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14370
> Subject : ext4 corruptions
> Submitter : Alexey Fisher <[email protected]>
> Date : 2009-10-09 19:20 (3 days old)
> References : http://marc.info/?l=linux-kernel&m=125511643504864&w=4
>
> Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14354
> Subject : Bad corruption with 2.6.32-rc1 and upwards
> Submitter : Holger Freyther <[email protected]>
> Date : 2009-10-09 15:42 (3 days old)

These two look like duplicates to me. No?

Linus

2009-10-12 00:03:29

by Theodore Ts'o

[permalink] [raw]
Subject: Re: 2.6.32-rc4: Reported regressions from 2.6.31

On Sun, Oct 11, 2009 at 04:11:20PM -0700, Linus Torvalds wrote:
>
> On Mon, 12 Oct 2009, Rafael J. Wysocki wrote:
> >
> > Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14370
> > Subject : ext4 corruptions
> > Submitter : Alexey Fisher <[email protected]>
> > Date : 2009-10-09 19:20 (3 days old)
> > References : http://marc.info/?l=linux-kernel&m=125511643504864&w=4
> >
> > Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14354
> > Subject : Bad corruption with 2.6.32-rc1 and upwards
> > Submitter : Holger Freyther <[email protected]>
> > Date : 2009-10-09 15:42 (3 days old)
>
> These two look like duplicates to me. No?

I'm not sure. I'm not able to replicate either failure. According to
Holger, his failure happens after a clean shutdown. Alexey's happens
after a crash/forced power down. So they could possibly be two
separate failures --- although obviously I hope not!

However, Alexey has been posting his failure information in bug
#14354, so we might as well close #14370. What I really need is hints
about how to replicate the failure.

- Ted

2009-10-12 03:14:47

by Justin P. Mattock

[permalink] [raw]
Subject: Re: [Bug #14379] ACPI Warning for _SB_.BAT0._BIF: Converted Buffer to expected String

On Sun, Oct 11, 2009 at 3:22 PM, Rafael J. Wysocki <[email protected]> wrote:
> This message has been generated automatically as a part of a report
> of recent regressions.
>
> The following bug entry is on the current list of known regressions
> from 2.6.31. ?Please verify if it still should be listed and let me know
> (either way).
>
>
> Bug-Entry ? ? ? : http://bugzilla.kernel.org/show_bug.cgi?id=14379
> Subject ? ? ? ? : ACPI Warning for _SB_.BAT0._BIF: Converted Buffer to expected String
> Submitter ? ? ? : Justin Mattock <[email protected]>
> Date ? ? ? ? ? ?: 2009-10-08 21:46 (4 days old)
> References ? ? ?: http://marc.info/?l=linux-kernel&m=125504031328941&w=4
>
>
>

pulled rc4 today and the warning is still present.
here is the results of git bisect:

d9adc2e031bd22d5d9607a53a8d3b30e0b675f39 is the first bad commit
commit d9adc2e031bd22d5d9607a53a8d3b30e0b675f39
Author: Lin Ming <[email protected]>
Date: Mon Jul 27 11:31:10 2009 +0800

ACPICA: reformat predefined method table, no functional change

Reformatted the methods that return package objects.

Signed-off-by: Lin Ming <[email protected]>
Signed-off-by: Len Brown <[email protected]>

:040000 040000 17c30a6e75ce2fafc677b32bdcf3fbecae8458ab
f43e38a2a4034bcc73ef14943e61187f11b08731 M drivers

I will go and attach the git log to the bug report.

--
Justin P. Mattock

2009-10-12 03:51:07

by Lin Ming

[permalink] [raw]
Subject: Re: [Bug #14379] ACPI Warning for _SB_.BAT0._BIF: Converted Buffer to expected String

On Mon, 2009-10-12 at 11:14 +0800, Justin Mattock wrote:
> On Sun, Oct 11, 2009 at 3:22 PM, Rafael J. Wysocki <[email protected]> wrote:
> > This message has been generated automatically as a part of a report
> > of recent regressions.
> >
> > The following bug entry is on the current list of known regressions
> > from 2.6.31. Please verify if it still should be listed and let me know
> > (either way).
> >
> >
> > Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14379
> > Subject : ACPI Warning for _SB_.BAT0._BIF: Converted Buffer to expected String
> > Submitter : Justin Mattock <[email protected]>
> > Date : 2009-10-08 21:46 (4 days old)
> > References : http://marc.info/?l=linux-kernel&m=125504031328941&w=4
> >
> >
> >
>
> pulled rc4 today and the warning is still present.
> here is the results of git bisect:

Patch at http://bugzilla.kernel.org/show_bug.cgi?id=14379#c1 fixes the
warning.

Actually, the _BIF warning was fixed at commit
e899b6485c332aa2d7510739507ab5e5d7b28e59.
See http://bugzilla.kernel.org/show_bug.cgi?id=11822#c14

I made a mistake at the cleanup of predefined method table so the
warning shows up again.

Lin Ming

>
> d9adc2e031bd22d5d9607a53a8d3b30e0b675f39 is the first bad commit
> commit d9adc2e031bd22d5d9607a53a8d3b30e0b675f39
> Author: Lin Ming <[email protected]>
> Date: Mon Jul 27 11:31:10 2009 +0800
>
> ACPICA: reformat predefined method table, no functional change
>
> Reformatted the methods that return package objects.
>
> Signed-off-by: Lin Ming <[email protected]>
> Signed-off-by: Len Brown <[email protected]>
>
> :040000 040000 17c30a6e75ce2fafc677b32bdcf3fbecae8458ab
> f43e38a2a4034bcc73ef14943e61187f11b08731 M drivers
>
> I will go and attach the git log to the bug report.
>

2009-10-12 03:59:21

by Justin P. Mattock

[permalink] [raw]
Subject: Re: [Bug #14379] ACPI Warning for _SB_.BAT0._BIF: Converted Buffer to expected String

Lin Ming wrote:
> On Mon, 2009-10-12 at 11:14 +0800, Justin Mattock wrote:
>
>> On Sun, Oct 11, 2009 at 3:22 PM, Rafael J. Wysocki<[email protected]> wrote:
>>
>>> This message has been generated automatically as a part of a report
>>> of recent regressions.
>>>
>>> The following bug entry is on the current list of known regressions
>>> from 2.6.31. Please verify if it still should be listed and let me know
>>> (either way).
>>>
>>>
>>> Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14379
>>> Subject : ACPI Warning for _SB_.BAT0._BIF: Converted Buffer to expected String
>>> Submitter : Justin Mattock<[email protected]>
>>> Date : 2009-10-08 21:46 (4 days old)
>>> References : http://marc.info/?l=linux-kernel&m=125504031328941&w=4
>>>
>>>
>>>
>>>
>> pulled rc4 today and the warning is still present.
>> here is the results of git bisect:
>>
>
> Patch at http://bugzilla.kernel.org/show_bug.cgi?id=14379#c1 fixes the
> warning.
>
> Actually, the _BIF warning was fixed at commit
> e899b6485c332aa2d7510739507ab5e5d7b28e59.
> See http://bugzilla.kernel.org/show_bug.cgi?id=11822#c14
>
> I made a mistake at the cleanup of predefined method table so the
> warning shows up again.
>
> Lin Ming
>
>
Cool, I can give that a try.
then I guess just leave this open(bugreport) until this patch
http://bugzilla.kernel.org/show_bug.cgi?id=14379#c1
gets pushed into the main tree then close the bugreport.

Justin P. Mattock

2009-10-12 06:51:47

by Oleksij Rempel

[permalink] [raw]
Subject: Re: [Bug #14370] ext4 corruptions

Am Montag, den 12.10.2009, 00:22 +0200 schrieb Rafael J. Wysocki:
> This message has been generated automatically as a part of a report
> of recent regressions.
>
> The following bug entry is on the current list of known regressions
> from 2.6.31. Please verify if it still should be listed and let me know
> (either way).
>
>
> Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14370
> Subject : ext4 corruptions
> Submitter : Alexey Fisher <[email protected]>
> Date : 2009-10-09 19:20 (3 days old)
> References : http://marc.info/?l=linux-kernel&m=125511643504864&w=4
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe kernel-testers" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html

This can be removed, it is duplicate of this bug:
http://bugzilla.kernel.org/show_bug.cgi?id=14354

2009-10-12 08:10:37

by Jan Beulich

[permalink] [raw]
Subject: Re: [Bug #14279] Suspend to RAM freeze totally since 2.6.32-rc1 - Acer Aspire 1511Lmi laptop

The issue is still present, albeit a patch (available in the bugzilla entry) had
been posted on Oct 6th. Jan

>>> "Rafael J. Wysocki" <[email protected]> 12.10.09 00:22 >>>
This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31. Please verify if it still should be listed and let me know
(either way).


Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14279
Subject : Suspend to RAM freeze totally since 2.6.32-rc1 - Acer Aspire 1511Lmi laptop
Submitter : Christian Casteyde <[email protected]>
Date : 2009-09-30 18:14 (12 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=5f68563996e812f9ca35b3939ad2a42e5d254d66


2009-10-12 10:19:41

by Woodhouse, David

[permalink] [raw]
Subject: Re: 2.6.32-rc4: Reported regressions from 2.6.31

On Mon, 2009-10-12 at 00:07 +0100, Linus Torvalds wrote:
> The IOMMU should not be reprogrammed until _after_ we have handled and
> re-initialized all the devices, so I really think the problem is the IOMMU
> code, not the USB code.

The IOMMU init is happening in the right place, I think. It has to be
before device_initcall(), because it has to be set up before we actually
start initialising devices -- we have to have the IOMMU in place before
any real drivers try to set up DMA mappings.

It's currently fs_initcall(), with an explicit comment saying that it
has to be after PCI initialisation. The only way it could be later is if
we move it to rootfs_initcall().

> (Of course, the underlying problem is that USB legacy mode handling by
> BIOS with SMM is some seriously insane crap, no question about that. At
> the same time, the IOMMU code is just clearly wrong in removing the legacy
> mappings before we have initialized all devices, so the revert is simply
> the right thing to do)

Well, according to the design, the IOMMU code is doing the right thing¹.

The theory is that the BIOS _tells_ us about the legacy mappings it
needs by putting them in an ACPI table (the RMRR table). Of course, the
reality is that BIOSes are universally shit and often fail to do this.

If the BIOSes were only _slightly_ shit, there'd be a few harmless DMA
faults and the legacy USB would stop of its own accord. But no, a bunch
of them also manage to lock up in SMM mode when their DMA goes AWOL.

The only viable solution (short of an open source BIOS written by sober
people) was to quiesce the USB controllers before enabling the IOMMU.

The final PCI quirks are currently run from pci_init() which is a
device_initcall(), which is too late -- in fact, it could actually be
_after_ some of the real device drivers have taken control of the same
hardware.

So the better fix is probably just to fix that problem -- move the final
PCI quirks so they happen a little earlier. If we move them to
fs_initcall_sync() and then move the IOMMU init to rootfs_initcall(),
then everything ought to work, I think...

(Let's rename 'pci_init()' and move it to quirks.c while we're at it,
since it only actually does that one thing. If/when I submit this
properly, I'll do that in a separate commit.)

diff --git a/arch/x86/kernel/pci-dma.c b/arch/x86/kernel/pci-dma.c
index 64b838e..e0d9199 100644
--- a/arch/x86/kernel/pci-dma.c
+++ b/arch/x86/kernel/pci-dma.c
@@ -311,7 +311,7 @@ void pci_iommu_shutdown(void)
amd_iommu_shutdown();
}
/* Must execute after PCI subsystem */
-fs_initcall(pci_iommu_init);
+rootfs_initcall(pci_iommu_init);

#ifdef CONFIG_PCI
/* Many VIA bridges seem to corrupt data for DAC. Disable it here */
diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index 6edecff..2b575cf 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -2719,17 +2719,6 @@ int __attribute__ ((weak)) pci_ext_cfg_avail(struct pci_dev *dev)
return 1;
}

-static int __devinit pci_init(void)
-{
- struct pci_dev *dev = NULL;
-
- while ((dev = pci_get_device(PCI_ANY_ID, PCI_ANY_ID, dev)) != NULL) {
- pci_fixup_device(pci_fixup_final, dev);
- }
-
- return 0;
-}
-
static int __init pci_setup(char *str)
{
while (str) {
@@ -2767,8 +2756,6 @@ static int __init pci_setup(char *str)
}
early_param("pci", pci_setup);

-device_initcall(pci_init);
-
EXPORT_SYMBOL(pci_reenable_device);
EXPORT_SYMBOL(pci_enable_device_io);
EXPORT_SYMBOL(pci_enable_device_mem);
diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
index 6099fac..d99c9b4 100644
--- a/drivers/pci/quirks.c
+++ b/drivers/pci/quirks.c
@@ -2572,6 +2572,19 @@ void pci_fixup_device(enum pci_fixup_pass pass, struct pci_dev *dev)
}
pci_do_fixups(dev, start, end);
}
+
+static int __devinit pci_apply_final_quirks(void)
+{
+ struct pci_dev *dev = NULL;
+
+ while ((dev = pci_get_device(PCI_ANY_ID, PCI_ANY_ID, dev)) != NULL) {
+ pci_fixup_device(pci_fixup_final, dev);
+ }
+
+ return 0;
+}
+
+fs_initcall_sync(pci_apply_final_quirks);
#else
void pci_fixup_device(enum pci_fixup_pass pass, struct pci_dev *dev) {}
#endif


--
David Woodhouse Open Source Technology Centre
[email protected] Intel Corporation

¹ Yes, the design is completely broken. Anyone with any real-world
experience would have known that the BIOS would never get this right.
Personally, I find that it's useful to assume malice on the part of
BIOS authors. I'm not suggesting that it's actually _true_, but it
does tend to give a useful predictor of _just_ how badly they're going
to screw things up.


2009-10-12 10:42:31

by David Miller

[permalink] [raw]
Subject: Re: [Bug #14378] Problems with net/core/skbuff.c

From: "Rafael J. Wysocki" <[email protected]>
Date: Mon, 12 Oct 2009 00:22:04 +0200 (CEST)

> Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14378
> Subject : Problems with net/core/skbuff.c
> Submitter : Massimo Cetra <[email protected]>
> Date : 2009-10-08 14:51 (4 days old)
> References : http://marc.info/?l=linux-kernel&m=125501488220358&w=4

I don't know what to do about this one.

The user indicates that they have the vserver patches applied,
so maybe there is some interaction with that stuff.

2009-10-12 12:07:22

by David Woodhouse

[permalink] [raw]
Subject: [PATCH 1/4] Rename pci_init() to pci_apply_final_quirks(), move it to quirks.c

This function may have done more in the past, but all it does now is
apply the PCI_FIXUP_FINAL quirks. So name it sensibly and put it where
it belongs.

Signed-off-by: David Woodhouse <[email protected]>
---
drivers/pci/pci.c | 13 -------------
drivers/pci/quirks.c | 13 +++++++++++++
2 files changed, 13 insertions(+), 13 deletions(-)

diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index 6edecff..2b575cf 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -2719,17 +2719,6 @@ int __attribute__ ((weak)) pci_ext_cfg_avail(struct pci_dev *dev)
return 1;
}

-static int __devinit pci_init(void)
-{
- struct pci_dev *dev = NULL;
-
- while ((dev = pci_get_device(PCI_ANY_ID, PCI_ANY_ID, dev)) != NULL) {
- pci_fixup_device(pci_fixup_final, dev);
- }
-
- return 0;
-}
-
static int __init pci_setup(char *str)
{
while (str) {
@@ -2767,8 +2756,6 @@ static int __init pci_setup(char *str)
}
early_param("pci", pci_setup);

-device_initcall(pci_init);
-
EXPORT_SYMBOL(pci_reenable_device);
EXPORT_SYMBOL(pci_enable_device_io);
EXPORT_SYMBOL(pci_enable_device_mem);
diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
index 6099fac..0142275 100644
--- a/drivers/pci/quirks.c
+++ b/drivers/pci/quirks.c
@@ -2572,6 +2572,19 @@ void pci_fixup_device(enum pci_fixup_pass pass, struct pci_dev *dev)
}
pci_do_fixups(dev, start, end);
}
+
+static int __devinit pci_apply_final_quirks(void)
+{
+ struct pci_dev *dev = NULL;
+
+ while ((dev = pci_get_device(PCI_ANY_ID, PCI_ANY_ID, dev)) != NULL) {
+ pci_fixup_device(pci_fixup_final, dev);
+ }
+
+ return 0;
+}
+
+device_initcall(pci_apply_final_quirks);
#else
void pci_fixup_device(enum pci_fixup_pass pass, struct pci_dev *dev) {}
#endif
--
1.6.5.rc2


2009-10-12 12:07:22

by David Woodhouse

[permalink] [raw]
Subject: [PATCH 2/4] Mark pci_apply_final_quirks() __init rather than __devinit

It doesn't get invoked on hotplug; it can be thrown away after init.

Signed-off-by: David Woodhouse <[email protected]>
---
drivers/pci/quirks.c | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
index 0142275..77bf620 100644
--- a/drivers/pci/quirks.c
+++ b/drivers/pci/quirks.c
@@ -2573,7 +2573,7 @@ void pci_fixup_device(enum pci_fixup_pass pass, struct pci_dev *dev)
pci_do_fixups(dev, start, end);
}

-static int __devinit pci_apply_final_quirks(void)
+static int __init pci_apply_final_quirks(void)
{
struct pci_dev *dev = NULL;

--
1.6.5.rc2


2009-10-12 12:07:10

by David Woodhouse

[permalink] [raw]
Subject: [PATCH 3/4] Run pci_apply_final_quirks() sooner.

Having this as a device_initcall() means that some real device drivers
can actually initialise _before_ the quirks are run, which is wrong.

We want it to run _before_ device_initcall(), but _after_ fs_initcall(),
since some arch-specific PCI initialisation like pcibios_assign_resources()
is done at fs_initcall().

We could use rootfs_initcall() but I actually want to use that for the
IOMMU initialisation, which has to come after the quirks, but still
before the real devices. So use fs_initcall_sync() instead -- since this
is entirely synchronous, it doesn't hurt that it'll escape the
synchronisation.

Signed-off-by: David Woodhouse <[email protected]>
---
drivers/pci/quirks.c | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
index 77bf620..4e1b1b8 100644
--- a/drivers/pci/quirks.c
+++ b/drivers/pci/quirks.c
@@ -2584,7 +2584,7 @@ static int __init pci_apply_final_quirks(void)
return 0;
}

-device_initcall(pci_apply_final_quirks);
+fs_initcall_sync(pci_apply_final_quirks);
#else
void pci_fixup_device(enum pci_fixup_pass pass, struct pci_dev *dev) {}
#endif
--
1.6.5.rc2


2009-10-12 12:07:14

by David Woodhouse

[permalink] [raw]
Subject: [PATCH 4/4] x86: Move pci_iommu_init to rootfs_initcall()

We want this to happen after the PCI quirks, which are now running at
the very end of the fs_initcalls.

This works around the BIOS problems which were originally addressed by
commit db8be50c4307dac2b37305fc59c8dc0f978d09ea ('USB: Work around BIOS
bugs by quiescing USB controllers earlier'), which was reverted in
commit d93a8f829fe1d2f3002f2c6ddb553d12db420412.

Signed-off-by: David Woodhouse <[email protected]>
---
arch/x86/kernel/pci-dma.c | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/arch/x86/kernel/pci-dma.c b/arch/x86/kernel/pci-dma.c
index d20009b..b2a71dc 100644
--- a/arch/x86/kernel/pci-dma.c
+++ b/arch/x86/kernel/pci-dma.c
@@ -311,7 +311,7 @@ void pci_iommu_shutdown(void)
amd_iommu_shutdown();
}
/* Must execute after PCI subsystem */
-fs_initcall(pci_iommu_init);
+rootfs_initcall(pci_iommu_init);

#ifdef CONFIG_PCI
/* Many VIA bridges seem to corrupt data for DAC. Disable it here */
--
1.6.5.rc2

2009-10-12 13:16:30

by Michael Büsch

[permalink] [raw]
Subject: Re: [Bug #14278] New message "NOHZ: local_softirq_pending 08" at each ping request

On Monday 12 October 2009 00:22:00 Rafael J. Wysocki wrote:
> This message has been generated automatically as a part of a report
> of recent regressions.
>
> The following bug entry is on the current list of known regressions
> from 2.6.31. Please verify if it still should be listed and let me know
> (either way).
>
>
> Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14278
> Subject : New message "NOHZ: local_softirq_pending 08" at each ping request
> Submitter : Christian Casteyde <[email protected]>
> Date : 2009-09-30 18:12 (12 days old)
> Handled-By : Michael Buesch <[email protected]>

I think John or Johannes is handling it, as my patch was rejected.

> Patch : http://bugzilla.kernel.org/attachment.cgi?id=23220

--
Greetings, Michael.

2009-10-12 13:36:25

by Mikael Pettersson

[permalink] [raw]
Subject: Re: [Bug #14374] MCEs caused by commit db8be50c4307dac2b37305fc59c8dc0f978d09ea

Rafael J. Wysocki wrote:
> The following bug entry is on the current list of known regressions
> from 2.6.31. Please verify if it still should be listed and let me know
> (either way).
>
>
> Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=3D14374
> Subject : MCEs caused by commit db8be50c4307dac2b37305fc59c8dc0f978d09ea
> Submitter : Nick Piggin <[email protected]>
> Date : 2009-10-02 7:34 (10 days old)
> First-Bad-Commit: http://git.kernel.org/?p=3Dlinux/kernel/git/torvalds/linu=
> x-2.6.git;a=3Dcommit;h=3Ddb8be50c4307dac2b37305fc59c8dc0f978d09ea
> References : http://marc.info/?l=3Dlinux-kernel&m=3D125446885705223&w=3D4

Fixed for 2.6.32-rc5 by d93a8f829fe1d2f3002f2c6ddb553d12db420412
'Revert "USB: Work around BIOS bugs by quiescing USB controllers earlier"'.

2009-10-12 14:27:46

by David Woodhouse

[permalink] [raw]
Subject: Re: 2.6.32-rc4: Reported regressions from 2.6.31

On Mon, 2009-10-12 at 11:19 +0100, David Woodhouse wrote:
> So the better fix is probably just to fix that problem -- move the final
> PCI quirks so they happen a little earlier. If we move them to
> fs_initcall_sync() and then move the IOMMU init to rootfs_initcall(),
> then everything ought to work, I think...

I've sent that series of patches for review. Assuming you like it,
please pull those patches and a couple of other fixes -- one cosmetic
and one serious crash fix (another BIOS workaround, in fact), from

git://git.infradead.org/~dwmw2/iommu-2.6.32.git

David Woodhouse (5):
intel-iommu: Yet another BIOS workaround: Isoch DMAR unit with no TLB space
Rename pci_init() to pci_apply_final_quirks(), move it to quirks.c
Mark pci_apply_final_quirks() __init rather than __devinit
Run pci_apply_final_quirks() sooner.
x86: Move pci_iommu_init to rootfs_initcall()

Roland Dreier (2):
intel-iommu: Make "Unknown DMAR structure" message more informative
intel-iommu: Decode (and ignore) RHSA entries

arch/x86/kernel/pci-dma.c | 2 +-
drivers/pci/dmar.c | 13 ++++++-
drivers/pci/intel-iommu.c | 82 ++++++++++++++++++++++++++++++++++++++++++---
drivers/pci/pci.c | 13 -------
drivers/pci/quirks.c | 13 +++++++
5 files changed, 103 insertions(+), 20 deletions(-)


--
David Woodhouse Open Source Technology Centre
[email protected] Intel Corporation

2009-10-12 15:24:26

by CtRiX

[permalink] [raw]
Subject: Re: [Bug #14378] Problems with net/core/skbuff.c

David Miller ha scritto:
> From: "Rafael J. Wysocki" <[email protected]>
> Date: Mon, 12 Oct 2009 00:22:04 +0200 (CEST)
>
>> Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14378
>> Subject : Problems with net/core/skbuff.c
>> Submitter : Massimo Cetra <[email protected]>
>> Date : 2009-10-08 14:51 (4 days old)
>> References : http://marc.info/?l=linux-kernel&m=125501488220358&w=4
>
> I don't know what to do about this one.
>
> The user indicates that they have the vserver patches applied,
> so maybe there is some interaction with that stuff.

Actually i don't think that it's Vserver related.

I'm not a kernel hacker but the vserver patch doesn't seems to have much
relation with networking (for who don't know vserver, it's a sort of
advanced chroot).

I'm using OCFS2 and drbd. Both of those 2 things make heavy use of
networking to share data between two hosts.

The backtrace seems to begin in the "world" of OFCS2.
I'm CC'ing ocfs mailing list where someome may have some clues.

Thanks,
Max




2009-10-12 15:27:35

by Linus Torvalds

[permalink] [raw]
Subject: Re: 2.6.32-rc4: Reported regressions from 2.6.31



On Mon, 12 Oct 2009, David Woodhouse wrote:
>
> Well, according to the design, the IOMMU code is doing the right thing?.
>
> The theory is that the BIOS _tells_

There is no "theory". There is only crap BIOSes. Stop living in a dream
world, and stop making arguments that are only relevant in that dream
world.

> The only viable solution (short of an open source BIOS written by sober
> people)

Again, you're living in that dream world. Wake up, sheeple.

BIOS writers write crap, because it's a crap job. It's that simple. Yes,
they're probably drunk or drugged up, but they need it to deal with the
hand they have been dealt.

There are absolutely _zero_ incentives to write good firmware for any
particular device (and nothing else matters), because the projects are all
(a) largely "throw-over-the-fence-and-forget-about-it" for any particular
machine (ie any fixes are mostly relevant for the _next_ generation of
machines), and (b) they have no actual user incentives or feedback, since
nobody really "runs" the firmware anyway - it's supposed to be invisible
by design and just sets things up.

So outside of the testing that it gets (_before_ it ever hits any users
hands) there is never _ever_ going to be any QA. Once it is in user
hands, it is what it is. Almost nobody upgrades their firmware in
practice. Yeah, geeks may do. Normal people? Not so much.

And that would _not_ change even with an open source BIOS. Why? Because
the _code_ generally isn't the problem. The problem tends to be all the
localization for any particular machine.

Even if the code was open source and perfect, we'd still have trouble with
firmware - because 99% of it is driven by tables that by design have to be
changed for each machine (yes, yes, a BIOS is easily a megabyte of
compressed code too, but a lot of it is the nice GUI for setup etc - the
actual code that runs on _bootup_ is rather small, and is all about those
tables).

We might have a few _fewer_ problems if the code was perfect, and maybe
we'd have missed this particular issue (but I doubt it, it's still a
localized table). But it wouldn't really change any of the fundamentals in
the issue.

BTW, in this case, it's not even necessarily the firmware people who are
to blame. It's a combination of (a) USB is crazy polled DMA and (b) IOMMU
is a whole new thing and (c) ACPI is too f*cking complex so nobody will
_ever_ get it right anyway, even if the firmware people didn't have
everything against them.

So your arguments about RMRR tables and "should do" are POINTLESS. Just
give it up.

The sane thing to do is to have a legacy IOMMU mapping until all devices
are initialized, so that things work _despite_ the inevitable BIOS
crapola. End of story.

So stop blaming the BIOS. We _know_ firmware is crap - there is no point
in blaming it. The response to "firmware bug" should be "oh, of course -
and our code was too fragile, since it didn't take that into account".

And stop saying these problems would magically go away with open-source
firmware. That just shows that you don't understand the realities of the
situation. Even an open-source bios would end up having buggy tables, and
even with an opensource bios, users generally wouldn't upgrade it.

"If wishes were horses, beggars would ride"

> was to quiesce the USB controllers before enabling the IOMMU.

The other solution would be to just _enable_ (and do all the setup) of the
IOMMU early. And then just leave a legacy mapping for the IOMMU, and then
_after_all_devices_are_set_up_ can you then remove the legacy mapping.

I don't much care. Just as long as the DMA works for the whole bus setup.

> The final PCI quirks are currently run from pci_init() which is a
> device_initcall(), which is too late -- in fact, it could actually be
> _after_ some of the real device drivers have taken control of the same
> hardware.

Now, this I can certainly agree 100% with. We can and should move the
final quirks up a bit. And yes, I absolutely think we should at least
_guarantee_ that those quirks are run before any other device_initcalls,
regardless of any IOMMU issues (ie now it looks like it depends on link
ordering whether a built-in driver might run before or after pci_init()).

So I agree that we may well be able to fix this by changing the ordering.
What I disagree with is your continual "I wish things were different".
I've seen you make that "open source firmware" argument before. It's
pointless. Stop doing it.

But your patch looks fine.

That said, I think your choice of initcall is odd, even if it is
understandable. Right now we have, at least on x86:

fs_initcall(pcibios_assign_resources);

and I assume you picked fs_initcall_sync() so that it happens after that
one. No?

Which makes sense, but at the same time, it all looks just random. And
different architectures actually do it in different places (some seem to
do it inside pcibios_init() at subsys_initcall time). So I'm not even sure
fs_initcall_sync() will do it for other architectures, although it looks
like most others do their final PCI setup _earlier_ rather than later.

I'm wondering if we should not get _rid_ of that whole pci_init() (you
renamed it, and I agree - it's clearly not really "pci_init()"), and move
it to just be the last thing that pci_subsys_init() does, or perhaps all
the way into pcibios_resource_survey().

But I guess there could be random ACPI initcalls etc involved too, and
subtle ordering constraints with _those_. And we have way too many
arch-specific details here. So your patch may be the simplest one, but I
wish we could also make some of this be less of a jungle of different
initcalls.

Basically, it seems silly to have this kind of subtlety for just the final
quirk, when the _other_ quirks are all handled by generic code in very
well-defined places.

For example, one of the effects of this mess is that as far as I can tell,
PCI hotplug (or cardbus, which is a special case of PCI hotplug) will
never run the pci_fixup_late fixups at all, even though it _will_ run the
pci_fixup_header/early/enable ones (because those are done by generic
code: pci_setup_device(), pci_device_add() and pci_enable_device()).

Linus

2009-10-12 15:56:43

by Woodhouse, David

[permalink] [raw]
Subject: Re: 2.6.32-rc4: Reported regressions from 2.6.31

On Mon, 2009-10-12 at 16:24 +0100, Linus Torvalds wrote:
>
> On Mon, 12 Oct 2009, David Woodhouse wrote:
> >
> > Well, according to the design, the IOMMU code is doing the right thing¹.
> >
> > The theory is that the BIOS _tells_
>
> There is no "theory". There is only crap BIOSes. Stop living in a dream
> world, and stop making arguments that are only relevant in that dream
> world.

You're preaching to the choir, Linus.

> > The only viable solution (short of an open source BIOS written by sober
> > people)
>
> Again, you're living in that dream world. Wake up, sheeple.

...

> So your arguments about RMRR tables and "should do" are POINTLESS. Just
> give it up.

I didn't intend that as an argument. That was just a statement of how
the architecture specifies it's _supposed_ to work. I _said_ it was
nonsense, in my own footnote.

> The sane thing to do is to have a legacy IOMMU mapping until all devices
> are initialized, so that things work _despite_ the inevitable BIOS
> crapola. End of story.

Damn right.

That's precisely what the original "kill USB DMA early" patch was
intending to do -- and that's why I've been telling the people
responsible for the RMRR design that it was a stupid idea in the first
place.

The problem with the original patch was just that I didn't realise that
some architectures didn't have ioremap() working at the time that
PCI_FIXUP_HEADER() runs.

> So stop blaming the BIOS. We _know_ firmware is crap - there is no point
> in blaming it. The response to "firmware bug" should be "oh, of course -
> and our code was too fragile, since it didn't take that into account".

Yes. That's why our code has grown a metric shitload of BIOS
workarounds, and been refactored to be less fragile, since I've taken
over responsibility for it.

You won't stop me hating the BIOS for it though. And although I agree
that it wouldn't be perfect if we had an open source BIOS, it _would_ be
a whole lot better.

> The other solution would be to just _enable_ (and do all the setup) of the
> IOMMU early. And then just leave a legacy mapping for the IOMMU, and then
> _after_all_devices_are_set_up_ can you then remove the legacy mapping.

That involves allocating a _shitload_ of page tables for a 1:1 mapping
of all of physical memory. We used to do that to pander to broken
graphics drivers, and the AMD guys asked me to stop because it was
actually quite painful for them to do the same.

> But your patch looks fine.
>
> That said, I think your choice of initcall is odd, even if it is
> understandable. Right now we have, at least on x86:
>
> fs_initcall(pcibios_assign_resources);
>
> and I assume you picked fs_initcall_sync() so that it happens after that
> one. No?

Right.

> Which makes sense, but at the same time, it all looks just random.

No more random than the fact that pcibios_assign_resources() is marked
as a FILE SYSTEM initcall in the first place, surely?

We have a very limited method of expressing ordering constraints for
initcalls; we make do with what we have. Some people around here have
been known to argue _very_ vocally against a more comprehensive
framework for ordering them.

Yes, sometimes it looks odd :)

> And
> different architectures actually do it in different places (some seem to
> do it inside pcibios_init() at subsys_initcall time). So I'm not even sure
> fs_initcall_sync() will do it for other architectures, although it looks
> like most others do their final PCI setup _earlier_ rather than later.

They will do it earlier; never later. It would be completely broken to
do it any later -- fs_initcall_sync() and rootfs_initcall() are new
inventions, so such an architecture would have to be doing it in
device_initcall(). And that would just be totally broken.

> But I guess there could be random ACPI initcalls etc involved too, and
> subtle ordering constraints with _those_. And we have way too many
> arch-specific details here. So your patch may be the simplest one, but I
> wish we could also make some of this be less of a jungle of different
> initcalls.

Yeah, it's a complete mess of generic and arch-specific initcalls, all
invoked at different times. It would be good to clean that up, perhaps,
but maybe not for 2.6.32.

> Basically, it seems silly to have this kind of subtlety for just the final
> quirk, when the _other_ quirks are all handled by generic code in very
> well-defined places.

I did think briefly about adding it to pci_subsys_init(), but moving it
into arch-specific code didn't seem like a great idea. And the
pcibios_assign_resources() initcall happens _after_ that, anyway.

--
David Woodhouse Open Source Technology Centre
[email protected] Intel Corporation

2009-10-12 16:20:50

by Alan

[permalink] [raw]
Subject: Re: 2.6.32-rc4: Reported regressions from 2.6.31

> That involves allocating a _shitload_ of page tables for a 1:1 mapping
> of all of physical memory. We used to do that to pander to broken
> graphics drivers, and the AMD guys asked me to stop because it was
> actually quite painful for them to do the same.

SMM isn't going to be touching high pages. In fact I wouldn't be the
least suprised if simply mapping the low 1MB including EBDA/ISA space and
the ACPI reserved areas worked simply because if it touches any other RAM
in SMM mode its going to be peeing over our memory spaces.

And if I remember rightly the magic SMM remapped memory area is quite
small (64K) and low down too plus not DMAable (because otherwise DMA in
the real OS to the pages SMM aliases over will go kersplat if an SMM
occurs)

Alan

2009-10-12 17:36:54

by Linus Torvalds

[permalink] [raw]
Subject: Re: 2.6.32-rc4: Reported regressions from 2.6.31



On Mon, 12 Oct 2009, David Woodhouse wrote:
>
> > The other solution would be to just _enable_ (and do all the setup) of the
> > IOMMU early. And then just leave a legacy mapping for the IOMMU, and then
> > _after_all_devices_are_set_up_ can you then remove the legacy mapping.
>
> That involves allocating a _shitload_ of page tables for a 1:1 mapping
> of all of physical memory.

I don't think that's true.

Nobody cares about "all physical memory". For one thing, we know that
anything that we consider to be normal memory (ie it's listed in the e820
tables as RAM) can't be all that interesting, since the BIOS definitely
released that to us.

That said, as long as the IOMMU is clearly enabled after the quirks have
run, for this particular case we don't much care.

But I could also imagine something similar happening for some BIOS-enabled
ethernet device being set up to listen to packets into some BIOS data
areas (left-overs from whatever bootp thing or other), which doesn't have
a quirk, and which ends up doing DMA until we actually load the driver.

Of course, we'd hope that the DMA just fails and nothing bad really
happens (hopefully the driver re-init will clear up any hung device). But
I can also imagine the hardware simply being really really unhappy, and
not recovering.

So in many ways it would be safest to leave memory that we don't know
about and we don't own as DMA'able in the IOMMU.

And no, I don't think it would be a shitload of pages. Quite the reverse.
It's probably not very many at all.

Linus

2009-10-12 18:49:34

by Fabio Comolli

[permalink] [raw]
Subject: Re: [Bug #14372] Wireless not working after suspend-resume

Still present with -rc4.

Error is:

[ 690.547660] wlan0: direct probe to AP 00:11:95:09:2c:e0 (try 1)
[ 690.550064] wlan0: direct probe responded
[ 690.550074] wlan0: authenticate with AP 00:11:95:09:2c:e0 (try 1)
[ 690.551629] wlan0: authenticated
[ 690.551663] wlan0: associate with AP 00:11:95:09:2c:e0 (try 1)
[ 690.553289] wlan0: RX AssocResp from 00:11:95:09:2c:e0 (capab=0x451
status=12 aid=1)
[ 690.553298] wlan0: AP denied association (code=12)

rfkill-ing the device off and then on fixes the problem.

Fabio




On Mon, Oct 12, 2009 at 12:22 AM, Rafael J. Wysocki <[email protected]> wrote:
> This message has been generated automatically as a part of a report
> of recent regressions.
>
> The following bug entry is on the current list of known regressions
> from 2.6.31.  Please verify if it still should be listed and let me know
> (either way).
>
>
> Bug-Entry       : http://bugzilla.kernel.org/show_bug.cgi?id=14372
> Subject         : Wireless not working after suspend-resume
> Submitter       : Fabio Comolli <[email protected]>
> Date            : 2009-10-03 15:36 (9 days old)
> References      : http://lkml.org/lkml/2009/10/3/91
>
>
>

2009-10-12 20:30:00

by Woodhouse, David

[permalink] [raw]
Subject: Re: 2.6.32-rc4: Reported regressions from 2.6.31

On Mon, 2009-10-12 at 18:35 +0100, Linus Torvalds wrote:
>
> On Mon, 12 Oct 2009, David Woodhouse wrote:
> >
> > > The other solution would be to just _enable_ (and do all the setup) of the
> > > IOMMU early. And then just leave a legacy mapping for the IOMMU, and then
> > > _after_all_devices_are_set_up_ can you then remove the legacy mapping.
> >
> > That involves allocating a _shitload_ of page tables for a 1:1 mapping
> > of all of physical memory.
>
> I don't think that's true.
>
> Nobody cares about "all physical memory". For one thing, we know that
> anything that we consider to be normal memory (ie it's listed in the e820
> tables as RAM) can't be all that interesting, since the BIOS definitely
> released that to us.

Yeah, that's probably true. I was thinking of the 1:1 mappings we set up
for crappy drivers under Linux, which do need access to all of memory
and don't use the DMA API correctly. This is different, though.

We have talked about dma-mapping all of E820-reserved memory, but that
would allow a rogue device to scribble over arbitrary bits of memory
belonging to the BIOS, which isn't necessarily such a good thing. Even
if we did tear it down as soon as a real driver came along, the TXT
folks would still object -- and rightly so.

As you said, it's sufficient just to enable the IOMMU after the quirks
have run. So the patches I posted earlier should be just fine.

--
David Woodhouse Open Source Technology Centre
[email protected] Intel Corporation

2009-10-12 21:48:41

by Rafael J. Wysocki

[permalink] [raw]
Subject: Re: [Bug #14278] New message "NOHZ: local_softirq_pending 08" at each ping request

On Monday 12 October 2009, Michael Buesch wrote:
> On Monday 12 October 2009 00:22:00 Rafael J. Wysocki wrote:
> > This message has been generated automatically as a part of a report
> > of recent regressions.
> >
> > The following bug entry is on the current list of known regressions
> > from 2.6.31. Please verify if it still should be listed and let me know
> > (either way).
> >
> >
> > Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14278
> > Subject : New message "NOHZ: local_softirq_pending 08" at each ping request
> > Submitter : Christian Casteyde <[email protected]>
> > Date : 2009-09-30 18:12 (12 days old)
> > Handled-By : Michael Buesch <[email protected]>
>
> I think John or Johannes is handling it, as my patch was rejected.
>
> > Patch : http://bugzilla.kernel.org/attachment.cgi?id=23220

Johannes, what's the status here?

Rafael

2009-10-12 21:49:55

by Rafael J. Wysocki

[permalink] [raw]
Subject: Re: [Bug #14279] Suspend to RAM freeze totally since 2.6.32-rc1 - Acer Aspire 1511Lmi laptop

On Monday 12 October 2009, Jan Beulich wrote:
> The issue is still present, albeit a patch (available in the bugzilla entry) had
> been posted on Oct 6th. Jan

Thanks for the update. I saw the patch go into the tip tree a few moments ago.

Best,
Rafael

2009-10-12 21:52:01

by Rafael J. Wysocki

[permalink] [raw]
Subject: Re: [Bug #14370] ext4 corruptions

On Monday 12 October 2009, Alexey Fisher wrote:
> Am Montag, den 12.10.2009, 00:22 +0200 schrieb Rafael J. Wysocki:
> > This message has been generated automatically as a part of a report
> > of recent regressions.
> >
> > The following bug entry is on the current list of known regressions
> > from 2.6.31. Please verify if it still should be listed and let me know
> > (either way).
> >
> >
> > Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14370
> > Subject : ext4 corruptions
> > Submitter : Alexey Fisher <[email protected]>
> > Date : 2009-10-09 19:20 (3 days old)
> > References : http://marc.info/?l=linux-kernel&m=125511643504864&w=4
> >
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe kernel-testers" in
> > the body of a message to [email protected]
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
>
> This can be removed, it is duplicate of this bug:
> http://bugzilla.kernel.org/show_bug.cgi?id=14354

Thanks, I closed #14370 as a duplicate of #14354.

Best,
Rafael

2009-10-12 21:54:55

by Rafael J. Wysocki

[permalink] [raw]
Subject: Re: [Bug #14372] Wireless not working after suspend-resume

On Monday 12 October 2009, Fabio Comolli wrote:
> Still present with -rc4.
>
> Error is:
>
> [ 690.547660] wlan0: direct probe to AP 00:11:95:09:2c:e0 (try 1)
> [ 690.550064] wlan0: direct probe responded
> [ 690.550074] wlan0: authenticate with AP 00:11:95:09:2c:e0 (try 1)
> [ 690.551629] wlan0: authenticated
> [ 690.551663] wlan0: associate with AP 00:11:95:09:2c:e0 (try 1)
> [ 690.553289] wlan0: RX AssocResp from 00:11:95:09:2c:e0 (capab=0x451
> status=12 aid=1)
> [ 690.553298] wlan0: AP denied association (code=12)
>
> rfkill-ing the device off and then on fixes the problem.

Thanks for the update.

Rafael

2009-10-12 21:55:10

by Rafael J. Wysocki

[permalink] [raw]
Subject: Re: [Bug #14374] MCEs caused by commit db8be50c4307dac2b37305fc59c8dc0f978d09ea

On Monday 12 October 2009, Mikael Pettersson wrote:
> Rafael J. Wysocki wrote:
> > The following bug entry is on the current list of known regressions
> > from 2.6.31. Please verify if it still should be listed and let me know
> > (either way).
> >
> >
> > Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=3D14374
> > Subject : MCEs caused by commit db8be50c4307dac2b37305fc59c8dc0f978d09ea
> > Submitter : Nick Piggin <[email protected]>
> > Date : 2009-10-02 7:34 (10 days old)
> > First-Bad-Commit: http://git.kernel.org/?p=3Dlinux/kernel/git/torvalds/linu=
> > x-2.6.git;a=3Dcommit;h=3Ddb8be50c4307dac2b37305fc59c8dc0f978d09ea
> > References : http://marc.info/?l=3Dlinux-kernel&m=3D125446885705223&w=3D4
>
> Fixed for 2.6.32-rc5 by d93a8f829fe1d2f3002f2c6ddb553d12db420412
> 'Revert "USB: Work around BIOS bugs by quiescing USB controllers earlier"'.

Thanks, closed.

Rafael

2009-10-12 21:57:44

by Rafael J. Wysocki

[permalink] [raw]
Subject: Re: [Bug #14379] ACPI Warning for _SB_.BAT0._BIF: Converted Buffer to expected String

On Monday 12 October 2009, Justin P. Mattock wrote:
> Lin Ming wrote:
> > On Mon, 2009-10-12 at 11:14 +0800, Justin Mattock wrote:
> >
> >> On Sun, Oct 11, 2009 at 3:22 PM, Rafael J. Wysocki<[email protected]> wrote:
> >>
> >>> This message has been generated automatically as a part of a report
> >>> of recent regressions.
> >>>
> >>> The following bug entry is on the current list of known regressions
> >>> from 2.6.31. Please verify if it still should be listed and let me know
> >>> (either way).
> >>>
> >>>
> >>> Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14379
> >>> Subject : ACPI Warning for _SB_.BAT0._BIF: Converted Buffer to expected String
> >>> Submitter : Justin Mattock<[email protected]>
> >>> Date : 2009-10-08 21:46 (4 days old)
> >>> References : http://marc.info/?l=linux-kernel&m=125504031328941&w=4
> >>>
> >>>
> >>>
> >>>
> >> pulled rc4 today and the warning is still present.
> >> here is the results of git bisect:
> >>
> >
> > Patch at http://bugzilla.kernel.org/show_bug.cgi?id=14379#c1 fixes the
> > warning.
> >
> > Actually, the _BIF warning was fixed at commit
> > e899b6485c332aa2d7510739507ab5e5d7b28e59.
> > See http://bugzilla.kernel.org/show_bug.cgi?id=11822#c14
> >
> > I made a mistake at the cleanup of predefined method table so the
> > warning shows up again.
> >
> > Lin Ming
> >
> >
> Cool, I can give that a try.
> then I guess just leave this open(bugreport) until this patch
> http://bugzilla.kernel.org/show_bug.cgi?id=14379#c1
> gets pushed into the main tree then close the bugreport.

I updated the bug entry with the patch information.

Thanks,
Rafael

2009-10-13 03:12:43

by Justin P. Mattock

[permalink] [raw]
Subject: Re: [Bug #14379] ACPI Warning for _SB_.BAT0._BIF: Converted Buffer to expected String

Rafael J. Wysocki wrote:
> On Monday 12 October 2009, Justin P. Mattock wrote:
>
>> Lin Ming wrote:
>>
>>> On Mon, 2009-10-12 at 11:14 +0800, Justin Mattock wrote:
>>>
>>>
>>>> On Sun, Oct 11, 2009 at 3:22 PM, Rafael J. Wysocki<[email protected]> wrote:
>>>>
>>>>
>>>>> This message has been generated automatically as a part of a report
>>>>> of recent regressions.
>>>>>
>>>>> The following bug entry is on the current list of known regressions
>>>>> from 2.6.31. Please verify if it still should be listed and let me know
>>>>> (either way).
>>>>>
>>>>>
>>>>> Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14379
>>>>> Subject : ACPI Warning for _SB_.BAT0._BIF: Converted Buffer to expected String
>>>>> Submitter : Justin Mattock<[email protected]>
>>>>> Date : 2009-10-08 21:46 (4 days old)
>>>>> References : http://marc.info/?l=linux-kernel&m=125504031328941&w=4
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>> pulled rc4 today and the warning is still present.
>>>> here is the results of git bisect:
>>>>
>>>>
>>> Patch at http://bugzilla.kernel.org/show_bug.cgi?id=14379#c1 fixes the
>>> warning.
>>>
>>> Actually, the _BIF warning was fixed at commit
>>> e899b6485c332aa2d7510739507ab5e5d7b28e59.
>>> See http://bugzilla.kernel.org/show_bug.cgi?id=11822#c14
>>>
>>> I made a mistake at the cleanup of predefined method table so the
>>> warning shows up again.
>>>
>>> Lin Ming
>>>
>>>
>>>
>> Cool, I can give that a try.
>> then I guess just leave this open(bugreport) until this patch
>> http://bugzilla.kernel.org/show_bug.cgi?id=14379#c1
>> gets pushed into the main tree then close the bugreport.
>>
>
> I updated the bug entry with the patch information.
>
> Thanks,
> Rafael
>
>
o.k. after somehow blowing my system up, I'm up and
running. Anyways I applied that patch and yes it works.

Justin P. Mattock

2009-10-13 07:08:22

by Maciej Rutecki

[permalink] [raw]
Subject: Re: [Bug #14372] Wireless not working after suspend-resume

CC added

2009/10/12 Fabio Comolli <[email protected]>:

[...]
> [  690.553298] wlan0: AP denied association (code=12)
[...]

I have the same code in different situation:
http://bugzilla.intellinuxwireless.org/show_bug.cgi?id=2089

Regards
--
Maciej Rutecki
http://www.maciek.unixy.pl

2009-10-13 09:24:33

by Massimo Cetra

[permalink] [raw]
Subject: Re: [Bug #14378] Problems with net/core/skbuff.c

[60257.728500] ------------[ cut here ]------------
[60257.728500] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60257.728500] Hardware name:
[60257.728500] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60257.728500] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1
[60257.728500] Call Trace:
[60257.728500] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60257.728500] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60257.728500] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60257.728500] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60257.728500] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60257.728500] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60257.728500] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60257.728500] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500] [<ffffffff81308309>] ? printk+0x4e/0x5d
[60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60257.728500] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60257.728500] [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60257.728500] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60257.728500] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60257.728500] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60257.728500] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60257.728500] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60257.728500] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60257.728500] [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60257.728500] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60257.728500] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60257.728500] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60257.728500] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60257.728500] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60257.728500] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60257.728500] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60257.728500] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60257.728500] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60257.728500] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60257.728500] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60257.728500] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60257.728500] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60257.728500] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60257.728500] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60257.728500] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60257.728500] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60257.728500] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60257.728500] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60257.728500] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60257.728500] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60257.728500] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60257.728500] [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60257.728500] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60257.728500] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60257.728500] [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60257.728500] ---[ end trace 694817acca794f2c ]---
[60257.728500] ------------[ cut here ]------------
[60257.728500] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60257.728500] Hardware name:
[60257.728500] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60257.728500] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1
[60257.728500] Call Trace:
[60257.728500] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60257.728500] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60257.728500] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60257.728500] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60257.728500] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60257.728500] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60257.728500] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60257.728500] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500] [<ffffffff81308309>] ? printk+0x4e/0x5d
[60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60257.728500] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60257.728500] [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60257.728500] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60257.728500] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60257.728500] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60257.728500] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60257.728500] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60257.728500] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60257.728500] [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60257.728500] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60257.728500] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60257.728500] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60257.728500] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60257.728500] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60257.728500] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60257.728500] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60257.728500] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60257.728500] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60257.728500] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60257.728500] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60257.728500] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60257.728500] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60257.728500] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60257.728500] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60257.728500] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60257.728500] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60257.728500] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60257.728500] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60257.728500] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60257.728500] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60257.728500] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60257.728500] [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60257.728500] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60257.728500] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60257.728500] [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60257.728500] ---[ end trace 694817acca794f2d ]---
[60257.728500] ------------[ cut here ]------------
[60257.728500] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60257.728500] Hardware name:
[60257.728500] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60257.728500] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1
[60257.728500] Call Trace:
[60257.728500] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60257.728500] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60257.728500] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60257.728500] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60257.728500] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60257.728500] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60257.728500] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60257.728500] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500] [<ffffffff81308309>] ? printk+0x4e/0x5d
[60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60257.728500] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60257.728500] [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60257.728500] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60257.728500] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60257.728500] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60257.728500] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60257.728500] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60257.728500] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60257.728500] [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60257.728500] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60257.728500] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60257.728500] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60257.728500] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60257.728500] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60257.728500] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60257.728500] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60257.728500] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60257.728500] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60257.728500] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60257.728500] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60257.728500] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60257.728500] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60257.728500] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60257.728500] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60257.728500] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60257.728500] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60257.728500] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60257.728500] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60257.728500] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60257.728500] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60257.728500] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60257.728500] [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60257.728500] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60257.728500] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60257.728500] [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60257.728500] ---[ end trace 694817acca794f2e ]---
[60257.728500] ------------[ cut here ]------------
[60257.728500] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60257.728500] Hardware name:
[60257.728500] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60257.728500] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1
[60257.728500] Call Trace:
[60257.728500] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60257.728500] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60257.728500] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60257.728500] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60257.728500] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60257.728500] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60257.728500] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60257.728500] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500] [<ffffffff81308309>] ? printk+0x4e/0x5d
[60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60257.728500] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60257.728500] [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60257.728500] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60257.728500] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60257.728500] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60257.728500] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60257.728500] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60257.728500] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60257.728500] [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60257.728500] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60257.728500] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60257.728500] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60257.728500] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60257.728500] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60257.728500] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60257.728500] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60257.728500] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60257.728500] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60257.728500] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60257.728500] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60257.728500] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60257.728500] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60257.728500] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60257.728500] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60257.728500] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60257.728500] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60257.728500] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60257.728500] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60257.728500] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60257.728500] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60257.728500] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60257.728500] [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60257.728500] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60257.728500] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60257.728500] [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60257.728500] ---[ end trace 694817acca794f2f ]---
[60257.728500] ------------[ cut here ]------------
[60257.728500] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60257.728500] Hardware name:
[60257.728500] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60257.728500] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1
[60257.728500] Call Trace:
[60257.728500] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60257.728500] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60257.728500] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60257.728500] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60257.728500] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60257.728500] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60257.728500] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60257.728500] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500] [<ffffffff81308309>] ? printk+0x4e/0x5d
[60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60257.728500] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60257.728500] [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60257.728500] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60257.728500] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60257.728500] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60257.728500] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60257.728500] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60257.728500] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60257.728500] [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60257.728500] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60257.728500] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60257.728500] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60257.728500] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60257.728500] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60257.728500] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60257.728500] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60257.728500] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60257.728500] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60257.728500] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60257.728500] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60257.728500] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60257.728500] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60257.728500] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60257.728500] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60257.728500] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60257.728500] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60257.728500] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60257.728500] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60257.728500] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60257.728500] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60257.728500] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60257.728500] [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60257.728500] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60257.728500] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60257.728500] [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60257.728500] ---[ end trace 694817acca794f30 ]---
[60257.728500] ------------[ cut here ]------------
[60257.728500] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60257.728500] Hardware name:
[60257.728500] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60257.728500] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1
[60257.728500] Call Trace:
[60257.728500] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60257.728500] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60257.728500] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60257.728500] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60257.728500] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60257.728500] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60257.728500] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60257.728500] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500] [<ffffffff81308309>] ? printk+0x4e/0x5d
[60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60257.728500] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60257.728500] [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60257.728500] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60257.728500] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60257.728500] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60257.728500] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60257.728500] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60257.728500] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60257.728500] [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60257.728500] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60257.728500] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60257.728500] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60257.728500] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60257.728500] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60257.728500] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60257.728500] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60257.728500] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60257.728500] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60257.728500] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60257.728500] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60257.728500] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60257.728500] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60257.728500] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60257.728500] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60257.728500] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60257.728500] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60257.728500] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60257.728500] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60257.728500] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60257.728500] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60257.728500] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60257.728500] [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60257.728500] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60257.728500] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60257.728500] [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60257.728500] ---[ end trace 694817acca794f31 ]---
[60257.728500] ------------[ cut here ]------------
[60257.728500] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60257.728500] Hardware name:
[60257.728500] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60257.728500] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1
[60257.728500] Call Trace:
[60257.728500] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60257.728500] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60257.728500] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60257.728500] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60257.728500] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60257.728500] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60257.728500] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60257.728500] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500] [<ffffffff81308309>] ? printk+0x4e/0x5d
[60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60257.728500] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60257.728500] [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60257.728500] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60257.728500] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60257.728500] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60257.728500] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60257.728500] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60257.728500] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60257.728500] [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60257.728500] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60257.728500] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60257.728500] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60257.728500] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60257.728500] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60257.728500] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60257.728500] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60257.728500] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60257.728500] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60257.728500] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60257.728500] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60257.728500] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60257.728500] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60257.728500] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60257.728500] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60257.728500] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60257.728500] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60257.728500] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60257.728500] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60257.728500] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60257.728500] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60257.728500] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60257.728500] [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60257.728500] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60257.728500] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60257.728500] [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60257.728500] ---[ end trace 694817acca794f32 ]---
[60257.728500] ------------[ cut here ]------------
[60257.728500] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60257.728500] Hardware name:
[60257.728500] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60257.728500] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1
[60257.728500] Call Trace:
[60257.728500] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60257.728500] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60257.728500] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60257.728500] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60257.728500] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60257.728500] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60257.728500] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60257.728500] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500] [<ffffffff81308309>] ? printk+0x4e/0x5d
[60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60257.728500] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60257.728500] [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60257.728500] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60257.728500] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60257.728500] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60257.728500] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60257.728500] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60257.728500] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60257.728500] [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60257.728500] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60257.728500] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60257.728500] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60257.728500] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60257.728500] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60257.728500] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60257.728500] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60257.728500] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60257.728500] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60257.728500] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60257.728500] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60257.728500] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60257.728500] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60257.728500] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60257.728500] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60257.728500] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60257.728500] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60257.728500] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60257.728500] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60257.728500] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60257.728500] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60257.728500] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60257.728500] [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60257.728500] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60257.728500] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60257.728500] [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60257.728500] ---[ end trace 694817acca794f33 ]---
[60257.728500] ------------[ cut here ]------------
[60257.728500] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60257.728500] Hardware name:
[60257.728500] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60257.728500] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1
[60257.728500] Call Trace:
[60257.728500] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60257.728500] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60257.728500] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60257.728500] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60257.728500] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60257.728500] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60257.728500] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60257.728500] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500] [<ffffffff81308309>] ? printk+0x4e/0x5d
[60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60257.728500] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60257.728500] [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60257.728500] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60257.728500] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60257.728500] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60257.728500] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60257.728500] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60257.728500] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60257.728500] [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60257.728500] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60257.728500] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60257.728500] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60257.728500] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60257.728500] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60257.728500] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60257.728500] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60257.728500] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60257.728500] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60257.728500] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60257.728500] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60257.728500] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60257.728500] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60257.728500] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60257.728500] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60257.728500] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60257.728500] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60257.728500] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60257.728500] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60257.728500] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60257.728500] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60257.728500] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60257.728500] [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60257.728500] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60257.728500] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60257.728500] [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60257.728500] ---[ end trace 694817acca794f34 ]---
[60257.728500] ------------[ cut here ]------------
[60257.728500] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60257.728500] Hardware name:
[60257.728500] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60257.728500] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1
[60257.728500] Call Trace:
[60257.728500] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60257.728500] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60257.728500] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60257.728500] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60257.728500] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60257.728500] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60257.728500] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60257.728500] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500] [<ffffffff81308309>] ? printk+0x4e/0x5d
[60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60257.728500] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60257.728500] [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60257.728500] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60257.728500] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60257.728500] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60257.728500] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60257.728500] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60257.728500] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60257.728500] [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60257.728500] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60257.728500] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60257.728500] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60257.728500] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60257.728500] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60257.728500] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60257.728500] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60257.728500] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60257.728500] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60257.728500] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60257.728500] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60257.728500] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60257.728500] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60257.728500] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60257.728500] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60257.728500] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60257.728500] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60257.728500] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60257.728500] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60257.728500] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60257.728500] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60257.728500] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60257.728500] [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60257.728500] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60257.728500] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60257.728500] [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60257.728500] ---[ end trace 694817acca794f35 ]---
[60258.036152] ------------[ cut here ]------------
[60258.036152] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60258.036152] Hardware name:
[60258.036152] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60258.036152] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1
[60258.036152] Call Trace:
[60258.036152] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60258.036152] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60258.036152] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60258.036152] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60258.036152] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60258.036152] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60258.036152] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60258.036152] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60258.036152] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60258.036152] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60258.036152] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60258.036152] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60258.036152] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60258.036152] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60258.036152] [<ffffffff81308309>] ? printk+0x4e/0x5d
[60258.036152] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60258.036152] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60258.036152] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60258.036152] [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60258.036152] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60258.036152] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60258.036152] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60258.036152] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60258.036152] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60258.036152] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60258.036152] [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60258.036152] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60258.036152] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60258.036152] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60258.036152] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60258.036152] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60258.036152] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60258.036152] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60258.036152] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60258.036152] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60258.036152] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60258.036152] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60258.036152] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60258.036152] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60258.036152] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60258.036152] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60258.036152] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60258.036152] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60258.036152] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60258.036152] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60258.036152] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60258.036152] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60258.036152] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60258.036152] [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60258.036152] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60258.036152] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60258.036152] [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60258.036152] ---[ end trace 694817acca794f36 ]---
[60258.036152] ------------[ cut here ]------------
[60258.036152] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60258.036152] Hardware name:
[60258.036152] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60258.036152] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1
[60258.036152] Call Trace:
[60258.036152] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60258.036152] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60258.036152] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60258.036152] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60258.036152] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60258.036152] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60258.036152] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60258.036152] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60258.036152] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60258.036152] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60258.036152] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60258.036152] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60258.036152] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60258.036152] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60258.036152] [<ffffffff81308309>] ? printk+0x4e/0x5d
[60258.036152] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60258.036152] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60258.036152] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60258.036152] [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60258.036152] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60258.036152] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60258.036152] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60258.036152] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60258.036152] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60258.036152] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60258.036152] [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60258.036152] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60258.036152] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60258.036152] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60258.036152] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60258.036152] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60258.036152] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60258.036152] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60258.036152] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60258.036152] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60258.036152] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60258.036152] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60258.036152] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60258.036152] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60258.036152] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60258.036152] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60258.036152] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60258.036152] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60258.036152] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60258.036152] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60258.036152] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60258.036152] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60258.036152] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60258.036152] [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60258.036152] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60258.036152] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60258.036152] [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60258.036152] ---[ end trace 694817acca794f37 ]---
[60258.036152] ------------[ cut here ]------------
[60258.036152] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60258.036152] Hardware name:
[60258.036152] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60258.036152] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1
[60258.036152] Call Trace:
[60258.036152] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60258.036152] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60258.036152] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60258.036152] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60258.036152] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60258.036152] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60258.036152] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60258.036152] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60258.036152] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60258.036152] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60258.036152] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60258.036152] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60258.036152] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60258.036152] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60258.036152] [<ffffffff81308309>] ? printk+0x4e/0x5d
[60258.036152] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60258.036152] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60258.036152] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60258.036152] [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60258.036152] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60258.036152] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60258.036152] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60258.036152] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60258.036152] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60258.036152] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60258.036152] [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60258.036152] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60258.036152] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60258.036152] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60258.036152] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60258.036152] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60258.036152] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60258.036152] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60258.036152] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60258.036152] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60258.036152] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60258.036152] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60258.036152] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60258.036152] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60258.036152] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60258.036152] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60258.036152] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60258.036152] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60258.036152] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60258.036152] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60258.036152] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60258.036152] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60258.036152] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60258.036152] [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60258.036152] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60258.036152] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60258.036152] [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60258.036152] ---[ end trace 694817acca794f38 ]---
[60258.036152] ------------[ cut here ]------------
[60258.036152] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60258.036152] Hardware name:
[60258.036152] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60258.036152] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1
[60258.036152] Call Trace:
[60258.036152] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60258.036152] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60258.036152] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60258.036152] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60258.036152] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60258.036152] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60258.036152] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60258.036152] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60258.036152] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60258.036152] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60258.036152] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60258.036152] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60258.036152] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60258.036152] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60258.036152] [<ffffffff81308309>] ? printk+0x4e/0x5d
[60258.036152] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60258.036152] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60258.036152] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60258.036152] [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60258.036152] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60258.036152] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60258.036152] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60258.036152] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60258.036152] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60258.036152] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60258.036152] [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60258.036152] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60258.036152] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60258.036152] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60258.036152] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60258.036152] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60258.036152] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60258.036152] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60258.036152] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60258.036152] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60258.036152] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60258.036152] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60258.036152] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60258.036152] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60258.036152] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60258.036152] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60258.036152] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60258.036152] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60258.036152] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60258.036152] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60258.036152] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60258.036152] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60258.036152] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60258.036152] [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60258.036152] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60258.036152] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60258.036152] [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60258.036152] ---[ end trace 694817acca794f39 ]---
[60258.036152] ------------[ cut here ]------------
[60258.036152] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60258.036152] Hardware name:
[60258.036152] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60258.036152] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1
[60258.036152] Call Trace:
[60258.036152] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60258.036152] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60258.036152] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60258.036152] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60258.036152] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60258.036152] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60258.036152] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60258.036152] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60258.036152] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60258.036152] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60258.036152] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60258.036152] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60258.036152] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60258.036152] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60258.036152] [<ffffffff81308309>] ? printk+0x4e/0x5d
[60258.036152] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60258.036152] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60258.036152] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60258.036152] [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60258.036152] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60258.036152] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60258.036152] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60258.036152] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60258.036152] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60258.036152] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60258.036152] [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60258.036152] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60258.036152] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60258.036152] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60258.036152] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60258.036152] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60258.036152] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60258.036152] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60258.036152] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60258.036152] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60258.036152] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60258.036152] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60258.036152] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60258.036152] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60258.036152] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60258.036152] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60258.036152] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60258.036152] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60258.036152] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60258.036152] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60258.036152] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60258.036152] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60258.036152] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60258.036152] [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60258.036152] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60258.036152] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60258.036152] [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60258.036152] ---[ end trace 694817acca794f3a ]---
[60258.036152] ------------[ cut here ]------------
[60258.036152] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60258.036152] Hardware name:
[60258.036152] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60258.036152] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1
[60258.036152] Call Trace:
[60258.036152] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60258.036152] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60258.036152] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60258.036152] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60258.036152] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60258.036152] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60258.036152] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60258.036152] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60258.036152] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60258.036152] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60258.036152] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60258.036152] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60258.036152] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60258.036152] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60258.036152] [<ffffffff81308309>] ? printk+0x4e/0x5d
[60258.036152] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60258.036152] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60258.036152] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60258.036152] [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60258.036152] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60258.036152] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60258.036152] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60258.036152] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60258.036152] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60258.036152] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60258.036152] [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60258.036152] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60258.036152] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60258.036152] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60258.036152] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60258.036152] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60258.036152] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60258.036152] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60258.036152] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60258.036152] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60258.036152] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60258.036152] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60258.036152] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60258.036152] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60258.036152] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60258.036152] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60258.036152] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60258.036152] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60258.036152] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60258.036152] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60258.036152] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60258.036152] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60258.036152] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60258.036152] [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60258.036152] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60258.036152] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60258.036152] [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60258.036152] ---[ end trace 694817acca794f3b ]---
[60258.036152] ------------[ cut here ]------------
[60258.036152] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60258.036152] Hardware name:
[60258.036152] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60258.036152] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1
[60258.036152] Call Trace:
[60258.036152] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60258.036152] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60258.036152] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60258.036152] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60258.036152] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60258.036152] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60258.036152] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60258.036152] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60258.036152] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60258.036152] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60258.036152] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60258.036152] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60258.036152] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60258.036152] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60258.036152] [<ffffffff81308309>] ? printk+0x4e/0x5d
[60258.036152] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60258.036152] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60258.036152] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60258.036152] [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60258.036152] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60258.036152] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60258.036152] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60258.036152] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60258.036152] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60258.036152] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60258.036152] [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60258.036152] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60258.036152] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60258.036152] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60258.036152] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60258.036152] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60258.036152] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60258.036152] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60258.036152] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60258.036152] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60258.036152] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60258.036152] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60258.036152] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60258.036152] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60258.036152] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60258.036152] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60258.036152] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60258.036152] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60258.036152] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60258.036152] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60258.036152] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60258.036152] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60258.036152] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60258.036152] [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60258.036152] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60258.036152] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60258.036152] [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60258.036152] ---[ end trace 694817acca794f3c ]---
[60259.116162] ------------[ cut here ]------------
[60259.116162] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60259.116162] Hardware name:
[60259.116162] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60259.116162] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1
[60259.116162] Call Trace:
[60259.116162] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60259.116162] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60259.116162] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60259.116162] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60259.116162] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60259.116162] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60259.116162] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60259.116162] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162] [<ffffffff81308309>] ? printk+0x4e/0x5d
[60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60259.116162] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60259.116162] [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60259.116162] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60259.116162] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60259.116162] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60259.116162] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60259.116162] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60259.116162] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60259.116162] [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60259.116162] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60259.116162] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60259.116162] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60259.116162] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60259.116162] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60259.116162] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60259.116162] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60259.116162] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60259.116162] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60259.116162] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60259.116162] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60259.116162] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60259.116162] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60259.116162] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60259.116162] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60259.116162] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60259.116162] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60259.116162] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60259.116162] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60259.116162] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60259.116162] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60259.116162] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60259.116162] [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60259.116162] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60259.116162] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60259.116162] [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60259.116162] ---[ end trace 694817acca794f3d ]---
[60259.116162] ------------[ cut here ]------------
[60259.116162] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60259.116162] Hardware name:
[60259.116162] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60259.116162] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1
[60259.116162] Call Trace:
[60259.116162] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60259.116162] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60259.116162] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60259.116162] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60259.116162] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60259.116162] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60259.116162] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60259.116162] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162] [<ffffffff81308309>] ? printk+0x4e/0x5d
[60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60259.116162] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60259.116162] [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60259.116162] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60259.116162] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60259.116162] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60259.116162] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60259.116162] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60259.116162] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60259.116162] [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60259.116162] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60259.116162] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60259.116162] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60259.116162] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60259.116162] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60259.116162] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60259.116162] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60259.116162] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60259.116162] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60259.116162] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60259.116162] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60259.116162] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60259.116162] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60259.116162] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60259.116162] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60259.116162] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60259.116162] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60259.116162] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60259.116162] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60259.116162] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60259.116162] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60259.116162] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60259.116162] [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60259.116162] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60259.116162] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60259.116162] [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60259.116162] ---[ end trace 694817acca794f3e ]---
[60259.116162] ------------[ cut here ]------------
[60259.116162] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60259.116162] Hardware name:
[60259.116162] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60259.116162] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1
[60259.116162] Call Trace:
[60259.116162] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60259.116162] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60259.116162] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60259.116162] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60259.116162] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60259.116162] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60259.116162] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60259.116162] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162] [<ffffffff81308309>] ? printk+0x4e/0x5d
[60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60259.116162] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60259.116162] [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60259.116162] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60259.116162] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60259.116162] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60259.116162] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60259.116162] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60259.116162] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60259.116162] [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60259.116162] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60259.116162] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60259.116162] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60259.116162] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60259.116162] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60259.116162] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60259.116162] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60259.116162] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60259.116162] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60259.116162] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60259.116162] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60259.116162] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60259.116162] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60259.116162] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60259.116162] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60259.116162] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60259.116162] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60259.116162] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60259.116162] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60259.116162] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60259.116162] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60259.116162] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60259.116162] [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60259.116162] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60259.116162] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60259.116162] [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60259.116162] ---[ end trace 694817acca794f3f ]---
[60259.116162] ------------[ cut here ]------------
[60259.116162] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60259.116162] Hardware name:
[60259.116162] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60259.116162] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1
[60259.116162] Call Trace:
[60259.116162] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60259.116162] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60259.116162] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60259.116162] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60259.116162] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60259.116162] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60259.116162] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60259.116162] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162] [<ffffffff81308309>] ? printk+0x4e/0x5d
[60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60259.116162] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60259.116162] [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60259.116162] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60259.116162] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60259.116162] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60259.116162] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60259.116162] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60259.116162] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60259.116162] [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60259.116162] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60259.116162] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60259.116162] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60259.116162] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60259.116162] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60259.116162] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60259.116162] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60259.116162] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60259.116162] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60259.116162] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60259.116162] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60259.116162] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60259.116162] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60259.116162] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60259.116162] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60259.116162] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60259.116162] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60259.116162] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60259.116162] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60259.116162] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60259.116162] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60259.116162] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60259.116162] [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60259.116162] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60259.116162] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60259.116162] [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60259.116162] ---[ end trace 694817acca794f40 ]---
[60259.116162] ------------[ cut here ]------------
[60259.116162] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60259.116162] Hardware name:
[60259.116162] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60259.116162] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1
[60259.116162] Call Trace:
[60259.116162] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60259.116162] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60259.116162] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60259.116162] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60259.116162] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60259.116162] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60259.116162] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60259.116162] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162] [<ffffffff81308309>] ? printk+0x4e/0x5d
[60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60259.116162] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60259.116162] [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60259.116162] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60259.116162] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60259.116162] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60259.116162] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60259.116162] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60259.116162] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60259.116162] [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60259.116162] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60259.116162] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60259.116162] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60259.116162] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60259.116162] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60259.116162] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60259.116162] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60259.116162] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60259.116162] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60259.116162] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60259.116162] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60259.116162] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60259.116162] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60259.116162] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60259.116162] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60259.116162] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60259.116162] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60259.116162] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60259.116162] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60259.116162] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60259.116162] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60259.116162] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60259.116162] [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60259.116162] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60259.116162] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60259.116162] [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60259.116162] ---[ end trace 694817acca794f41 ]---
[60259.116162] ------------[ cut here ]------------
[60259.116162] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60259.116162] Hardware name:
[60259.116162] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60259.116162] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1
[60259.116162] Call Trace:
[60259.116162] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60259.116162] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60259.116162] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60259.116162] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60259.116162] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60259.116162] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60259.116162] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60259.116162] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162] [<ffffffff81308309>] ? printk+0x4e/0x5d
[60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60259.116162] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60259.116162] [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60259.116162] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60259.116162] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60259.116162] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60259.116162] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60259.116162] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60259.116162] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60259.116162] [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60259.116162] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60259.116162] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60259.116162] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60259.116162] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60259.116162] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60259.116162] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60259.116162] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60259.116162] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60259.116162] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60259.116162] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60259.116162] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60259.116162] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60259.116162] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60259.116162] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60259.116162] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60259.116162] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60259.116162] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60259.116162] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60259.116162] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60259.116162] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60259.116162] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60259.116162] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60259.116162] [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60259.116162] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60259.116162] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60259.116162] [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60259.116162] ---[ end trace 694817acca794f42 ]---
[60259.116162] ------------[ cut here ]------------
[60259.116162] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60259.116162] Hardware name:
[60259.116162] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60259.116162] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1
[60259.116162] Call Trace:
[60259.116162] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60259.116162] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60259.116162] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60259.116162] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60259.116162] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60259.116162] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60259.116162] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60259.116162] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162] [<ffffffff81308309>] ? printk+0x4e/0x5d
[60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60259.116162] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60259.116162] [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60259.116162] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60259.116162] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60259.116162] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60259.116162] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60259.116162] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60259.116162] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60259.116162] [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60259.116162] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60259.116162] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60259.116162] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60259.116162] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60259.116162] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60259.116162] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60259.116162] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60259.116162] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60259.116162] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60259.116162] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60259.116162] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60259.116162] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60259.116162] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60259.116162] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60259.116162] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60259.116162] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60259.116162] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60259.116162] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60259.116162] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60259.116162] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60259.116162] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60259.116162] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60259.116162] [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60259.116162] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60259.116162] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60259.116162] [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60259.116162] ---[ end trace 694817acca794f43 ]---
[60259.116162] ------------[ cut here ]------------
[60259.116162] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60259.116162] Hardware name:
[60259.116162] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60259.116162] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1
[60259.116162] Call Trace:
[60259.116162] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60259.116162] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60259.116162] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60259.116162] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60259.116162] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60259.116162] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60259.116162] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60259.116162] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60259.116162] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162] [<ffffffff81308309>] ? printk+0x4e/0x5d
[60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60259.116162] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60259.116162] [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60259.116162] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60259.116162] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60259.116162] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60259.116162] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60259.116162] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60259.116162] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60259.116162] [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60259.116162] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60259.116162] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60259.116162] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60259.116162] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60259.116162] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60259.116162] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60259.116162] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60259.116162] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60259.116162] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60259.116162] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60259.116162] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60259.116162] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60259.116162] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60259.116162] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60259.116162] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60259.116162] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60259.116162] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60259.116162] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60259.116162] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60259.116162] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60259.116162] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60259.116162] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60259.116162] [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60259.116162] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60259.116162] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60259.116162] [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60259.116162] ---[ end trace 694817acca794f44 ]---
[60259.116162] ------------[ cut here ]------------
[60259.116162] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60259.116162] Hardware name:
[60259.116162] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60259.116162] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1
[60259.116162] Call Trace:
[60259.116162] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60259.116162] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60259.116162] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60259.116162] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60259.116162] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60259.116162] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60259.116162] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60259.116162] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60259.116162] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162] [<ffffffff81308309>] ? printk+0x4e/0x5d
[60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60259.116162] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60259.116162] [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60259.116162] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60259.116162] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60259.116162] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60259.116162] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60259.116162] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60259.116162] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60259.116162] [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60259.116162] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60259.116162] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60259.116162] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60259.116162] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60259.116162] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60259.116162] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60259.116162] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60259.116162] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60259.116162] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60259.116162] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60259.116162] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60259.116162] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60259.116162] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60259.116162] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60259.116162] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60259.116162] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60259.116162] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60259.116162] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60259.116162] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60259.116162] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60259.116162] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60259.116162] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60259.116162] [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60259.116162] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60259.116162] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60259.116162] [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60259.116162] ---[ end trace 694817acca794f45 ]---
[60259.116162] ------------[ cut here ]------------
[60259.116162] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60259.116162] Hardware name:
[60259.116162] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60259.116162] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1
[60259.116162] Call Trace:
[60259.116162] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60259.116162] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60259.116162] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60259.116162] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60259.116162] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60259.116162] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60259.116162] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60259.116162] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60259.116162] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162] [<ffffffff81308309>] ? printk+0x4e/0x5d
[60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60259.116162] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60259.116162] [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60259.116162] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60259.116162] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60259.116162] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60259.116162] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60259.116162] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60259.116162] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60259.116162] [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60259.116162] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60259.116162] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60259.116162] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60259.116162] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60259.116162] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60259.116162] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60259.116162] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60259.116162] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60259.116162] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60259.116162] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60259.116162] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60259.116162] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60259.116162] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60259.116162] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60259.116162] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60259.116162] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60259.116162] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60259.116162] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60259.116162] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60259.116162] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60259.116162] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60259.116162] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60259.116162] [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60259.116162] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60259.116162] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60259.116162] [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60259.116162] ---[ end trace 694817acca794f46 ]---
[60259.116162] ------------[ cut here ]------------
[60259.116162] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60259.116162] Hardware name:
[60259.116162] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60259.116162] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1
[60259.116162] Call Trace:
[60259.116162] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60259.116162] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60259.116162] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60259.116162] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60259.116162] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60259.116162] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60259.116162] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60259.116162] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60259.116162] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162] [<ffffffff81308309>] ? printk+0x4e/0x5d
[60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60259.116162] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60259.116162] [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60259.116162] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60259.116162] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60259.116162] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60259.116162] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60259.116162] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60259.116162] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60259.116162] [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60259.116162] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60259.116162] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60259.116162] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60259.116162] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60259.116162] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60259.116162] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60259.116162] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60259.116162] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60259.116162] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60259.116162] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60259.116162] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60259.116162] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60259.116162] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60259.116162] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60259.116162] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60259.116162] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60259.116162] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60259.116162] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60259.116162] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60259.116162] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60259.116162] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60259.116162] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60259.116162] [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60259.116162] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60259.116162] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60259.116162] [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60259.116162] ---[ end trace 694817acca794f47 ]---


Attachments:
oops2.txt (120.04 kB)

2009-10-13 09:24:35

by Massimo Cetra

[permalink] [raw]
Subject: Re: [Bug #14378] Problems with net/core/skbuff.c

Massimo Cetra ha scritto:
> David Miller ha scritto:
>> From: "Rafael J. Wysocki" <[email protected]>
>> Date: Mon, 12 Oct 2009 00:22:04 +0200 (CEST)
>>
>>
>>> Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14378
>>> Subject : Problems with net/core/skbuff.c
>>> Submitter : Massimo Cetra <[email protected]>
>>> Date : 2009-10-08 14:51 (4 days old)
>>> References : http://marc.info/?l=linux-kernel&m=125501488220358&w=4
>>>
>>
>> I don't know what to do about this one.
>>
>> The user indicates that they have the vserver patches applied,
>> so maybe there is some interaction with that stuff.
>>
> Actually i found another oops which is very similar to the previous one.

And here it is another one, this time triggered by postfix, where mor
drbd nor vserver are involved.
This is not the same server where the other oopses were grabbed.

Max


2009-10-13 10:21:07

by Eric Dumazet

[permalink] [raw]
Subject: Re: [Bug #14378] Problems with net/core/skbuff.c

Massimo Cetra a ?crit :
> David Miller ha scritto:
>> From: "Rafael J. Wysocki" <[email protected]>
>> Date: Mon, 12 Oct 2009 00:22:04 +0200 (CEST)
>>
>>
>>> Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14378
>>> Subject : Problems with net/core/skbuff.c
>>> Submitter : Massimo Cetra <[email protected]>
>>> Date : 2009-10-08 14:51 (4 days old)
>>> References : http://marc.info/?l=linux-kernel&m=125501488220358&w=4
>>>
>>
>> I don't know what to do about this one.
>>
>> The user indicates that they have the vserver patches applied,
>> so maybe there is some interaction with that stuff.
>>
> Actually i found another oops which is very similar to the previous one.
> Here, vserver is not involved, and the problem starts at drbd which
> lives in kernel space (the other oops started at ocfs2).
>
> Both ocfs2 and drbd make heavy use of network I/O so i guess the problem
> is something in the network layer.
>
> Anything i can do to help to debugging and solving this issue ?
>
> Thanks
> Max
>

Problem is kfree_skb() is called from irq context, wich is not allowed.

static void skb_release_head_state(struct sk_buff *skb)
{
...
if (skb->destructor) {
WARN_ON(in_irq());
skb->destructor();
}
...
}

virtio_net start_xmit() function calls free_old_xmit_skbs() and
free_old_xmit_skbs() ultimately calls kfree_skb()

Quick fix would be to use dev_kfree_skb_any() instead,
because netpoll can definitly calls start_xmit()
with irq disabled.

Could you please following patch ?

diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index 8d00976..54bf091 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -454,7 +454,7 @@ static unsigned int free_old_xmit_skbs(struct virtnet_info *vi)
vi->dev->stats.tx_bytes += skb->len;
vi->dev->stats.tx_packets++;
tot_sgs += skb_vnet_hdr(skb)->num_sg;
- kfree_skb(skb);
+ dev_kfree_skb_any(skb);
}
return tot_sgs;
}

2009-10-13 10:22:43

by David Miller

[permalink] [raw]
Subject: Re: [Bug #14378] Problems with net/core/skbuff.c

From: Massimo Cetra <[email protected]>
Date: Tue, 13 Oct 2009 11:11:17 +0200

> Here, vserver is not involved, and the problem starts at drbd which
> lives in kernel space (the other oops started at ocfs2).
>
> Both ocfs2 and drbd make heavy use of network I/O so i guess the
> problem is something in the network layer.
>
> Anything i can do to help to debugging and solving this issue ?

Is all of your traffic going over virtio_net?

2009-10-13 10:25:44

by David Miller

[permalink] [raw]
Subject: Re: [Bug #14378] Problems with net/core/skbuff.c

From: Eric Dumazet <[email protected]>
Date: Tue, 13 Oct 2009 12:19:51 +0200

> Quick fix would be to use dev_kfree_skb_any() instead,
> because netpoll can definitly calls start_xmit()
> with irq disabled.

Indeed, because of netpoll(), that is something drivers
must be able to cope with.

> Could you please following patch ?

Indeed, let us know how Eric's patch works.

Thanks Eric!

2009-10-13 10:27:29

by Massimo Cetra

[permalink] [raw]
Subject: Re: [Bug #14378] Problems with net/core/skbuff.c

Eric Dumazet ha scritto:
> Could you please following patch ?
>
> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> index 8d00976..54bf091 100644
> --- a/drivers/net/virtio_net.c
> +++ b/drivers/net/virtio_net.c
> @@ -454,7 +454,7 @@ static unsigned int free_old_xmit_skbs(struct virtnet_info *vi)
> vi->dev->stats.tx_bytes += skb->len;
> vi->dev->stats.tx_packets++;
> tot_sgs += skb_vnet_hdr(skb)->num_sg;
> - kfree_skb(skb);
> + dev_kfree_skb_any(skb);
> }
> return tot_sgs;
> }
>
>
>
Thank you very much.

Compiling.
It' will be in production in a few minutes.
I'll let you know if the problem arises again.
give me a couple of days because this problem is randomly triggered.
Sometimes it happens multiple times a day, sometimes none.

Max

2009-10-13 10:28:21

by Massimo Cetra

[permalink] [raw]
Subject: Re: [Bug #14378] Problems with net/core/skbuff.c

David Miller ha scritto:
> From: Massimo Cetra <[email protected]>
> Date: Tue, 13 Oct 2009 11:11:17 +0200
>
>
>> Here, vserver is not involved, and the problem starts at drbd which
>> lives in kernel space (the other oops started at ocfs2).
>>
>> Both ocfs2 and drbd make heavy use of network I/O so i guess the
>> problem is something in the network layer.
>>
>> Anything i can do to help to debugging and solving this issue ?
>>
>
> Is all of your traffic going over virtio_net?
>
Yes, indeed.
I'm compiling the patch and i'll let you know.

Thanks,
Max

2009-10-13 14:31:51

by John W. Linville

[permalink] [raw]
Subject: Re: [Bug #14278] New message "NOHZ: local_softirq_pending 08" at each ping request

On Mon, Oct 12, 2009 at 11:49:39PM +0200, Rafael J. Wysocki wrote:
> On Monday 12 October 2009, Michael Buesch wrote:
> > On Monday 12 October 2009 00:22:00 Rafael J. Wysocki wrote:
> > > This message has been generated automatically as a part of a report
> > > of recent regressions.
> > >
> > > The following bug entry is on the current list of known regressions
> > > from 2.6.31. Please verify if it still should be listed and let me know
> > > (either way).
> > >
> > >
> > > Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14278
> > > Subject : New message "NOHZ: local_softirq_pending 08" at each ping request
> > > Submitter : Christian Casteyde <[email protected]>
> > > Date : 2009-09-30 18:12 (12 days old)
> > > Handled-By : Michael Buesch <[email protected]>
> >
> > I think John or Johannes is handling it, as my patch was rejected.
> >
> > > Patch : http://bugzilla.kernel.org/attachment.cgi?id=23220
>
> Johannes, what's the status here?

Johannes posted a patch here:

http://marc.info/?l=linux-wireless&m=125525657120789&w=2

It is on my merge train, but it hasn't gone to Linus (or Dave) just
yet -- probably today.

Hth!

John
--
John W. Linville Someday the world will need a hero, and you
[email protected] might be all we have. Be ready.

2009-10-13 18:53:54

by Sam Ravnborg

[permalink] [raw]
Subject: Re: [Bug #14389] Build system issue

On Mon, Oct 12, 2009 at 12:22:09AM +0200, Rafael J. Wysocki wrote:
> This message has been generated automatically as a part of a report
> of recent regressions.
>
> The following bug entry is on the current list of known regressions
> from 2.6.31. Please verify if it still should be listed and let me know
> (either way).
>
>
> Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14389
> Subject : Build system issue
> Submitter : Peter Zijlstra <[email protected]>
> Date : 2009-10-09 8:58 (3 days old)
> First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=575543347b5baed0ca927cb90ba8807396fe9cc9
> References : http://marc.info/?l=linux-kernel&m=125507914909152&w=4

It is fixed by: 2331d1a6cd3d6e580bc88b9a160066d9e1177fe1

I did not have the bug-entry when the changelog was written
so no references there.

Sam

2009-10-13 20:49:32

by Rafael J. Wysocki

[permalink] [raw]
Subject: Re: [Bug #14278] New message "NOHZ: local_softirq_pending 08" at each ping request

On Tuesday 13 October 2009, John W. Linville wrote:
> On Mon, Oct 12, 2009 at 11:49:39PM +0200, Rafael J. Wysocki wrote:
> > On Monday 12 October 2009, Michael Buesch wrote:
> > > On Monday 12 October 2009 00:22:00 Rafael J. Wysocki wrote:
> > > > This message has been generated automatically as a part of a report
> > > > of recent regressions.
> > > >
> > > > The following bug entry is on the current list of known regressions
> > > > from 2.6.31. Please verify if it still should be listed and let me know
> > > > (either way).
> > > >
> > > >
> > > > Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14278
> > > > Subject : New message "NOHZ: local_softirq_pending 08" at each ping request
> > > > Submitter : Christian Casteyde <[email protected]>
> > > > Date : 2009-09-30 18:12 (12 days old)
> > > > Handled-By : Michael Buesch <[email protected]>
> > >
> > > I think John or Johannes is handling it, as my patch was rejected.
> > >
> > > > Patch : http://bugzilla.kernel.org/attachment.cgi?id=23220
> >
> > Johannes, what's the status here?
>
> Johannes posted a patch here:
>
> http://marc.info/?l=linux-wireless&m=125525657120789&w=2
>
> It is on my merge train, but it hasn't gone to Linus (or Dave) just
> yet -- probably today.
>
> Hth!

Yes, it does, I've updated the bug entry with the link to the Johannes' patch.

Thanks,
Rafael

2009-10-13 20:51:57

by Rafael J. Wysocki

[permalink] [raw]
Subject: Re: [Bug #14389] Build system issue

On Tuesday 13 October 2009, Sam Ravnborg wrote:
> On Mon, Oct 12, 2009 at 12:22:09AM +0200, Rafael J. Wysocki wrote:
> > This message has been generated automatically as a part of a report
> > of recent regressions.
> >
> > The following bug entry is on the current list of known regressions
> > from 2.6.31. Please verify if it still should be listed and let me know
> > (either way).
> >
> >
> > Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14389
> > Subject : Build system issue
> > Submitter : Peter Zijlstra <[email protected]>
> > Date : 2009-10-09 8:58 (3 days old)
> > First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=575543347b5baed0ca927cb90ba8807396fe9cc9
> > References : http://marc.info/?l=linux-kernel&m=125507914909152&w=4
>
> It is fixed by: 2331d1a6cd3d6e580bc88b9a160066d9e1177fe1
>
> I did not have the bug-entry when the changelog was written
> so no references there.

Thanks, closing.

Rafael

2009-10-14 19:28:22

by Massimo Cetra

[permalink] [raw]
Subject: Re: [Bug #14378] Problems with net/core/skbuff.c

Massimo Cetra ha scritto:
> Eric Dumazet ha scritto:
>> Could you please following patch ?
>>
>> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
>> index 8d00976..54bf091 100644
>> --- a/drivers/net/virtio_net.c
>> +++ b/drivers/net/virtio_net.c
>> @@ -454,7 +454,7 @@ static unsigned int free_old_xmit_skbs(struct
>> virtnet_info *vi)
>> vi->dev->stats.tx_bytes += skb->len;
>> vi->dev->stats.tx_packets++;
>> tot_sgs += skb_vnet_hdr(skb)->num_sg;
>> - kfree_skb(skb);
>> + dev_kfree_skb_any(skb);
>> }
>> return tot_sgs;
>> }
>>
>>
>>
> Thank you very much.
>
> Compiling.
> It' will be in production in a few minutes.
> I'll let you know if the problem arises again.
> give me a couple of days because this problem is randomly triggered.
> Sometimes it happens multiple times a day, sometimes none.
>
Eric,
thanks for the patch.
The problem didn't arise again and i haven't seen any warning like that
on both servers where that problem was happening more frequently.

I would say that it's fixed and if it's not, i'll let you know as soon
as it happens again.

Thanks again,
Max


2009-10-15 00:37:58

by Eric Dumazet

[permalink] [raw]
Subject: [PATCH] virtio_net: use dev_kfree_skb_any() in free_old_xmit_skbs()

Massimo Cetra a ?crit :
> Eric,
> thanks for the patch.
> The problem didn't arise again and i haven't seen any warning like that
> on both servers where that problem was happening more frequently.
>
> I would say that it's fixed and if it's not, i'll let you know as soon
> as it happens again.
>

Thanks Massimo, I think patch is reasonably safe and should be taken as is :


[PATCH] virtio_net: use dev_kfree_skb_any() in free_old_xmit_skbs()

Because netpoll can call netdevice start_xmit() method with
irqs disabled, drivers should not call kfree_skb() from
their start_xmit(), but use dev_kfree_skb_any() instead.

Oct 8 11:16:52 172.30.1.31 [113074.791813] ------------[ cut here ]------------
Oct 8 11:16:52 172.30.1.31 [113074.791813] WARNING: at net/core/skbuff.c:398 \
skb_release_head_state+0x64/0xc8()
Oct 8 11:16:52 172.30.1.31 [113074.791813] Hardware name:
Oct 8 11:16:52 172.30.1.31 [113074.791813] Modules linked in: netconsole ocfs2 jbd2 quota_tree \
ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue configfs crc32c drbd cn loop \
serio_raw psmouse snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net pcspkr parport_pc parport \
i2c_piix4 i2c_core button processor evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot \
dm_mod ide_cd_mod cdrom ata_generic ata_piix virtio_blk libata scsi_mod piix ide_pci_generic ide_core \
virtio_pci virtio_ring virtio floppy thermal fan thermal_sys [last unloaded: netconsole]
Oct 8 11:16:52 172.30.1.31 [113074.791813] Pid: 11132, comm: php5-cgi Tainted: G W \
2.6.31.2-vserver #1
Oct 8 11:16:52 172.30.1.31 [113074.791813] Call Trace:
Oct 8 11:16:52 172.30.1.31 [113074.791813] <IRQ> [<ffffffff81253cd5>] ? \
skb_release_head_state+0x64/0xc8
Oct 8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
Oct 8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
Oct 8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
Oct 8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
Oct 8 11:16:52 172.30.1.31 [113074.791813] [<ffffffffa01cb139>] ? free_old_xmit_skbs+0x51/0x6e \
[virtio_net]
Oct 8 11:16:52 172.30.1.31 [113074.791813] [<ffffffffa01cbc85>] ? start_xmit+0x26/0xf2 [virtio_net]
Oct 8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
Oct 8 11:16:52 172.30.1.31 [113074.791813] [<ffffffffa0429216>] ? write_msg+0x90/0xeb [netconsole]
Oct 8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
Oct 8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
Oct 8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
Oct 8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
Oct 8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff8106b142>] ? vx_update_load+0x18/0x13e
Oct 8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff81308309>] ? printk+0x4e/0x5d
Oct 8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
Oct 8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
Oct 8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
Oct 8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff810626b7>] ? ktime_get+0xc/0x41
Oct 8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
Oct 8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
Oct 8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
Oct 8 11:16:52 172.30.1.31 [113074.791813] <EOI> [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31

Reported-and-tested-by: Massimo Cetra <[email protected]>
Signed-off-by: Eric Dumazet <[email protected]>
Bug-Entry: http://bugzilla.kernel.org/show_bug.cgi?id=14378
---

diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index 8d00976..54bf091 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -454,7 +454,7 @@ static unsigned int free_old_xmit_skbs(struct virtnet_info *vi)
vi->dev->stats.tx_bytes += skb->len;
vi->dev->stats.tx_packets++;
tot_sgs += skb_vnet_hdr(skb)->num_sg;
- kfree_skb(skb);
+ dev_kfree_skb_any(skb);
}
return tot_sgs;
}

2009-10-15 06:30:36

by David Miller

[permalink] [raw]
Subject: Re: [PATCH] virtio_net: use dev_kfree_skb_any() in free_old_xmit_skbs()

From: Eric Dumazet <[email protected]>
Date: Thu, 15 Oct 2009 02:36:43 +0200

> Massimo Cetra a ?crit :
>> Eric,
>> thanks for the patch.
>> The problem didn't arise again and i haven't seen any warning like that
>> on both servers where that problem was happening more frequently.
>>
>> I would say that it's fixed and if it's not, i'll let you know as soon
>> as it happens again.
>>
>
> Thanks Massimo, I think patch is reasonably safe and should be taken as is :
>
>
> [PATCH] virtio_net: use dev_kfree_skb_any() in free_old_xmit_skbs()

Applied, thanks Eric.