2011-05-31 01:24:14

by Brad Campbell

[permalink] [raw]
Subject: KVM induced panic on 2.6.38[2367] & 2.6.39

G'day all,

I'm running a pretty standard home server
x86_64 Phenom-II 6 Core
16GB DDR 3

I run some virtual machines under that. 3 x Debian 64 Bit, 1 x XP 32
Bit. These run at boot.

When I fire up another XP 32 bit instance and play with it for more than
about 2 minutes, I get the panics included in this mail.

I've included three of them here. The first and third are as booted. The
second was with ksmd disabled just as a data point.

The machine passes every load test and memory test I can throw at it,
but I still can't rule out this being a hardware issue.

Provided I don't start this XP VM the machine is quite stable, but
running this VM will kill it within minutes.

This was tested with qemu-kvm.
The last commit in the git tree was
commit c007db193eb6b2557acb5caf2dc4d7023639e6f3
Author: Avi Kivity <[email protected]>
Date: Sun May 29 09:00:42 2011 -0400
(I pulled it yesterday)

These panics were captured with netconsole to a remote syslog daemon,
and the formatting was ruined, so I've reformatted them by hand prior to
posting.

I've tested and reproduced this on 2.6.38.[2,3,6 & 7] and 2.6.39 obviously.

Can anyone help shed some light on this?

Regards,
Brad

[ 438.632061] general protection fault: 0000 [#1] SMP
[ 438.632196] last sysfs file: /sys/module/x_tables/initstate
[ 438.632242] CPU 4
[ 438.632282] Modules linked in: xt_iprange xt_DSCP xt_length
xt_CLASSIFY sch_sfq xt_CHECKSUM ipt_REJECT ipt_MASQUERADE ipt_REDIRECT
xt_recent xt_state iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack
nf_defrag_ipv4 iptable_filter xt_TCPMSS xt_tcpmss xt_tcpudp
iptable_mangle ip_tables x_tables pppoe pppox ppp_generic slhc cls_u32
sch_htb deflate zlib_deflate des_generic cbc ecb crypto_blkcipher
sha1_generic md5 hmac crypto_hash cryptomgr aead crypto_algapi af_key
fuse netconsole configfs= vhost_net powernow_k8 mperf i2c_nforce2
kvm_amd kvm pl2303 usbserial xhci_hcd k10temp i2c_piix4 usb_storage
usb_libusual ohci_hcd ehci_hcd usbcore ahci libahci r8169 mii sata_mv
megaraid_sas [last unloaded: scsi_wait_scan]
[ 438.634960]
[ 438.635006] Pid: 551, comm: ksmd Not tainted 2.6.39 #3 To Be Filled By
O.E.M. To Be Filled By O.E.M. /880G Extreme3
[ 438.635170] RIP: 0010:[<ffffffff810b4596>] [<ffffffff810b4596>]
remove_rmap_item_from_tree+0x96/0x150
[ 438.635268] RSP: 0018:ffff88041c065e20 EFLAGS: 00010282
[ 438.635314] RAX: ffff8804153bd8b0 RBX: ffff8804176c3fc0 RCX:
0000000000057754
[ 438.635362] RDX: 0000880415418030 RSI: ffff880414b65003 RDI:
ffffea000dde6030
[ 438.635410] RBP: ffff880414b65000 R08: 0000000000057755 R09:
000000001bbde1bd
[ 438.635458] R10: 000000001c28e0c3 R11: 0000000000000002 R12:
ffffea000dde6030
[ 438.635506] R13: ffff8804176c3f80 R14: ffff8804151ed7b0 R15:
ffff88041bf23be0
[ 438.635555] FS: 00007f617772e700(0000) GS:ffff88041fd00000(0000)
knlGS:0000000000000000
[ 438.635607] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 438.635654] CR2: 0000000000e70000 CR3: 0000000001583000 CR4:
00000000000006e0
[ 438.635703] DR0: 0000000000000045 DR1: 0000000000000000 DR2:
0000000000000000
[ 438.635750] DR3: 0000000000000005 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[ 438.635799] Process ksmd (pid: 551, threadinfo ffff88041c064000, task
ffff88041d8caa70)
[ 438.635851] Stack:
[ 438.635893] ffffea000c43c408 ffff8804176c3fc0 000000000000036c
ffffffff810b58f2
[ 438.636088] ffff88041c782a00 00007fc2d81b5000 ffff88041c064000
00000000003808ff
[ 438.636281] ffff88041c064000 ffff88041c065e98 ffff88041d8caa70
ffff8804165d4480
[ 438.636479] Call Trace:
[ 438.636528] [<ffffffff810b58f2>] ? ksm_scan_thread+0x4e2/0xc20
[ 438.636580] [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[ 438.636628] [<ffffffff810b5410>] ? try_to_merge_with_ksm_page+0x570/0x570
[ 438.636679] [<ffffffff810b5410>] ? try_to_merge_with_ksm_page+0x570/0x570
[ 438.636730] [<ffffffff810525b6>] ? kthread+0x96/0xa0
[ 438.636781] [<ffffffff813d1794>] ? kernel_thread_helper+0x4/0x10
[ 438.636832] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[ 438.636882] [<ffffffff813d1790>] ? gs_change+0xb/0xb
[ 438.636926] Code: 28 48 89 ef e8 6c fe ff ff 48 85 c0 49 89 c4 74 d2
f0 0f ba 28 00 19 c0 85 c0 0f 85 ae 00 00 00 00 48 8b 43 30 48 8b 53 38
48 85 c0
[ 438.638504] 89 02 74 04 48 89 50 08 48 ba 00 01 10 00 00 00 00 ad de 48 b8
[ 438.639329] RIP [<ffffffff810b4596>] remove_rmap_item_from_tree+0x96/0x150
[ 438.639414] RSP <ffff88041c065e20>
[ 438.639460] ---[ end trace c29fb871f6b874e3 ]---
[ 438.639506] Kernel panic - not syncing: Fatal exception
[ 438.639553] Pid: 551, comm: ksmd Tainted: G D 2.6.39 #3
[ 438.639598] Call Trace:
[ 438.639644] [<ffffffff813cd6f5>] ? panic+0x92/0x18a
[ 438.639693] [<ffffffff81038b61>] ? kmsg_dump+0x41/0xf0
[ 438.639745] [<ffffffff810050ad>] ? oops_end+0x8d/0xa0
[ 438.639794] [<ffffffff813d05ef>] ? general_protection+0x1f/0x30
[ 438.639842] [<ffffffff810b4596>] ? remove_rmap_item_from_tree+0x96/0x150
[ 438.639891] [<ffffffff810b58f2>] ? ksm_scan_thread+0x4e2/0xc20
[ 438.639941] [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[ 438.640045] [<ffffffff810b5410>] ? try_to_merge_with_ksm_page+0x570/0x570
[ 438.640096] [<ffffffff810b5410>] ? try_to_merge_with_ksm_page+0x570/0x570
[ 438.640147] [<ffffffff810525b6>] ? kthread+0x96/0xa0
[ 438.640196] [<ffffffff813d1794>] ? kernel_thread_helper+0x4/0x10
[ 438.640247] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[ 438.640297] [<ffffffff813d1790>] ? gs_change+0xb/0xb
[ 438.640349] Rebooting in 60 seconds..



[ 469.706013] general protection fault: 0000 [#1] SMP
[ 469.706125] last sysfs file:
/sys/devices/pci0000:00/0000:00:18.3/temp1_input
[ 469.706162] CPU 1
[ 469.706195] Modules linked in: xt_iprange xt_DSCP xt_length
xt_CLASSIFY sch_sfq xt_CHECKSUM ipt_REJECT ipt_MASQUERADE ipt_REDIRECT
xt_recent xt_state iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack
nf_defrag_ipv4 iptable_filter xt_TCPMSS xt_tcpmss xt_tcpudp
iptable_mangle ip_tables x_tables pppoe pppox ppp_generic slhc cls_u32
sch_htb deflate zlib_deflate des_generi cbc ecb crypto_blkcipher
sha1_generic md5 hmac crypto_hash cryptomgr aead crypto_algapi af_key
fuse netconsole configfs vhost_net powernow_k8 mperf i2c_nforce2 kvm_amd
kvm pl2303 usbserial xhci_hcd i2c_piix4 k10temp ahci usb_storage
usb_libusual ohci_hcd ehci_hcd r8169 libahci usbcore mii sata_mv
megaraid_sas [last unloaded: scsi_wait_scan]
[ 469.708573]
[ 469.708608] Pid: 4942, comm: runbot Not tainted 2.6.39 #3 To Be Filled
By O.E.M. To Be Filled By O.E.M. /880G Extreme3
[ 469.708753] RIP: 0010:[<ffffffff810db878>] [<ffffffff810db878>]
dup_fd+0x168/0x300
[ 469.708829] RSP: 0018:ffff880417279dd0 EFLAGS: 00010202
[ 469.708870] RAX: 00000000000007f8 RBX: ffff88041c1658c0 RCX:
bfffffffffffffff
[ 469.708907] RDX: 0000880413155540 RSI: 00000000000000ff RDI:
0000000000000800
[ 469.708945] RBP: ffff88041bf22000 R08: ffff880412839dc0 R09:
0000000000000003
[ 469.708981] R10: 0000000000000001 R11: 4000000000000000 R12:
0000000000000100
[ 469.709018] R13: ffff88041d3cc000 R14: ffff880412839d40 R15:
ffff88041c72f440
[ 469.709054] FS: 00007f510c0b4700(0000) GS:ffff88041fc40000(0000)
knlGS:0000000000000000
[ 469.709094] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 469.709130] CR2: 00007f510ba56810 CR3: 0000000417c93000 CR4:
00000000000006e0
[ 469.709167] DR0: 00000000000000a0 DR1: 0000000000000000 DR2:
0000000000000003
[ 469.709204] DR3: 00000000000000b0 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[ 469.709240] Process runbot (pid: 4942, threadinfo ffff880417278000,
task ffff880401351230)
[ 469.709279] Stack:
[ 469.709313] ffff880401351230 0000000000000020 ffff880412e9ca80
ffff88041c1658d0 ffff88041c1658d0
[ 469.709477] ffff88041723f330 0000000001200011 ffff88041723f330
0000000000000000
[ 469.709641] 00007f510c0b49d0 ffff880401351230 0000000000000000
ffffffff81037645
[ 469.709812] Call Trace:
[ 469.709849] [<ffffffff81037645>] ? copy_process+0xa75/0xfd0
[ 469.709885] [<ffffffff81037c0d>] ? do_fork+0x6d/0x2b0
[ 469.709921] [<ffffffff810dbc23>] ? alloc_fd+0x43/0x130
[ 469.709958] [<ffffffff810c08c5>] ? fd_install+0x35/0x70
[ 469.709998] [<ffffffff810457a9>] ? sigprocmask+0x69/0x100
[ 469.710042] [<ffffffff813d0ca3>] ? stub_clone+0x13/0x20
[ 469.710079] [<ffffffff813d0a3b>] ? system_call_fastpath+0x16/0x1b
[ 469.710114] Code: 4c 89 c2 e8 6b e5 0f 00 45 85 e4 74 78 41 8d 44 24
ff 31 f6 41 ba 01 00 00 00 00 48 8d 3c c5 08 00 00 00 00 31 c0 eb 1a 0f
1f 44 00 00 f0> 48 ff 42 30 48 89 54 05 00 48 83 c0
May 30 16:39:36 192.168.2.1 08 ff c6 48 39 f8 74 3b
[ 469.712247] RIP [<ffffffff810db878>] dup_fd+0x168/0x300
[ 469.712317] RSP <ffff880417279dd0>
[ 469.712391] ---[ end trace b80596608cd6bb6b ]---
[ 469.712432] Kernel panic - not syncing: Fatal exception
[ 469.712470] Pid: 4942, comm: runbot Tainted: G D 2.6.39 #3
[ 469.712508] Call Trace:
[ 469.712544] [<ffffffff813cd6f5>] ? panic+0x92/0x18a
[ 469.712581] [<ffffffff81038b61>] ? kmsg_dump+0x41/0xf0
[ 469.712618] [<ffffffff810050ad>] ? oops_end+0x8d/0xa0
[ 469.712654] [<ffffffff813d05ef>] ? general_protection+0x1f/0x30
[ 469.712691] [<ffffffff810db878>] ? dup_fd+0x168/0x300
[ 469.712727] [<ffffffff81037645>] ? copy_process+0xa75/0xfd0
[ 469.712767] [<ffffffff81037c0d>] ? do_fork+0x6d/0x2b0
[ 469.712803] [<ffffffff810dbc23>] ? alloc_fd+0x43/0x130
[ 469.712839] [<ffffffff810c08c5>] ? fd_install+0x35/0x70
[ 469.712875] [<ffffffff810457a9>] ? sigprocmask+0x69/0x100
[ 469.712912] [<ffffffff813d0ca3>] ? stub_clone+0x13/0x20
[ 469.712948] [<ffffffff813d0a3b>] ? system_call_fastpath+0x16/0x1b
[ 469.712989] Rebooting in 60 seconds..


[ 1100.473612] general protection fault: 0000 [#1] SMP
[ 1100.473718] last sysfs file:
/sys/devices/pci0000:00/0000:00:18.3/temp1_input
[ 1100.473755] CPU 0
[ 1100.473788] Modules linked in: xt_iprange xt_DSCP xt_length
xt_CLASSIFY sch_sfq xt_CHECKSUM ipt_REJECT ipt_MASQUERADE ipt_REDIRECT
xt_recent xt_state iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack
nf_defrag_ipv4 xt_TCPMSS xt_tcpmss xt_tcpudp iptable_mangle pppoe pppox
iptable_filter ip_tables x_tables ppp_generic slhc cls_u32 sch_htb
deflate zlib_deflate des_generic cbc ecb crypto_blkcipher sha1_generic
md5 hmac crypto_hash cryptomgr aead crypto_algapi af_key fuse netconsole
configfs vhost_net powernow_k8 mperf i2c_nforce2 kvm_amd kvm pl2303
xhci_hcd usbserial k10temp i2c_piix4 usb_storage usb_libusual ohci_hcd
ehci_hcd ahci usbcore sata_mv r8169 mii libahci megaraid_sas [last
unloaded: scsi_wait_scan]
[ 1100.476102]
[ 1100.476137] Pid: 551, comm: ksmd Not tainted 2.6.39 #3 To Be Filled
By O.E.M. To Be Filled By O.E.M. /880G Extreme3
[ 1100.476277] RIP: 0010:[<ffffffff810b5c71>] [<ffffffff810b5c71>]
ksm_scan_thread+0x861/0xc20
[ 1100.476351] RSP: 0018:ffff88041c065e40 EFLAGS: 00010282
[ 1100.476387] RAX: 0000000000000000 RBX: ffffea000a5c8468 RCX:
0000000000000f01
[ 1100.476424] RDX: 0000880403456828 RSI: ffff8802f5dcb002 RDI:
ffffea000ce7eac8
[ 1100.476460] RBP: ffff880418377fc0 R08: 6db6db6db6db6db7 R09:
0000160000000000
[ 1100.476498] R10: ffffea000ce7eac8 R11: 0000000080100073 R12:
00000000000000ab
[ 1100.476535] R13: ffff880418377ff8 R14: ffffea000ce7eac8 R15:
ffff880418377fe8
[ 1100.476572] FS: 00007f08c6498700(0000) GS:ffff88041fc00000(0000)
knlGS:0000000000000000
[ 1100.476611] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 1100.476646] CR2: 00000000e1dd3000 CR3: 00000004025f7000 CR4:
00000000000006f0
[ 1100.477007] DR0: 0000000000000045 DR1: 0000000000000000 DR2:
0000000000000000
[ 1100.477042] DR3: 0000000000000005 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[ 1100.477078] Process ksmd (pid: 551, threadinfo ffff88041c064000, task
ffff88041d0454e0)
[ 1100.477115] Stack:
[ 1100.477148] ffff880403a8b100 00007fc1863cb000 ffff88041c064000
00000000002f5dcb
[ 1100.477317] ffff88041c064000 ffff88041c065e98 ffff88041d0454e0
ffff88041267a580
[ 1100.477480] 0000000000000000 ffff88041d0454e0 ffffffff81052a20
ffff88041c065e98
[ 1100.477644] Call Trace:
[ 1100.477681] [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[ 1100.477717] [<ffffffff810b5410>] ?
try_to_merge_with_ksm_page+0x570/0x570
[ 1100.477754] [<ffffffff810b5410>] ?
try_to_merge_with_ksm_page+0x570/0x570
[ 1100.477791] [<ffffffff810525b6>] ? kthread+0x96/0xa0
[ 1100.477829] [<ffffffff813d1794>] ? kernel_thread_helper+0x4/0x10
[ 1100.477865] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[ 1100.477902] [<ffffffff813d1790>] ? gs_change+0xb/0xb
[ 1100.477937] Code: 05 99 6e 6e 00 4c 89 ff e8 ed 12 fa ff e9 6c f9 ff
ff 48 89 6c 24 38 49 83 7d 00 00 0f 84 63 01 00 00 e8 b3 8b 31 00 49 8b
55 00
[ 1100.479306] 8b 42 e8 4c 8b 72 f0 48 8d 6a d8 4c 8d 78 60 48 89 04 24 4c
[ 1100.480012] RIP [<ffffffff810b5c71>] ksm_scan_thread+0x861/0xc20
[ 1100.480015] RSP <ffff88041c065e40>
[ 1100.480141] ---[ end trace 7382777d9e74ea23 ]---
[ 1100.480178] Kernel panic - not syncing: Fatal exception
[ 1100.480216] Pid: 551, comm: ksmd Tainted: G D 2.6.39 #3
[ 1100.480257] Call Trace:
[ 1100.480297] [<ffffffff813cd6f5>] ? panic+0x92/0x18a
[ 1100.480340] [<ffffffff81038b61>] ? kmsg_dump+0x41/0xf0
[ 1100.480384] [<ffffffff810050ad>] ? oops_end+0x8d/0xa0
[ 1100.480425] [<ffffffff813d05ef>] ? general_protection+0x1f/0x30
[ 1100.480465] [<ffffffff810b5c71>] ? ksm_scan_thread+0x861/0xc20
[ 1100.480507] [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[ 1100.480545] [<ffffffff810b5410>] ?
try_to_merge_with_ksm_page+0x570/0x570
[ 1100.480584] [<ffffffff810b5410>] ?
try_to_merge_with_ksm_page+0x570/0x570
[ 1100.480624] [<ffffffff810525b6>] ? kthread+0x96/0xa0
[ 1100.480669] [<ffffffff813d1794>] ? kernel_thread_helper+0x4/0x10
[ 1100.480708] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[ 1100.480748] [<ffffffff813d1790>] ? gs_change+0xb/0xb
[ 1100.480794] Rebooting in 60 seconds..

brad@srv:~$ zcat /proc/config.gz | grep -v "#"
CONFIG_64BIT=y
CONFIG_X86_64=y
CONFIG_X86=y
CONFIG_INSTRUCTION_DECODER=y
CONFIG_OUTPUT_FORMAT="elf64-x86-64"
CONFIG_ARCH_DEFCONFIG="arch/x86/configs/x86_64_defconfig"
CONFIG_GENERIC_CMOS_UPDATE=y
CONFIG_CLOCKSOURCE_WATCHDOG=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_HAVE_LATENCYTOP_SUPPORT=y
CONFIG_MMU=y
CONFIG_ZONE_DMA=y
CONFIG_NEED_DMA_MAP_STATE=y
CONFIG_NEED_SG_DMA_LENGTH=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y
CONFIG_GENERIC_BUG=y
CONFIG_GENERIC_BUG_RELATIVE_POINTERS=y
CONFIG_GENERIC_HWEIGHT=y
CONFIG_ARCH_MAY_HAVE_PC_FDC=y
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
CONFIG_ARCH_HAS_CPU_IDLE_WAIT=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_GENERIC_TIME_VSYSCALL=y
CONFIG_ARCH_HAS_CPU_RELAX=y
CONFIG_ARCH_HAS_DEFAULT_IDLE=y
CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
CONFIG_HAVE_SETUP_PER_CPU_AREA=y
CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=y
CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=y
CONFIG_HAVE_CPUMASK_OF_CPU_MAP=y
CONFIG_ARCH_HIBERNATION_POSSIBLE=y
CONFIG_ARCH_SUSPEND_POSSIBLE=y
CONFIG_ZONE_DMA32=y
CONFIG_ARCH_POPULATES_NODE_MAP=y
CONFIG_AUDIT_ARCH=y
CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y
CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
CONFIG_X86_64_SMP=y
CONFIG_X86_HT=y
CONFIG_ARCH_HWEIGHT_CFLAGS="-fcall-saved-rdi -fcall-saved-rsi
-fcall-saved-rdx -fcall-saved-rcx -fcall-saved-r8 -fcall-saved-r9
-fcall-saved-r10 -fcall-saved-r11"
CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
CONFIG_CONSTRUCTORS=y
CONFIG_HAVE_IRQ_WORK=y
CONFIG_IRQ_WORK=y

CONFIG_EXPERIMENTAL=y
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_CROSS_COMPILE=""
CONFIG_LOCALVERSION=""
CONFIG_LOCALVERSION_AUTO=y
CONFIG_HAVE_KERNEL_GZIP=y
CONFIG_HAVE_KERNEL_BZIP2=y
CONFIG_HAVE_KERNEL_LZMA=y
CONFIG_HAVE_KERNEL_XZ=y
CONFIG_HAVE_KERNEL_LZO=y
CONFIG_KERNEL_GZIP=y
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_SYSVIPC_SYSCTL=y
CONFIG_BSD_PROCESS_ACCT=y
CONFIG_TASKSTATS=y
CONFIG_TASK_DELAY_ACCT=y
CONFIG_TASK_XACCT=y
CONFIG_TASK_IO_ACCOUNTING=y
CONFIG_HAVE_GENERIC_HARDIRQS=y

CONFIG_GENERIC_HARDIRQS=y
CONFIG_HAVE_SPARSE_IRQ=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_GENERIC_IRQ_SHOW=y
CONFIG_GENERIC_PENDING_IRQ=y
CONFIG_IRQ_FORCED_THREADING=y

CONFIG_TREE_RCU=y
CONFIG_RCU_FANOUT=64
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
CONFIG_LOG_BUF_SHIFT=20
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
CONFIG_NAMESPACES=y
CONFIG_BLK_DEV_INITRD=y
CONFIG_INITRAMFS_SOURCE=""
CONFIG_RD_GZIP=y
CONFIG_RD_BZIP2=y
CONFIG_RD_LZMA=y
CONFIG_RD_XZ=y
CONFIG_RD_LZO=y
CONFIG_SYSCTL=y
CONFIG_ANON_INODES=y
CONFIG_UID16=y
CONFIG_SYSCTL_SYSCALL=y
CONFIG_KALLSYMS=y
CONFIG_KALLSYMS_ALL=y
CONFIG_KALLSYMS_EXTRA_PASS=y
CONFIG_HOTPLUG=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_PCSPKR_PLATFORM=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_SIGNALFD=y
CONFIG_TIMERFD=y
CONFIG_EVENTFD=y
CONFIG_SHMEM=y
CONFIG_AIO=y
CONFIG_HAVE_PERF_EVENTS=y

CONFIG_PERF_EVENTS=y
CONFIG_VM_EVENT_COUNTERS=y
CONFIG_PCI_QUIRKS=y
CONFIG_SLUB_DEBUG=y
CONFIG_SLUB=y
CONFIG_HAVE_OPROFILE=y
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
CONFIG_USER_RETURN_NOTIFIER=y
CONFIG_HAVE_IOREMAP_PROT=y
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
CONFIG_HAVE_OPTPROBES=y
CONFIG_HAVE_ARCH_TRACEHOOK=y
CONFIG_HAVE_DMA_ATTRS=y
CONFIG_USE_GENERIC_SMP_HELPERS=y
CONFIG_HAVE_REGS_AND_STACK_ACCESS_API=y
CONFIG_HAVE_DMA_API_DEBUG=y
CONFIG_HAVE_HW_BREAKPOINT=y
CONFIG_HAVE_MIXED_BREAKPOINTS_REGS=y
CONFIG_HAVE_USER_RETURN_NOTIFIER=y
CONFIG_HAVE_PERF_EVENTS_NMI=y
CONFIG_HAVE_ARCH_JUMP_LABEL=y

CONFIG_SLABINFO=y
CONFIG_RT_MUTEXES=y
CONFIG_BASE_SMALL=0
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
CONFIG_STOP_MACHINE=y
CONFIG_BLOCK=y
CONFIG_BLK_DEV_BSG=y
CONFIG_BLOCK_COMPAT=y

CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
CONFIG_DEFAULT_CFQ=y
CONFIG_DEFAULT_IOSCHED="cfq"
CONFIG_PREEMPT_NOTIFIERS=y
CONFIG_INLINE_SPIN_UNLOCK=y
CONFIG_INLINE_SPIN_UNLOCK_IRQ=y
CONFIG_INLINE_READ_UNLOCK=y
CONFIG_INLINE_READ_UNLOCK_IRQ=y
CONFIG_INLINE_WRITE_UNLOCK=y
CONFIG_INLINE_WRITE_UNLOCK_IRQ=y
CONFIG_MUTEX_SPIN_ON_OWNER=y

CONFIG_TICK_ONESHOT=y
CONFIG_NO_HZ=y
CONFIG_HIGH_RES_TIMERS=y
CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
CONFIG_SMP=y
CONFIG_X86_SUPPORTS_MEMORY_FAILURE=y
CONFIG_SCHED_OMIT_FRAME_POINTER=y
CONFIG_NO_BOOTMEM=y
CONFIG_MK8=y
CONFIG_X86_INTERNODE_CACHE_SHIFT=6
CONFIG_X86_CMPXCHG=y
CONFIG_CMPXCHG_LOCAL=y
CONFIG_X86_L1_CACHE_SHIFT=6
CONFIG_X86_XADD=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_X86_TSC=y
CONFIG_X86_CMPXCHG64=y
CONFIG_X86_CMOV=y
CONFIG_X86_MINIMUM_CPU_FAMILY=64
CONFIG_X86_DEBUGCTLMSR=y
CONFIG_CPU_SUP_INTEL=y
CONFIG_CPU_SUP_AMD=y
CONFIG_CPU_SUP_CENTAUR=y
CONFIG_HPET_TIMER=y
CONFIG_HPET_EMULATE_RTC=y
CONFIG_DMI=y
CONFIG_GART_IOMMU=y
CONFIG_AMD_IOMMU=y
CONFIG_SWIOTLB=y
CONFIG_IOMMU_HELPER=y
CONFIG_IOMMU_API=y
CONFIG_NR_CPUS=8
CONFIG_SCHED_MC=y
CONFIG_PREEMPT_NONE=y
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_MCE=y
CONFIG_X86_MCE_AMD=y
CONFIG_X86_MCE_THRESHOLD=y
CONFIG_ARCH_PHYS_ADDR_T_64BIT=y
CONFIG_ARCH_DMA_ADDR_T_64BIT=y
CONFIG_DIRECT_GBPAGES=y
CONFIG_ARCH_SPARSEMEM_DEFAULT=y
CONFIG_ARCH_SPARSEMEM_ENABLE=y
CONFIG_ARCH_SELECT_MEMORY_MODEL=y
CONFIG_ARCH_MEMORY_PROBE=y
CONFIG_ILLEGAL_POINTER_VALUE=0xdead000000000000
CONFIG_SELECT_MEMORY_MODEL=y
CONFIG_SPARSEMEM_MANUAL=y
CONFIG_SPARSEMEM=y
CONFIG_HAVE_MEMORY_PRESENT=y
CONFIG_SPARSEMEM_EXTREME=y
CONFIG_SPARSEMEM_VMEMMAP_ENABLE=y
CONFIG_SPARSEMEM_ALLOC_MEM_MAP_TOGETHER=y
CONFIG_SPARSEMEM_VMEMMAP=y
CONFIG_HAVE_MEMBLOCK=y
CONFIG_MEMORY_HOTPLUG=y
CONFIG_MEMORY_HOTPLUG_SPARSE=y
CONFIG_PAGEFLAGS_EXTENDED=y
CONFIG_SPLIT_PTLOCK_CPUS=4
CONFIG_COMPACTION=y
CONFIG_MIGRATION=y
CONFIG_PHYS_ADDR_T_64BIT=y
CONFIG_ZONE_DMA_FLAG=1
CONFIG_BOUNCE=y
CONFIG_VIRT_TO_BUS=y
CONFIG_MMU_NOTIFIER=y
CONFIG_KSM=y
CONFIG_DEFAULT_MMAP_MIN_ADDR=4096
CONFIG_ARCH_SUPPORTS_MEMORY_FAILURE=y
CONFIG_TRANSPARENT_HUGEPAGE=y
CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=y
CONFIG_X86_RESERVE_LOW=64
CONFIG_MTRR=y
CONFIG_MTRR_SANITIZER=y
CONFIG_MTRR_SANITIZER_ENABLE_DEFAULT=0
CONFIG_MTRR_SANITIZER_SPARE_REG_NR_DEFAULT=1
CONFIG_X86_PAT=y
CONFIG_ARCH_USES_PG_UNCACHED=y
CONFIG_SECCOMP=y
CONFIG_HZ_100=y
CONFIG_HZ=100
CONFIG_SCHED_HRTICK=y
CONFIG_PHYSICAL_START=0x1000000
CONFIG_PHYSICAL_ALIGN=0x1000000
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
CONFIG_ARCH_ENABLE_MEMORY_HOTREMOVE=y

CONFIG_ACPI=y
CONFIG_ACPI_BUTTON=y
CONFIG_ACPI_DOCK=y
CONFIG_ACPI_PROCESSOR=y
CONFIG_ACPI_THERMAL=y
CONFIG_ACPI_BLACKLIST_YEAR=0
CONFIG_X86_PM_TIMER=y

CONFIG_CPU_FREQ=y
CONFIG_CPU_FREQ_TABLE=y
CONFIG_CPU_FREQ_STAT=y
CONFIG_CPU_FREQ_STAT_DETAILS=y
CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND=y
CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
CONFIG_CPU_FREQ_GOV_POWERSAVE=y
CONFIG_CPU_FREQ_GOV_USERSPACE=y
CONFIG_CPU_FREQ_GOV_ONDEMAND=y
CONFIG_CPU_FREQ_GOV_CONSERVATIVE=y

CONFIG_X86_ACPI_CPUFREQ=m
CONFIG_X86_POWERNOW_K8=m

CONFIG_CPU_IDLE=y
CONFIG_CPU_IDLE_GOV_LADDER=y
CONFIG_CPU_IDLE_GOV_MENU=y


CONFIG_PCI=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_MMCONFIG=y
CONFIG_PCI_DOMAINS=y
CONFIG_PCIEPORTBUS=y
CONFIG_PCIEAER=y
CONFIG_PCIEASPM=y
CONFIG_ARCH_SUPPORTS_MSI=y
CONFIG_PCI_MSI=y
CONFIG_HT_IRQ=y
CONFIG_PCI_IOAPIC=y
CONFIG_PCI_LABEL=y
CONFIG_ISA_DMA_API=y
CONFIG_AMD_NB=y

CONFIG_BINFMT_ELF=y
CONFIG_COMPAT_BINFMT_ELF=y
CONFIG_BINFMT_MISC=m
CONFIG_IA32_EMULATION=y
CONFIG_IA32_AOUT=m
CONFIG_COMPAT=y
CONFIG_COMPAT_FOR_U64_ALIGNMENT=y
CONFIG_SYSVIPC_COMPAT=y
CONFIG_HAVE_TEXT_POKE_SMP=y
CONFIG_NET=y

CONFIG_PACKET=y
CONFIG_UNIX=y
CONFIG_XFRM=y
CONFIG_XFRM_MIGRATE=y
CONFIG_XFRM_IPCOMP=m
CONFIG_NET_KEY=m
CONFIG_NET_KEY_MIGRATE=y
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_ROUTE_CLASSID=y
CONFIG_NET_IPGRE_DEMUX=m
CONFIG_INET_AH=m
CONFIG_INET_ESP=m
CONFIG_INET_IPCOMP=m
CONFIG_INET_XFRM_TUNNEL=m
CONFIG_INET_TUNNEL=m
CONFIG_INET_XFRM_MODE_TRANSPORT=m
CONFIG_INET_XFRM_MODE_TUNNEL=m
CONFIG_INET_XFRM_MODE_BEET=m
CONFIG_INET_DIAG=y
CONFIG_INET_TCP_DIAG=y
CONFIG_TCP_CONG_CUBIC=y
CONFIG_DEFAULT_TCP_CONG="cubic"
CONFIG_NETFILTER=y
CONFIG_NETFILTER_ADVANCED=y
CONFIG_BRIDGE_NETFILTER=y

CONFIG_NETFILTER_NETLINK=m
CONFIG_NETFILTER_NETLINK_QUEUE=m
CONFIG_NETFILTER_NETLINK_LOG=m
CONFIG_NF_CONNTRACK=m
CONFIG_NF_CONNTRACK_MARK=y
CONFIG_NF_CONNTRACK_ZONES=y
CONFIG_NF_CT_PROTO_DCCP=m
CONFIG_NF_CT_PROTO_GRE=m
CONFIG_NF_CT_PROTO_SCTP=m
CONFIG_NF_CT_PROTO_UDPLITE=m
CONFIG_NF_CONNTRACK_AMANDA=m
CONFIG_NF_CONNTRACK_FTP=m
CONFIG_NF_CONNTRACK_H323=m
CONFIG_NF_CONNTRACK_IRC=m
CONFIG_NF_CONNTRACK_BROADCAST=m
CONFIG_NF_CONNTRACK_NETBIOS_NS=m
CONFIG_NF_CONNTRACK_SNMP=m
CONFIG_NF_CONNTRACK_PPTP=m
CONFIG_NF_CONNTRACK_SANE=m
CONFIG_NF_CONNTRACK_SIP=m
CONFIG_NF_CONNTRACK_TFTP=m
CONFIG_NF_CT_NETLINK=m
CONFIG_NETFILTER_TPROXY=m
CONFIG_NETFILTER_XTABLES=m

CONFIG_NETFILTER_XT_MARK=m
CONFIG_NETFILTER_XT_CONNMARK=m

CONFIG_NETFILTER_XT_TARGET_CHECKSUM=m
CONFIG_NETFILTER_XT_TARGET_CLASSIFY=m
CONFIG_NETFILTER_XT_TARGET_CONNMARK=m
CONFIG_NETFILTER_XT_TARGET_CT=m
CONFIG_NETFILTER_XT_TARGET_DSCP=m
CONFIG_NETFILTER_XT_TARGET_HL=m
CONFIG_NETFILTER_XT_TARGET_IDLETIMER=m
CONFIG_NETFILTER_XT_TARGET_MARK=m
CONFIG_NETFILTER_XT_TARGET_NFLOG=m
CONFIG_NETFILTER_XT_TARGET_NFQUEUE=m
CONFIG_NETFILTER_XT_TARGET_NOTRACK=m
CONFIG_NETFILTER_XT_TARGET_RATEEST=m
CONFIG_NETFILTER_XT_TARGET_TEE=m
CONFIG_NETFILTER_XT_TARGET_TPROXY=m
CONFIG_NETFILTER_XT_TARGET_TRACE=m
CONFIG_NETFILTER_XT_TARGET_TCPMSS=m
CONFIG_NETFILTER_XT_TARGET_TCPOPTSTRIP=m

CONFIG_NETFILTER_XT_MATCH_ADDRTYPE=m
CONFIG_NETFILTER_XT_MATCH_CLUSTER=m
CONFIG_NETFILTER_XT_MATCH_COMMENT=m
CONFIG_NETFILTER_XT_MATCH_CONNBYTES=m
CONFIG_NETFILTER_XT_MATCH_CONNLIMIT=m
CONFIG_NETFILTER_XT_MATCH_CONNMARK=m
CONFIG_NETFILTER_XT_MATCH_CONNTRACK=m
CONFIG_NETFILTER_XT_MATCH_CPU=m
CONFIG_NETFILTER_XT_MATCH_DCCP=m
CONFIG_NETFILTER_XT_MATCH_DEVGROUP=m
CONFIG_NETFILTER_XT_MATCH_DSCP=m
CONFIG_NETFILTER_XT_MATCH_ESP=m
CONFIG_NETFILTER_XT_MATCH_HASHLIMIT=m
CONFIG_NETFILTER_XT_MATCH_HELPER=m
CONFIG_NETFILTER_XT_MATCH_HL=m
CONFIG_NETFILTER_XT_MATCH_IPRANGE=m
CONFIG_NETFILTER_XT_MATCH_LENGTH=m
CONFIG_NETFILTER_XT_MATCH_LIMIT=m
CONFIG_NETFILTER_XT_MATCH_MAC=m
CONFIG_NETFILTER_XT_MATCH_MARK=m
CONFIG_NETFILTER_XT_MATCH_MULTIPORT=m
CONFIG_NETFILTER_XT_MATCH_OSF=m
CONFIG_NETFILTER_XT_MATCH_OWNER=m
CONFIG_NETFILTER_XT_MATCH_POLICY=m
CONFIG_NETFILTER_XT_MATCH_PHYSDEV=m
CONFIG_NETFILTER_XT_MATCH_PKTTYPE=m
CONFIG_NETFILTER_XT_MATCH_QUOTA=m
CONFIG_NETFILTER_XT_MATCH_RATEEST=m
CONFIG_NETFILTER_XT_MATCH_REALM=m
CONFIG_NETFILTER_XT_MATCH_RECENT=m
CONFIG_NETFILTER_XT_MATCH_SCTP=m
CONFIG_NETFILTER_XT_MATCH_SOCKET=m
CONFIG_NETFILTER_XT_MATCH_STATE=m
CONFIG_NETFILTER_XT_MATCH_STATISTIC=m
CONFIG_NETFILTER_XT_MATCH_STRING=m
CONFIG_NETFILTER_XT_MATCH_TCPMSS=m
CONFIG_NETFILTER_XT_MATCH_TIME=m
CONFIG_NETFILTER_XT_MATCH_U32=m

CONFIG_NF_DEFRAG_IPV4=m
CONFIG_NF_CONNTRACK_IPV4=m
CONFIG_NF_CONNTRACK_PROC_COMPAT=y
CONFIG_IP_NF_QUEUE=m
CONFIG_IP_NF_IPTABLES=m
CONFIG_IP_NF_MATCH_AH=m
CONFIG_IP_NF_MATCH_ECN=m
CONFIG_IP_NF_MATCH_TTL=m
CONFIG_IP_NF_FILTER=m
CONFIG_IP_NF_TARGET_REJECT=m
CONFIG_IP_NF_TARGET_LOG=m
CONFIG_IP_NF_TARGET_ULOG=m
CONFIG_NF_NAT=m
CONFIG_NF_NAT_NEEDED=y
CONFIG_IP_NF_TARGET_MASQUERADE=m
CONFIG_IP_NF_TARGET_NETMAP=m
CONFIG_IP_NF_TARGET_REDIRECT=m
CONFIG_NF_NAT_SNMP_BASIC=m
CONFIG_NF_NAT_PROTO_DCCP=m
CONFIG_NF_NAT_PROTO_GRE=m
CONFIG_NF_NAT_PROTO_UDPLITE=m
CONFIG_NF_NAT_PROTO_SCTP=m
CONFIG_NF_NAT_FTP=m
CONFIG_NF_NAT_IRC=m
CONFIG_NF_NAT_TFTP=m
CONFIG_NF_NAT_AMANDA=m
CONFIG_NF_NAT_PPTP=m
CONFIG_NF_NAT_H323=m
CONFIG_NF_NAT_SIP=m
CONFIG_IP_NF_MANGLE=m
CONFIG_IP_NF_TARGET_CLUSTERIP=m
CONFIG_IP_NF_TARGET_ECN=m
CONFIG_IP_NF_TARGET_TTL=m
CONFIG_IP_NF_RAW=m
CONFIG_IP_NF_ARPTABLES=m
CONFIG_IP_NF_ARPFILTER=m
CONFIG_IP_NF_ARP_MANGLE=m
CONFIG_IP_SCTP=m
CONFIG_SCTP_HMAC_MD5=y
CONFIG_STP=y
CONFIG_BRIDGE=y
CONFIG_BRIDGE_IGMP_SNOOPING=y
CONFIG_VLAN_8021Q=y
CONFIG_LLC=y
CONFIG_NET_SCHED=y

CONFIG_NET_SCH_CBQ=m
CONFIG_NET_SCH_HTB=m
CONFIG_NET_SCH_HFSC=m
CONFIG_NET_SCH_PRIO=m
CONFIG_NET_SCH_RED=m
CONFIG_NET_SCH_SFB=m
CONFIG_NET_SCH_SFQ=m
CONFIG_NET_SCH_TEQL=m
CONFIG_NET_SCH_TBF=m
CONFIG_NET_SCH_GRED=m
CONFIG_NET_SCH_DSMARK=m
CONFIG_NET_SCH_NETEM=m
CONFIG_NET_SCH_MQPRIO=m
CONFIG_NET_SCH_CHOKE=m
CONFIG_NET_SCH_INGRESS=m

CONFIG_NET_CLS=y
CONFIG_NET_CLS_BASIC=m
CONFIG_NET_CLS_TCINDEX=m
CONFIG_NET_CLS_ROUTE4=m
CONFIG_NET_CLS_FW=m
CONFIG_NET_CLS_U32=m
CONFIG_NET_CLS_FLOW=m
CONFIG_NET_EMATCH=y
CONFIG_NET_EMATCH_STACK=32
CONFIG_NET_EMATCH_CMP=m
CONFIG_NET_EMATCH_NBYTE=m
CONFIG_NET_EMATCH_U32=m
CONFIG_NET_EMATCH_META=m
CONFIG_NET_EMATCH_TEXT=m
CONFIG_NET_CLS_ACT=y
CONFIG_NET_ACT_POLICE=m
CONFIG_NET_ACT_GACT=m
CONFIG_GACT_PROB=y
CONFIG_NET_ACT_MIRRED=m
CONFIG_NET_ACT_IPT=m
CONFIG_NET_ACT_NAT=m
CONFIG_NET_ACT_PEDIT=m
CONFIG_NET_ACT_SIMP=m
CONFIG_NET_ACT_SKBEDIT=m
CONFIG_NET_ACT_CSUM=m
CONFIG_NET_SCH_FIFO=y
CONFIG_RPS=y
CONFIG_RFS_ACCEL=y
CONFIG_XPS=y



CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
CONFIG_DEVTMPFS=y
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_FW_LOADER=y
CONFIG_FIRMWARE_IN_KERNEL=y
CONFIG_EXTRA_FIRMWARE=""
CONFIG_ARCH_NO_SYSDEV_OPS=y
CONFIG_PNP=y
CONFIG_PNP_DEBUG_MESSAGES=y

CONFIG_PNPACPI=y
CONFIG_BLK_DEV=y
CONFIG_BLK_DEV_LOOP=y

CONFIG_HAVE_IDE=y

CONFIG_SCSI_MOD=y
CONFIG_RAID_ATTRS=m
CONFIG_SCSI=y
CONFIG_SCSI_DMA=y
CONFIG_SCSI_PROC_FS=y

CONFIG_BLK_DEV_SD=y
CONFIG_BLK_DEV_SR=y
CONFIG_CHR_DEV_SG=y
CONFIG_SCSI_WAIT_SCAN=m

CONFIG_SCSI_SAS_ATTRS=m
CONFIG_SCSI_LOWLEVEL=y
CONFIG_MEGARAID_SAS=m
CONFIG_ATA=y
CONFIG_ATA_VERBOSE_ERROR=y
CONFIG_ATA_ACPI=y
CONFIG_SATA_PMP=y

CONFIG_SATA_AHCI=m
CONFIG_SATA_SIL24=m
CONFIG_ATA_SFF=y

CONFIG_ATA_BMDMA=y

CONFIG_SATA_MV=m
CONFIG_SATA_NV=m
CONFIG_SATA_PROMISE=m
CONFIG_SATA_SIL=m



CONFIG_MD=y
CONFIG_BLK_DEV_MD=y
CONFIG_MD_LINEAR=y
CONFIG_MD_RAID0=y
CONFIG_MD_RAID1=y
CONFIG_MD_RAID10=y
CONFIG_MD_RAID456=y
CONFIG_BLK_DEV_DM=y

CONFIG_NETDEVICES=y
CONFIG_DUMMY=m
CONFIG_BONDING=m
CONFIG_TUN=y
CONFIG_MII=m
CONFIG_NET_ETHERNET=y
CONFIG_NET_PCI=y
CONFIG_FORCEDETH=m
CONFIG_NETDEV_1000=y
CONFIG_R8169=m



CONFIG_PPP=m
CONFIG_PPP_MULTILINK=y
CONFIG_PPP_FILTER=y
CONFIG_PPP_ASYNC=m
CONFIG_PPP_SYNC_TTY=m
CONFIG_PPP_DEFLATE=m
CONFIG_PPP_BSDCOMP=m
CONFIG_PPP_MPPE=m
CONFIG_PPPOE=m
CONFIG_SLHC=m
CONFIG_NETCONSOLE=m
CONFIG_NETCONSOLE_DYNAMIC=y
CONFIG_NETPOLL=y
CONFIG_NET_POLL_CONTROLLER=y

CONFIG_INPUT=y

CONFIG_INPUT_MOUSEDEV=y
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768

CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y

CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_SERIO_LIBPS2=y

CONFIG_VT=y
CONFIG_CONSOLE_TRANSLATIONS=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
CONFIG_UNIX98_PTYS=y
CONFIG_LEGACY_PTYS=y
CONFIG_LEGACY_PTY_COUNT=256

CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_FIX_EARLYCON_MEM=y
CONFIG_SERIAL_8250_PCI=y
CONFIG_SERIAL_8250_PNP=y
CONFIG_SERIAL_8250_NR_UARTS=4
CONFIG_SERIAL_8250_RUNTIME_UARTS=4

CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
CONFIG_RTC=y
CONFIG_HPET=y
CONFIG_HPET_MMAP=y
CONFIG_DEVPORT=y
CONFIG_I2C=y
CONFIG_I2C_BOARDINFO=y
CONFIG_I2C_COMPAT=y
CONFIG_I2C_CHARDEV=y
CONFIG_I2C_HELPER_AUTO=y
CONFIG_I2C_ALGOBIT=y


CONFIG_I2C_AMD756=m
CONFIG_I2C_PIIX4=m
CONFIG_I2C_NFORCE2=m
CONFIG_I2C_VIA=m
CONFIG_I2C_VIAPRO=m

CONFIG_I2C_SCMI=m





CONFIG_ARCH_WANT_OPTIONAL_GPIOLIB=y
CONFIG_HWMON=y
CONFIG_HWMON_VID=m

CONFIG_SENSORS_K8TEMP=m
CONFIG_SENSORS_K10TEMP=m
CONFIG_SENSORS_W83781D=m
CONFIG_SENSORS_W83791D=m
CONFIG_SENSORS_W83792D=m
CONFIG_SENSORS_W83793=m
CONFIG_SENSORS_W83795=m
CONFIG_SENSORS_W83795_FANCTRL=y
CONFIG_SENSORS_W83L785TS=m
CONFIG_SENSORS_W83L786NG=m
CONFIG_SENSORS_W83627HF=m
CONFIG_SENSORS_W83627EHF=m

CONFIG_SENSORS_ATK0110=m
CONFIG_THERMAL=y
CONFIG_THERMAL_HWMON=y
CONFIG_SSB_POSSIBLE=y


CONFIG_VGA_ARB=y
CONFIG_VGA_ARB_MAX_GPUS=16
CONFIG_VGASTATE=y
CONFIG_FB=y
CONFIG_FB_DDC=y
CONFIG_FB_CFB_FILLRECT=y
CONFIG_FB_CFB_COPYAREA=y
CONFIG_FB_CFB_IMAGEBLIT=y
CONFIG_FB_MODE_HELPERS=y

CONFIG_FB_NVIDIA=y
CONFIG_FB_NVIDIA_I2C=y
CONFIG_FB_RADEON=y
CONFIG_FB_RADEON_I2C=y


CONFIG_VGA_CONSOLE=y
CONFIG_DUMMY_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE=y
CONFIG_FONT_8x8=y
CONFIG_FONT_8x16=y
CONFIG_LOGO=y
CONFIG_LOGO_LINUX_MONO=y
CONFIG_LOGO_LINUX_VGA16=y
CONFIG_LOGO_LINUX_CLUT224=y
CONFIG_HID_SUPPORT=y
CONFIG_HID=y
CONFIG_HIDRAW=y

CONFIG_USB_HID=m
CONFIG_USB_HIDDEV=y

CONFIG_HID_A4TECH=m
CONFIG_HID_APPLE=m
CONFIG_HID_BELKIN=m
CONFIG_HID_CHERRY=m
CONFIG_HID_CHICONY=m
CONFIG_HID_CYPRESS=m
CONFIG_HID_EZKEY=m
CONFIG_HID_KYE=m
CONFIG_HID_KENSINGTON=m
CONFIG_HID_LOGITECH=m
CONFIG_HID_MICROSOFT=m
CONFIG_HID_MONTEREY=m
CONFIG_USB_SUPPORT=y
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y
CONFIG_USB_ARCH_HAS_EHCI=y
CONFIG_USB=m


CONFIG_USB_XHCI_HCD=m
CONFIG_USB_EHCI_HCD=m
CONFIG_USB_OHCI_HCD=m
CONFIG_USB_OHCI_LITTLE_ENDIAN=y
CONFIG_USB_UHCI_HCD=m


CONFIG_USB_ACM=m


CONFIG_USB_STORAGE=m
CONFIG_USB_LIBUSUAL=y


CONFIG_USB_SERIAL=m
CONFIG_USB_SERIAL_GENERIC=y
CONFIG_USB_SERIAL_FTDI_SIO=m
CONFIG_USB_SERIAL_IPAQ=m
CONFIG_USB_SERIAL_PL2303=m



CONFIG_FIRMWARE_MEMMAP=y
CONFIG_DMIID=y

CONFIG_EXT2_FS=y
CONFIG_EXT2_FS_XATTR=y
CONFIG_EXT3_FS=y
CONFIG_EXT3_FS_XATTR=y
CONFIG_EXT4_FS=y
CONFIG_EXT4_FS_XATTR=y
CONFIG_JBD=y
CONFIG_JBD2=y
CONFIG_FS_MBCACHE=y
CONFIG_EXPORTFS=y
CONFIG_FILE_LOCKING=y
CONFIG_FSNOTIFY=y
CONFIG_DNOTIFY=y
CONFIG_INOTIFY_USER=y
CONFIG_FANOTIFY=y
CONFIG_FUSE_FS=m
CONFIG_CUSE=m

CONFIG_FSCACHE=m
CONFIG_FSCACHE_DEBUG=y
CONFIG_CACHEFILES=m

CONFIG_ISO9660_FS=m
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_UDF_FS=m
CONFIG_UDF_NLS=y

CONFIG_FAT_FS=m
CONFIG_MSDOS_FS=m
CONFIG_VFAT_FS=m
CONFIG_FAT_DEFAULT_CODEPAGE=437
CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
CONFIG_NTFS_FS=m

CONFIG_PROC_FS=y
CONFIG_PROC_SYSCTL=y
CONFIG_PROC_PAGE_MONITOR=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
CONFIG_HUGETLBFS=y
CONFIG_HUGETLB_PAGE=y
CONFIG_CONFIGFS_FS=m
CONFIG_NETWORK_FILESYSTEMS=y
CONFIG_NFS_FS=y
CONFIG_NFS_V3=y
CONFIG_NFSD=y
CONFIG_NFSD_DEPRECATED=y
CONFIG_NFSD_V3=y
CONFIG_LOCKD=y
CONFIG_LOCKD_V4=y
CONFIG_NFS_COMMON=y
CONFIG_SUNRPC=y

CONFIG_PARTITION_ADVANCED=y
CONFIG_MSDOS_PARTITION=y
CONFIG_EFI_PARTITION=y
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="iso8859-1"
CONFIG_NLS_CODEPAGE_437=y
CONFIG_NLS_ISO8859_1=y
CONFIG_DLM=m

CONFIG_TRACE_IRQFLAGS_SUPPORT=y
CONFIG_PRINTK_TIME=y
CONFIG_DEFAULT_MESSAGE_LOGLEVEL=4
CONFIG_ENABLE_WARN_DEPRECATED=y
CONFIG_ENABLE_MUST_CHECK=y
CONFIG_FRAME_WARN=2048
CONFIG_MAGIC_SYSRQ=y
CONFIG_DEBUG_FS=y
CONFIG_DEBUG_KERNEL=y
CONFIG_DETECT_HUNG_TASK=y
CONFIG_BOOTPARAM_HUNG_TASK_PANIC_VALUE=0
CONFIG_TIMER_STATS=y
CONFIG_DEBUG_BUGVERBOSE=y
CONFIG_DEBUG_MEMORY_INIT=y
CONFIG_ARCH_WANT_FRAME_POINTERS=y
CONFIG_SYSCTL_SYSCALL_CHECK=y
CONFIG_USER_STACKTRACE_SUPPORT=y
CONFIG_HAVE_FUNCTION_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_FP_TEST=y
CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
CONFIG_HAVE_SYSCALL_TRACEPOINTS=y
CONFIG_HAVE_C_RECORDMCOUNT=y
CONFIG_TRACING_SUPPORT=y
CONFIG_HAVE_ARCH_KGDB=y
CONFIG_HAVE_ARCH_KMEMCHECK=y
CONFIG_X86_VERBOSE_BOOTUP=y
CONFIG_EARLY_PRINTK=y
CONFIG_HAVE_MMIOTRACE_SUPPORT=y
CONFIG_IO_DELAY_TYPE_0X80=0
CONFIG_IO_DELAY_TYPE_0XED=1
CONFIG_IO_DELAY_TYPE_UDELAY=2
CONFIG_IO_DELAY_TYPE_NONE=3
CONFIG_IO_DELAY_0X80=y
CONFIG_DEFAULT_IO_DELAY_TYPE=0

CONFIG_DEFAULT_SECURITY_DAC=y
CONFIG_DEFAULT_SECURITY=""
CONFIG_XOR_BLOCKS=y
CONFIG_ASYNC_CORE=y
CONFIG_ASYNC_MEMCPY=y
CONFIG_ASYNC_XOR=y
CONFIG_ASYNC_PQ=y
CONFIG_ASYNC_RAID6_RECOV=y
CONFIG_CRYPTO=y

CONFIG_CRYPTO_ALGAPI=m
CONFIG_CRYPTO_ALGAPI2=m
CONFIG_CRYPTO_AEAD=m
CONFIG_CRYPTO_AEAD2=m
CONFIG_CRYPTO_BLKCIPHER=m
CONFIG_CRYPTO_BLKCIPHER2=m
CONFIG_CRYPTO_HASH=m
CONFIG_CRYPTO_HASH2=m
CONFIG_CRYPTO_RNG2=m
CONFIG_CRYPTO_PCOMP2=m
CONFIG_CRYPTO_MANAGER=m
CONFIG_CRYPTO_MANAGER2=m
CONFIG_CRYPTO_MANAGER_DISABLE_TESTS=y
CONFIG_CRYPTO_WORKQUEUE=m
CONFIG_CRYPTO_AUTHENC=m


CONFIG_CRYPTO_CBC=m
CONFIG_CRYPTO_ECB=m

CONFIG_CRYPTO_HMAC=m

CONFIG_CRYPTO_CRC32C=m
CONFIG_CRYPTO_MD5=m
CONFIG_CRYPTO_SHA1=m

CONFIG_CRYPTO_ARC4=m
CONFIG_CRYPTO_DES=m

CONFIG_CRYPTO_DEFLATE=m

CONFIG_HAVE_KVM=y
CONFIG_HAVE_KVM_IRQCHIP=y
CONFIG_HAVE_KVM_EVENTFD=y
CONFIG_KVM_APIC_ARCHITECTURE=y
CONFIG_KVM_MMIO=y
CONFIG_KVM_ASYNC_PF=y
CONFIG_VIRTUALIZATION=y
CONFIG_KVM=m
CONFIG_KVM_INTEL=m
CONFIG_KVM_AMD=m
CONFIG_VHOST_NET=m

CONFIG_RAID6_PQ=y
CONFIG_BITREVERSE=y
CONFIG_GENERIC_FIND_FIRST_BIT=y
CONFIG_GENERIC_FIND_NEXT_BIT=y
CONFIG_GENERIC_FIND_LAST_BIT=y
CONFIG_CRC_CCITT=m
CONFIG_CRC16=y
CONFIG_CRC_T10DIF=y
CONFIG_CRC_ITU_T=y
CONFIG_CRC32=y
CONFIG_LIBCRC32C=m
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=m
CONFIG_LZO_DECOMPRESS=y
CONFIG_XZ_DEC=y
CONFIG_XZ_DEC_X86=y
CONFIG_XZ_DEC_POWERPC=y
CONFIG_XZ_DEC_IA64=y
CONFIG_XZ_DEC_ARM=y
CONFIG_XZ_DEC_ARMTHUMB=y
CONFIG_XZ_DEC_SPARC=y
CONFIG_XZ_DEC_BCJ=y
CONFIG_DECOMPRESS_GZIP=y
CONFIG_DECOMPRESS_BZIP2=y
CONFIG_DECOMPRESS_LZMA=y
CONFIG_DECOMPRESS_XZ=y
CONFIG_DECOMPRESS_LZO=y
CONFIG_TEXTSEARCH=y
CONFIG_TEXTSEARCH_KMP=m
CONFIG_TEXTSEARCH_BM=m
CONFIG_TEXTSEARCH_FSM=m
CONFIG_HAS_IOMEM=y
CONFIG_HAS_IOPORT=y
CONFIG_HAS_DMA=y
CONFIG_CPU_RMAP=y
CONFIG_NLATTR=y

00:00.0 Host bridge: Advanced Micro Devices [AMD] RS880 Host Bridge
00:01.0 PCI bridge: ASRock Incorporation Device 9602
00:02.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI bridge
(ext gfx port 0)
00:03.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI bridge
(ext gfx port 1)
00:04.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI bridge
(PCIE port 0)
00:09.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI bridge
(PCIE port 4)
00:0a.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI bridge
(PCIE port 5)
00:11.0 SATA controller: ATI Technologies Inc SB700/SB800 SATA
Controller [AHCI mode] (rev 40)
00:12.0 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI0
Controller
00:12.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI Controller
00:13.0 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI0
Controller
00:13.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI Controller
00:14.0 SMBus: ATI Technologies Inc SBx00 SMBus Controller (rev 41)
00:14.2 Audio device: ATI Technologies Inc SBx00 Azalia (Intel HDA) (rev 40)
00:14.3 ISA bridge: ATI Technologies Inc SB700/SB800 LPC host controller
(rev 40)
00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge (rev 40)
00:14.5 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI2
Controller
00:16.0 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI0
Controller
00:16.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI Controller
00:18.0 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor
HyperTransport Configuration
00:18.1 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor
Address Map
00:18.2 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor
DRAM Controller
00:18.3 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor
Miscellaneous Control
00:18.4 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor
Link Control
01:05.0 VGA compatible controller: ATI Technologies Inc RS880 [Radeon HD
4250]
02:00.0 USB Controller: NEC Corporation uPD720200 USB 3.0 Host
Controller (rev 03)
03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd.
RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 06)
04:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 9240
(rev 02)
05:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 9240
(rev 02)
07:00.0 SCSI storage controller: Marvell Technology Group Ltd. 88SX7042
PCI-e 4-port SATA-II (rev 02)


2011-05-31 05:47:38

by Borislav Petkov

[permalink] [raw]
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39

Looks like a KSM issue. Disabling CONFIG_KSM should at least stop your
machine from oopsing.

Adding linux-mm.

On Tue, May 31, 2011 at 09:24:03AM +0800, Brad Campbell wrote:
> G'day all,
>
> I'm running a pretty standard home server
> x86_64 Phenom-II 6 Core
> 16GB DDR 3
>
> I run some virtual machines under that. 3 x Debian 64 Bit, 1 x XP 32
> Bit. These run at boot.
>
> When I fire up another XP 32 bit instance and play with it for more
> than about 2 minutes, I get the panics included in this mail.
>
> I've included three of them here. The first and third are as booted.
> The second was with ksmd disabled just as a data point.
>
> The machine passes every load test and memory test I can throw at
> it, but I still can't rule out this being a hardware issue.
>
> Provided I don't start this XP VM the machine is quite stable, but
> running this VM will kill it within minutes.
>
> This was tested with qemu-kvm.
> The last commit in the git tree was
> commit c007db193eb6b2557acb5caf2dc4d7023639e6f3
> Author: Avi Kivity <[email protected]>
> Date: Sun May 29 09:00:42 2011 -0400
> (I pulled it yesterday)
>
> These panics were captured with netconsole to a remote syslog
> daemon, and the formatting was ruined, so I've reformatted them by
> hand prior to posting.
>
> I've tested and reproduced this on 2.6.38.[2,3,6 & 7] and 2.6.39 obviously.
>
> Can anyone help shed some light on this?
>
> Regards,
> Brad
>
> [ 438.632061] general protection fault: 0000 [#1] SMP
> [ 438.632196] last sysfs file: /sys/module/x_tables/initstate
> [ 438.632242] CPU 4
> [ 438.632282] Modules linked in: xt_iprange xt_DSCP xt_length
> xt_CLASSIFY sch_sfq xt_CHECKSUM ipt_REJECT ipt_MASQUERADE
> ipt_REDIRECT xt_recent xt_state iptable_nat nf_nat nf_conntrack_ipv4
> nf_conntrack nf_defrag_ipv4 iptable_filter xt_TCPMSS xt_tcpmss
> xt_tcpudp iptable_mangle ip_tables x_tables pppoe pppox ppp_generic
> slhc cls_u32 sch_htb deflate zlib_deflate des_generic cbc ecb
> crypto_blkcipher sha1_generic md5 hmac crypto_hash cryptomgr aead
> crypto_algapi af_key fuse netconsole configfs= vhost_net powernow_k8
> mperf i2c_nforce2 kvm_amd kvm pl2303 usbserial xhci_hcd k10temp
> i2c_piix4 usb_storage usb_libusual ohci_hcd ehci_hcd usbcore ahci
> libahci r8169 mii sata_mv megaraid_sas [last unloaded:
> scsi_wait_scan]
> [ 438.634960]
> [ 438.635006] Pid: 551, comm: ksmd Not tainted 2.6.39 #3 To Be
> Filled By O.E.M. To Be Filled By O.E.M. /880G Extreme3
> [ 438.635170] RIP: 0010:[<ffffffff810b4596>] [<ffffffff810b4596>]
> remove_rmap_item_from_tree+0x96/0x150
> [ 438.635268] RSP: 0018:ffff88041c065e20 EFLAGS: 00010282
> [ 438.635314] RAX: ffff8804153bd8b0 RBX: ffff8804176c3fc0 RCX:
> 0000000000057754
> [ 438.635362] RDX: 0000880415418030 RSI: ffff880414b65003 RDI:
> ffffea000dde6030
> [ 438.635410] RBP: ffff880414b65000 R08: 0000000000057755 R09:
> 000000001bbde1bd
> [ 438.635458] R10: 000000001c28e0c3 R11: 0000000000000002 R12:
> ffffea000dde6030
> [ 438.635506] R13: ffff8804176c3f80 R14: ffff8804151ed7b0 R15:
> ffff88041bf23be0
> [ 438.635555] FS: 00007f617772e700(0000) GS:ffff88041fd00000(0000)
> knlGS:0000000000000000
> [ 438.635607] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 438.635654] CR2: 0000000000e70000 CR3: 0000000001583000 CR4:
> 00000000000006e0
> [ 438.635703] DR0: 0000000000000045 DR1: 0000000000000000 DR2:
> 0000000000000000
> [ 438.635750] DR3: 0000000000000005 DR6: 00000000ffff0ff0 DR7:
> 0000000000000400
> [ 438.635799] Process ksmd (pid: 551, threadinfo ffff88041c064000,
> task ffff88041d8caa70)
> [ 438.635851] Stack:
> [ 438.635893] ffffea000c43c408 ffff8804176c3fc0 000000000000036c
> ffffffff810b58f2
> [ 438.636088] ffff88041c782a00 00007fc2d81b5000 ffff88041c064000
> 00000000003808ff
> [ 438.636281] ffff88041c064000 ffff88041c065e98 ffff88041d8caa70
> ffff8804165d4480
> [ 438.636479] Call Trace:
> [ 438.636528] [<ffffffff810b58f2>] ? ksm_scan_thread+0x4e2/0xc20
> [ 438.636580] [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
> [ 438.636628] [<ffffffff810b5410>] ? try_to_merge_with_ksm_page+0x570/0x570
> [ 438.636679] [<ffffffff810b5410>] ? try_to_merge_with_ksm_page+0x570/0x570
> [ 438.636730] [<ffffffff810525b6>] ? kthread+0x96/0xa0
> [ 438.636781] [<ffffffff813d1794>] ? kernel_thread_helper+0x4/0x10
> [ 438.636832] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
> [ 438.636882] [<ffffffff813d1790>] ? gs_change+0xb/0xb
> [ 438.636926] Code: 28 48 89 ef e8 6c fe ff ff 48 85 c0 49 89 c4 74
> d2 f0 0f ba 28 00 19 c0 85 c0 0f 85 ae 00 00 00 00 48 8b 43 30 48 8b
> 53 38 48 85 c0
> [ 438.638504] 89 02 74 04 48 89 50 08 48 ba 00 01 10 00 00 00 00 ad de 48 b8
> [ 438.639329] RIP [<ffffffff810b4596>] remove_rmap_item_from_tree+0x96/0x150
> [ 438.639414] RSP <ffff88041c065e20>
> [ 438.639460] ---[ end trace c29fb871f6b874e3 ]---
> [ 438.639506] Kernel panic - not syncing: Fatal exception
> [ 438.639553] Pid: 551, comm: ksmd Tainted: G D 2.6.39 #3
> [ 438.639598] Call Trace:
> [ 438.639644] [<ffffffff813cd6f5>] ? panic+0x92/0x18a
> [ 438.639693] [<ffffffff81038b61>] ? kmsg_dump+0x41/0xf0
> [ 438.639745] [<ffffffff810050ad>] ? oops_end+0x8d/0xa0
> [ 438.639794] [<ffffffff813d05ef>] ? general_protection+0x1f/0x30
> [ 438.639842] [<ffffffff810b4596>] ? remove_rmap_item_from_tree+0x96/0x150
> [ 438.639891] [<ffffffff810b58f2>] ? ksm_scan_thread+0x4e2/0xc20
> [ 438.639941] [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
> [ 438.640045] [<ffffffff810b5410>] ? try_to_merge_with_ksm_page+0x570/0x570
> [ 438.640096] [<ffffffff810b5410>] ? try_to_merge_with_ksm_page+0x570/0x570
> [ 438.640147] [<ffffffff810525b6>] ? kthread+0x96/0xa0
> [ 438.640196] [<ffffffff813d1794>] ? kernel_thread_helper+0x4/0x10
> [ 438.640247] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
> [ 438.640297] [<ffffffff813d1790>] ? gs_change+0xb/0xb
> [ 438.640349] Rebooting in 60 seconds..
>
>
>
> [ 469.706013] general protection fault: 0000 [#1] SMP
> [ 469.706125] last sysfs file:
> /sys/devices/pci0000:00/0000:00:18.3/temp1_input
> [ 469.706162] CPU 1
> [ 469.706195] Modules linked in: xt_iprange xt_DSCP xt_length
> xt_CLASSIFY sch_sfq xt_CHECKSUM ipt_REJECT ipt_MASQUERADE
> ipt_REDIRECT xt_recent xt_state iptable_nat nf_nat nf_conntrack_ipv4
> nf_conntrack nf_defrag_ipv4 iptable_filter xt_TCPMSS xt_tcpmss
> xt_tcpudp iptable_mangle ip_tables x_tables pppoe pppox ppp_generic
> slhc cls_u32 sch_htb deflate zlib_deflate des_generi cbc ecb
> crypto_blkcipher sha1_generic md5 hmac crypto_hash cryptomgr aead
> crypto_algapi af_key fuse netconsole configfs vhost_net powernow_k8
> mperf i2c_nforce2 kvm_amd kvm pl2303 usbserial xhci_hcd i2c_piix4
> k10temp ahci usb_storage usb_libusual ohci_hcd ehci_hcd r8169
> libahci usbcore mii sata_mv megaraid_sas [last unloaded:
> scsi_wait_scan]
> [ 469.708573]
> [ 469.708608] Pid: 4942, comm: runbot Not tainted 2.6.39 #3 To Be
> Filled By O.E.M. To Be Filled By O.E.M. /880G Extreme3
> [ 469.708753] RIP: 0010:[<ffffffff810db878>] [<ffffffff810db878>]
> dup_fd+0x168/0x300
> [ 469.708829] RSP: 0018:ffff880417279dd0 EFLAGS: 00010202
> [ 469.708870] RAX: 00000000000007f8 RBX: ffff88041c1658c0 RCX:
> bfffffffffffffff
> [ 469.708907] RDX: 0000880413155540 RSI: 00000000000000ff RDI:
> 0000000000000800
> [ 469.708945] RBP: ffff88041bf22000 R08: ffff880412839dc0 R09:
> 0000000000000003
> [ 469.708981] R10: 0000000000000001 R11: 4000000000000000 R12:
> 0000000000000100
> [ 469.709018] R13: ffff88041d3cc000 R14: ffff880412839d40 R15:
> ffff88041c72f440
> [ 469.709054] FS: 00007f510c0b4700(0000) GS:ffff88041fc40000(0000)
> knlGS:0000000000000000
> [ 469.709094] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 469.709130] CR2: 00007f510ba56810 CR3: 0000000417c93000 CR4:
> 00000000000006e0
> [ 469.709167] DR0: 00000000000000a0 DR1: 0000000000000000 DR2:
> 0000000000000003
> [ 469.709204] DR3: 00000000000000b0 DR6: 00000000ffff0ff0 DR7:
> 0000000000000400
> [ 469.709240] Process runbot (pid: 4942, threadinfo
> ffff880417278000, task ffff880401351230)
> [ 469.709279] Stack:
> [ 469.709313] ffff880401351230 0000000000000020 ffff880412e9ca80
> ffff88041c1658d0 ffff88041c1658d0
> [ 469.709477] ffff88041723f330 0000000001200011 ffff88041723f330
> 0000000000000000
> [ 469.709641] 00007f510c0b49d0 ffff880401351230 0000000000000000
> ffffffff81037645
> [ 469.709812] Call Trace:
> [ 469.709849] [<ffffffff81037645>] ? copy_process+0xa75/0xfd0
> [ 469.709885] [<ffffffff81037c0d>] ? do_fork+0x6d/0x2b0
> [ 469.709921] [<ffffffff810dbc23>] ? alloc_fd+0x43/0x130
> [ 469.709958] [<ffffffff810c08c5>] ? fd_install+0x35/0x70
> [ 469.709998] [<ffffffff810457a9>] ? sigprocmask+0x69/0x100
> [ 469.710042] [<ffffffff813d0ca3>] ? stub_clone+0x13/0x20
> [ 469.710079] [<ffffffff813d0a3b>] ? system_call_fastpath+0x16/0x1b
> [ 469.710114] Code: 4c 89 c2 e8 6b e5 0f 00 45 85 e4 74 78 41 8d 44
> 24 ff 31 f6 41 ba 01 00 00 00 00 48 8d 3c c5 08 00 00 00 00 31 c0 eb
> 1a 0f 1f 44 00 00 f0> 48 ff 42 30 48 89 54 05 00 48 83 c0
> May 30 16:39:36 192.168.2.1 08 ff c6 48 39 f8 74 3b
> [ 469.712247] RIP [<ffffffff810db878>] dup_fd+0x168/0x300
> [ 469.712317] RSP <ffff880417279dd0>
> [ 469.712391] ---[ end trace b80596608cd6bb6b ]---
> [ 469.712432] Kernel panic - not syncing: Fatal exception
> [ 469.712470] Pid: 4942, comm: runbot Tainted: G D 2.6.39 #3
> [ 469.712508] Call Trace:
> [ 469.712544] [<ffffffff813cd6f5>] ? panic+0x92/0x18a
> [ 469.712581] [<ffffffff81038b61>] ? kmsg_dump+0x41/0xf0
> [ 469.712618] [<ffffffff810050ad>] ? oops_end+0x8d/0xa0
> [ 469.712654] [<ffffffff813d05ef>] ? general_protection+0x1f/0x30
> [ 469.712691] [<ffffffff810db878>] ? dup_fd+0x168/0x300
> [ 469.712727] [<ffffffff81037645>] ? copy_process+0xa75/0xfd0
> [ 469.712767] [<ffffffff81037c0d>] ? do_fork+0x6d/0x2b0
> [ 469.712803] [<ffffffff810dbc23>] ? alloc_fd+0x43/0x130
> [ 469.712839] [<ffffffff810c08c5>] ? fd_install+0x35/0x70
> [ 469.712875] [<ffffffff810457a9>] ? sigprocmask+0x69/0x100
> [ 469.712912] [<ffffffff813d0ca3>] ? stub_clone+0x13/0x20
> [ 469.712948] [<ffffffff813d0a3b>] ? system_call_fastpath+0x16/0x1b
> [ 469.712989] Rebooting in 60 seconds..
>
>
> [ 1100.473612] general protection fault: 0000 [#1] SMP
> [ 1100.473718] last sysfs file:
> /sys/devices/pci0000:00/0000:00:18.3/temp1_input
> [ 1100.473755] CPU 0
> [ 1100.473788] Modules linked in: xt_iprange xt_DSCP xt_length
> xt_CLASSIFY sch_sfq xt_CHECKSUM ipt_REJECT ipt_MASQUERADE
> ipt_REDIRECT xt_recent xt_state iptable_nat nf_nat nf_conntrack_ipv4
> nf_conntrack nf_defrag_ipv4 xt_TCPMSS xt_tcpmss xt_tcpudp
> iptable_mangle pppoe pppox iptable_filter ip_tables x_tables
> ppp_generic slhc cls_u32 sch_htb deflate zlib_deflate des_generic
> cbc ecb crypto_blkcipher sha1_generic md5 hmac crypto_hash cryptomgr
> aead crypto_algapi af_key fuse netconsole configfs vhost_net
> powernow_k8 mperf i2c_nforce2 kvm_amd kvm pl2303 xhci_hcd usbserial
> k10temp i2c_piix4 usb_storage usb_libusual ohci_hcd ehci_hcd ahci
> usbcore sata_mv r8169 mii libahci megaraid_sas [last unloaded:
> scsi_wait_scan]
> [ 1100.476102]
> [ 1100.476137] Pid: 551, comm: ksmd Not tainted 2.6.39 #3 To Be
> Filled By O.E.M. To Be Filled By O.E.M. /880G Extreme3
> [ 1100.476277] RIP: 0010:[<ffffffff810b5c71>] [<ffffffff810b5c71>]
> ksm_scan_thread+0x861/0xc20
> [ 1100.476351] RSP: 0018:ffff88041c065e40 EFLAGS: 00010282
> [ 1100.476387] RAX: 0000000000000000 RBX: ffffea000a5c8468 RCX:
> 0000000000000f01
> [ 1100.476424] RDX: 0000880403456828 RSI: ffff8802f5dcb002 RDI:
> ffffea000ce7eac8
> [ 1100.476460] RBP: ffff880418377fc0 R08: 6db6db6db6db6db7 R09:
> 0000160000000000
> [ 1100.476498] R10: ffffea000ce7eac8 R11: 0000000080100073 R12:
> 00000000000000ab
> [ 1100.476535] R13: ffff880418377ff8 R14: ffffea000ce7eac8 R15:
> ffff880418377fe8
> [ 1100.476572] FS: 00007f08c6498700(0000) GS:ffff88041fc00000(0000)
> knlGS:0000000000000000
> [ 1100.476611] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 1100.476646] CR2: 00000000e1dd3000 CR3: 00000004025f7000 CR4:
> 00000000000006f0
> [ 1100.477007] DR0: 0000000000000045 DR1: 0000000000000000 DR2:
> 0000000000000000
> [ 1100.477042] DR3: 0000000000000005 DR6: 00000000ffff0ff0 DR7:
> 0000000000000400
> [ 1100.477078] Process ksmd (pid: 551, threadinfo ffff88041c064000,
> task ffff88041d0454e0)
> [ 1100.477115] Stack:
> [ 1100.477148] ffff880403a8b100 00007fc1863cb000 ffff88041c064000
> 00000000002f5dcb
> [ 1100.477317] ffff88041c064000 ffff88041c065e98 ffff88041d0454e0
> ffff88041267a580
> [ 1100.477480] 0000000000000000 ffff88041d0454e0 ffffffff81052a20
> ffff88041c065e98
> [ 1100.477644] Call Trace:
> [ 1100.477681] [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
> [ 1100.477717] [<ffffffff810b5410>] ?
> try_to_merge_with_ksm_page+0x570/0x570
> [ 1100.477754] [<ffffffff810b5410>] ?
> try_to_merge_with_ksm_page+0x570/0x570
> [ 1100.477791] [<ffffffff810525b6>] ? kthread+0x96/0xa0
> [ 1100.477829] [<ffffffff813d1794>] ? kernel_thread_helper+0x4/0x10
> [ 1100.477865] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
> [ 1100.477902] [<ffffffff813d1790>] ? gs_change+0xb/0xb
> [ 1100.477937] Code: 05 99 6e 6e 00 4c 89 ff e8 ed 12 fa ff e9 6c f9
> ff ff 48 89 6c 24 38 49 83 7d 00 00 0f 84 63 01 00 00 e8 b3 8b 31 00
> 49 8b 55 00
> [ 1100.479306] 8b 42 e8 4c 8b 72 f0 48 8d 6a d8 4c 8d 78 60 48 89 04 24 4c
> [ 1100.480012] RIP [<ffffffff810b5c71>] ksm_scan_thread+0x861/0xc20
> [ 1100.480015] RSP <ffff88041c065e40>
> [ 1100.480141] ---[ end trace 7382777d9e74ea23 ]---
> [ 1100.480178] Kernel panic - not syncing: Fatal exception
> [ 1100.480216] Pid: 551, comm: ksmd Tainted: G D 2.6.39 #3
> [ 1100.480257] Call Trace:
> [ 1100.480297] [<ffffffff813cd6f5>] ? panic+0x92/0x18a
> [ 1100.480340] [<ffffffff81038b61>] ? kmsg_dump+0x41/0xf0
> [ 1100.480384] [<ffffffff810050ad>] ? oops_end+0x8d/0xa0
> [ 1100.480425] [<ffffffff813d05ef>] ? general_protection+0x1f/0x30
> [ 1100.480465] [<ffffffff810b5c71>] ? ksm_scan_thread+0x861/0xc20
> [ 1100.480507] [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
> [ 1100.480545] [<ffffffff810b5410>] ?
> try_to_merge_with_ksm_page+0x570/0x570
> [ 1100.480584] [<ffffffff810b5410>] ?
> try_to_merge_with_ksm_page+0x570/0x570
> [ 1100.480624] [<ffffffff810525b6>] ? kthread+0x96/0xa0
> [ 1100.480669] [<ffffffff813d1794>] ? kernel_thread_helper+0x4/0x10
> [ 1100.480708] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
> [ 1100.480748] [<ffffffff813d1790>] ? gs_change+0xb/0xb
> [ 1100.480794] Rebooting in 60 seconds..
>
> brad@srv:~$ zcat /proc/config.gz | grep -v "#"
> CONFIG_64BIT=y
> CONFIG_X86_64=y
> CONFIG_X86=y
> CONFIG_INSTRUCTION_DECODER=y
> CONFIG_OUTPUT_FORMAT="elf64-x86-64"
> CONFIG_ARCH_DEFCONFIG="arch/x86/configs/x86_64_defconfig"
> CONFIG_GENERIC_CMOS_UPDATE=y
> CONFIG_CLOCKSOURCE_WATCHDOG=y
> CONFIG_GENERIC_CLOCKEVENTS=y
> CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
> CONFIG_LOCKDEP_SUPPORT=y
> CONFIG_STACKTRACE_SUPPORT=y
> CONFIG_HAVE_LATENCYTOP_SUPPORT=y
> CONFIG_MMU=y
> CONFIG_ZONE_DMA=y
> CONFIG_NEED_DMA_MAP_STATE=y
> CONFIG_NEED_SG_DMA_LENGTH=y
> CONFIG_GENERIC_ISA_DMA=y
> CONFIG_GENERIC_IOMAP=y
> CONFIG_GENERIC_BUG=y
> CONFIG_GENERIC_BUG_RELATIVE_POINTERS=y
> CONFIG_GENERIC_HWEIGHT=y
> CONFIG_ARCH_MAY_HAVE_PC_FDC=y
> CONFIG_RWSEM_XCHGADD_ALGORITHM=y
> CONFIG_ARCH_HAS_CPU_IDLE_WAIT=y
> CONFIG_GENERIC_CALIBRATE_DELAY=y
> CONFIG_GENERIC_TIME_VSYSCALL=y
> CONFIG_ARCH_HAS_CPU_RELAX=y
> CONFIG_ARCH_HAS_DEFAULT_IDLE=y
> CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
> CONFIG_HAVE_SETUP_PER_CPU_AREA=y
> CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=y
> CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=y
> CONFIG_HAVE_CPUMASK_OF_CPU_MAP=y
> CONFIG_ARCH_HIBERNATION_POSSIBLE=y
> CONFIG_ARCH_SUSPEND_POSSIBLE=y
> CONFIG_ZONE_DMA32=y
> CONFIG_ARCH_POPULATES_NODE_MAP=y
> CONFIG_AUDIT_ARCH=y
> CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y
> CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
> CONFIG_X86_64_SMP=y
> CONFIG_X86_HT=y
> CONFIG_ARCH_HWEIGHT_CFLAGS="-fcall-saved-rdi -fcall-saved-rsi
> -fcall-saved-rdx -fcall-saved-rcx -fcall-saved-r8 -fcall-saved-r9
> -fcall-saved-r10 -fcall-saved-r11"
> CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
> CONFIG_CONSTRUCTORS=y
> CONFIG_HAVE_IRQ_WORK=y
> CONFIG_IRQ_WORK=y
>
> CONFIG_EXPERIMENTAL=y
> CONFIG_INIT_ENV_ARG_LIMIT=32
> CONFIG_CROSS_COMPILE=""
> CONFIG_LOCALVERSION=""
> CONFIG_LOCALVERSION_AUTO=y
> CONFIG_HAVE_KERNEL_GZIP=y
> CONFIG_HAVE_KERNEL_BZIP2=y
> CONFIG_HAVE_KERNEL_LZMA=y
> CONFIG_HAVE_KERNEL_XZ=y
> CONFIG_HAVE_KERNEL_LZO=y
> CONFIG_KERNEL_GZIP=y
> CONFIG_SWAP=y
> CONFIG_SYSVIPC=y
> CONFIG_SYSVIPC_SYSCTL=y
> CONFIG_BSD_PROCESS_ACCT=y
> CONFIG_TASKSTATS=y
> CONFIG_TASK_DELAY_ACCT=y
> CONFIG_TASK_XACCT=y
> CONFIG_TASK_IO_ACCOUNTING=y
> CONFIG_HAVE_GENERIC_HARDIRQS=y
>
> CONFIG_GENERIC_HARDIRQS=y
> CONFIG_HAVE_SPARSE_IRQ=y
> CONFIG_GENERIC_IRQ_PROBE=y
> CONFIG_GENERIC_IRQ_SHOW=y
> CONFIG_GENERIC_PENDING_IRQ=y
> CONFIG_IRQ_FORCED_THREADING=y
>
> CONFIG_TREE_RCU=y
> CONFIG_RCU_FANOUT=64
> CONFIG_IKCONFIG=y
> CONFIG_IKCONFIG_PROC=y
> CONFIG_LOG_BUF_SHIFT=20
> CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
> CONFIG_NAMESPACES=y
> CONFIG_BLK_DEV_INITRD=y
> CONFIG_INITRAMFS_SOURCE=""
> CONFIG_RD_GZIP=y
> CONFIG_RD_BZIP2=y
> CONFIG_RD_LZMA=y
> CONFIG_RD_XZ=y
> CONFIG_RD_LZO=y
> CONFIG_SYSCTL=y
> CONFIG_ANON_INODES=y
> CONFIG_UID16=y
> CONFIG_SYSCTL_SYSCALL=y
> CONFIG_KALLSYMS=y
> CONFIG_KALLSYMS_ALL=y
> CONFIG_KALLSYMS_EXTRA_PASS=y
> CONFIG_HOTPLUG=y
> CONFIG_PRINTK=y
> CONFIG_BUG=y
> CONFIG_ELF_CORE=y
> CONFIG_PCSPKR_PLATFORM=y
> CONFIG_BASE_FULL=y
> CONFIG_FUTEX=y
> CONFIG_EPOLL=y
> CONFIG_SIGNALFD=y
> CONFIG_TIMERFD=y
> CONFIG_EVENTFD=y
> CONFIG_SHMEM=y
> CONFIG_AIO=y
> CONFIG_HAVE_PERF_EVENTS=y
>
> CONFIG_PERF_EVENTS=y
> CONFIG_VM_EVENT_COUNTERS=y
> CONFIG_PCI_QUIRKS=y
> CONFIG_SLUB_DEBUG=y
> CONFIG_SLUB=y
> CONFIG_HAVE_OPROFILE=y
> CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
> CONFIG_USER_RETURN_NOTIFIER=y
> CONFIG_HAVE_IOREMAP_PROT=y
> CONFIG_HAVE_KPROBES=y
> CONFIG_HAVE_KRETPROBES=y
> CONFIG_HAVE_OPTPROBES=y
> CONFIG_HAVE_ARCH_TRACEHOOK=y
> CONFIG_HAVE_DMA_ATTRS=y
> CONFIG_USE_GENERIC_SMP_HELPERS=y
> CONFIG_HAVE_REGS_AND_STACK_ACCESS_API=y
> CONFIG_HAVE_DMA_API_DEBUG=y
> CONFIG_HAVE_HW_BREAKPOINT=y
> CONFIG_HAVE_MIXED_BREAKPOINTS_REGS=y
> CONFIG_HAVE_USER_RETURN_NOTIFIER=y
> CONFIG_HAVE_PERF_EVENTS_NMI=y
> CONFIG_HAVE_ARCH_JUMP_LABEL=y
>
> CONFIG_SLABINFO=y
> CONFIG_RT_MUTEXES=y
> CONFIG_BASE_SMALL=0
> CONFIG_MODULES=y
> CONFIG_MODULE_UNLOAD=y
> CONFIG_STOP_MACHINE=y
> CONFIG_BLOCK=y
> CONFIG_BLK_DEV_BSG=y
> CONFIG_BLOCK_COMPAT=y
>
> CONFIG_IOSCHED_NOOP=y
> CONFIG_IOSCHED_DEADLINE=y
> CONFIG_IOSCHED_CFQ=y
> CONFIG_DEFAULT_CFQ=y
> CONFIG_DEFAULT_IOSCHED="cfq"
> CONFIG_PREEMPT_NOTIFIERS=y
> CONFIG_INLINE_SPIN_UNLOCK=y
> CONFIG_INLINE_SPIN_UNLOCK_IRQ=y
> CONFIG_INLINE_READ_UNLOCK=y
> CONFIG_INLINE_READ_UNLOCK_IRQ=y
> CONFIG_INLINE_WRITE_UNLOCK=y
> CONFIG_INLINE_WRITE_UNLOCK_IRQ=y
> CONFIG_MUTEX_SPIN_ON_OWNER=y
>
> CONFIG_TICK_ONESHOT=y
> CONFIG_NO_HZ=y
> CONFIG_HIGH_RES_TIMERS=y
> CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
> CONFIG_SMP=y
> CONFIG_X86_SUPPORTS_MEMORY_FAILURE=y
> CONFIG_SCHED_OMIT_FRAME_POINTER=y
> CONFIG_NO_BOOTMEM=y
> CONFIG_MK8=y
> CONFIG_X86_INTERNODE_CACHE_SHIFT=6
> CONFIG_X86_CMPXCHG=y
> CONFIG_CMPXCHG_LOCAL=y
> CONFIG_X86_L1_CACHE_SHIFT=6
> CONFIG_X86_XADD=y
> CONFIG_X86_WP_WORKS_OK=y
> CONFIG_X86_INTEL_USERCOPY=y
> CONFIG_X86_USE_PPRO_CHECKSUM=y
> CONFIG_X86_TSC=y
> CONFIG_X86_CMPXCHG64=y
> CONFIG_X86_CMOV=y
> CONFIG_X86_MINIMUM_CPU_FAMILY=64
> CONFIG_X86_DEBUGCTLMSR=y
> CONFIG_CPU_SUP_INTEL=y
> CONFIG_CPU_SUP_AMD=y
> CONFIG_CPU_SUP_CENTAUR=y
> CONFIG_HPET_TIMER=y
> CONFIG_HPET_EMULATE_RTC=y
> CONFIG_DMI=y
> CONFIG_GART_IOMMU=y
> CONFIG_AMD_IOMMU=y
> CONFIG_SWIOTLB=y
> CONFIG_IOMMU_HELPER=y
> CONFIG_IOMMU_API=y
> CONFIG_NR_CPUS=8
> CONFIG_SCHED_MC=y
> CONFIG_PREEMPT_NONE=y
> CONFIG_X86_LOCAL_APIC=y
> CONFIG_X86_IO_APIC=y
> CONFIG_X86_MCE=y
> CONFIG_X86_MCE_AMD=y
> CONFIG_X86_MCE_THRESHOLD=y
> CONFIG_ARCH_PHYS_ADDR_T_64BIT=y
> CONFIG_ARCH_DMA_ADDR_T_64BIT=y
> CONFIG_DIRECT_GBPAGES=y
> CONFIG_ARCH_SPARSEMEM_DEFAULT=y
> CONFIG_ARCH_SPARSEMEM_ENABLE=y
> CONFIG_ARCH_SELECT_MEMORY_MODEL=y
> CONFIG_ARCH_MEMORY_PROBE=y
> CONFIG_ILLEGAL_POINTER_VALUE=0xdead000000000000
> CONFIG_SELECT_MEMORY_MODEL=y
> CONFIG_SPARSEMEM_MANUAL=y
> CONFIG_SPARSEMEM=y
> CONFIG_HAVE_MEMORY_PRESENT=y
> CONFIG_SPARSEMEM_EXTREME=y
> CONFIG_SPARSEMEM_VMEMMAP_ENABLE=y
> CONFIG_SPARSEMEM_ALLOC_MEM_MAP_TOGETHER=y
> CONFIG_SPARSEMEM_VMEMMAP=y
> CONFIG_HAVE_MEMBLOCK=y
> CONFIG_MEMORY_HOTPLUG=y
> CONFIG_MEMORY_HOTPLUG_SPARSE=y
> CONFIG_PAGEFLAGS_EXTENDED=y
> CONFIG_SPLIT_PTLOCK_CPUS=4
> CONFIG_COMPACTION=y
> CONFIG_MIGRATION=y
> CONFIG_PHYS_ADDR_T_64BIT=y
> CONFIG_ZONE_DMA_FLAG=1
> CONFIG_BOUNCE=y
> CONFIG_VIRT_TO_BUS=y
> CONFIG_MMU_NOTIFIER=y
> CONFIG_KSM=y
> CONFIG_DEFAULT_MMAP_MIN_ADDR=4096
> CONFIG_ARCH_SUPPORTS_MEMORY_FAILURE=y
> CONFIG_TRANSPARENT_HUGEPAGE=y
> CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=y
> CONFIG_X86_RESERVE_LOW=64
> CONFIG_MTRR=y
> CONFIG_MTRR_SANITIZER=y
> CONFIG_MTRR_SANITIZER_ENABLE_DEFAULT=0
> CONFIG_MTRR_SANITIZER_SPARE_REG_NR_DEFAULT=1
> CONFIG_X86_PAT=y
> CONFIG_ARCH_USES_PG_UNCACHED=y
> CONFIG_SECCOMP=y
> CONFIG_HZ_100=y
> CONFIG_HZ=100
> CONFIG_SCHED_HRTICK=y
> CONFIG_PHYSICAL_START=0x1000000
> CONFIG_PHYSICAL_ALIGN=0x1000000
> CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
> CONFIG_ARCH_ENABLE_MEMORY_HOTREMOVE=y
>
> CONFIG_ACPI=y
> CONFIG_ACPI_BUTTON=y
> CONFIG_ACPI_DOCK=y
> CONFIG_ACPI_PROCESSOR=y
> CONFIG_ACPI_THERMAL=y
> CONFIG_ACPI_BLACKLIST_YEAR=0
> CONFIG_X86_PM_TIMER=y
>
> CONFIG_CPU_FREQ=y
> CONFIG_CPU_FREQ_TABLE=y
> CONFIG_CPU_FREQ_STAT=y
> CONFIG_CPU_FREQ_STAT_DETAILS=y
> CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND=y
> CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
> CONFIG_CPU_FREQ_GOV_POWERSAVE=y
> CONFIG_CPU_FREQ_GOV_USERSPACE=y
> CONFIG_CPU_FREQ_GOV_ONDEMAND=y
> CONFIG_CPU_FREQ_GOV_CONSERVATIVE=y
>
> CONFIG_X86_ACPI_CPUFREQ=m
> CONFIG_X86_POWERNOW_K8=m
>
> CONFIG_CPU_IDLE=y
> CONFIG_CPU_IDLE_GOV_LADDER=y
> CONFIG_CPU_IDLE_GOV_MENU=y
>
>
> CONFIG_PCI=y
> CONFIG_PCI_DIRECT=y
> CONFIG_PCI_MMCONFIG=y
> CONFIG_PCI_DOMAINS=y
> CONFIG_PCIEPORTBUS=y
> CONFIG_PCIEAER=y
> CONFIG_PCIEASPM=y
> CONFIG_ARCH_SUPPORTS_MSI=y
> CONFIG_PCI_MSI=y
> CONFIG_HT_IRQ=y
> CONFIG_PCI_IOAPIC=y
> CONFIG_PCI_LABEL=y
> CONFIG_ISA_DMA_API=y
> CONFIG_AMD_NB=y
>
> CONFIG_BINFMT_ELF=y
> CONFIG_COMPAT_BINFMT_ELF=y
> CONFIG_BINFMT_MISC=m
> CONFIG_IA32_EMULATION=y
> CONFIG_IA32_AOUT=m
> CONFIG_COMPAT=y
> CONFIG_COMPAT_FOR_U64_ALIGNMENT=y
> CONFIG_SYSVIPC_COMPAT=y
> CONFIG_HAVE_TEXT_POKE_SMP=y
> CONFIG_NET=y
>
> CONFIG_PACKET=y
> CONFIG_UNIX=y
> CONFIG_XFRM=y
> CONFIG_XFRM_MIGRATE=y
> CONFIG_XFRM_IPCOMP=m
> CONFIG_NET_KEY=m
> CONFIG_NET_KEY_MIGRATE=y
> CONFIG_INET=y
> CONFIG_IP_MULTICAST=y
> CONFIG_IP_ROUTE_CLASSID=y
> CONFIG_NET_IPGRE_DEMUX=m
> CONFIG_INET_AH=m
> CONFIG_INET_ESP=m
> CONFIG_INET_IPCOMP=m
> CONFIG_INET_XFRM_TUNNEL=m
> CONFIG_INET_TUNNEL=m
> CONFIG_INET_XFRM_MODE_TRANSPORT=m
> CONFIG_INET_XFRM_MODE_TUNNEL=m
> CONFIG_INET_XFRM_MODE_BEET=m
> CONFIG_INET_DIAG=y
> CONFIG_INET_TCP_DIAG=y
> CONFIG_TCP_CONG_CUBIC=y
> CONFIG_DEFAULT_TCP_CONG="cubic"
> CONFIG_NETFILTER=y
> CONFIG_NETFILTER_ADVANCED=y
> CONFIG_BRIDGE_NETFILTER=y
>
> CONFIG_NETFILTER_NETLINK=m
> CONFIG_NETFILTER_NETLINK_QUEUE=m
> CONFIG_NETFILTER_NETLINK_LOG=m
> CONFIG_NF_CONNTRACK=m
> CONFIG_NF_CONNTRACK_MARK=y
> CONFIG_NF_CONNTRACK_ZONES=y
> CONFIG_NF_CT_PROTO_DCCP=m
> CONFIG_NF_CT_PROTO_GRE=m
> CONFIG_NF_CT_PROTO_SCTP=m
> CONFIG_NF_CT_PROTO_UDPLITE=m
> CONFIG_NF_CONNTRACK_AMANDA=m
> CONFIG_NF_CONNTRACK_FTP=m
> CONFIG_NF_CONNTRACK_H323=m
> CONFIG_NF_CONNTRACK_IRC=m
> CONFIG_NF_CONNTRACK_BROADCAST=m
> CONFIG_NF_CONNTRACK_NETBIOS_NS=m
> CONFIG_NF_CONNTRACK_SNMP=m
> CONFIG_NF_CONNTRACK_PPTP=m
> CONFIG_NF_CONNTRACK_SANE=m
> CONFIG_NF_CONNTRACK_SIP=m
> CONFIG_NF_CONNTRACK_TFTP=m
> CONFIG_NF_CT_NETLINK=m
> CONFIG_NETFILTER_TPROXY=m
> CONFIG_NETFILTER_XTABLES=m
>
> CONFIG_NETFILTER_XT_MARK=m
> CONFIG_NETFILTER_XT_CONNMARK=m
>
> CONFIG_NETFILTER_XT_TARGET_CHECKSUM=m
> CONFIG_NETFILTER_XT_TARGET_CLASSIFY=m
> CONFIG_NETFILTER_XT_TARGET_CONNMARK=m
> CONFIG_NETFILTER_XT_TARGET_CT=m
> CONFIG_NETFILTER_XT_TARGET_DSCP=m
> CONFIG_NETFILTER_XT_TARGET_HL=m
> CONFIG_NETFILTER_XT_TARGET_IDLETIMER=m
> CONFIG_NETFILTER_XT_TARGET_MARK=m
> CONFIG_NETFILTER_XT_TARGET_NFLOG=m
> CONFIG_NETFILTER_XT_TARGET_NFQUEUE=m
> CONFIG_NETFILTER_XT_TARGET_NOTRACK=m
> CONFIG_NETFILTER_XT_TARGET_RATEEST=m
> CONFIG_NETFILTER_XT_TARGET_TEE=m
> CONFIG_NETFILTER_XT_TARGET_TPROXY=m
> CONFIG_NETFILTER_XT_TARGET_TRACE=m
> CONFIG_NETFILTER_XT_TARGET_TCPMSS=m
> CONFIG_NETFILTER_XT_TARGET_TCPOPTSTRIP=m
>
> CONFIG_NETFILTER_XT_MATCH_ADDRTYPE=m
> CONFIG_NETFILTER_XT_MATCH_CLUSTER=m
> CONFIG_NETFILTER_XT_MATCH_COMMENT=m
> CONFIG_NETFILTER_XT_MATCH_CONNBYTES=m
> CONFIG_NETFILTER_XT_MATCH_CONNLIMIT=m
> CONFIG_NETFILTER_XT_MATCH_CONNMARK=m
> CONFIG_NETFILTER_XT_MATCH_CONNTRACK=m
> CONFIG_NETFILTER_XT_MATCH_CPU=m
> CONFIG_NETFILTER_XT_MATCH_DCCP=m
> CONFIG_NETFILTER_XT_MATCH_DEVGROUP=m
> CONFIG_NETFILTER_XT_MATCH_DSCP=m
> CONFIG_NETFILTER_XT_MATCH_ESP=m
> CONFIG_NETFILTER_XT_MATCH_HASHLIMIT=m
> CONFIG_NETFILTER_XT_MATCH_HELPER=m
> CONFIG_NETFILTER_XT_MATCH_HL=m
> CONFIG_NETFILTER_XT_MATCH_IPRANGE=m
> CONFIG_NETFILTER_XT_MATCH_LENGTH=m
> CONFIG_NETFILTER_XT_MATCH_LIMIT=m
> CONFIG_NETFILTER_XT_MATCH_MAC=m
> CONFIG_NETFILTER_XT_MATCH_MARK=m
> CONFIG_NETFILTER_XT_MATCH_MULTIPORT=m
> CONFIG_NETFILTER_XT_MATCH_OSF=m
> CONFIG_NETFILTER_XT_MATCH_OWNER=m
> CONFIG_NETFILTER_XT_MATCH_POLICY=m
> CONFIG_NETFILTER_XT_MATCH_PHYSDEV=m
> CONFIG_NETFILTER_XT_MATCH_PKTTYPE=m
> CONFIG_NETFILTER_XT_MATCH_QUOTA=m
> CONFIG_NETFILTER_XT_MATCH_RATEEST=m
> CONFIG_NETFILTER_XT_MATCH_REALM=m
> CONFIG_NETFILTER_XT_MATCH_RECENT=m
> CONFIG_NETFILTER_XT_MATCH_SCTP=m
> CONFIG_NETFILTER_XT_MATCH_SOCKET=m
> CONFIG_NETFILTER_XT_MATCH_STATE=m
> CONFIG_NETFILTER_XT_MATCH_STATISTIC=m
> CONFIG_NETFILTER_XT_MATCH_STRING=m
> CONFIG_NETFILTER_XT_MATCH_TCPMSS=m
> CONFIG_NETFILTER_XT_MATCH_TIME=m
> CONFIG_NETFILTER_XT_MATCH_U32=m
>
> CONFIG_NF_DEFRAG_IPV4=m
> CONFIG_NF_CONNTRACK_IPV4=m
> CONFIG_NF_CONNTRACK_PROC_COMPAT=y
> CONFIG_IP_NF_QUEUE=m
> CONFIG_IP_NF_IPTABLES=m
> CONFIG_IP_NF_MATCH_AH=m
> CONFIG_IP_NF_MATCH_ECN=m
> CONFIG_IP_NF_MATCH_TTL=m
> CONFIG_IP_NF_FILTER=m
> CONFIG_IP_NF_TARGET_REJECT=m
> CONFIG_IP_NF_TARGET_LOG=m
> CONFIG_IP_NF_TARGET_ULOG=m
> CONFIG_NF_NAT=m
> CONFIG_NF_NAT_NEEDED=y
> CONFIG_IP_NF_TARGET_MASQUERADE=m
> CONFIG_IP_NF_TARGET_NETMAP=m
> CONFIG_IP_NF_TARGET_REDIRECT=m
> CONFIG_NF_NAT_SNMP_BASIC=m
> CONFIG_NF_NAT_PROTO_DCCP=m
> CONFIG_NF_NAT_PROTO_GRE=m
> CONFIG_NF_NAT_PROTO_UDPLITE=m
> CONFIG_NF_NAT_PROTO_SCTP=m
> CONFIG_NF_NAT_FTP=m
> CONFIG_NF_NAT_IRC=m
> CONFIG_NF_NAT_TFTP=m
> CONFIG_NF_NAT_AMANDA=m
> CONFIG_NF_NAT_PPTP=m
> CONFIG_NF_NAT_H323=m
> CONFIG_NF_NAT_SIP=m
> CONFIG_IP_NF_MANGLE=m
> CONFIG_IP_NF_TARGET_CLUSTERIP=m
> CONFIG_IP_NF_TARGET_ECN=m
> CONFIG_IP_NF_TARGET_TTL=m
> CONFIG_IP_NF_RAW=m
> CONFIG_IP_NF_ARPTABLES=m
> CONFIG_IP_NF_ARPFILTER=m
> CONFIG_IP_NF_ARP_MANGLE=m
> CONFIG_IP_SCTP=m
> CONFIG_SCTP_HMAC_MD5=y
> CONFIG_STP=y
> CONFIG_BRIDGE=y
> CONFIG_BRIDGE_IGMP_SNOOPING=y
> CONFIG_VLAN_8021Q=y
> CONFIG_LLC=y
> CONFIG_NET_SCHED=y
>
> CONFIG_NET_SCH_CBQ=m
> CONFIG_NET_SCH_HTB=m
> CONFIG_NET_SCH_HFSC=m
> CONFIG_NET_SCH_PRIO=m
> CONFIG_NET_SCH_RED=m
> CONFIG_NET_SCH_SFB=m
> CONFIG_NET_SCH_SFQ=m
> CONFIG_NET_SCH_TEQL=m
> CONFIG_NET_SCH_TBF=m
> CONFIG_NET_SCH_GRED=m
> CONFIG_NET_SCH_DSMARK=m
> CONFIG_NET_SCH_NETEM=m
> CONFIG_NET_SCH_MQPRIO=m
> CONFIG_NET_SCH_CHOKE=m
> CONFIG_NET_SCH_INGRESS=m
>
> CONFIG_NET_CLS=y
> CONFIG_NET_CLS_BASIC=m
> CONFIG_NET_CLS_TCINDEX=m
> CONFIG_NET_CLS_ROUTE4=m
> CONFIG_NET_CLS_FW=m
> CONFIG_NET_CLS_U32=m
> CONFIG_NET_CLS_FLOW=m
> CONFIG_NET_EMATCH=y
> CONFIG_NET_EMATCH_STACK=32
> CONFIG_NET_EMATCH_CMP=m
> CONFIG_NET_EMATCH_NBYTE=m
> CONFIG_NET_EMATCH_U32=m
> CONFIG_NET_EMATCH_META=m
> CONFIG_NET_EMATCH_TEXT=m
> CONFIG_NET_CLS_ACT=y
> CONFIG_NET_ACT_POLICE=m
> CONFIG_NET_ACT_GACT=m
> CONFIG_GACT_PROB=y
> CONFIG_NET_ACT_MIRRED=m
> CONFIG_NET_ACT_IPT=m
> CONFIG_NET_ACT_NAT=m
> CONFIG_NET_ACT_PEDIT=m
> CONFIG_NET_ACT_SIMP=m
> CONFIG_NET_ACT_SKBEDIT=m
> CONFIG_NET_ACT_CSUM=m
> CONFIG_NET_SCH_FIFO=y
> CONFIG_RPS=y
> CONFIG_RFS_ACCEL=y
> CONFIG_XPS=y
>
>
>
> CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
> CONFIG_DEVTMPFS=y
> CONFIG_STANDALONE=y
> CONFIG_PREVENT_FIRMWARE_BUILD=y
> CONFIG_FW_LOADER=y
> CONFIG_FIRMWARE_IN_KERNEL=y
> CONFIG_EXTRA_FIRMWARE=""
> CONFIG_ARCH_NO_SYSDEV_OPS=y
> CONFIG_PNP=y
> CONFIG_PNP_DEBUG_MESSAGES=y
>
> CONFIG_PNPACPI=y
> CONFIG_BLK_DEV=y
> CONFIG_BLK_DEV_LOOP=y
>
> CONFIG_HAVE_IDE=y
>
> CONFIG_SCSI_MOD=y
> CONFIG_RAID_ATTRS=m
> CONFIG_SCSI=y
> CONFIG_SCSI_DMA=y
> CONFIG_SCSI_PROC_FS=y
>
> CONFIG_BLK_DEV_SD=y
> CONFIG_BLK_DEV_SR=y
> CONFIG_CHR_DEV_SG=y
> CONFIG_SCSI_WAIT_SCAN=m
>
> CONFIG_SCSI_SAS_ATTRS=m
> CONFIG_SCSI_LOWLEVEL=y
> CONFIG_MEGARAID_SAS=m
> CONFIG_ATA=y
> CONFIG_ATA_VERBOSE_ERROR=y
> CONFIG_ATA_ACPI=y
> CONFIG_SATA_PMP=y
>
> CONFIG_SATA_AHCI=m
> CONFIG_SATA_SIL24=m
> CONFIG_ATA_SFF=y
>
> CONFIG_ATA_BMDMA=y
>
> CONFIG_SATA_MV=m
> CONFIG_SATA_NV=m
> CONFIG_SATA_PROMISE=m
> CONFIG_SATA_SIL=m
>
>
>
> CONFIG_MD=y
> CONFIG_BLK_DEV_MD=y
> CONFIG_MD_LINEAR=y
> CONFIG_MD_RAID0=y
> CONFIG_MD_RAID1=y
> CONFIG_MD_RAID10=y
> CONFIG_MD_RAID456=y
> CONFIG_BLK_DEV_DM=y
>
> CONFIG_NETDEVICES=y
> CONFIG_DUMMY=m
> CONFIG_BONDING=m
> CONFIG_TUN=y
> CONFIG_MII=m
> CONFIG_NET_ETHERNET=y
> CONFIG_NET_PCI=y
> CONFIG_FORCEDETH=m
> CONFIG_NETDEV_1000=y
> CONFIG_R8169=m
>
>
>
> CONFIG_PPP=m
> CONFIG_PPP_MULTILINK=y
> CONFIG_PPP_FILTER=y
> CONFIG_PPP_ASYNC=m
> CONFIG_PPP_SYNC_TTY=m
> CONFIG_PPP_DEFLATE=m
> CONFIG_PPP_BSDCOMP=m
> CONFIG_PPP_MPPE=m
> CONFIG_PPPOE=m
> CONFIG_SLHC=m
> CONFIG_NETCONSOLE=m
> CONFIG_NETCONSOLE_DYNAMIC=y
> CONFIG_NETPOLL=y
> CONFIG_NET_POLL_CONTROLLER=y
>
> CONFIG_INPUT=y
>
> CONFIG_INPUT_MOUSEDEV=y
> CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
> CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
>
> CONFIG_INPUT_KEYBOARD=y
> CONFIG_KEYBOARD_ATKBD=y
>
> CONFIG_SERIO=y
> CONFIG_SERIO_I8042=y
> CONFIG_SERIO_LIBPS2=y
>
> CONFIG_VT=y
> CONFIG_CONSOLE_TRANSLATIONS=y
> CONFIG_VT_CONSOLE=y
> CONFIG_HW_CONSOLE=y
> CONFIG_UNIX98_PTYS=y
> CONFIG_LEGACY_PTYS=y
> CONFIG_LEGACY_PTY_COUNT=256
>
> CONFIG_SERIAL_8250=y
> CONFIG_SERIAL_8250_CONSOLE=y
> CONFIG_FIX_EARLYCON_MEM=y
> CONFIG_SERIAL_8250_PCI=y
> CONFIG_SERIAL_8250_PNP=y
> CONFIG_SERIAL_8250_NR_UARTS=4
> CONFIG_SERIAL_8250_RUNTIME_UARTS=4
>
> CONFIG_SERIAL_CORE=y
> CONFIG_SERIAL_CORE_CONSOLE=y
> CONFIG_RTC=y
> CONFIG_HPET=y
> CONFIG_HPET_MMAP=y
> CONFIG_DEVPORT=y
> CONFIG_I2C=y
> CONFIG_I2C_BOARDINFO=y
> CONFIG_I2C_COMPAT=y
> CONFIG_I2C_CHARDEV=y
> CONFIG_I2C_HELPER_AUTO=y
> CONFIG_I2C_ALGOBIT=y
>
>
> CONFIG_I2C_AMD756=m
> CONFIG_I2C_PIIX4=m
> CONFIG_I2C_NFORCE2=m
> CONFIG_I2C_VIA=m
> CONFIG_I2C_VIAPRO=m
>
> CONFIG_I2C_SCMI=m
>
>
>
>
>
> CONFIG_ARCH_WANT_OPTIONAL_GPIOLIB=y
> CONFIG_HWMON=y
> CONFIG_HWMON_VID=m
>
> CONFIG_SENSORS_K8TEMP=m
> CONFIG_SENSORS_K10TEMP=m
> CONFIG_SENSORS_W83781D=m
> CONFIG_SENSORS_W83791D=m
> CONFIG_SENSORS_W83792D=m
> CONFIG_SENSORS_W83793=m
> CONFIG_SENSORS_W83795=m
> CONFIG_SENSORS_W83795_FANCTRL=y
> CONFIG_SENSORS_W83L785TS=m
> CONFIG_SENSORS_W83L786NG=m
> CONFIG_SENSORS_W83627HF=m
> CONFIG_SENSORS_W83627EHF=m
>
> CONFIG_SENSORS_ATK0110=m
> CONFIG_THERMAL=y
> CONFIG_THERMAL_HWMON=y
> CONFIG_SSB_POSSIBLE=y
>
>
> CONFIG_VGA_ARB=y
> CONFIG_VGA_ARB_MAX_GPUS=16
> CONFIG_VGASTATE=y
> CONFIG_FB=y
> CONFIG_FB_DDC=y
> CONFIG_FB_CFB_FILLRECT=y
> CONFIG_FB_CFB_COPYAREA=y
> CONFIG_FB_CFB_IMAGEBLIT=y
> CONFIG_FB_MODE_HELPERS=y
>
> CONFIG_FB_NVIDIA=y
> CONFIG_FB_NVIDIA_I2C=y
> CONFIG_FB_RADEON=y
> CONFIG_FB_RADEON_I2C=y
>
>
> CONFIG_VGA_CONSOLE=y
> CONFIG_DUMMY_CONSOLE=y
> CONFIG_FRAMEBUFFER_CONSOLE=y
> CONFIG_FONT_8x8=y
> CONFIG_FONT_8x16=y
> CONFIG_LOGO=y
> CONFIG_LOGO_LINUX_MONO=y
> CONFIG_LOGO_LINUX_VGA16=y
> CONFIG_LOGO_LINUX_CLUT224=y
> CONFIG_HID_SUPPORT=y
> CONFIG_HID=y
> CONFIG_HIDRAW=y
>
> CONFIG_USB_HID=m
> CONFIG_USB_HIDDEV=y
>
> CONFIG_HID_A4TECH=m
> CONFIG_HID_APPLE=m
> CONFIG_HID_BELKIN=m
> CONFIG_HID_CHERRY=m
> CONFIG_HID_CHICONY=m
> CONFIG_HID_CYPRESS=m
> CONFIG_HID_EZKEY=m
> CONFIG_HID_KYE=m
> CONFIG_HID_KENSINGTON=m
> CONFIG_HID_LOGITECH=m
> CONFIG_HID_MICROSOFT=m
> CONFIG_HID_MONTEREY=m
> CONFIG_USB_SUPPORT=y
> CONFIG_USB_ARCH_HAS_HCD=y
> CONFIG_USB_ARCH_HAS_OHCI=y
> CONFIG_USB_ARCH_HAS_EHCI=y
> CONFIG_USB=m
>
>
> CONFIG_USB_XHCI_HCD=m
> CONFIG_USB_EHCI_HCD=m
> CONFIG_USB_OHCI_HCD=m
> CONFIG_USB_OHCI_LITTLE_ENDIAN=y
> CONFIG_USB_UHCI_HCD=m
>
>
> CONFIG_USB_ACM=m
>
>
> CONFIG_USB_STORAGE=m
> CONFIG_USB_LIBUSUAL=y
>
>
> CONFIG_USB_SERIAL=m
> CONFIG_USB_SERIAL_GENERIC=y
> CONFIG_USB_SERIAL_FTDI_SIO=m
> CONFIG_USB_SERIAL_IPAQ=m
> CONFIG_USB_SERIAL_PL2303=m
>
>
>
> CONFIG_FIRMWARE_MEMMAP=y
> CONFIG_DMIID=y
>
> CONFIG_EXT2_FS=y
> CONFIG_EXT2_FS_XATTR=y
> CONFIG_EXT3_FS=y
> CONFIG_EXT3_FS_XATTR=y
> CONFIG_EXT4_FS=y
> CONFIG_EXT4_FS_XATTR=y
> CONFIG_JBD=y
> CONFIG_JBD2=y
> CONFIG_FS_MBCACHE=y
> CONFIG_EXPORTFS=y
> CONFIG_FILE_LOCKING=y
> CONFIG_FSNOTIFY=y
> CONFIG_DNOTIFY=y
> CONFIG_INOTIFY_USER=y
> CONFIG_FANOTIFY=y
> CONFIG_FUSE_FS=m
> CONFIG_CUSE=m
>
> CONFIG_FSCACHE=m
> CONFIG_FSCACHE_DEBUG=y
> CONFIG_CACHEFILES=m
>
> CONFIG_ISO9660_FS=m
> CONFIG_JOLIET=y
> CONFIG_ZISOFS=y
> CONFIG_UDF_FS=m
> CONFIG_UDF_NLS=y
>
> CONFIG_FAT_FS=m
> CONFIG_MSDOS_FS=m
> CONFIG_VFAT_FS=m
> CONFIG_FAT_DEFAULT_CODEPAGE=437
> CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
> CONFIG_NTFS_FS=m
>
> CONFIG_PROC_FS=y
> CONFIG_PROC_SYSCTL=y
> CONFIG_PROC_PAGE_MONITOR=y
> CONFIG_SYSFS=y
> CONFIG_TMPFS=y
> CONFIG_HUGETLBFS=y
> CONFIG_HUGETLB_PAGE=y
> CONFIG_CONFIGFS_FS=m
> CONFIG_NETWORK_FILESYSTEMS=y
> CONFIG_NFS_FS=y
> CONFIG_NFS_V3=y
> CONFIG_NFSD=y
> CONFIG_NFSD_DEPRECATED=y
> CONFIG_NFSD_V3=y
> CONFIG_LOCKD=y
> CONFIG_LOCKD_V4=y
> CONFIG_NFS_COMMON=y
> CONFIG_SUNRPC=y
>
> CONFIG_PARTITION_ADVANCED=y
> CONFIG_MSDOS_PARTITION=y
> CONFIG_EFI_PARTITION=y
> CONFIG_NLS=y
> CONFIG_NLS_DEFAULT="iso8859-1"
> CONFIG_NLS_CODEPAGE_437=y
> CONFIG_NLS_ISO8859_1=y
> CONFIG_DLM=m
>
> CONFIG_TRACE_IRQFLAGS_SUPPORT=y
> CONFIG_PRINTK_TIME=y
> CONFIG_DEFAULT_MESSAGE_LOGLEVEL=4
> CONFIG_ENABLE_WARN_DEPRECATED=y
> CONFIG_ENABLE_MUST_CHECK=y
> CONFIG_FRAME_WARN=2048
> CONFIG_MAGIC_SYSRQ=y
> CONFIG_DEBUG_FS=y
> CONFIG_DEBUG_KERNEL=y
> CONFIG_DETECT_HUNG_TASK=y
> CONFIG_BOOTPARAM_HUNG_TASK_PANIC_VALUE=0
> CONFIG_TIMER_STATS=y
> CONFIG_DEBUG_BUGVERBOSE=y
> CONFIG_DEBUG_MEMORY_INIT=y
> CONFIG_ARCH_WANT_FRAME_POINTERS=y
> CONFIG_SYSCTL_SYSCALL_CHECK=y
> CONFIG_USER_STACKTRACE_SUPPORT=y
> CONFIG_HAVE_FUNCTION_TRACER=y
> CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
> CONFIG_HAVE_FUNCTION_GRAPH_FP_TEST=y
> CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST=y
> CONFIG_HAVE_DYNAMIC_FTRACE=y
> CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
> CONFIG_HAVE_SYSCALL_TRACEPOINTS=y
> CONFIG_HAVE_C_RECORDMCOUNT=y
> CONFIG_TRACING_SUPPORT=y
> CONFIG_HAVE_ARCH_KGDB=y
> CONFIG_HAVE_ARCH_KMEMCHECK=y
> CONFIG_X86_VERBOSE_BOOTUP=y
> CONFIG_EARLY_PRINTK=y
> CONFIG_HAVE_MMIOTRACE_SUPPORT=y
> CONFIG_IO_DELAY_TYPE_0X80=0
> CONFIG_IO_DELAY_TYPE_0XED=1
> CONFIG_IO_DELAY_TYPE_UDELAY=2
> CONFIG_IO_DELAY_TYPE_NONE=3
> CONFIG_IO_DELAY_0X80=y
> CONFIG_DEFAULT_IO_DELAY_TYPE=0
>
> CONFIG_DEFAULT_SECURITY_DAC=y
> CONFIG_DEFAULT_SECURITY=""
> CONFIG_XOR_BLOCKS=y
> CONFIG_ASYNC_CORE=y
> CONFIG_ASYNC_MEMCPY=y
> CONFIG_ASYNC_XOR=y
> CONFIG_ASYNC_PQ=y
> CONFIG_ASYNC_RAID6_RECOV=y
> CONFIG_CRYPTO=y
>
> CONFIG_CRYPTO_ALGAPI=m
> CONFIG_CRYPTO_ALGAPI2=m
> CONFIG_CRYPTO_AEAD=m
> CONFIG_CRYPTO_AEAD2=m
> CONFIG_CRYPTO_BLKCIPHER=m
> CONFIG_CRYPTO_BLKCIPHER2=m
> CONFIG_CRYPTO_HASH=m
> CONFIG_CRYPTO_HASH2=m
> CONFIG_CRYPTO_RNG2=m
> CONFIG_CRYPTO_PCOMP2=m
> CONFIG_CRYPTO_MANAGER=m
> CONFIG_CRYPTO_MANAGER2=m
> CONFIG_CRYPTO_MANAGER_DISABLE_TESTS=y
> CONFIG_CRYPTO_WORKQUEUE=m
> CONFIG_CRYPTO_AUTHENC=m
>
>
> CONFIG_CRYPTO_CBC=m
> CONFIG_CRYPTO_ECB=m
>
> CONFIG_CRYPTO_HMAC=m
>
> CONFIG_CRYPTO_CRC32C=m
> CONFIG_CRYPTO_MD5=m
> CONFIG_CRYPTO_SHA1=m
>
> CONFIG_CRYPTO_ARC4=m
> CONFIG_CRYPTO_DES=m
>
> CONFIG_CRYPTO_DEFLATE=m
>
> CONFIG_HAVE_KVM=y
> CONFIG_HAVE_KVM_IRQCHIP=y
> CONFIG_HAVE_KVM_EVENTFD=y
> CONFIG_KVM_APIC_ARCHITECTURE=y
> CONFIG_KVM_MMIO=y
> CONFIG_KVM_ASYNC_PF=y
> CONFIG_VIRTUALIZATION=y
> CONFIG_KVM=m
> CONFIG_KVM_INTEL=m
> CONFIG_KVM_AMD=m
> CONFIG_VHOST_NET=m
>
> CONFIG_RAID6_PQ=y
> CONFIG_BITREVERSE=y
> CONFIG_GENERIC_FIND_FIRST_BIT=y
> CONFIG_GENERIC_FIND_NEXT_BIT=y
> CONFIG_GENERIC_FIND_LAST_BIT=y
> CONFIG_CRC_CCITT=m
> CONFIG_CRC16=y
> CONFIG_CRC_T10DIF=y
> CONFIG_CRC_ITU_T=y
> CONFIG_CRC32=y
> CONFIG_LIBCRC32C=m
> CONFIG_ZLIB_INFLATE=y
> CONFIG_ZLIB_DEFLATE=m
> CONFIG_LZO_DECOMPRESS=y
> CONFIG_XZ_DEC=y
> CONFIG_XZ_DEC_X86=y
> CONFIG_XZ_DEC_POWERPC=y
> CONFIG_XZ_DEC_IA64=y
> CONFIG_XZ_DEC_ARM=y
> CONFIG_XZ_DEC_ARMTHUMB=y
> CONFIG_XZ_DEC_SPARC=y
> CONFIG_XZ_DEC_BCJ=y
> CONFIG_DECOMPRESS_GZIP=y
> CONFIG_DECOMPRESS_BZIP2=y
> CONFIG_DECOMPRESS_LZMA=y
> CONFIG_DECOMPRESS_XZ=y
> CONFIG_DECOMPRESS_LZO=y
> CONFIG_TEXTSEARCH=y
> CONFIG_TEXTSEARCH_KMP=m
> CONFIG_TEXTSEARCH_BM=m
> CONFIG_TEXTSEARCH_FSM=m
> CONFIG_HAS_IOMEM=y
> CONFIG_HAS_IOPORT=y
> CONFIG_HAS_DMA=y
> CONFIG_CPU_RMAP=y
> CONFIG_NLATTR=y
>
> 00:00.0 Host bridge: Advanced Micro Devices [AMD] RS880 Host Bridge
> 00:01.0 PCI bridge: ASRock Incorporation Device 9602
> 00:02.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI
> bridge (ext gfx port 0)
> 00:03.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI
> bridge (ext gfx port 1)
> 00:04.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI
> bridge (PCIE port 0)
> 00:09.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI
> bridge (PCIE port 4)
> 00:0a.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI
> bridge (PCIE port 5)
> 00:11.0 SATA controller: ATI Technologies Inc SB700/SB800 SATA
> Controller [AHCI mode] (rev 40)
> 00:12.0 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI0
> Controller
> 00:12.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI Controller
> 00:13.0 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI0
> Controller
> 00:13.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI Controller
> 00:14.0 SMBus: ATI Technologies Inc SBx00 SMBus Controller (rev 41)
> 00:14.2 Audio device: ATI Technologies Inc SBx00 Azalia (Intel HDA) (rev 40)
> 00:14.3 ISA bridge: ATI Technologies Inc SB700/SB800 LPC host
> controller (rev 40)
> 00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge (rev 40)
> 00:14.5 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI2
> Controller
> 00:16.0 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI0
> Controller
> 00:16.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI Controller
> 00:18.0 Host bridge: Advanced Micro Devices [AMD] Family 10h
> Processor HyperTransport Configuration
> 00:18.1 Host bridge: Advanced Micro Devices [AMD] Family 10h
> Processor Address Map
> 00:18.2 Host bridge: Advanced Micro Devices [AMD] Family 10h
> Processor DRAM Controller
> 00:18.3 Host bridge: Advanced Micro Devices [AMD] Family 10h
> Processor Miscellaneous Control
> 00:18.4 Host bridge: Advanced Micro Devices [AMD] Family 10h
> Processor Link Control
> 01:05.0 VGA compatible controller: ATI Technologies Inc RS880
> [Radeon HD 4250]
> 02:00.0 USB Controller: NEC Corporation uPD720200 USB 3.0 Host
> Controller (rev 03)
> 03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd.
> RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 06)
> 04:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS
> 9240 (rev 02)
> 05:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS
> 9240 (rev 02)
> 07:00.0 SCSI storage controller: Marvell Technology Group Ltd.
> 88SX7042 PCI-e 4-port SATA-II (rev 02)
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/

--
Regards/Gruss,
Boris.

2011-05-31 09:26:21

by Brad Campbell

[permalink] [raw]
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39

On 31/05/11 13:47, Borislav Petkov wrote:
> Looks like a KSM issue. Disabling CONFIG_KSM should at least stop your
> machine from oopsing.
>
> Adding linux-mm.
>

I initially thought that, so the second panic was produced with KSM
disabled from boot.

echo 0 > /sys/kernel/mm/ksm/run

If you still think that compiling ksm out of the kernel will prevent it
then I'm willing to give it a go.

It's a production server, so I can only really bounce it around after
about 9PM - GMT+8.

Regards,
Brad

2011-05-31 10:37:54

by Borislav Petkov

[permalink] [raw]
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39

On Tue, May 31, 2011 at 05:26:10PM +0800, Brad Campbell wrote:
> On 31/05/11 13:47, Borislav Petkov wrote:
> >Looks like a KSM issue. Disabling CONFIG_KSM should at least stop your
> >machine from oopsing.
> >
> >Adding linux-mm.
> >
>
> I initially thought that, so the second panic was produced with KSM
> disabled from boot.
>
> echo 0 > /sys/kernel/mm/ksm/run
>
> If you still think that compiling ksm out of the kernel will prevent
> it then I'm willing to give it a go.

Ok, from looking at the code, when KSM inits, it starts the ksm kernel
thread and it looks like your oops comes from the function that is run
in the kernel thread - ksm_scan_thread.

So even if you disable it from sysfs, it runs at least once.

Let's add some more people to Cc and see what happens :).

--
Regards/Gruss,
Boris.

2011-05-31 14:24:51

by Brad Campbell

[permalink] [raw]
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39

On 31/05/11 18:38, Borislav Petkov wrote:
> On Tue, May 31, 2011 at 05:26:10PM +0800, Brad Campbell wrote:
>> On 31/05/11 13:47, Borislav Petkov wrote:
>>> Looks like a KSM issue. Disabling CONFIG_KSM should at least stop your
>>> machine from oopsing.
>>>
>>> Adding linux-mm.
>>>
>>
>> I initially thought that, so the second panic was produced with KSM
>> disabled from boot.
>>
>> echo 0> /sys/kernel/mm/ksm/run
>>
>> If you still think that compiling ksm out of the kernel will prevent
>> it then I'm willing to give it a go.
>
> Ok, from looking at the code, when KSM inits, it starts the ksm kernel
> thread and it looks like your oops comes from the function that is run
> in the kernel thread - ksm_scan_thread.
>
> So even if you disable it from sysfs, it runs at least once.
>

Just to confirm, I recompiled 2.6.38.7 without KSM enabled and I've been
unable to reproduce the bug, so it looks like you were on the money.

I've moved back to 2.6.38.7 as 2.6.39 has a painful SCSI bug that panics
about 75% of boots, and the reboot cycle required to get luck my way
into a working kernel is just too much hassle.

It would appear that XP zero's its memory space on bootup, so there
would be lots of pages to merge with a couple of relatively freshly
booted XP machines running.

Regards,
Brad.

2011-05-31 22:32:00

by Hugh Dickins

[permalink] [raw]
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39

On Tue, 31 May 2011, Brad Campbell wrote:
> On 31/05/11 18:38, Borislav Petkov wrote:
> > On Tue, May 31, 2011 at 05:26:10PM +0800, Brad Campbell wrote:
> > > On 31/05/11 13:47, Borislav Petkov wrote:
> > > > Looks like a KSM issue. Disabling CONFIG_KSM should at least stop your
> > > > machine from oopsing.
> > > >
> > > > Adding linux-mm.
> > > >
> > >
> > > I initially thought that, so the second panic was produced with KSM
> > > disabled from boot.
> > >
> > > echo 0> /sys/kernel/mm/ksm/run
> > >
> > > If you still think that compiling ksm out of the kernel will prevent
> > > it then I'm willing to give it a go.
> >
> > Ok, from looking at the code, when KSM inits, it starts the ksm kernel
> > thread and it looks like your oops comes from the function that is run
> > in the kernel thread - ksm_scan_thread.
> >
> > So even if you disable it from sysfs, it runs at least once.
> >
>
> Just to confirm, I recompiled 2.6.38.7 without KSM enabled and I've been
> unable to reproduce the bug, so it looks like you were on the money.
>
> I've moved back to 2.6.38.7 as 2.6.39 has a painful SCSI bug that panics
> about 75% of boots, and the reboot cycle required to get luck my way into a
> working kernel is just too much hassle.
>
> It would appear that XP zero's its memory space on bootup, so there would be
> lots of pages to merge with a couple of relatively freshly booted XP
> machines running.

Thanks for the Cc, Borislav.

Brad, my suspicion is that in each case the top 16 bits of RDX have been
mysteriously corrupted from ffff to 0000, causing the general protection
faults. I don't understand what that has to do with KSM.

But it's only a suspicion, because I can't make sense of the "Code:"
lines in your traces, they have more than the expected 64 bytes, and
only one of them has a ">" (with no "<") to mark faulting instruction.

I did try compiling the 2.6.39 kernel from your config, but of course
we have different compilers, so although I got close, it wasn't exact.

Would you mind mailing me privately (it's about 73MB) the "objdump -trd"
output for your original vmlinux (with KSM on)? (Those -trd options are
the ones I'm used to typing, I bet not they're not all relevant.)

Of course, it's only a tiny fraction of that output that I need,
might be better to cut it down to remove_rmap_item_from_tree and
dup_fd and ksm_scan_thread, if you have the time to do so.

Thanks,
Hugh

2011-06-01 00:18:18

by Brad Campbell

[permalink] [raw]
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39

On 01/06/11 06:31, Hugh Dickins wrote:
>
> Brad, my suspicion is that in each case the top 16 bits of RDX have been
> mysteriously corrupted from ffff to 0000, causing the general protection
> faults. I don't understand what that has to do with KSM.

No, nor do I. The panic I reproduced with KSM off was in a completely
unrelated code path. To be honest I would not be surprised if it turns
out I have dodgy RAM, although it has passed multiple memtests and I've
tried clocking it down. Just a gut feeling.

> But it's only a suspicion, because I can't make sense of the "Code:"
> lines in your traces, they have more than the expected 64 bytes, and
> only one of them has a ">" (with no"<") to mark faulting instruction.

Yeah, with hindsight I must have removed them when I re-formatted the
code from the oops. Each byte was one line in the syslog so there was a
lot of deleting to get it to a postable format.

> I did try compiling the 2.6.39 kernel from your config, but of course
> we have different compilers, so although I got close, it wasn't exact.
>
> Would you mind mailing me privately (it's about 73MB) the "objdump -trd"
> output for your original vmlinux (with KSM on)? (Those -trd options are
> the ones I'm used to typing, I bet not they're not all relevant.)
>
> Of course, it's only a tiny fraction of that output that I need,
> might be better to cut it down to remove_rmap_item_from_tree and
> dup_fd and ksm_scan_thread, if you have the time to do so.

Ok, so since my initial posting I've figured out how to get a clean oops
out of netconsole, so tonight (after 9PM GMT+8) I'll reproduce the oops
a couple of times. What about I upload the oops, plus the vmlinux, plus
.config and System.map to a server with a fat pipe and give you a link
to it?

At least I can reproduce it quickly and easily.

2011-06-01 00:37:40

by Brad Campbell

[permalink] [raw]
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39

On 01/06/11 06:31, Hugh Dickins wrote:
> Brad, my suspicion is that in each case the top 16 bits of RDX have been
> mysteriously corrupted from ffff to 0000, causing the general protection
> faults. I don't understand what that has to do with KSM.
>
> But it's only a suspicion, because I can't make sense of the "Code:"
> lines in your traces, they have more than the expected 64 bytes, and
> only one of them has a ">" (with no"<") to mark faulting instruction.
>
> I did try compiling the 2.6.39 kernel from your config, but of course
> we have different compilers, so although I got close, it wasn't exact.
>
> Would you mind mailing me privately (it's about 73MB) the "objdump -trd"
> output for your original vmlinux (with KSM on)? (Those -trd options are
> the ones I'm used to typing, I bet not they're not all relevant.)
>
> Of course, it's only a tiny fraction of that output that I need,
> might be better to cut it down to remove_rmap_item_from_tree and
> dup_fd and ksm_scan_thread, if you have the time to do so.

Would you believe about 20 seconds after I pressed send the kernel oopsed.

http://www.fnarfbargle.com/private/003_kernel_oops/

oops reproduced here, but an un-munged version is in that directory
alongside the kernel.

[36542.880228] general protection fault: 0000 [#1] SMP
[36542.880271] last sysfs file:
/sys/devices/pci0000:00/0000:00:18.3/temp1_input
[36542.880290] CPU 4
[36542.880301] Modules linked in: xt_iprange xt_DSCP xt_length
xt_CLASSIFY sch_sfq xt_CHECKSUM ipt_REJECT ipt_MASQUER
ADE ipt_REDIRECT xt_recent xt_state iptable_filter iptable_nat nf_nat
nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 x
t_TCPMSS xt_tcpmss xt_tcpudp iptable_mangle ip_tables x_tables pppoe
pppox ppp_generic slhc cls_u32 sch_htb deflate z
lib_deflate des_generic cbc ecb crypto_blkcipher sha1_generic md5 hmac
crypto_hash cryptomgr aead crypto_algapi af_ke
y fuse hwmon_vid netconsole configfs vhost_net powernow_k8 mperf kvm_amd
kvm pl2303 usbserial xhci_hcd k10temp i2c_pi
ix4 ahci usb_storage usb_libusual ohci_hcd ehci_hcd r8169 libahci
usbcore mii sata_mv megaraid_sas [last unloaded: sc
si_wait_scan]
[36542.880842]
[36542.880858] Pid: 13346, comm: bash Not tainted 2.6.38.7 #29 To Be
Filled By O.E.M. To Be Filled By O.E.M./880G Ext
reme3
[36542.880911] RIP: 0010:[<ffffffff810cf0de>] [<ffffffff810cf0de>]
do_vfs_ioctl+0x5e/0x510
[36542.880948] RSP: 0018:ffff8802d25a1ec8 EFLAGS: 00010206
[36542.880965] RAX: fffffffffffffff7 RBX: 000088040eb12840 RCX:
00007fff4fe4a4c0
[36542.880984] RDX: 0000000000005413 RSI: 0000000000005413 RDI:
00000000000000ff
[36542.881002] RBP: 00000000000000ff R08: 00007fff4fe4a400 R09:
0000000000000000
[36542.881020] R10: 00007fff4fe4a380 R11: 0000000000000246 R12:
00007fff4fe4a4c0
[36542.881038] R13: 00007fff4fe4a4c0 R14: 0000000000000000 R15:
0000000000000001
[36542.881058] FS: 00007f65f725b700(0000) GS:ffff8800dbd00000(0000)
knlGS:0000000000000000
[36542.881081] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[36542.881098] CR2: 0000000001f01008 CR3: 00000002d25c3000 CR4:
00000000000006e0
[36542.881116] DR0: 00000000000000a0 DR1: 0000000000000000 DR2:
0000000000000003
[36542.881133] DR3: 00000000000000b0 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[36542.881152] Process bash (pid: 13346, threadinfo ffff8802d25a0000,
task ffff88041df88000)
[36542.881172] Stack:
[36542.881183] 0000000000000000 ffff88041df88218 0000000000100000
0000000000000001
[36542.881225] 0000000000000002 00007fff4fe4a2c0 00007fff4fe4a220
0000000000000002
[36542.881268] 0000000000000000 ffffffff81046d6a 000088040eb12840
00000000000000ff
[36542.881312] Call Trace:
[36542.881333] [<ffffffff81046d6a>] ? sys_rt_sigaction+0x8a/0xc0
[36542.881351] [<ffffffff810cf5d9>] ? sys_ioctl+0x49/0x80
[36542.881373] [<ffffffff810023fb>] ? system_call_fastpath+0x16/0x1b
[36542.881389] Code: 76 7b 81 fa 77 58 04 c0 0f 84 77 01 00 00 0f 1f 80
00 00 00 00 0f 87 a2 00 00 00 81 fa 60 54 00 00 0f 1f 40 00 0f 84 ba 01
00 00 <48> 8b 43 18 48 8b 50 30 0f b7 02 25 00 f0 00 00 3d 00 80 00 00
[36542.881793] RIP [<ffffffff810cf0de>] do_vfs_ioctl+0x5e/0x510
[36542.881818] RSP <ffff8802d25a1ec8>
[36542.882082] ---[ end trace 1b8d730cd479e388 ]---
[36542.882126] Kernel panic - not syncing: Fatal exception
[36542.882175] Pid: 13346, comm: bash Tainted: G D 2.6.38.7 #29
[36542.882222] Call Trace:
[36542.882269] [<ffffffff813c7f42>] ? panic+0x92/0x18a
[36542.882318] [<ffffffff81039a41>] ? kmsg_dump+0x41/0xf0
[36542.882366] [<ffffffff810062bd>] ? oops_end+0x8d/0xa0
[36542.882414] [<ffffffff813caeef>] ? general_protection+0x1f/0x30
[36542.882463] [<ffffffff810cf0de>] ? do_vfs_ioctl+0x5e/0x510
[36542.882511] [<ffffffff81046d6a>] ? sys_rt_sigaction+0x8a/0xc0
[36542.882560] [<ffffffff810cf5d9>] ? sys_ioctl+0x49/0x80
[36542.882608] [<ffffffff810023fb>] ? system_call_fastpath+0x16/0x1b
[36542.882688] Rebooting in 60 seconds..[ 33.104725] fuse init (API
version 7.16)

2011-06-01 01:15:53

by Andrea Arcangeli

[permalink] [raw]
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39

Hello,

On Wed, Jun 01, 2011 at 08:37:25AM +0800, Brad Campbell wrote:
> On 01/06/11 06:31, Hugh Dickins wrote:
> > Brad, my suspicion is that in each case the top 16 bits of RDX have been
> > mysteriously corrupted from ffff to 0000, causing the general protection
> > faults. I don't understand what that has to do with KSM.
> >
> > But it's only a suspicion, because I can't make sense of the "Code:"
> > lines in your traces, they have more than the expected 64 bytes, and
> > only one of them has a ">" (with no"<") to mark faulting instruction.
> >
> > I did try compiling the 2.6.39 kernel from your config, but of course
> > we have different compilers, so although I got close, it wasn't exact.
> >
> > Would you mind mailing me privately (it's about 73MB) the "objdump -trd"
> > output for your original vmlinux (with KSM on)? (Those -trd options are
> > the ones I'm used to typing, I bet not they're not all relevant.)
> >
> > Of course, it's only a tiny fraction of that output that I need,
> > might be better to cut it down to remove_rmap_item_from_tree and
> > dup_fd and ksm_scan_thread, if you have the time to do so.
>
> Would you believe about 20 seconds after I pressed send the kernel oopsed.
>
> http://www.fnarfbargle.com/private/003_kernel_oops/
>
> oops reproduced here, but an un-munged version is in that directory
> alongside the kernel.
>
> [36542.880228] general protection fault: 0000 [#1] SMP

Reminds me of another oops that was reported on the kvm list for
2.6.38.1 with message id 4D8C6110.6090204. There the top 16 bits of
rsi were flipped and it was a general protection too because of
hitting on the not mappable virtual range.

http://www.virtall.com/files/temp/kvm.txt
http://www.virtall.com/files/temp/config-2.6.38.1
http://virtall.com/files/temp/mmu-objdump.txt

That oops happened in kvm_unmap_rmapp though, but it looked memory
corruption (Avi suggested use after free) but it was a production
system so we couldn't debug it further.

I recommend next thing to reproduce again with 2.6.39 or
3.0.0-rc1. Let's fix your scsi trouble if needed but it's better you
test with 2.6.39.

We'd need chmod +r vmlinux on private/003_kernel_oops/

Thanks,
Andrea

2011-06-01 02:03:30

by Brad Campbell

[permalink] [raw]
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39

On 01/06/11 09:15, Andrea Arcangeli wrote:
> Hello,
>
> On Wed, Jun 01, 2011 at 08:37:25AM +0800, Brad Campbell wrote:
>> On 01/06/11 06:31, Hugh Dickins wrote:
>>> Brad, my suspicion is that in each case the top 16 bits of RDX have been
>>> mysteriously corrupted from ffff to 0000, causing the general protection
>>> faults. I don't understand what that has to do with KSM.
>>>
>>> But it's only a suspicion, because I can't make sense of the "Code:"
>>> lines in your traces, they have more than the expected 64 bytes, and
>>> only one of them has a ">" (with no"<") to mark faulting instruction.
>>>
>>> I did try compiling the 2.6.39 kernel from your config, but of course
>>> we have different compilers, so although I got close, it wasn't exact.
>>>
>>> Would you mind mailing me privately (it's about 73MB) the "objdump -trd"
>>> output for your original vmlinux (with KSM on)? (Those -trd options are
>>> the ones I'm used to typing, I bet not they're not all relevant.)
>>>
>>> Of course, it's only a tiny fraction of that output that I need,
>>> might be better to cut it down to remove_rmap_item_from_tree and
>>> dup_fd and ksm_scan_thread, if you have the time to do so.
>>
>> Would you believe about 20 seconds after I pressed send the kernel oopsed.
>>
>> http://www.fnarfbargle.com/private/003_kernel_oops/
>>
>> oops reproduced here, but an un-munged version is in that directory
>> alongside the kernel.
>>
>> [36542.880228] general protection fault: 0000 [#1] SMP
>
> Reminds me of another oops that was reported on the kvm list for
> 2.6.38.1 with message id 4D8C6110.6090204. There the top 16 bits of
> rsi were flipped and it was a general protection too because of
> hitting on the not mappable virtual range.
>
> http://www.virtall.com/files/temp/kvm.txt
> http://www.virtall.com/files/temp/config-2.6.38.1
> http://virtall.com/files/temp/mmu-objdump.txt
>
> That oops happened in kvm_unmap_rmapp though, but it looked memory
> corruption (Avi suggested use after free) but it was a production
> system so we couldn't debug it further.
>
> I recommend next thing to reproduce again with 2.6.39 or
> 3.0.0-rc1. Let's fix your scsi trouble if needed but it's better you
> test with 2.6.39.
>
> We'd need chmod +r vmlinux on private/003_kernel_oops/

Ok, here we go then.

http://www.fnarfbargle.com/private/004_kernel_oops/

The permissions are right this time.
2.6.39 + KSM

[ 694.227866] general protection fault: 0000 [#1] SMP
[ 694.228001] last sysfs file: /sys/devices/platform/w83627ehf.656/cpu0_vid
[ 694.228050] CPU 3
[ 694.228091] Modules linked in: xt_iprange xt_DSCP xt_length
xt_CLASSIFY sch_sfq xt_CHECKSUM ipt_REJECT ipt_MASQUERADE ipt_REDIRECT
xt_recent xt_state iptable_filter iptable_nat nf_nat nf_conntrack_ipv4
nf_conntrack nf_defrag_ipv4 xt_TCPMSS xt_tcpmss xt_tcpudp iptable_mangle
ip_tables x_tables pppoe pppox ppp_generic slhc cls_u32 sch_htb deflate
zlib_deflate des_generic cbc ecb crypto_blkcipher sha1_generic md5 hmac
crypto_hash cryptomgr aead crypto_algapi af_key fuse w83627ehf hwmon_vid
netconsole configfs vhost_net powernow_k8 mperf kvm_amd kvm pl2303
usbserial i2c_piix4 k10temp xhci_hcd usb_storage usb_libusual ohci_hcd
r8169 ehci_hcd ahci usbcore sata_mv mii libahci megaraid_sas [last
unloaded: scsi_wait_scan]
[ 694.230897]
[ 694.230944] Pid: 11841, comm: keepalive Not tainted 2.6.39 #3 To Be
Filled By O.E.M. To Be Filled By O.E.M./880G Extreme3
[ 694.231111] RIP: 0010:[<ffffffff810db878>] [<ffffffff810db878>]
dup_fd+0x168/0x300
[ 694.231210] RSP: 0018:ffff8802f524fdd0 EFLAGS: 00010206
[ 694.231258] RAX: 00000000000007f8 RBX: ffff8802f5721b80 RCX:
bfffffffffffffff
[ 694.231308] RDX: 00008802f51cacc0 RSI: 00000000000000ff RDI:
0000000000000800
[ 694.231358] RBP: ffff8803bf419800 R08: ffff88030167f6c0 R09:
0000000000000003
[ 694.231407] R10: 0000000000000001 R11: 4000000000000000 R12:
0000000000000100
[ 694.231457] R13: ffff880417aa9800 R14: ffff88030167f440 R15:
ffff8803bd8c1600
[ 694.231507] FS: 00007f02cfc32700(0000) GS:ffff88041fcc0000(0000)
knlGS:0000000000000000
[ 694.231560] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 694.231609] CR2: 00007f02cf5d4810 CR3: 00000002f52c3000 CR4:
00000000000006e0
[ 694.231657] DR0: 0000000000000045 DR1: 0000000000000000 DR2:
0000000000000000
[ 694.231707] DR3: 0000000000000005 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[ 694.231757] Process keepalive (pid: 11841, threadinfo
ffff8802f524e000, task ffff8802f5143690)
[ 694.231809] Stack:
[ 694.231852] ffff8802f5143690 0000000000000020 ffff8802f56badc0
ffff8802f5721b90
[ 694.232050] ffff880417aa54e0 0000000001200011 ffff880417aa54e0
0000000000000000
[ 694.232248] 00007f02cfc329d0 ffff8802f5143690 0000000000000000
ffffffff81037645
[ 694.232448] Call Trace:
[ 694.232499] [<ffffffff81037645>] ? copy_process+0xa75/0xfd0
[ 694.232549] [<ffffffff81037c0d>] ? do_fork+0x6d/0x2b0
[ 694.232599] [<ffffffff810457a9>] ? sigprocmask+0x69/0x100
[ 694.232651] [<ffffffff813d0ca3>] ? stub_clone+0x13/0x20
[ 694.232699] [<ffffffff813d0a3b>] ? system_call_fastpath+0x16/0x1b
[ 694.232745] Code: 4c 89 c2 e8 6b e5 0f 00 45 85 e4 74 78 41 8d 44 24
ff 31 f6 41 ba 01 00 00 00 48 8d 3c c5 08 00 00 00 31 c0 eb 1a 0f 1f 44
00 00 <f0> 48 ff 42 30 48 89 54 05 00 48 83 c0 08 ff c6 48 39 f8 74 3b
[ 694.235190] RIP [<ffffffff810db878>] dup_fd+0x168/0x300
[ 694.235282] RSP <ffff8802f524fdd0>
[ 694.235379] ---[ end trace 949fad05591fcdb3 ]---
[ 694.235428] Kernel panic - not syncing: Fatal exception
[ 694.235478] Pid: 11841, comm: keepalive Tainted: G D 2.6.39 #3
[ 694.235525] Call Trace:
[ 694.235573] [<ffffffff813cd6f5>] ? panic+0x92/0x18a
[ 694.235624] [<ffffffff81038b61>] ? kmsg_dump+0x41/0xf0
[ 694.235676] [<ffffffff810050ad>] ? oops_end+0x8d/0xa0
[ 694.235726] [<ffffffff813d05ef>] ? general_protection+0x1f/0x30
[ 694.235778] [<ffffffff810db878>] ? dup_fd+0x168/0x300
[ 694.235827] [<ffffffff81037645>] ? copy_process+0xa75/0xfd0
[ 694.235877] [<ffffffff81037c0d>] ? do_fork+0x6d/0x2b0
[ 694.235926] [<ffffffff810457a9>] ? sigprocmask+0x69/0x100
[ 694.235978] [<ffffffff813d0ca3>] ? stub_clone+0x13/0x20
[ 694.236028] [<ffffffff813d0a3b>] ? system_call_fastpath+0x16/0x1b
[ 694.236083] Rebooting in 60 seconds..

2011-06-01 04:53:13

by Hugh Dickins

[permalink] [raw]
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39

On Wed, 1 Jun 2011, Andrea Arcangeli wrote:
> On Wed, Jun 01, 2011 at 08:37:25AM +0800, Brad Campbell wrote:
> > On 01/06/11 06:31, Hugh Dickins wrote:
> > > Brad, my suspicion is that in each case the top 16 bits of RDX have been
> > > mysteriously corrupted from ffff to 0000, causing the general protection
> > > faults. I don't understand what that has to do with KSM.
> > >
> > > But it's only a suspicion, because I can't make sense of the "Code:"
> > > lines in your traces, they have more than the expected 64 bytes, and
> > > only one of them has a ">" (with no"<") to mark faulting instruction.
> > >
> > > I did try compiling the 2.6.39 kernel from your config, but of course
> > > we have different compilers, so although I got close, it wasn't exact.
> > >
> > > Would you mind mailing me privately (it's about 73MB) the "objdump -trd"
> > > output for your original vmlinux (with KSM on)? (Those -trd options are
> > > the ones I'm used to typing, I bet not they're not all relevant.)
> > >
> > > Of course, it's only a tiny fraction of that output that I need,
> > > might be better to cut it down to remove_rmap_item_from_tree and
> > > dup_fd and ksm_scan_thread, if you have the time to do so.
> >
> > Would you believe about 20 seconds after I pressed send the kernel oopsed.
> >
> > http://www.fnarfbargle.com/private/003_kernel_oops/
> >
> > oops reproduced here, but an un-munged version is in that directory
> > alongside the kernel.
> >
> > [36542.880228] general protection fault: 0000 [#1] SMP
>
> Reminds me of another oops that was reported on the kvm list for
> 2.6.38.1 with message id 4D8C6110.6090204. There the top 16 bits of
> rsi were flipped and it was a general protection too because of
> hitting on the not mappable virtual range.
>
> http://www.virtall.com/files/temp/kvm.txt
> http://www.virtall.com/files/temp/config-2.6.38.1
> http://virtall.com/files/temp/mmu-objdump.txt
>
> That oops happened in kvm_unmap_rmapp though, but it looked memory
> corruption (Avi suggested use after free) but it was a production
> system so we couldn't debug it further.
>
> I recommend next thing to reproduce again with 2.6.39 or
> 3.0.0-rc1. Let's fix your scsi trouble if needed but it's better you
> test with 2.6.39.

Brad, thanks for this and the other further crash, with vmlinux etc:
very helpful info.

Andrea, I'm pretty sure you're right to connect Brad's report with
the one above.

In four out of five of Brad's reports (cannot tell in the fifth),
the bad pointer (with top 16 bits 0000 instead of ffff) had been
loaded from SLUB memory at an address offset 0x7f8 (1 case) or
0xff8 (3 cases) i.e. it's the short at 0x7fe or 0xffe that has
been zeroed.

No reason to suspect KSM's rmap_item code, or file table handling:
they just seem to be the victims of corruption from elsewhere.

I notice %rax and %rsi, the corrupted pointer in your kvm.txt
case, is itself a ...7f8 address; and %r13 an ...ff8 address.
I've not even glanced at the code, but I wonder if that implies
that KVM is close to the origin of the corruption.

I doubt I'll be able to spend more time on this, hope you can
take over.

I guess Brad could try SLUB debugging, boot with slub_debug=P
for poisoning perhaps; though it might upset alignments and
drive the problem underground. Or see if the same happens
with SLAB instead of SLUB.

But I rather hope that you or someone will understand the 7fe clue.

Thanks,
Hugh

2011-06-01 06:31:17

by Brad Campbell

[permalink] [raw]
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39

On 01/06/11 12:52, Hugh Dickins wrote:

>
> I guess Brad could try SLUB debugging, boot with slub_debug=P
> for poisoning perhaps; though it might upset alignments and
> drive the problem underground. Or see if the same happens
> with SLAB instead of SLUB.

Not much use I'm afraid.
This is all I get in the log

[ 3161.300073]
=============================================================================
[ 3161.300147] BUG kmalloc-512: Freechain corrupt

The qemu process is then frozen, unkillable but reported in state "R"

13881 ? R 3:27 /usr/bin/qemu -S -M pc-0.13 -enable-kvm -m
1024 -smp 2,sockets=2,cores=1,threads=1 -nam

The machine then progressively dies until it's frozen solid with no
further error messages.

I stupidly forgot to do an alt-sysrq-t prior to doing an alt-sysrq-b,
but at least it responded to that.

On the bright side I can reproduce it at will.

2011-06-01 06:57:00

by Avi Kivity

[permalink] [raw]
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39

On 06/01/2011 09:31 AM, Brad Campbell wrote:
> On 01/06/11 12:52, Hugh Dickins wrote:
>
>>
>> I guess Brad could try SLUB debugging, boot with slub_debug=P
>> for poisoning perhaps; though it might upset alignments and
>> drive the problem underground. Or see if the same happens
>> with SLAB instead of SLUB.
>
> Not much use I'm afraid.
> This is all I get in the log
>
> [ 3161.300073]
> =============================================================================
> [ 3161.300147] BUG kmalloc-512: Freechain corrupt
>
> The qemu process is then frozen, unkillable but reported in state "R"
>
> 13881 ? R 3:27 /usr/bin/qemu -S -M pc-0.13 -enable-kvm -m
> 1024 -smp 2,sockets=2,cores=1,threads=1 -nam
>
> The machine then progressively dies until it's frozen solid with no
> further error messages.
>
> I stupidly forgot to do an alt-sysrq-t prior to doing an alt-sysrq-b,
> but at least it responded to that.
>
> On the bright side I can reproduce it at will.

Please try slub_debug=FZPU; that should point the finger (hopefully at
somebody else).

--
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

2011-06-01 09:30:05

by Brad Campbell

[permalink] [raw]
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39

On 01/06/11 14:56, Avi Kivity wrote:
> On 06/01/2011 09:31 AM, Brad Campbell wrote:
>> On 01/06/11 12:52, Hugh Dickins wrote:
>>
>>>
>>> I guess Brad could try SLUB debugging, boot with slub_debug=P
>>> for poisoning perhaps; though it might upset alignments and
>>> drive the problem underground. Or see if the same happens
>>> with SLAB instead of SLUB.
>>
>> Not much use I'm afraid.
>> This is all I get in the log
>>
>> [ 3161.300073]
>> =============================================================================
>>
>> [ 3161.300147] BUG kmalloc-512: Freechain corrupt
>>
>> The qemu process is then frozen, unkillable but reported in state "R"
>>
>> 13881 ? R 3:27 /usr/bin/qemu -S -M pc-0.13 -enable-kvm -m 1024 -smp
>> 2,sockets=2,cores=1,threads=1 -nam
>>
>> The machine then progressively dies until it's frozen solid with no
>> further error messages.
>>
>> I stupidly forgot to do an alt-sysrq-t prior to doing an alt-sysrq-b,
>> but at least it responded to that.
>>
>> On the bright side I can reproduce it at will.
>
> Please try slub_debug=FZPU; that should point the finger (hopefully at
> somebody else).
>

Well the first attempt locked the machine solid. No network, no console..

I saw
"=========================================================================="

on the console.. nothing after that. Would not respond to sysrq-t or any
other sysrq combination other than -b, which rebooted the box.


No output on netconsole at all, I had to walk to the other building to
look at the monitor and reboot it.

The second attempt jammed netconsole again, but I managed to get this
from an ssh session I already had established. The machine died a slow
and horrible death, but remained interactive enough for me to reboot it with

echo b > /proc/sysrq-trigger

Nothing else worked.



[ 376.269051]
=============================================================================
[ 413.755328]
=============================================================================
[ 413.755337] BUG kmalloc-1024: Object padding overwritten
[ 413.755341]
-----------------------------------------------------------------------------
[ 413.755344]
[ 413.755349] INFO: 0xffff880417179566-0xffff880417179567. First byte
0x0 instead of 0x5a
[ 413.755363] INFO: Allocated in tcp_send_ack+0x26/0x120 age=5320 cpu=5
pid=0
[ 413.755374] INFO: Freed in __kfree_skb+0x11/0x90 age=5320 cpu=5 pid=0
[ 413.755380] INFO: Slab 0xffffea000e50d240 objects=29 used=5
fp=0xffff880417179120 flags=0x80000000000040c1
[ 413.755386] INFO: Object 0xffff880417179120 @offset=4384
fp=0xffff8804171799b0
[ 413.755389]
[ 413.755392] Bytes b4 0xffff880417179110: d9 2b 00 00 01 00 00 00 5a
5a 5a 5a 5a 5a 5a 5a Ù+......ZZZZZZZZ
[ 413.755408] Object 0xffff880417179120: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755423] Object 0xffff880417179130: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755438] Object 0xffff880417179140: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755452] Object 0xffff880417179150: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755466] Object 0xffff880417179160: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755480] Object 0xffff880417179170: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755494] Object 0xffff880417179180: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755508] Object 0xffff880417179190: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755522] Object 0xffff8804171791a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755536] Object 0xffff8804171791b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755551] Object 0xffff8804171791c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755565] Object 0xffff8804171791d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755579] Object 0xffff8804171791e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755593] Object 0xffff8804171791f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755607] Object 0xffff880417179200: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755621] Object 0xffff880417179210: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755635] Object 0xffff880417179220: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755650] Object 0xffff880417179230: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755664] Object 0xffff880417179240: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755678] Object 0xffff880417179250: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755692] Object 0xffff880417179260: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755706] Object 0xffff880417179270: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755720] Object 0xffff880417179280: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755734] Object 0xffff880417179290: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755749] Object 0xffff8804171792a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755763] Object 0xffff8804171792b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755777] Object 0xffff8804171792c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755791] Object 0xffff8804171792d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755805] Object 0xffff8804171792e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755819] Object 0xffff8804171792f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755834] Object 0xffff880417179300: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755848] Object 0xffff880417179310: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755862] Object 0xffff880417179320: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755876] Object 0xffff880417179330: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755890] Object 0xffff880417179340: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755904] Object 0xffff880417179350: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755919] Object 0xffff880417179360: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755933] Object 0xffff880417179370: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755947] Object 0xffff880417179380: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755961] Object 0xffff880417179390: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755975] Object 0xffff8804171793a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755989] Object 0xffff8804171793b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756004] Object 0xffff8804171793c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756018] Object 0xffff8804171793d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756032] Object 0xffff8804171793e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756046] Object 0xffff8804171793f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756060] Object 0xffff880417179400: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756074] Object 0xffff880417179410: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756089] Object 0xffff880417179420: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756103] Object 0xffff880417179430: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756117] Object 0xffff880417179440: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756131] Object 0xffff880417179450: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756145] Object 0xffff880417179460: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756160] Object 0xffff880417179470: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756174] Object 0xffff880417179480: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756188] Object 0xffff880417179490: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756202] Object 0xffff8804171794a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756216] Object 0xffff8804171794b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756230] Object 0xffff8804171794c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756245] Object 0xffff8804171794d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756259] Object 0xffff8804171794e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756273] Object 0xffff8804171794f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756287] Object 0xffff880417179500: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756301] Object 0xffff880417179510: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[ 413.756316] Redzone 0xffff880417179520: bb bb bb bb bb bb bb bb
»»»»»»»»
[ 413.756329] Padding 0xffff880417179560: 5a 5a 5a 5a 5a 5a 00 00
ZZZZZZ..
[ 413.756345] Pid: 5247, comm: qemu Not tainted 2.6.39 #2
[ 413.756349] Call Trace:
[ 413.756353] <IRQ> [<ffffffff810b7ccd>] ?
check_bytes_and_report+0x10d/0x150
[ 413.756372] [<ffffffff81318f1c>] ? pskb_expand_head+0x15c/0x250
[ 413.756379] [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[ 413.756387] [<ffffffff81318f1c>] ? pskb_expand_head+0x15c/0x250
[ 413.756393] [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[ 413.756400] [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[ 413.756409] [<ffffffff810ba59d>] ? __kmalloc+0x10d/0x160
[ 413.756416] [<ffffffff81318f1c>] ? pskb_expand_head+0x15c/0x250
[ 413.756424] [<ffffffff813a6c45>] ? nf_bridge_copy_header+0x145/0x160
[ 413.756431] [<ffffffff8139f78d>] ? br_dev_queue_push_xmit+0x6d/0x80
[ 413.756439] [<ffffffff813a55a0>] ? br_nf_post_routing+0x2a0/0x2f0
[ 413.756447] [<ffffffff81346bc4>] ? nf_iterate+0x84/0xb0
[ 413.756453] [<ffffffff8139f720>] ? br_flood_deliver+0x20/0x20
[ 413.756459] [<ffffffff81346c64>] ? nf_hook_slow+0x74/0x120
[ 413.756465] [<ffffffff8139f720>] ? br_flood_deliver+0x20/0x20
[ 413.756472] [<ffffffff8139f7da>] ? br_forward_finish+0x3a/0x60
[ 413.756479] [<ffffffff813a5758>] ? br_nf_forward_finish+0x168/0x170
[ 413.756487] [<ffffffff813a5c90>] ? br_nf_forward_ip+0x360/0x3a0
[ 413.756492] [<ffffffff81346bc4>] ? nf_iterate+0x84/0xb0
[ 413.756498] [<ffffffff8139f7a0>] ? br_dev_queue_push_xmit+0x80/0x80
[ 413.756504] [<ffffffff81346c64>] ? nf_hook_slow+0x74/0x120
[ 413.756510] [<ffffffff8139f7a0>] ? br_dev_queue_push_xmit+0x80/0x80
[ 413.756516] [<ffffffff8139f800>] ? br_forward_finish+0x60/0x60
[ 413.756522] [<ffffffff8139f800>] ? br_forward_finish+0x60/0x60
[ 413.756528] [<ffffffff8139f875>] ? __br_forward+0x75/0xc0
[ 413.756534] [<ffffffff8139f426>] ? deliver_clone+0x36/0x60
[ 413.756540] [<ffffffff8139f69d>] ? br_flood+0xbd/0x100
[ 413.756546] [<ffffffff813a05b0>] ? br_handle_local_finish+0x40/0x40
[ 413.756552] [<ffffffff813a080e>] ? br_handle_frame_finish+0x25e/0x280
[ 413.756560] [<ffffffff813a60f0>] ? br_nf_pre_routing_finish+0x1a0/0x330
[ 413.756568] [<ffffffff813a6958>] ? br_nf_pre_routing+0x6d8/0x800
[ 413.756577] [<ffffffff8102d46a>] ? enqueue_task+0x3a/0x90
[ 413.756582] [<ffffffff81346bc4>] ? nf_iterate+0x84/0xb0
[ 413.756589] [<ffffffff813a05b0>] ? br_handle_local_finish+0x40/0x40
[ 413.756594] [<ffffffff81346c64>] ? nf_hook_slow+0x74/0x120
[ 413.756600] [<ffffffff813a05b0>] ? br_handle_local_finish+0x40/0x40
[ 413.756607] [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[ 413.756613] [<ffffffff813a09d9>] ? br_handle_frame+0x1a9/0x280
[ 413.756620] [<ffffffff813a0830>] ? br_handle_frame_finish+0x280/0x280
[ 413.756627] [<ffffffff81320ef7>] ? __netif_receive_skb+0x157/0x5c0
[ 413.756634] [<ffffffff81321443>] ? process_backlog+0xe3/0x1d0
[ 413.756641] [<ffffffff81321da5>] ? net_rx_action+0xc5/0x1d0
[ 413.756650] [<ffffffff8103df11>] ? __do_softirq+0x91/0x120
[ 413.756657] [<ffffffff813d838c>] ? call_softirq+0x1c/0x30
[ 413.756660] <EOI> [<ffffffff81003cbd>] ? do_softirq+0x4d/0x80
[ 413.756673] [<ffffffff81321ece>] ? netif_rx_ni+0x1e/0x30
[ 413.756681] [<ffffffff812b3ae2>] ? tun_chr_aio_write+0x332/0x4e0
[ 413.756688] [<ffffffff812b37b0>] ? tun_sendmsg+0x4d0/0x4d0
[ 413.756697] [<ffffffff810c24e9>] ? do_sync_readv_writev+0xa9/0xf0
[ 413.756704] [<ffffffff81063f9c>] ? do_futex+0x13c/0xa70
[ 413.756711] [<ffffffff811d6730>] ? timerqueue_add+0x60/0xb0
[ 413.756719] [<ffffffff81056ab7>] ? __hrtimer_start_range_ns+0x1e7/0x410
[ 413.756726] [<ffffffff810c231b>] ? rw_copy_check_uvector+0x7b/0x140
[ 413.756734] [<ffffffff810c2bcf>] ? do_readv_writev+0xdf/0x210
[ 413.756742] [<ffffffff810c2e7e>] ? sys_writev+0x4e/0xc0
[ 413.756750] [<ffffffff813d753b>] ? system_call_fastpath+0x16/0x1b
[ 413.756756] FIX kmalloc-1024: Restoring
0xffff880417179566-0xffff880417179567=0x5a
[ 413.756760]
[ 556.640033]
=============================================================================
[ 556.640041] BUG kmalloc-512: Object padding overwritten
[ 556.640045]
-----------------------------------------------------------------------------
[ 556.640048]
[ 556.640053] INFO: 0xffff880403bf091e-0xffff880403bf091f. First byte
0x0 instead of 0x5a
[ 556.640069] INFO: Allocated in sock_alloc_send_pskb+0x1d0/0x320
age=21401 cpu=2 pid=5630
[ 556.640080] INFO: Freed in __kfree_skb+0x11/0x90 age=21386 cpu=2 pid=3753
[ 556.640086] INFO: Slab 0xffffea000e0d1c80 objects=28 used=10
fp=0xffff880403bf06d8 flags=0x80000000000040c1
[ 556.640092] INFO: Object 0xffff880403bf06d8 @offset=1752
fp=0xffff880403bf1488
[ 556.640096]
[ 556.640099] Bytes b4 0xffff880403bf06c8: 90 c4 ff ff 00 00 00 00 5a
5a 5a 5a 5a 5a 5a 5a .Äÿÿ....ZZZZZZZZ
[ 556.640115] Object 0xffff880403bf06d8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640130] Object 0xffff880403bf06e8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640144] Object 0xffff880403bf06f8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640158] Object 0xffff880403bf0708: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640172] Object 0xffff880403bf0718: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640187] Object 0xffff880403bf0728: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640201] Object 0xffff880403bf0738: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640215] Object 0xffff880403bf0748: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640229] Object 0xffff880403bf0758: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640243] Object 0xffff880403bf0768: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640257] Object 0xffff880403bf0778: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640271] Object 0xffff880403bf0788: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640286] Object 0xffff880403bf0798: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640300] Object 0xffff880403bf07a8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640314] Object 0xffff880403bf07b8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640328] Object 0xffff880403bf07c8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640343] Object 0xffff880403bf07d8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640357] Object 0xffff880403bf07e8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640371] Object 0xffff880403bf07f8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640385] Object 0xffff880403bf0808: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640399] Object 0xffff880403bf0818: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640413] Object 0xffff880403bf0828: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640428] Object 0xffff880403bf0838: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640442] Object 0xffff880403bf0848: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640456] Object 0xffff880403bf0858: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640471] Object 0xffff880403bf0868: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640485] Object 0xffff880403bf0878: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640499] Object 0xffff880403bf0888: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640513] Object 0xffff880403bf0898: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640527] Object 0xffff880403bf08a8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640542] Object 0xffff880403bf08b8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640556] Object 0xffff880403bf08c8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[ 556.640570] Redzone 0xffff880403bf08d8: bb bb bb bb bb bb bb bb
»»»»»»»»
[ 556.640583] Padding 0xffff880403bf0918: 5a 5a 5a 5a 5a 5a 00 00
ZZZZZZ..
[ 556.640599] Pid: 4809, comm: qemu Not tainted 2.6.39 #2
[ 556.640603] Call Trace:
[ 556.640607] <IRQ> [<ffffffff810b7ccd>] ?
check_bytes_and_report+0x10d/0x150
[ 556.640626] [<ffffffff8131aae7>] ? __netdev_alloc_skb+0x17/0x40
[ 556.640632] [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[ 556.640640] [<ffffffff8131aae7>] ? __netdev_alloc_skb+0x17/0x40
[ 556.640647] [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[ 556.640654] [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[ 556.640661] [<ffffffff810bb70d>] ? __kmalloc_track_caller+0x10d/0x160
[ 556.640668] [<ffffffff813198c2>] ? __alloc_skb+0x72/0x160
[ 556.640676] [<ffffffff8131aae7>] ? __netdev_alloc_skb+0x17/0x40
[ 556.640684] [<ffffffff813a7b93>] ?
br_ip4_multicast_alloc_query+0x23/0x1c0
[ 556.640690] [<ffffffff813a7e76>] ? br_multicast_send_query+0x76/0x130
[ 556.640698] [<ffffffff8104dba8>] ? wq_worker_waking_up+0x8/0x30
[ 556.640706] [<ffffffff8102ca81>] ? sched_slice+0x51/0x80
[ 556.640711] [<ffffffff813a7fb5>] ?
br_multicast_port_query_expired+0x45/0x70
[ 556.640719] [<ffffffff81043f77>] ? run_timer_softirq+0x137/0x270
[ 556.640725] [<ffffffff81033fe9>] ? scheduler_tick+0x289/0x2d0
[ 556.640731] [<ffffffff813a7f70>] ? br_multicast_query_expired+0x40/0x40
[ 556.640740] [<ffffffff8103df11>] ? __do_softirq+0x91/0x120
[ 556.640747] [<ffffffff813d838c>] ? call_softirq+0x1c/0x30
[ 556.640755] [<ffffffff81003cbd>] ? do_softirq+0x4d/0x80
[ 556.640762] [<ffffffff8103ddbe>] ? irq_exit+0x8e/0xb0
[ 556.640770] [<ffffffff81019d1a>] ? smp_apic_timer_interrupt+0x6a/0xa0
[ 556.640776] [<ffffffff813d7e53>] ? apic_timer_interrupt+0x13/0x20
[ 556.640780] <EOI> [<ffffffff8102965a>] ?
flush_tlb_others_ipi+0x11a/0x130
[ 556.640791] [<ffffffff81029647>] ? flush_tlb_others_ipi+0x107/0x130
[ 556.640799] [<ffffffff810aadab>] ? ptep_clear_flush+0xb/0x10
[ 556.640806] [<ffffffff8109a40b>] ? do_wp_page+0x34b/0x7a0
[ 556.640836] [<ffffffffa00a188a>] ? kvm_read_guest_page+0x5a/0x70 [kvm]
[ 556.640863] [<ffffffffa00b7db4>] ?
paging64_walk_addr_generic+0x264/0x4c0 [kvm]
[ 556.640871] [<ffffffff8109acad>] ? handle_pte_fault+0x44d/0x990
[ 556.640878] [<ffffffff8109b4e8>] ? follow_page+0x268/0x440
[ 556.640886] [<ffffffff8109bd4d>] ? __get_user_pages+0x12d/0x530
[ 556.640892] [<ffffffff81028dbb>] ? gup_pud_range+0x12b/0x1b0
[ 556.640914] [<ffffffffa00a2d2e>] ? get_user_page_nowait+0x2e/0x40 [kvm]
[ 556.640935] [<ffffffffa00a2e7b>] ? hva_to_pfn+0x13b/0x440 [kvm]
[ 556.640956] [<ffffffffa00a31e9>] ? __gfn_to_pfn+0x39/0xf0 [kvm]
[ 556.640979] [<ffffffffa00b825d>] ? try_async_pf+0x4d/0x190 [kvm]
[ 556.641002] [<ffffffffa00b92be>] ? tdp_page_fault+0x10e/0x200 [kvm]
[ 556.641026] [<ffffffffa00b968c>] ? kvm_mmu_page_fault+0x1c/0x80 [kvm]
[ 556.641050] [<ffffffffa00b0a7a>] ?
kvm_arch_vcpu_ioctl_run+0x3fa/0xcf0 [kvm]
[ 556.641058] [<ffffffff8106239f>] ? futex_wake+0x10f/0x120
[ 556.641065] [<ffffffff81063f7b>] ? do_futex+0x11b/0xa70
[ 556.641086] [<ffffffffa00a517f>] ? kvm_vcpu_ioctl+0x4df/0x5e0 [kvm]
[ 556.641097] [<ffffffff81046671>] ? __dequeue_signal+0xe1/0x170
[ 556.641104] [<ffffffff8104821c>] ? do_send_sig_info+0x6c/0x90
[ 556.641112] [<ffffffff81046aac>] ? dequeue_signal+0x3c/0x170
[ 556.641119] [<ffffffff8104551f>] ? copy_siginfo_to_user+0xef/0x1d0
[ 556.641125] [<ffffffff810d26db>] ? do_vfs_ioctl+0x9b/0x4f0
[ 556.641132] [<ffffffff8106494a>] ? sys_futex+0x7a/0x180
[ 556.641137] [<ffffffff810d2b79>] ? sys_ioctl+0x49/0x80
[ 556.641145] [<ffffffff813d753b>] ? system_call_fastpath+0x16/0x1b
[ 556.641152] FIX kmalloc-512: Restoring
0xffff880403bf091e-0xffff880403bf091f=0x5a
[ 556.641155]
[ 602.610062] INFO: task ksmd:552 blocked for more than 120 seconds.
[ 602.610068] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 602.610073] ksmd D 0000000000000000 0 552 2
0x00000000
[ 602.610083] ffff88041db68ca0 0000000000000046 000212d01fc11d00
ffff880400000000
[ 602.610090] ffffffff81593020 ffff88041c1a9fd8 0000000000004000
ffff88041c1a8010
[ 602.610097] ffff88041c1a9fd8 ffff88041db68ca0 0000000000000000
0000000200000002
[ 602.610104] Call Trace:
[ 602.610121] [<ffffffff813d5595>] ? schedule_timeout+0x1c5/0x230
[ 602.610132] [<ffffffff8102f69c>] ? enqueue_task_fair+0x14c/0x190
[ 602.610141] [<ffffffff8102db27>] ? task_rq_lock+0x47/0x90
[ 602.610148] [<ffffffff813d4562>] ? wait_for_common+0xd2/0x180
[ 602.610154] [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[ 602.610162] [<ffffffff8104cb94>] ? flush_work+0x24/0x30
[ 602.610167] [<ffffffff8104be10>] ? do_work_for_cpu+0x20/0x20
[ 602.610174] [<ffffffff8104e0ab>] ? schedule_on_each_cpu+0xab/0xe0
[ 602.610181] [<ffffffff810b5c05>] ? ksm_scan_thread+0x7f5/0xc20
[ 602.610189] [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[ 602.610194] [<ffffffff810b5410>] ?
try_to_merge_with_ksm_page+0x570/0x570
[ 602.610200] [<ffffffff810b5410>] ?
try_to_merge_with_ksm_page+0x570/0x570
[ 602.610207] [<ffffffff810525b6>] ? kthread+0x96/0xa0
[ 602.610214] [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[ 602.610221] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[ 602.610227] [<ffffffff813d8290>] ? gs_change+0xb/0xb
[ 602.610233] INFO: task fsnotify_mark:662 blocked for more than 120
seconds.
[ 602.610237] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 602.610241] fsnotify_mark D 0000000000000000 0 662 2
0x00000000
[ 602.610248] ffff88041c31d860 0000000000000046 0000000000000000
0000000000000000
[ 602.610255] ffffffff81593020 ffff88041c365fd8 0000000000004000
ffff88041c364010
[ 602.610262] ffff88041c365fd8 ffff88041c31d860 0000000000000000
0000000000000000
[ 602.610268] Call Trace:
[ 602.610275] [<ffffffff81035521>] ? load_balance+0x91/0x5e0
[ 602.610282] [<ffffffff813d5595>] ? schedule_timeout+0x1c5/0x230
[ 602.610289] [<ffffffff8102f323>] ? pick_next_task_fair+0x103/0x190
[ 602.610295] [<ffffffff813d4a6d>] ? schedule+0x28d/0x910
[ 602.610302] [<ffffffff813d4562>] ? wait_for_common+0xd2/0x180
[ 602.610307] [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[ 602.610314] [<ffffffff81074420>] ? synchronize_rcu_bh+0x50/0x50
[ 602.610320] [<ffffffff8107446a>] ? synchronize_sched+0x4a/0x50
[ 602.610326] [<ffffffff8104f940>] ? find_ge_pid+0x40/0x40
[ 602.610333] [<ffffffff8105742b>] ? __synchronize_srcu+0x5b/0xc0
[ 602.610342] [<ffffffff810f5f63>] ? fsnotify_mark_destroy+0x83/0x150
[ 602.610348] [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[ 602.610356] [<ffffffff810f5ee0>] ?
fsnotify_set_mark_ignored_mask_locked+0x20/0x20
[ 602.610364] [<ffffffff810f5ee0>] ?
fsnotify_set_mark_ignored_mask_locked+0x20/0x20
[ 602.610371] [<ffffffff810525b6>] ? kthread+0x96/0xa0
[ 602.610377] [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[ 602.610385] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[ 602.610390] [<ffffffff813d8290>] ? gs_change+0xb/0xb
[ 644.689408]
=============================================================================
[ 644.689416] BUG kmalloc-512: Object padding overwritten
[ 644.689420]
-----------------------------------------------------------------------------
[ 644.689423]
[ 644.689428] INFO: 0xffff88041411f476-0xffff88041411f477. First byte
0x0 instead of 0x5a
[ 644.689444] INFO: Allocated in load_elf_binary+0xa6c/0x1c00 age=18648
cpu=3 pid=5915
[ 644.689454] INFO: Freed in load_elf_binary+0xab8/0x1c00 age=18648
cpu=3 pid=5915
[ 644.689461] INFO: Slab 0xffffea000e463e20 objects=28 used=10
fp=0xffff88041411f230 flags=0x80000000000040c1
[ 644.689467] INFO: Object 0xffff88041411f230 @offset=12848
fp=0xffff88041411eda0
[ 644.689470]
[ 644.689474] Bytes b4 0xffff88041411f220: 22 6f 00 00 01 00 00 00 5a
5a 5a 5a 5a 5a 5a 5a "o......ZZZZZZZZ
[ 644.689490] Object 0xffff88041411f230: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689505] Object 0xffff88041411f240: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689519] Object 0xffff88041411f250: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689533] Object 0xffff88041411f260: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689547] Object 0xffff88041411f270: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689562] Object 0xffff88041411f280: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689576] Object 0xffff88041411f290: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689590] Object 0xffff88041411f2a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689604] Object 0xffff88041411f2b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689618] Object 0xffff88041411f2c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689632] Object 0xffff88041411f2d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689647] Object 0xffff88041411f2e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689661] Object 0xffff88041411f2f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689675] Object 0xffff88041411f300: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689689] Object 0xffff88041411f310: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689703] Object 0xffff88041411f320: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689717] Object 0xffff88041411f330: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689732] Object 0xffff88041411f340: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689746] Object 0xffff88041411f350: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689760] Object 0xffff88041411f360: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689774] Object 0xffff88041411f370: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689788] Object 0xffff88041411f380: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689802] Object 0xffff88041411f390: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689817] Object 0xffff88041411f3a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689831] Object 0xffff88041411f3b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689845] Object 0xffff88041411f3c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689859] Object 0xffff88041411f3d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689873] Object 0xffff88041411f3e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689888] Object 0xffff88041411f3f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689902] Object 0xffff88041411f400: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689916] Object 0xffff88041411f410: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689930] Object 0xffff88041411f420: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[ 644.689945] Redzone 0xffff88041411f430: bb bb bb bb bb bb bb bb
»»»»»»»»
[ 644.689957] Padding 0xffff88041411f470: 5a 5a 5a 5a 5a 5a 00 00
ZZZZZZ..
[ 644.689974] Pid: 6597, comm: awk Not tainted 2.6.39 #2
[ 644.689978] Call Trace:
[ 644.689989] [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[ 644.689998] [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[ 644.690005] [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[ 644.690012] [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[ 644.690019] [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[ 644.690026] [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[ 644.690032] [<ffffffff810ba59d>] ? __kmalloc+0x10d/0x160
[ 644.690039] [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[ 644.690047] [<ffffffff8109bd9d>] ? __get_user_pages+0x17d/0x530
[ 644.690055] [<ffffffff810c9556>] ? get_arg_page+0x56/0x100
[ 644.690062] [<ffffffff810c8070>] ? search_binary_handler+0x90/0x240
[ 644.690069] [<ffffffff810c9e9f>] ? do_execve+0x22f/0x2f0
[ 644.690076] [<ffffffff810094a6>] ? sys_execve+0x36/0x60
[ 644.690085] [<ffffffff813d78dc>] ? stub_execve+0x6c/0xc0
[ 644.690092] FIX kmalloc-512: Restoring
0xffff88041411f476-0xffff88041411f477=0x5a
[ 644.690095]
[ 704.979481]
=============================================================================
[ 704.979488] BUG kmalloc-512: Object padding overwritten
[ 704.979492]
-----------------------------------------------------------------------------
[ 704.979496]
[ 704.979501] INFO: 0xffff8804168c1fee-0xffff8804168c1fef. First byte
0x0 instead of 0x5a
[ 704.979517] INFO: Allocated in load_elf_binary+0xa6c/0x1c00 age=24184
cpu=0 pid=5999
[ 704.979527] INFO: Freed in load_elf_binary+0xab8/0x1c00 age=24184
cpu=0 pid=5999
[ 704.979534] INFO: Slab 0xffffea000e4eea00 objects=28 used=10
fp=0xffff8804168c1da8 flags=0x80000000000040c1
[ 704.979540] INFO: Object 0xffff8804168c1da8 @offset=7592
fp=0xffff8804168c2910
[ 704.979543]
[ 704.979547] Bytes b4 0xffff8804168c1d98: 93 98 ff ff 00 00 00 00 5a
5a 5a 5a 5a 5a 5a 5a ..ÿÿ....ZZZZZZZZ
[ 704.979563] Object 0xffff8804168c1da8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979578] Object 0xffff8804168c1db8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979592] Object 0xffff8804168c1dc8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979606] Object 0xffff8804168c1dd8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979620] Object 0xffff8804168c1de8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979634] Object 0xffff8804168c1df8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979648] Object 0xffff8804168c1e08: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979662] Object 0xffff8804168c1e18: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979676] Object 0xffff8804168c1e28: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979690] Object 0xffff8804168c1e38: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979704] Object 0xffff8804168c1e48: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979719] Object 0xffff8804168c1e58: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979733] Object 0xffff8804168c1e68: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979747] Object 0xffff8804168c1e78: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979761] Object 0xffff8804168c1e88: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979775] Object 0xffff8804168c1e98: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979789] Object 0xffff8804168c1ea8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979803] Object 0xffff8804168c1eb8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979817] Object 0xffff8804168c1ec8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979831] Object 0xffff8804168c1ed8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979845] Object 0xffff8804168c1ee8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979859] Object 0xffff8804168c1ef8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979873] Object 0xffff8804168c1f08: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979888] Object 0xffff8804168c1f18: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979902] Object 0xffff8804168c1f28: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979916] Object 0xffff8804168c1f38: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979930] Object 0xffff8804168c1f48: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979944] Object 0xffff8804168c1f58: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979958] Object 0xffff8804168c1f68: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979972] Object 0xffff8804168c1f78: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979986] Object 0xffff8804168c1f88: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.980000] Object 0xffff8804168c1f98: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[ 704.980015] Redzone 0xffff8804168c1fa8: bb bb bb bb bb bb bb bb
»»»»»»»»
[ 704.980028] Padding 0xffff8804168c1fe8: 5a 5a 5a 5a 5a 5a 00 00
ZZZZZZ..
[ 704.980044] Pid: 6812, comm: get-rrdtool-dat Not tainted 2.6.39 #2
[ 704.980048] Call Trace:
[ 704.980060] [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[ 704.980069] [<ffffffff81106e49>] ? load_elf_binary+0x179/0x1c00
[ 704.980075] [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[ 704.980083] [<ffffffff81106e49>] ? load_elf_binary+0x179/0x1c00
[ 704.980090] [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[ 704.980096] [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[ 704.980102] [<ffffffff810ba59d>] ? __kmalloc+0x10d/0x160
[ 704.980109] [<ffffffff81106e49>] ? load_elf_binary+0x179/0x1c00
[ 704.980117] [<ffffffff8109b782>] ? __pte_alloc+0x42/0x130
[ 704.980123] [<ffffffff8109b5a5>] ? follow_page+0x325/0x440
[ 704.980130] [<ffffffff8109b5a5>] ? follow_page+0x325/0x440
[ 704.980137] [<ffffffff8109bd9d>] ? __get_user_pages+0x17d/0x530
[ 704.980145] [<ffffffff810c9556>] ? get_arg_page+0x56/0x100
[ 704.980152] [<ffffffff810c8070>] ? search_binary_handler+0x90/0x240
[ 704.980158] [<ffffffff810c9e9f>] ? do_execve+0x22f/0x2f0
[ 704.980166] [<ffffffff810094a6>] ? sys_execve+0x36/0x60
[ 704.980175] [<ffffffff813d78dc>] ? stub_execve+0x6c/0xc0
[ 704.980182] FIX kmalloc-512: Restoring
0xffff8804168c1fee-0xffff8804168c1fef=0x5a
[ 704.980185]
[ 722.610058] INFO: task ksmd:552 blocked for more than 120 seconds.
[ 722.610063] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 722.610069] ksmd D 0000000000000000 0 552 2
0x00000000
[ 722.610078] ffff88041db68ca0 0000000000000046 000212d01fc11d00
ffff880400000000
[ 722.610085] ffffffff81593020 ffff88041c1a9fd8 0000000000004000
ffff88041c1a8010
[ 722.610092] ffff88041c1a9fd8 ffff88041db68ca0 0000000000000000
0000000200000002
[ 722.610099] Call Trace:
[ 722.610115] [<ffffffff813d5595>] ? schedule_timeout+0x1c5/0x230
[ 722.610127] [<ffffffff8102f69c>] ? enqueue_task_fair+0x14c/0x190
[ 722.610135] [<ffffffff8102db27>] ? task_rq_lock+0x47/0x90
[ 722.610142] [<ffffffff813d4562>] ? wait_for_common+0xd2/0x180
[ 722.610148] [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[ 722.610156] [<ffffffff8104cb94>] ? flush_work+0x24/0x30
[ 722.610162] [<ffffffff8104be10>] ? do_work_for_cpu+0x20/0x20
[ 722.610168] [<ffffffff8104e0ab>] ? schedule_on_each_cpu+0xab/0xe0
[ 722.610175] [<ffffffff810b5c05>] ? ksm_scan_thread+0x7f5/0xc20
[ 722.610182] [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[ 722.610188] [<ffffffff810b5410>] ?
try_to_merge_with_ksm_page+0x570/0x570
[ 722.610194] [<ffffffff810b5410>] ?
try_to_merge_with_ksm_page+0x570/0x570
[ 722.610201] [<ffffffff810525b6>] ? kthread+0x96/0xa0
[ 722.610208] [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[ 722.610215] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[ 722.610221] [<ffffffff813d8290>] ? gs_change+0xb/0xb
[ 722.610226] INFO: task fsnotify_mark:662 blocked for more than 120
seconds.
[ 722.610230] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 722.610235] fsnotify_mark D 0000000000000000 0 662 2
0x00000000
[ 722.610241] ffff88041c31d860 0000000000000046 0000000000000000
0000000000000000
[ 722.610248] ffffffff81593020 ffff88041c365fd8 0000000000004000
ffff88041c364010
[ 722.610255] ffff88041c365fd8 ffff88041c31d860 0000000000000000
0000000000000000
[ 722.610261] Call Trace:
[ 722.610268] [<ffffffff81035521>] ? load_balance+0x91/0x5e0
[ 722.610275] [<ffffffff813d5595>] ? schedule_timeout+0x1c5/0x230
[ 722.610282] [<ffffffff8102f323>] ? pick_next_task_fair+0x103/0x190
[ 722.610289] [<ffffffff813d4a6d>] ? schedule+0x28d/0x910
[ 722.610295] [<ffffffff813d4562>] ? wait_for_common+0xd2/0x180
[ 722.610301] [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[ 722.610308] [<ffffffff81074420>] ? synchronize_rcu_bh+0x50/0x50
[ 722.610313] [<ffffffff8107446a>] ? synchronize_sched+0x4a/0x50
[ 722.610319] [<ffffffff8104f940>] ? find_ge_pid+0x40/0x40
[ 722.610326] [<ffffffff8105742b>] ? __synchronize_srcu+0x5b/0xc0
[ 722.610335] [<ffffffff810f5f63>] ? fsnotify_mark_destroy+0x83/0x150
[ 722.610342] [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[ 722.610350] [<ffffffff810f5ee0>] ?
fsnotify_set_mark_ignored_mask_locked+0x20/0x20
[ 722.610358] [<ffffffff810f5ee0>] ?
fsnotify_set_mark_ignored_mask_locked+0x20/0x20
[ 722.610365] [<ffffffff810525b6>] ? kthread+0x96/0xa0
[ 722.610371] [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[ 722.610378] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[ 722.610384] [<ffffffff813d8290>] ? gs_change+0xb/0xb
[ 722.610392] INFO: task jbd2/md0-8:2528 blocked for more than 120 seconds.
[ 722.610396] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 722.610400] jbd2/md0-8 D 0000000000000005 0 2528 2
0x00000000
[ 722.610407] ffff88041d91c570 0000000000000046 ffff88041baeea28
ffffea0000000000
[ 722.610413] ffff88041d91a5e0 ffff880419f9bfd8 0000000000004000
ffff880419f9a010
[ 722.610420] ffff880419f9bfd8 ffff88041d91c570 ffff88041baeea28
ffffffff810b8495
[ 722.610426] Call Trace:
[ 722.610432] [<ffffffff810b8495>] ? init_object+0x85/0xa0
[ 722.610438] [<ffffffff810b8916>] ? free_debug_processing+0x196/0x250
[ 722.610447] [<ffffffff8105adae>] ? ktime_get_ts+0x6e/0xf0
[ 722.610455] [<ffffffff810810f0>] ? __lock_page+0x70/0x70
[ 722.610461] [<ffffffff813d5174>] ? io_schedule+0x84/0xd0
[ 722.610469] [<ffffffff811d4403>] ?
radix_tree_gang_lookup_tag_slot+0x93/0xf0
[ 722.610476] [<ffffffff810810f9>] ? sleep_on_page+0x9/0x10
[ 722.610482] [<ffffffff813d57df>] ? __wait_on_bit+0x4f/0x80
[ 722.610489] [<ffffffff810812eb>] ? wait_on_page_bit+0x6b/0x80
[ 722.610496] [<ffffffff81052a50>] ? autoremove_wake_function+0x30/0x30
[ 722.610504] [<ffffffff8108a458>] ? pagevec_lookup_tag+0x18/0x20
[ 722.610509] [<ffffffff81081f2a>] ? filemap_fdatawait_range+0xfa/0x180
[ 722.610518] [<ffffffff811be09f>] ? submit_bio+0x6f/0xf0
[ 722.610526] [<ffffffff81176276>] ?
jbd2_journal_commit_transaction+0x796/0x1270
[ 722.610536] [<ffffffff81179ed1>] ? kjournald2+0xb1/0x1e0
[ 722.610542] [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[ 722.610549] [<ffffffff81179e20>] ? commit_timeout+0x10/0x10
[ 722.610556] [<ffffffff81179e20>] ? commit_timeout+0x10/0x10
[ 722.610563] [<ffffffff810525b6>] ? kthread+0x96/0xa0
[ 722.610569] [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[ 722.610576] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[ 722.610582] [<ffffffff813d8290>] ? gs_change+0xb/0xb
[ 722.610591] INFO: task nfsd:4326 blocked for more than 120 seconds.
[ 722.610594] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 722.610598] nfsd D 0000000000000000 0 4326 2
0x00000000
[ 722.610605] ffff88041b604bc0 0000000000000046 0000000000000016
ffffffff00000000
[ 722.610611] ffff88041d8fa5e0 ffff88041c527fd8 0000000000004000
ffff88041c526010
[ 722.610617] ffff88041c527fd8 ffff88041b604bc0 000000000000009c
ffff880000000000
[ 722.610623] Call Trace:
[ 722.610630] [<ffffffff8101e845>] ? amd_flush_garts+0x105/0x140
[ 722.610637] [<ffffffff8101fa80>] ? gart_map_sg+0x480/0x480
[ 722.610643] [<ffffffff8101f5d3>] ? flush_gart+0x23/0x50
[ 722.610650] [<ffffffff81080f98>] ? find_get_page+0x18/0x90
[ 722.610657] [<ffffffff81174b95>] ? do_get_write_access+0x265/0x4a0
[ 722.610665] [<ffffffff81052a50>] ? autoremove_wake_function+0x30/0x30
[ 722.610672] [<ffffffff81174ef9>] ?
jbd2_journal_get_write_access+0x29/0x50
[ 722.610680] [<ffffffff8115f122>] ?
__ext4_journal_get_write_access+0x32/0x80
[ 722.610689] [<ffffffff81143908>] ? ext4_reserve_inode_write+0x78/0xa0
[ 722.610696] [<ffffffff81143970>] ? ext4_mark_inode_dirty+0x40/0x1e0
[ 722.610703] [<ffffffff81156c0b>] ? ext4_journal_start_sb+0x6b/0x160
[ 722.610711] [<ffffffff81322645>] ? dev_hard_start_xmit+0x305/0x5f0
[ 722.610719] [<ffffffff81352fd0>] ? ip_finish_output2+0x290/0x290
[ 722.610726] [<ffffffff81143c65>] ? ext4_dirty_inode+0x35/0x70
[ 722.610733] [<ffffffff810e4a08>] ? __mark_inode_dirty+0x38/0x210
[ 722.610741] [<ffffffff810d9317>] ? file_update_time+0xf7/0x180
[ 722.610747] [<ffffffff81082458>] ? __generic_file_aio_write+0x1f8/0x430
[ 722.610755] [<ffffffff81373ef9>] ? udp_sendmsg+0x3c9/0x7e0
[ 722.610763] [<ffffffff81314260>] ? sock_alloc_send_pskb+0x1d0/0x320
[ 722.610769] [<ffffffff81082703>] ? generic_file_aio_write+0x73/0xf0
[ 722.610776] [<ffffffff8113f74e>] ? ext4_file_write+0x6e/0x2b0
[ 722.610783] [<ffffffff810da8ac>] ? iget_locked+0x4c/0x140
[ 722.610789] [<ffffffff8119a590>] ? fh_compose+0x4c0/0x4c0
[ 722.610795] [<ffffffff8113f6e0>] ? ext4_llseek+0x110/0x110
[ 722.610803] [<ffffffff810c24e9>] ? do_sync_readv_writev+0xa9/0xf0
[ 722.610811] [<ffffffff810c231b>] ? rw_copy_check_uvector+0x7b/0x140
[ 722.610819] [<ffffffff810c2bcf>] ? do_readv_writev+0xdf/0x210
[ 722.610825] [<ffffffff810b8735>] ? alloc_debug_processing+0x145/0x190
[ 722.610831] [<ffffffff810b9bc5>] ? kmem_cache_alloc+0xa5/0xb0
[ 722.610838] [<ffffffff8113f463>] ? ext4_file_open+0x63/0x180
[ 722.610844] [<ffffffff8119c21d>] ? nfsd_vfs_write+0xed/0x3a0
[ 722.610851] [<ffffffff810c1127>] ? __dentry_open+0x1f7/0x2b0
[ 722.610857] [<ffffffff8119c892>] ? nfsd_open+0xf2/0x1b0
[ 722.610862] [<ffffffff8119cd34>] ? nfsd_write+0xf4/0x110
[ 722.610868] [<ffffffff81199930>] ? nfsd_proc_write+0xb0/0x120
[ 722.610876] [<ffffffff811971c5>] ? nfsd_dispatch+0xf5/0x230
[ 722.610882] [<ffffffff813b456f>] ? svc_process+0x4af/0x820
[ 722.610887] [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[ 722.610894] [<ffffffff811977a0>] ? nfsd_svc+0x1b0/0x1b0
[ 722.610901] [<ffffffff8119784d>] ? nfsd+0xad/0x150
[ 722.610907] [<ffffffff810525b6>] ? kthread+0x96/0xa0
[ 722.610913] [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[ 722.610921] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[ 722.610926] [<ffffffff813d8290>] ? gs_change+0xb/0xb
[ 765.241265]
=============================================================================
[ 765.241267] BUG kmalloc-512: Object padding overwritten
[ 765.241268]
-----------------------------------------------------------------------------
[ 765.241269]
[ 765.241271] INFO: 0xffff8804168c1486-0xffff8804168c1487. First byte
0x0 instead of 0x5a
[ 765.241277] INFO: Allocated in bio_kmalloc+0x2d/0x70 age=73010 cpu=5
pid=2495
[ 765.241281] INFO: Freed in r10buf_pool_free+0x71/0x90 age=72999 cpu=5
pid=1728
[ 765.241283] INFO: Slab 0xffffea000e4eea00 objects=28 used=13
fp=0xffff8804168c1240 flags=0x80000000000040c1
[ 765.241284] INFO: Object 0xffff8804168c1240 @offset=4672
fp=0xffff8804168c0ff8
[ 765.241285]
[ 765.241286] Bytes b4 0xffff8804168c1230: 95 98 ff ff 00 00 00 00 5a
5a 5a 5a 5a 5a 5a 5a ..ÿÿ....ZZZZZZZZ
[ 765.241291] Object 0xffff8804168c1240: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241294] Object 0xffff8804168c1250: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241298] Object 0xffff8804168c1260: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241301] Object 0xffff8804168c1270: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241305] Object 0xffff8804168c1280: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241308] Object 0xffff8804168c1290: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241312] Object 0xffff8804168c12a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241315] Object 0xffff8804168c12b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241319] Object 0xffff8804168c12c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241322] Object 0xffff8804168c12d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241326] Object 0xffff8804168c12e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241329] Object 0xffff8804168c12f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241332] Object 0xffff8804168c1300: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241336] Object 0xffff8804168c1310: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241339] Object 0xffff8804168c1320: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241343] Object 0xffff8804168c1330: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241346] Object 0xffff8804168c1340: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241350] Object 0xffff8804168c1350: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241353] Object 0xffff8804168c1360: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241357] Object 0xffff8804168c1370: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241360] Object 0xffff8804168c1380: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241364] Object 0xffff8804168c1390: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241367] Object 0xffff8804168c13a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241371] Object 0xffff8804168c13b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241374] Object 0xffff8804168c13c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241378] Object 0xffff8804168c13d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241381] Object 0xffff8804168c13e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241385] Object 0xffff8804168c13f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241388] Object 0xffff8804168c1400: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241392] Object 0xffff8804168c1410: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241395] Object 0xffff8804168c1420: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241399] Object 0xffff8804168c1430: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[ 765.241402] Redzone 0xffff8804168c1440: bb bb bb bb bb bb bb bb
»»»»»»»»
[ 765.241406] Padding 0xffff8804168c1480: 5a 5a 5a 5a 5a 5a 00 00
ZZZZZZ..
[ 765.241410] Pid: 7147, comm: apcaccess Not tainted 2.6.39 #2
[ 765.241411] Call Trace:
[ 765.241416] [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[ 765.241419] [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[ 765.241421] [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[ 765.241423] [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[ 765.241424] [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[ 765.241426] [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[ 765.241428] [<ffffffff810ba59d>] ? __kmalloc+0x10d/0x160
[ 765.241429] [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[ 765.241432] [<ffffffff8109bd9d>] ? __get_user_pages+0x17d/0x530
[ 765.241434] [<ffffffff810c9556>] ? get_arg_page+0x56/0x100
[ 765.241437] [<ffffffff811da4ad>] ? strnlen_user+0x2d/0x80
[ 765.241439] [<ffffffff810c8070>] ? search_binary_handler+0x90/0x240
[ 765.241440] [<ffffffff810c9e9f>] ? do_execve+0x22f/0x2f0
[ 765.241443] [<ffffffff810094a6>] ? sys_execve+0x36/0x60
[ 765.241446] [<ffffffff813d78dc>] ? stub_execve+0x6c/0xc0
[ 765.241448] FIX kmalloc-512: Restoring
0xffff8804168c1486-0xffff8804168c1487=0x5a
[ 765.241449]
[ 789.536436]
=============================================================================
[ 789.536444] BUG kmalloc-2048: Object padding overwritten
[ 789.536448]
-----------------------------------------------------------------------------
[ 789.536451]
[ 789.536456] INFO: 0xffff8802c8b1a11e-0xffff8802c8b1a11f. First byte
0x0 instead of 0x5a
[ 789.536471] INFO: Allocated in sk_stream_alloc_skb+0x3a/0x110 age=63
cpu=3 pid=5169
[ 789.536482] INFO: Freed in __kfree_skb+0x11/0x90 age=63 cpu=5 pid=0
[ 789.536488] INFO: Slab 0xffffea0009be6d40 objects=15 used=4
fp=0xffff8802c8b198d8 flags=0x80000000000040c1
[ 789.536494] INFO: Object 0xffff8802c8b198d8 @offset=6360
fp=0xffff8802c8b1a968
[ 789.536497]
[ 789.536501] Bytes b4 0xffff8802c8b198c8: cc be 00 00 01 00 00 00 5a
5a 5a 5a 5a 5a 5a 5a ÌŸ......ZZZZZZZZ
[ 789.536517] Object 0xffff8802c8b198d8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536532] Object 0xffff8802c8b198e8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536546] Object 0xffff8802c8b198f8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536560] Object 0xffff8802c8b19908: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536575] Object 0xffff8802c8b19918: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536589] Object 0xffff8802c8b19928: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536603] Object 0xffff8802c8b19938: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536617] Object 0xffff8802c8b19948: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536631] Object 0xffff8802c8b19958: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536645] Object 0xffff8802c8b19968: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536660] Object 0xffff8802c8b19978: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536674] Object 0xffff8802c8b19988: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536688] Object 0xffff8802c8b19998: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536702] Object 0xffff8802c8b199a8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536716] Object 0xffff8802c8b199b8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536730] Object 0xffff8802c8b199c8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536745] Object 0xffff8802c8b199d8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536759] Object 0xffff8802c8b199e8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536773] Object 0xffff8802c8b199f8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536787] Object 0xffff8802c8b19a08: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536801] Object 0xffff8802c8b19a18: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536816] Object 0xffff8802c8b19a28: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536830] Object 0xffff8802c8b19a38: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536844] Object 0xffff8802c8b19a48: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536858] Object 0xffff8802c8b19a58: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536872] Object 0xffff8802c8b19a68: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536887] Object 0xffff8802c8b19a78: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536901] Object 0xffff8802c8b19a88: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536915] Object 0xffff8802c8b19a98: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536929] Object 0xffff8802c8b19aa8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536943] Object 0xffff8802c8b19ab8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536958] Object 0xffff8802c8b19ac8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536972] Object 0xffff8802c8b19ad8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536986] Object 0xffff8802c8b19ae8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537000] Object 0xffff8802c8b19af8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537014] Object 0xffff8802c8b19b08: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537029] Object 0xffff8802c8b19b18: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537043] Object 0xffff8802c8b19b28: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537057] Object 0xffff8802c8b19b38: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537071] Object 0xffff8802c8b19b48: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537085] Object 0xffff8802c8b19b58: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537100] Object 0xffff8802c8b19b68: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537114] Object 0xffff8802c8b19b78: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537128] Object 0xffff8802c8b19b88: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537143] Object 0xffff8802c8b19b98: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537158] Object 0xffff8802c8b19ba8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537172] Object 0xffff8802c8b19bb8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537187] Object 0xffff8802c8b19bc8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537201] Object 0xffff8802c8b19bd8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537215] Object 0xffff8802c8b19be8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537229] Object 0xffff8802c8b19bf8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537244] Object 0xffff8802c8b19c08: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537258] Object 0xffff8802c8b19c18: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537272] Object 0xffff8802c8b19c28: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537286] Object 0xffff8802c8b19c38: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537301] Object 0xffff8802c8b19c48: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537315] Object 0xffff8802c8b19c58: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537329] Object 0xffff8802c8b19c68: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537343] Object 0xffff8802c8b19c78: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537357] Object 0xffff8802c8b19c88: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537372] Object 0xffff8802c8b19c98: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537386] Object 0xffff8802c8b19ca8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537400] Object 0xffff8802c8b19cb8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537414] Object 0xffff8802c8b19cc8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537428] Object 0xffff8802c8b19cd8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537443] Object 0xffff8802c8b19ce8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537457] Object 0xffff8802c8b19cf8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537471] Object 0xffff8802c8b19d08: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537485] Object 0xffff8802c8b19d18: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537500] Object 0xffff8802c8b19d28: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537514] Object 0xffff8802c8b19d38: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537528] Object 0xffff8802c8b19d48: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537542] Object 0xffff8802c8b19d58: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537557] Object 0xffff8802c8b19d68: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537571] Object 0xffff8802c8b19d78: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537585] Object 0xffff8802c8b19d88: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537599] Object 0xffff8802c8b19d98: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537613] Object 0xffff8802c8b19da8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537628] Object 0xffff8802c8b19db8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537642] Object 0xffff8802c8b19dc8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537656] Object 0xffff8802c8b19dd8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537670] Object 0xffff8802c8b19de8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537685] Object 0xffff8802c8b19df8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537699] Object 0xffff8802c8b19e08: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537713] Object 0xffff8802c8b19e18: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537727] Object 0xffff8802c8b19e28: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537742] Object 0xffff8802c8b19e38: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537756] Object 0xffff8802c8b19e48: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537770] Object 0xffff8802c8b19e58: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537784] Object 0xffff8802c8b19e68: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537799] Object 0xffff8802c8b19e78: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537813] Object 0xffff8802c8b19e88: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537827] Object 0xffff8802c8b19e98: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537841] Object 0xffff8802c8b19ea8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537856] Object 0xffff8802c8b19eb8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537870] Object 0xffff8802c8b19ec8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537884] Object 0xffff8802c8b19ed8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537898] Object 0xffff8802c8b19ee8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537912] Object 0xffff8802c8b19ef8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537927] Object 0xffff8802c8b19f08: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537941] Object 0xffff8802c8b19f18: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537955] Object 0xffff8802c8b19f28: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537969] Object 0xffff8802c8b19f38: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537984] Object 0xffff8802c8b19f48: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537998] Object 0xffff8802c8b19f58: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538012] Object 0xffff8802c8b19f68: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538026] Object 0xffff8802c8b19f78: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538041] Object 0xffff8802c8b19f88: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538055] Object 0xffff8802c8b19f98: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538069] Object 0xffff8802c8b19fa8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538083] Object 0xffff8802c8b19fb8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538097] Object 0xffff8802c8b19fc8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538112] Object 0xffff8802c8b19fd8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538126] Object 0xffff8802c8b19fe8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538140] Object 0xffff8802c8b19ff8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538154] Object 0xffff8802c8b1a008: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538169] Object 0xffff8802c8b1a018: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538183] Object 0xffff8802c8b1a028: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538197] Object 0xffff8802c8b1a038: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538211] Object 0xffff8802c8b1a048: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538226] Object 0xffff8802c8b1a058: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538240] Object 0xffff8802c8b1a068: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538254] Object 0xffff8802c8b1a078: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538268] Object 0xffff8802c8b1a088: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538283] Object 0xffff8802c8b1a098: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538297] Object 0xffff8802c8b1a0a8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538311] Object 0xffff8802c8b1a0b8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538325] Object 0xffff8802c8b1a0c8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[ 789.538340] Redzone 0xffff8802c8b1a0d8: bb bb bb bb bb bb bb bb
»»»»»»»»
[ 789.538353] Padding 0xffff8802c8b1a118: 5a 5a 5a 5a 5a 5a 00 00
ZZZZZZ..
[ 789.538369] Pid: 5247, comm: qemu Not tainted 2.6.39 #2
[ 789.538373] Call Trace:
[ 789.538385] [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[ 789.538394] [<ffffffff81314260>] ? sock_alloc_send_pskb+0x1d0/0x320
[ 789.538400] [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[ 789.538407] [<ffffffff81314260>] ? sock_alloc_send_pskb+0x1d0/0x320
[ 789.538414] [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[ 789.538420] [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[ 789.538427] [<ffffffff810bb70d>] ? __kmalloc_track_caller+0x10d/0x160
[ 789.538434] [<ffffffff813198c2>] ? __alloc_skb+0x72/0x160
[ 789.538441] [<ffffffff81314260>] ? sock_alloc_send_pskb+0x1d0/0x320
[ 789.538450] [<ffffffff8105af7b>] ? getnstimeofday+0x4b/0xd0
[ 789.538457] [<ffffffff81320d32>] ? netif_rx+0xb2/0x120
[ 789.538465] [<ffffffff812b393e>] ? tun_chr_aio_write+0x18e/0x4e0
[ 789.538472] [<ffffffff812b37b0>] ? tun_sendmsg+0x4d0/0x4d0
[ 789.538481] [<ffffffff810c24e9>] ? do_sync_readv_writev+0xa9/0xf0
[ 789.538489] [<ffffffff810c231b>] ? rw_copy_check_uvector+0x7b/0x140
[ 789.538496] [<ffffffff810c2bcf>] ? do_readv_writev+0xdf/0x210
[ 789.538504] [<ffffffff810c2e7e>] ? sys_writev+0x4e/0xc0
[ 789.538513] [<ffffffff813d753b>] ? system_call_fastpath+0x16/0x1b
[ 789.538519] FIX kmalloc-2048: Restoring
0xffff8802c8b1a11e-0xffff8802c8b1a11f=0x5a
[ 789.538522]
[ 825.601291]
=============================================================================
[ 825.601298] BUG kmalloc-512: Object padding overwritten
[ 825.601302]
-----------------------------------------------------------------------------
[ 825.601306]
[ 825.601311] INFO: 0xffff88041411efe6-0xffff88041411efe7. First byte
0x0 instead of 0x5a
[ 825.601327] INFO: Allocated in sock_alloc_send_pskb+0x1d0/0x320
age=36742 cpu=3 pid=4846
[ 825.601338] INFO: Freed in __kfree_skb+0x11/0x90 age=36742 cpu=3 pid=3781
[ 825.601344] INFO: Slab 0xffffea000e463e20 objects=28 used=11
fp=0xffff88041411eda0 flags=0x80000000000040c1
[ 825.601350] INFO: Object 0xffff88041411eda0 @offset=11680
fp=0xffff88041411cdb0
[ 825.601353]
[ 825.601357] Bytes b4 0xffff88041411ed90: e2 96 ff ff 00 00 00 00 5a
5a 5a 5a 5a 5a 5a 5a â.ÿÿ....ZZZZZZZZ
[ 825.601373] Object 0xffff88041411eda0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601388] Object 0xffff88041411edb0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601402] Object 0xffff88041411edc0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601416] Object 0xffff88041411edd0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601430] Object 0xffff88041411ede0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601444] Object 0xffff88041411edf0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601459] Object 0xffff88041411ee00: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601473] Object 0xffff88041411ee10: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601487] Object 0xffff88041411ee20: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601501] Object 0xffff88041411ee30: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601515] Object 0xffff88041411ee40: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601529] Object 0xffff88041411ee50: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601543] Object 0xffff88041411ee60: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601557] Object 0xffff88041411ee70: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601571] Object 0xffff88041411ee80: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601586] Object 0xffff88041411ee90: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601600] Object 0xffff88041411eea0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601614] Object 0xffff88041411eeb0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601628] Object 0xffff88041411eec0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601642] Object 0xffff88041411eed0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601656] Object 0xffff88041411eee0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601670] Object 0xffff88041411eef0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601684] Object 0xffff88041411ef00: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601699] Object 0xffff88041411ef10: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601713] Object 0xffff88041411ef20: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601727] Object 0xffff88041411ef30: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601741] Object 0xffff88041411ef40: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601755] Object 0xffff88041411ef50: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601769] Object 0xffff88041411ef60: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601783] Object 0xffff88041411ef70: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601797] Object 0xffff88041411ef80: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601812] Object 0xffff88041411ef90: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[ 825.601826] Redzone 0xffff88041411efa0: bb bb bb bb bb bb bb bb
»»»»»»»»
[ 825.601839] Padding 0xffff88041411efe0: 5a 5a 5a 5a 5a 5a 00 00
ZZZZZZ..
[ 825.601855] Pid: 7353, comm: apcaccess Not tainted 2.6.39 #2
[ 825.601859] Call Trace:
[ 825.601871] [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[ 825.601881] [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[ 825.601887] [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[ 825.601895] [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[ 825.601902] [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[ 825.601908] [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[ 825.601914] [<ffffffff810ba59d>] ? __kmalloc+0x10d/0x160
[ 825.601922] [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[ 825.601930] [<ffffffff8109bd9d>] ? __get_user_pages+0x17d/0x530
[ 825.601938] [<ffffffff810c9556>] ? get_arg_page+0x56/0x100
[ 825.601945] [<ffffffff810c8070>] ? search_binary_handler+0x90/0x240
[ 825.601951] [<ffffffff810c9e9f>] ? do_execve+0x22f/0x2f0
[ 825.601959] [<ffffffff810094a6>] ? sys_execve+0x36/0x60
[ 825.601968] [<ffffffff813d78dc>] ? stub_execve+0x6c/0xc0
[ 825.601975] FIX kmalloc-512: Restoring
0xffff88041411efe6-0xffff88041411efe7=0x5a
[ 825.601978]
[ 842.610057] INFO: task ksmd:552 blocked for more than 120 seconds.
[ 842.610063] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 842.610069] ksmd D 0000000000000000 0 552 2
0x00000000
[ 842.610078] ffff88041db68ca0 0000000000000046 000212d01fc11d00
ffff880400000000
[ 842.610086] ffffffff81593020 ffff88041c1a9fd8 0000000000004000
ffff88041c1a8010
[ 842.610093] ffff88041c1a9fd8 ffff88041db68ca0 0000000000000000
0000000200000002
[ 842.610100] Call Trace:
[ 842.610116] [<ffffffff813d5595>] ? schedule_timeout+0x1c5/0x230
[ 842.610128] [<ffffffff8102f69c>] ? enqueue_task_fair+0x14c/0x190
[ 842.610137] [<ffffffff8102db27>] ? task_rq_lock+0x47/0x90
[ 842.610143] [<ffffffff813d4562>] ? wait_for_common+0xd2/0x180
[ 842.610150] [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[ 842.610157] [<ffffffff8104cb94>] ? flush_work+0x24/0x30
[ 842.610163] [<ffffffff8104be10>] ? do_work_for_cpu+0x20/0x20
[ 842.610170] [<ffffffff8104e0ab>] ? schedule_on_each_cpu+0xab/0xe0
[ 842.610177] [<ffffffff810b5c05>] ? ksm_scan_thread+0x7f5/0xc20
[ 842.610184] [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[ 842.610190] [<ffffffff810b5410>] ?
try_to_merge_with_ksm_page+0x570/0x570
[ 842.610196] [<ffffffff810b5410>] ?
try_to_merge_with_ksm_page+0x570/0x570
[ 842.610203] [<ffffffff810525b6>] ? kthread+0x96/0xa0
[ 842.610210] [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[ 842.610218] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[ 842.610223] [<ffffffff813d8290>] ? gs_change+0xb/0xb
[ 842.610229] INFO: task fsnotify_mark:662 blocked for more than 120
seconds.
[ 842.610233] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 842.610237] fsnotify_mark D 0000000000000000 0 662 2
0x00000000
[ 842.610244] ffff88041c31d860 0000000000000046 0000000000000000
0000000000000000
[ 842.610251] ffffffff81593020 ffff88041c365fd8 0000000000004000
ffff88041c364010
[ 842.610257] ffff88041c365fd8 ffff88041c31d860 0000000000000000
0000000000000000
[ 842.610263] Call Trace:
[ 842.610270] [<ffffffff81035521>] ? load_balance+0x91/0x5e0
[ 842.610278] [<ffffffff813d5595>] ? schedule_timeout+0x1c5/0x230
[ 842.610285] [<ffffffff8102f323>] ? pick_next_task_fair+0x103/0x190
[ 842.610291] [<ffffffff813d4a6d>] ? schedule+0x28d/0x910
[ 842.610297] [<ffffffff813d4562>] ? wait_for_common+0xd2/0x180
[ 842.610303] [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[ 842.610310] [<ffffffff81074420>] ? synchronize_rcu_bh+0x50/0x50
[ 842.610316] [<ffffffff8107446a>] ? synchronize_sched+0x4a/0x50
[ 842.610322] [<ffffffff8104f940>] ? find_ge_pid+0x40/0x40
[ 842.610329] [<ffffffff8105742b>] ? __synchronize_srcu+0x5b/0xc0
[ 842.610338] [<ffffffff810f5f63>] ? fsnotify_mark_destroy+0x83/0x150
[ 842.610344] [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[ 842.610352] [<ffffffff810f5ee0>] ?
fsnotify_set_mark_ignored_mask_locked+0x20/0x20
[ 842.610360] [<ffffffff810f5ee0>] ?
fsnotify_set_mark_ignored_mask_locked+0x20/0x20
[ 842.610367] [<ffffffff810525b6>] ? kthread+0x96/0xa0
[ 842.610373] [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[ 842.610381] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[ 842.610386] [<ffffffff813d8290>] ? gs_change+0xb/0xb
[ 842.610395] INFO: task jbd2/md0-8:2528 blocked for more than 120 seconds.
[ 842.610399] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 842.610403] jbd2/md0-8 D 0000000000000005 0 2528 2
0x00000000
[ 842.610410] ffff88041d91c570 0000000000000046 ffff88041baeea28
ffffea0000000000
[ 842.610417] ffff88041d91a5e0 ffff880419f9bfd8 0000000000004000
ffff880419f9a010
[ 842.610423] ffff880419f9bfd8 ffff88041d91c570 ffff88041baeea28
ffffffff810b8495
[ 842.610429] Call Trace:
[ 842.610435] [<ffffffff810b8495>] ? init_object+0x85/0xa0
[ 842.610442] [<ffffffff810b8916>] ? free_debug_processing+0x196/0x250
[ 842.610450] [<ffffffff8105adae>] ? ktime_get_ts+0x6e/0xf0
[ 842.610458] [<ffffffff810810f0>] ? __lock_page+0x70/0x70
[ 842.610464] [<ffffffff813d5174>] ? io_schedule+0x84/0xd0
[ 842.610473] [<ffffffff811d4403>] ?
radix_tree_gang_lookup_tag_slot+0x93/0xf0
[ 842.610480] [<ffffffff810810f9>] ? sleep_on_page+0x9/0x10
[ 842.610486] [<ffffffff813d57df>] ? __wait_on_bit+0x4f/0x80
[ 842.610492] [<ffffffff810812eb>] ? wait_on_page_bit+0x6b/0x80
[ 842.610499] [<ffffffff81052a50>] ? autoremove_wake_function+0x30/0x30
[ 842.610507] [<ffffffff8108a458>] ? pagevec_lookup_tag+0x18/0x20
[ 842.610512] [<ffffffff81081f2a>] ? filemap_fdatawait_range+0xfa/0x180
[ 842.610521] [<ffffffff811be09f>] ? submit_bio+0x6f/0xf0
[ 842.610530] [<ffffffff81176276>] ?
jbd2_journal_commit_transaction+0x796/0x1270
[ 842.610539] [<ffffffff81179ed1>] ? kjournald2+0xb1/0x1e0
[ 842.610546] [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[ 842.610553] [<ffffffff81179e20>] ? commit_timeout+0x10/0x10
[ 842.610560] [<ffffffff81179e20>] ? commit_timeout+0x10/0x10
[ 842.610566] [<ffffffff810525b6>] ? kthread+0x96/0xa0
[ 842.610572] [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[ 842.610580] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[ 842.610585] [<ffffffff813d8290>] ? gs_change+0xb/0xb
[ 842.610594] INFO: task nfsd:4326 blocked for more than 120 seconds.
[ 842.610598] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 842.610602] nfsd D 0000000000000000 0 4326 2
0x00000000
[ 842.610608] ffff88041b604bc0 0000000000000046 0000000000000016
ffffffff00000000
[ 842.610614] ffff88041d8fa5e0 ffff88041c527fd8 0000000000004000
ffff88041c526010
[ 842.610621] ffff88041c527fd8 ffff88041b604bc0 000000000000009c
ffff880000000000
[ 842.610626] Call Trace:
[ 842.610634] [<ffffffff8101e845>] ? amd_flush_garts+0x105/0x140
[ 842.610640] [<ffffffff8101fa80>] ? gart_map_sg+0x480/0x480
[ 842.610646] [<ffffffff8101f5d3>] ? flush_gart+0x23/0x50
[ 842.610653] [<ffffffff81080f98>] ? find_get_page+0x18/0x90
[ 842.610660] [<ffffffff81174b95>] ? do_get_write_access+0x265/0x4a0
[ 842.610668] [<ffffffff81052a50>] ? autoremove_wake_function+0x30/0x30
[ 842.610675] [<ffffffff81174ef9>] ?
jbd2_journal_get_write_access+0x29/0x50
[ 842.610684] [<ffffffff8115f122>] ?
__ext4_journal_get_write_access+0x32/0x80
[ 842.610692] [<ffffffff81143908>] ? ext4_reserve_inode_write+0x78/0xa0
[ 842.610700] [<ffffffff81143970>] ? ext4_mark_inode_dirty+0x40/0x1e0
[ 842.610706] [<ffffffff81156c0b>] ? ext4_journal_start_sb+0x6b/0x160
[ 842.610715] [<ffffffff81322645>] ? dev_hard_start_xmit+0x305/0x5f0
[ 842.610723] [<ffffffff81352fd0>] ? ip_finish_output2+0x290/0x290
[ 842.610730] [<ffffffff81143c65>] ? ext4_dirty_inode+0x35/0x70
[ 842.610738] [<ffffffff810e4a08>] ? __mark_inode_dirty+0x38/0x210
[ 842.610745] [<ffffffff810d9317>] ? file_update_time+0xf7/0x180
[ 842.610751] [<ffffffff81082458>] ? __generic_file_aio_write+0x1f8/0x430
[ 842.610760] [<ffffffff81373ef9>] ? udp_sendmsg+0x3c9/0x7e0
[ 842.610767] [<ffffffff81314260>] ? sock_alloc_send_pskb+0x1d0/0x320
[ 842.610774] [<ffffffff81082703>] ? generic_file_aio_write+0x73/0xf0
[ 842.610781] [<ffffffff8113f74e>] ? ext4_file_write+0x6e/0x2b0
[ 842.610788] [<ffffffff810da8ac>] ? iget_locked+0x4c/0x140
[ 842.610794] [<ffffffff8119a590>] ? fh_compose+0x4c0/0x4c0
[ 842.610800] [<ffffffff8113f6e0>] ? ext4_llseek+0x110/0x110
[ 842.610808] [<ffffffff810c24e9>] ? do_sync_readv_writev+0xa9/0xf0
[ 842.610816] [<ffffffff810c231b>] ? rw_copy_check_uvector+0x7b/0x140
[ 842.610824] [<ffffffff810c2bcf>] ? do_readv_writev+0xdf/0x210
[ 842.610830] [<ffffffff810b8735>] ? alloc_debug_processing+0x145/0x190
[ 842.610836] [<ffffffff810b9bc5>] ? kmem_cache_alloc+0xa5/0xb0
[ 842.610843] [<ffffffff8113f463>] ? ext4_file_open+0x63/0x180
[ 842.610849] [<ffffffff8119c21d>] ? nfsd_vfs_write+0xed/0x3a0
[ 842.610856] [<ffffffff810c1127>] ? __dentry_open+0x1f7/0x2b0
[ 842.610862] [<ffffffff8119c892>] ? nfsd_open+0xf2/0x1b0
[ 842.610867] [<ffffffff8119cd34>] ? nfsd_write+0xf4/0x110
[ 842.610873] [<ffffffff81199930>] ? nfsd_proc_write+0xb0/0x120
[ 842.610880] [<ffffffff811971c5>] ? nfsd_dispatch+0xf5/0x230
[ 842.610886] [<ffffffff813b456f>] ? svc_process+0x4af/0x820
[ 842.610892] [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[ 842.610899] [<ffffffff811977a0>] ? nfsd_svc+0x1b0/0x1b0
[ 842.610906] [<ffffffff8119784d>] ? nfsd+0xad/0x150
[ 842.610912] [<ffffffff810525b6>] ? kthread+0x96/0xa0
[ 842.610918] [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[ 842.610926] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[ 842.610931] [<ffffffff813d8290>] ? gs_change+0xb/0xb
[ 854.241450]
=============================================================================
[ 854.241459] BUG kmalloc-1024: Object padding overwritten
[ 854.241463]
-----------------------------------------------------------------------------
[ 854.241466]
[ 854.241471] INFO: 0xffff8802c86059e6-0xffff8802c86059e7. First byte
0x0 instead of 0x5a
[ 854.241488] INFO: Allocated in __blockdev_direct_IO+0x16c/0xa90
age=518 cpu=5 pid=7094
[ 854.241500] INFO: Freed in __kfree_skb+0x11/0x90 age=519 cpu=5 pid=5247
[ 854.241507] INFO: Slab 0xffffea0009bd5000 objects=29 used=22
fp=0xffff8802c8606f50 flags=0x80000000000040c1
[ 854.241513] INFO: Object 0xffff8802c86055a0 @offset=21920
fp=0xffff8802c86066c0
[ 854.241516]
[ 854.241520] Bytes b4 0xffff8802c8605590: 00 00 00 00 00 00 00 00 5a
5a 5a 5a 5a 5a 5a 5a ........ZZZZZZZZ
[ 854.241536] Object 0xffff8802c86055a0: 00 00 00 00 00 00 00 00 e0
61 29 fa 03 88 ff ff ........àa)ú..ÿÿ
[ 854.241551] Object 0xffff8802c86055b0: 00 00 00 00 00 00 00 00 00
00 41 ce 04 00 00 00 ..........AÎ....
[ 854.241565] Object 0xffff8802c86055c0: 03 00 00 00 0c 00 00 00 00
00 00 00 01 00 00 00 ................
[ 854.241579] Object 0xffff8802c86055d0: 00 00 00 00 00 00 00 00 00
60 00 00 00 00 00 00 .........`......
[ 854.241593] Object 0xffff8802c86055e0: b7 2c 0a 00 00 00 00 00 00
00 00 00 00 00 00 00 ·,..............
[ 854.241607] Object 0xffff8802c86055f0: b7 2c 0a 00 00 00 00 00 00
00 00 00 00 00 00 00 ·,..............
[ 854.241621] Object 0xffff8802c8605600: 01 00 00 00 00 00 00 00 f0
77 14 81 ff ff ff ff ........ðw..ÿÿÿÿ
[ 854.241635] Object 0xffff8802c8605610: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 854.241649] Object 0xffff8802c8605620: 00 00 00 00 00 00 00 00 b7
54 3b 02 00 00 00 00 ........·T;.....
[ 854.241663] Object 0xffff8802c8605630: b7 54 3b 02 00 00 00 00 20
00 00 00 00 00 00 00 ·T;.............
[ 854.241677] Object 0xffff8802c8605640: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 854.241691] Object 0xffff8802c8605650: b6 54 3b 02 00 00 00 00 00
10 00 00 00 00 00 00 ¶T;.............
[ 854.241705] Object 0xffff8802c8605660: 00 00 00 00 00 00 00 00 00
80 8e 1a 04 88 ff ff ..............ÿÿ
[ 854.241719] Object 0xffff8802c8605670: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 854.241733] Object 0xffff8802c8605680: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 854.241747] Object 0xffff8802c8605690: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 854.241761] Object 0xffff8802c86056a0: 00 00 00 00 00 00 00 00 00
00 00 00 00 10 00 00 ................
[ 854.241775] Object 0xffff8802c86056b0: b6 54 3b 02 00 00 00 00 00
60 cb a2 00 00 00 00 ¶T;......`Ë¢....
[ 854.241789] Object 0xffff8802c86056c0: 06 06 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 854.241803] Object 0xffff8802c86056d0: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 854.241817] Object 0xffff8802c86056e0: 68 fd 1c c9 02 88 ff ff 00
00 00 00 00 00 00 00 hý.É..ÿÿ........
[ 854.241831] Object 0xffff8802c86056f0: 00 60 00 00 00 00 00 00 01
00 00 00 01 00 00 00 .`..............
[ 854.241845] Object 0xffff8802c8605700: 00 80 c9 dd 39 7f 00 00 01
00 00 00 01 00 00 00 ..ÉÝ9...........
[ 854.241860] Object 0xffff8802c8605710: 00 00 00 00 00 00 00 00 08
81 bb 0c 00 ea ff ff ..........»..êÿÿ
[ 854.241874] Object 0xffff8802c8605720: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.241888] Object 0xffff8802c8605730: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.241902] Object 0xffff8802c8605740: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.241916] Object 0xffff8802c8605750: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.241930] Object 0xffff8802c8605760: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.241944] Object 0xffff8802c8605770: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.241959] Object 0xffff8802c8605780: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.241973] Object 0xffff8802c8605790: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.241987] Object 0xffff8802c86057a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242001] Object 0xffff8802c86057b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242015] Object 0xffff8802c86057c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242029] Object 0xffff8802c86057d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242043] Object 0xffff8802c86057e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242058] Object 0xffff8802c86057f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242072] Object 0xffff8802c8605800: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242086] Object 0xffff8802c8605810: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242100] Object 0xffff8802c8605820: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242114] Object 0xffff8802c8605830: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242128] Object 0xffff8802c8605840: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242142] Object 0xffff8802c8605850: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242157] Object 0xffff8802c8605860: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242171] Object 0xffff8802c8605870: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242185] Object 0xffff8802c8605880: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242199] Object 0xffff8802c8605890: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242213] Object 0xffff8802c86058a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242227] Object 0xffff8802c86058b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242241] Object 0xffff8802c86058c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242256] Object 0xffff8802c86058d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242270] Object 0xffff8802c86058e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242284] Object 0xffff8802c86058f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242298] Object 0xffff8802c8605900: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242312] Object 0xffff8802c8605910: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242326] Object 0xffff8802c8605920: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242340] Object 0xffff8802c8605930: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242355] Object 0xffff8802c8605940: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242369] Object 0xffff8802c8605950: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242383] Object 0xffff8802c8605960: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242397] Object 0xffff8802c8605970: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242411] Object 0xffff8802c8605980: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242425] Object 0xffff8802c8605990: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[ 854.242440] Redzone 0xffff8802c86059a0: cc cc cc cc cc cc cc cc
ÌÌÌÌÌÌÌÌ
[ 854.242453] Padding 0xffff8802c86059e0: 5a 5a 5a 5a 5a 5a 00 00
ZZZZZZ..
[ 854.242470] Pid: 7094, comm: qemu Not tainted 2.6.39 #2
[ 854.242474] Call Trace:
[ 854.242485] [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[ 854.242492] [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[ 854.242500] [<ffffffff810f3126>] ? __blockdev_direct_IO+0xa16/0xa90
[ 854.242507] [<ffffffff810b88b3>] ? free_debug_processing+0x133/0x250
[ 854.242513] [<ffffffff810b8afb>] ? __slab_free+0x12b/0x140
[ 854.242520] [<ffffffff810f3126>] ? __blockdev_direct_IO+0xa16/0xa90
[ 854.242530] [<ffffffff81144a17>] ? ext4_ind_direct_IO+0xf7/0x410
[ 854.242536] [<ffffffff811477f0>] ? noalloc_get_block_write+0x30/0x30
[ 854.242543] [<ffffffff81082029>] ? __filemap_fdatawrite_range+0x49/0x50
[ 854.242550] [<ffffffff810830c3>] ? generic_file_aio_read+0x653/0x6d0
[ 854.242558] [<ffffffff810567a2>] ? hrtimer_cancel+0x12/0x20
[ 854.242565] [<ffffffff81062137>] ? futex_wait+0x197/0x240
[ 854.242572] [<ffffffff81082a70>] ? delete_from_page_cache+0x70/0x70
[ 854.242580] [<ffffffff810c24e9>] ? do_sync_readv_writev+0xa9/0xf0
[ 854.242587] [<ffffffff8106239f>] ? futex_wake+0x10f/0x120
[ 854.242594] [<ffffffff81063f7b>] ? do_futex+0x11b/0xa70
[ 854.242601] [<ffffffff810471cc>] ? T.680+0x13c/0x280
[ 854.242608] [<ffffffff810c231b>] ? rw_copy_check_uvector+0x7b/0x140
[ 854.242616] [<ffffffff810c2bcf>] ? do_readv_writev+0xdf/0x210
[ 854.242624] [<ffffffff810c3003>] ? sys_preadv+0xc3/0xd0
[ 854.242633] [<ffffffff813d753b>] ? system_call_fastpath+0x16/0x1b
[ 854.242640] FIX kmalloc-1024: Restoring
0xffff8802c86059e6-0xffff8802c86059e7=0x5a
[ 854.242643]
[ 860.203918]
=============================================================================
[ 860.203927] BUG kmalloc-1024: Object padding overwritten
[ 860.203931]
-----------------------------------------------------------------------------
[ 860.203934]
[ 860.203939] INFO: 0xffff88041ad15156-0xffff88041ad15157. First byte
0x0 instead of 0x5a
[ 860.203955] INFO: Allocated in __blockdev_direct_IO+0x16c/0xa90
age=1609 cpu=3 pid=7094
[ 860.203965] INFO: Freed in __blockdev_direct_IO+0xa16/0xa90 age=1609
cpu=3 pid=7094
[ 860.203972] INFO: Slab 0xffffea000e5ddb80 objects=29 used=25
fp=0xffff88041ad14d10 flags=0x80000000000040c1
[ 860.203977] INFO: Object 0xffff88041ad14d10 @offset=19728
fp=0xffff88041ad10000
[ 860.203981]
[ 860.203984] Bytes b4 0xffff88041ad14d00: 62 28 00 00 01 00 00 00 5a
5a 5a 5a 5a 5a 5a 5a b(......ZZZZZZZZ
[ 860.204000] Object 0xffff88041ad14d10: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204015] Object 0xffff88041ad14d20: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204030] Object 0xffff88041ad14d30: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204044] Object 0xffff88041ad14d40: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204058] Object 0xffff88041ad14d50: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204073] Object 0xffff88041ad14d60: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204087] Object 0xffff88041ad14d70: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204101] Object 0xffff88041ad14d80: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204116] Object 0xffff88041ad14d90: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204130] Object 0xffff88041ad14da0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204144] Object 0xffff88041ad14db0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204159] Object 0xffff88041ad14dc0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204173] Object 0xffff88041ad14dd0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204187] Object 0xffff88041ad14de0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204202] Object 0xffff88041ad14df0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204216] Object 0xffff88041ad14e00: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204230] Object 0xffff88041ad14e10: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204245] Object 0xffff88041ad14e20: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204259] Object 0xffff88041ad14e30: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204273] Object 0xffff88041ad14e40: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204288] Object 0xffff88041ad14e50: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204302] Object 0xffff88041ad14e60: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204316] Object 0xffff88041ad14e70: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204331] Object 0xffff88041ad14e80: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204345] Object 0xffff88041ad14e90: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204359] Object 0xffff88041ad14ea0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204374] Object 0xffff88041ad14eb0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204388] Object 0xffff88041ad14ec0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204402] Object 0xffff88041ad14ed0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204417] Object 0xffff88041ad14ee0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204431] Object 0xffff88041ad14ef0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204445] Object 0xffff88041ad14f00: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204460] Object 0xffff88041ad14f10: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204474] Object 0xffff88041ad14f20: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204488] Object 0xffff88041ad14f30: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204503] Object 0xffff88041ad14f40: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204517] Object 0xffff88041ad14f50: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204531] Object 0xffff88041ad14f60: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204546] Object 0xffff88041ad14f70: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204560] Object 0xffff88041ad14f80: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204574] Object 0xffff88041ad14f90: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204589] Object 0xffff88041ad14fa0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204603] Object 0xffff88041ad14fb0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204618] Object 0xffff88041ad14fc0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204632] Object 0xffff88041ad14fd0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204646] Object 0xffff88041ad14fe0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204661] Object 0xffff88041ad14ff0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204675] Object 0xffff88041ad15000: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204689] Object 0xffff88041ad15010: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204704] Object 0xffff88041ad15020: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204718] Object 0xffff88041ad15030: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204732] Object 0xffff88041ad15040: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204747] Object 0xffff88041ad15050: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204761] Object 0xffff88041ad15060: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204775] Object 0xffff88041ad15070: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204790] Object 0xffff88041ad15080: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204804] Object 0xffff88041ad15090: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204818] Object 0xffff88041ad150a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204833] Object 0xffff88041ad150b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204847] Object 0xffff88041ad150c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204861] Object 0xffff88041ad150d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204876] Object 0xffff88041ad150e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204890] Object 0xffff88041ad150f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204904] Object 0xffff88041ad15100: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[ 860.204919] Redzone 0xffff88041ad15110: bb bb bb bb bb bb bb bb
»»»»»»»»
[ 860.204932] Padding 0xffff88041ad15150: 5a 5a 5a 5a 5a 5a 00 00
ZZZZZZ..
[ 860.204949] Pid: 7100, comm: qemu Not tainted 2.6.39 #2
[ 860.204953] Call Trace:
[ 860.204964] [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[ 860.204973] [<ffffffff810f287c>] ? __blockdev_direct_IO+0x16c/0xa90
[ 860.204979] [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[ 860.204986] [<ffffffff810f287c>] ? __blockdev_direct_IO+0x16c/0xa90
[ 860.204993] [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[ 860.204999] [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[ 860.205006] [<ffffffff810b9bc5>] ? kmem_cache_alloc+0xa5/0xb0
[ 860.205013] [<ffffffff810f287c>] ? __blockdev_direct_IO+0x16c/0xa90
[ 860.205022] [<ffffffff8108a475>] ? pagevec_lookup+0x15/0x20
[ 860.205029] [<ffffffff8108b5b7>] ?
invalidate_inode_pages2_range+0x87/0x300
[ 860.205039] [<ffffffff81144e41>] ? ext4_direct_IO+0x111/0x1f0
[ 860.205045] [<ffffffff811477a0>] ? _ext4_get_block+0x160/0x160
[ 860.205052] [<ffffffff81144f20>] ? ext4_direct_IO+0x1f0/0x1f0
[ 860.205058] [<ffffffff81082182>] ? generic_file_direct_write+0xd2/0x1b0
[ 860.205064] [<ffffffff8108252b>] ? __generic_file_aio_write+0x2cb/0x430
[ 860.205071] [<ffffffff81082703>] ? generic_file_aio_write+0x73/0xf0
[ 860.205079] [<ffffffff8113f74e>] ? ext4_file_write+0x6e/0x2b0
[ 860.205087] [<ffffffff810c25ef>] ? do_sync_write+0xbf/0x100
[ 860.205095] [<ffffffff810471cc>] ? T.680+0x13c/0x280
[ 860.205103] [<ffffffff810485a8>] ? kill_pid_info+0x38/0x60
[ 860.205110] [<ffffffff81048854>] ? sys_kill+0x94/0x1b0
[ 860.205117] [<ffffffff810c3198>] ? vfs_write+0xc8/0x190
[ 860.205124] [<ffffffff810c32fb>] ? sys_pwrite64+0x9b/0xb0
[ 860.205134] [<ffffffff813d753b>] ? system_call_fastpath+0x16/0x1b
[ 860.205140] FIX kmalloc-1024: Restoring
0xffff88041ad15156-0xffff88041ad15157=0x5a
[ 860.205143]

2011-06-01 09:41:20

by Avi Kivity

[permalink] [raw]
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39

On 06/01/2011 12:29 PM, Brad Campbell wrote:
> On 01/06/11 14:56, Avi Kivity wrote:
>> On 06/01/2011 09:31 AM, Brad Campbell wrote:
>>> On 01/06/11 12:52, Hugh Dickins wrote:
>>>
>>>>
>>>> I guess Brad could try SLUB debugging, boot with slub_debug=P
>>>> for poisoning perhaps; though it might upset alignments and
>>>> drive the problem underground. Or see if the same happens
>>>> with SLAB instead of SLUB.
>>>
>>> Not much use I'm afraid.
>>> This is all I get in the log
>>>
>>> [ 3161.300073]
>>> =============================================================================
>>>
>>>
>>> [ 3161.300147] BUG kmalloc-512: Freechain corrupt
>>>
>>> The qemu process is then frozen, unkillable but reported in state "R"
>>>
>>> 13881 ? R 3:27 /usr/bin/qemu -S -M pc-0.13 -enable-kvm -m 1024 -smp
>>> 2,sockets=2,cores=1,threads=1 -nam
>>>
>>> The machine then progressively dies until it's frozen solid with no
>>> further error messages.
>>>
>>> I stupidly forgot to do an alt-sysrq-t prior to doing an alt-sysrq-b,
>>> but at least it responded to that.
>>>
>>> On the bright side I can reproduce it at will.
>>
>> Please try slub_debug=FZPU; that should point the finger (hopefully at
>> somebody else).
>>
>
> Well the first attempt locked the machine solid. No network, no console..
>
> I saw
> "=========================================================================="
>
> on the console.. nothing after that. Would not respond to sysrq-t or
> any other sysrq combination other than -b, which rebooted the box.
>
>
> No output on netconsole at all, I had to walk to the other building to
> look at the monitor and reboot it.
>
> The second attempt jammed netconsole again, but I managed to get this
> from an ssh session I already had established. The machine died a slow
> and horrible death, but remained interactive enough for me to reboot
> it with
>
> echo b > /proc/sysrq-trigger
>
> Nothing else worked.
>
>
> [ 413.756416] [<ffffffff81318f1c>] ? pskb_expand_head+0x15c/0x250
> [ 413.756424] [<ffffffff813a6c45>] ? nf_bridge_copy_header+0x145/0x160
> [ 413.756431] [<ffffffff8139f78d>] ? br_dev_queue_push_xmit+0x6d/0x80
> [ 413.756439] [<ffffffff813a55a0>] ? br_nf_post_routing+0x2a0/0x2f0
> [ 413.756447] [<ffffffff81346bc4>] ? nf_iterate+0x84/0xb0
> [ 413.756453] [<ffffffff8139f720>] ? br_flood_deliver+0x20/0x20
> [ 413.756459] [<ffffffff81346c64>] ? nf_hook_slow+0x74/0x120
> [ 413.756465] [<ffffffff8139f720>] ? br_flood_deliver+0x20/0x20
> [ 413.756472] [<ffffffff8139f7da>] ? br_forward_finish+0x3a/0x60
> [ 413.756479] [<ffffffff813a5758>] ? br_nf_forward_finish+0x168/0x170
> [ 413.756487] [<ffffffff813a5c90>] ? br_nf_forward_ip+0x360/0x3a0
> [ 413.756492] [<ffffffff81346bc4>] ? nf_iterate+0x84/0xb0
> [ 413.756498] [<ffffffff8139f7a0>] ? br_dev_queue_push_xmit+0x80/0x80
> [ 413.756504] [<ffffffff81346c64>] ? nf_hook_slow+0x74/0x120
> [ 413.756510] [<ffffffff8139f7a0>] ? br_dev_queue_push_xmit+0x80/0x80
> [ 413.756516] [<ffffffff8139f800>] ? br_forward_finish+0x60/0x60
> [ 413.756522] [<ffffffff8139f800>] ? br_forward_finish+0x60/0x60
> [ 413.756528] [<ffffffff8139f875>] ? __br_forward+0x75/0xc0
> [ 413.756534] [<ffffffff8139f426>] ? deliver_clone+0x36/0x60
> [ 413.756540] [<ffffffff8139f69d>] ? br_flood+0xbd/0x100
> [ 413.756546] [<ffffffff813a05b0>] ? br_handle_local_finish+0x40/0x40
> [ 413.756552] [<ffffffff813a080e>] ? br_handle_frame_finish+0x25e/0x280
> [ 413.756560] [<ffffffff813a60f0>] ?
> br_nf_pre_routing_finish+0x1a0/0x330
> [ 413.756568] [<ffffffff813a6958>] ? br_nf_pre_routing+0x6d8/0x800
> [ 413.756577] [<ffffffff8102d46a>] ? enqueue_task+0x3a/0x90
> [ 413.756582] [<ffffffff81346bc4>] ? nf_iterate+0x84/0xb0
> [ 413.756589] [<ffffffff813a05b0>] ? br_handle_local_finish+0x40/0x40
> [ 413.756594] [<ffffffff81346c64>] ? nf_hook_slow+0x74/0x120
> [ 413.756600] [<ffffffff813a05b0>] ? br_handle_local_finish+0x40/0x40
> [ 413.756607] [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
> [ 413.756613] [<ffffffff813a09d9>] ? br_handle_frame+0x1a9/0x280
> [ 413.756620] [<ffffffff813a0830>] ? br_handle_frame_finish+0x280/0x280
> [ 413.756627] [<ffffffff81320ef7>] ? __netif_receive_skb+0x157/0x5c0
> [ 413.756634] [<ffffffff81321443>] ? process_backlog+0xe3/0x1d0
> [ 413.756641] [<ffffffff81321da5>] ? net_rx_action+0xc5/0x1d0
> [ 413.756650] [<ffffffff8103df11>] ? __do_softirq+0x91/0x120
> [ 413.756657] [<ffffffff813d838c>] ? call_softirq+0x1c/0x30
> [ 413.756660] <EOI> [<ffffffff81003cbd>] ? do_softirq+0x4d/0x80
> [ 413.756673] [<ffffffff81321ece>] ? netif_rx_ni+0x1e/0x30
> [ 413.756681] [<ffffffff812b3ae2>] ? tun_chr_aio_write+0x332/0x4e0
> [ 413.756688] [<ffffffff812b37b0>] ? tun_sendmsg+0x4d0/0x4d0
> [ 413.756697] [<ffffffff810c24e9>] ? do_sync_readv_writev+0xa9/0xf0
> [ 413.756704] [<ffffffff81063f9c>] ? do_futex+0x13c/0xa70
> [ 413.756711] [<ffffffff811d6730>] ? timerqueue_add+0x60/0xb0
> [ 413.756719] [<ffffffff81056ab7>] ?
> __hrtimer_start_range_ns+0x1e7/0x410
> [ 413.756726] [<ffffffff810c231b>] ? rw_copy_check_uvector+0x7b/0x140
> [ 413.756734] [<ffffffff810c2bcf>] ? do_readv_writev+0xdf/0x210
> [ 413.756742] [<ffffffff810c2e7e>] ? sys_writev+0x4e/0xc0
> [ 413.756750] [<ffffffff813d753b>] ? system_call_fastpath+0x16/0x1b
> [ 413.756756] FIX kmalloc-1024: Restoring
> 0xffff880417179566-0xffff880417179567=0x5a

bridge and netfilter, IIRC this was also the problem last time.

Do you have any ebtables loaded?

Can you try building a kernel without ebtables? Without netfilter at all?

Please run all tests with slub_debug=FZPU.

--
error compiling committee.c: too many arguments to function

2011-06-01 09:41:50

by Avi Kivity

[permalink] [raw]
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39

On 06/01/2011 12:40 PM, Avi Kivity wrote:
>
> bridge and netfilter, IIRC this was also the problem last time.
>
> Do you have any ebtables loaded?
>
> Can you try building a kernel without ebtables? Without netfilter at
> all?
>
> Please run all tests with slub_debug=FZPU.
>

--
error compiling committee.c: too many arguments to function

2011-06-01 10:53:45

by Brad Campbell

[permalink] [raw]
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39

On 01/06/11 17:41, Avi Kivity wrote:
> On 06/01/2011 12:40 PM, Avi Kivity wrote:
>>
>> bridge and netfilter, IIRC this was also the problem last time.
>>
>> Do you have any ebtables loaded?

Never heard of them, but making a cursory check just in case..

brad@srv:/raid10/src/linux-2.6.39$ grep EBTABLE .config
# CONFIG_BRIDGE_NF_EBTABLES is not set

>> Can you try building a kernel without ebtables? Without netfilter at all?

Well, without netfilter I can't get it to crash. The problem is without
netfilter I can't actually use it the way I use it to get it to crash.

I rebooted into a netfilter kernel, and did all the steps I'd used on
the no-netfilter kernel and it ticked along happily.

So the result of the experiment is inconclusive. Having said that, the
backtraces certainly smell networky.

To get it to crash, I have to start IE in the VM and https to the public
address of the machine, which is then redirected by netfilter back into
another of the VM's.

I can https directly to the other VM's address, but that does not cause
it to crash, however without netfilter loaded I can't bounce off the
public IP. It's all rather confusing really.

What next Sherlock?

2011-06-01 11:09:57

by Avi Kivity

[permalink] [raw]
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39

On 06/01/2011 01:53 PM, Brad Campbell wrote:
> On 01/06/11 17:41, Avi Kivity wrote:
>> On 06/01/2011 12:40 PM, Avi Kivity wrote:
>>>
>>> bridge and netfilter, IIRC this was also the problem last time.
>>>
>>> Do you have any ebtables loaded?
>
> Never heard of them, but making a cursory check just in case..
>
> brad@srv:/raid10/src/linux-2.6.39$ grep EBTABLE .config
> # CONFIG_BRIDGE_NF_EBTABLES is not set
>
>>> Can you try building a kernel without ebtables? Without netfilter at
>>> all?
>
> Well, without netfilter I can't get it to crash. The problem is
> without netfilter I can't actually use it the way I use it to get it
> to crash.
>
> I rebooted into a netfilter kernel, and did all the steps I'd used on
> the no-netfilter kernel and it ticked along happily.
>
> So the result of the experiment is inconclusive. Having said that, the
> backtraces certainly smell networky.
>
> To get it to crash, I have to start IE in the VM and https to the
> public address of the machine, which is then redirected by netfilter
> back into another of the VM's.
>
> I can https directly to the other VM's address, but that does not
> cause it to crash, however without netfilter loaded I can't bounce off
> the public IP. It's all rather confusing really.
>
> What next Sherlock?
>

Maybe the Sherlocks at netdev@ can tell.

--
error compiling committee.c: too many arguments to function

2011-06-01 11:19:51

by CaT

[permalink] [raw]
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39

On Wed, Jun 01, 2011 at 06:53:31PM +0800, Brad Campbell wrote:
> I rebooted into a netfilter kernel, and did all the steps I'd used
> on the no-netfilter kernel and it ticked along happily.
>
> So the result of the experiment is inconclusive. Having said that,
> the backtraces certainly smell networky.
>
> To get it to crash, I have to start IE in the VM and https to the
> public address of the machine, which is then redirected by netfilter
> back into another of the VM's.
>
> I can https directly to the other VM's address, but that does not
> cause it to crash, however without netfilter loaded I can't bounce
> off the public IP. It's all rather confusing really.
>
> What next Sherlock?

I think you're hitting something I've seen. Can you try rewriting
your firewall rules so that it does not reference any bridge
interfaces at all. Instead, reference the real interface names
in their place. I'm betting it wont crash.

(netdev added to CC since we're aleady bouncing there)

--
"A search of his car uncovered pornography, a homemade sex aid, women's
stockings and a Jack Russell terrier."
- http://www.dailytelegraph.com.au/news/wacky/indeed/story-e6frev20-1111118083480

2011-06-01 11:52:49

by Brad Campbell

[permalink] [raw]
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39

On 01/06/11 19:18, CaT wrote:
> On Wed, Jun 01, 2011 at 06:53:31PM +0800, Brad Campbell wrote:
>> I rebooted into a netfilter kernel, and did all the steps I'd used
>> on the no-netfilter kernel and it ticked along happily.
>>
>> So the result of the experiment is inconclusive. Having said that,
>> the backtraces certainly smell networky.
>>
>> To get it to crash, I have to start IE in the VM and https to the
>> public address of the machine, which is then redirected by netfilter
>> back into another of the VM's.
>>
>> I can https directly to the other VM's address, but that does not
>> cause it to crash, however without netfilter loaded I can't bounce
>> off the public IP. It's all rather confusing really.
>>
>> What next Sherlock?
>
> I think you're hitting something I've seen. Can you try rewriting
> your firewall rules so that it does not reference any bridge
> interfaces at all. Instead, reference the real interface names
> in their place. I'm betting it wont crash.
>

Unfortunately the only interface that is mentioned by name anywhere in
my firewall is $DMZ (which is ppp0 and not part of any bridge).

All of the nat/dnat and other horrible hacks are based on IP addresses.

2011-06-01 23:04:06

by CaT

[permalink] [raw]
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39

On Wed, Jun 01, 2011 at 07:52:33PM +0800, Brad Campbell wrote:
> Unfortunately the only interface that is mentioned by name anywhere
> in my firewall is $DMZ (which is ppp0 and not part of any bridge).
>
> All of the nat/dnat and other horrible hacks are based on IP addresses.

Damn. Not referencing the bridge interfaces at all stopped our host from
going down in flames when we passed it a few packets. These are two
of the oopses we got from it. Whilst the kernel here is .35 we got the
same issue from a range of kernels. Seems related.

The oopses may be a bit weird. Copy and paste from an ipmi terminal.


slab error in cache_alloc_debugcheck_after(): cache `size-64': double
free, or n
Pid: 2431, comm: kvm Tainted: G D
2.6.35.9-local.20110314-141930 #1
Call Trace:
<IRQ> [<ffffffff810fb8bf>] ? __slab_error+0x1f/0x30
[<ffffffff810fc22b>] ? cache_alloc_debugcheck_after+0x6b/0x1f0
[<ffffffff81530a00>] ? br_nf_pre_routing_finish+0x0/0x370
[<ffffffff8153106b>] ? br_nf_pre_routing+0x2fb/0x980
[<ffffffff810fdd3d>] ? kmem_cache_alloc_notrace+0x7d/0xf0

[<ffffffff8153106b>] ? br_nf_pre_routing+0x2fb/0x980
[<ffffffff81466e66>] ? nf_iterate+0x66/0xb0
[<ffffffff8152b9f0>] ? br_handle_frame_finish+0x0/0x1c0
[<ffffffff81466f14>] ? nf_hook_slow+0x64/0xf0
[<ffffffff8152b9f0>] ? br_handle_frame_finish+0x0/0x1c0
[<ffffffff8152bd3c>] ? br_handle_frame+0x18c/0x250
[<ffffffff81445459>] ? __netif_receive_skb+0x169/0x2a0
[<ffffffff81445673>] ? process_backlog+0xe3/0x1d0
[<ffffffff81446347>] ? net_rx_action+0x87/0x1c0
[<ffffffff810793f7>] ? __do_softirq+0xa7/0x1d0
[<ffffffff81035b8c>] ? call_softirq+0x1c/0x30
<EOI> [<ffffffff81037c6d>] ? do_softirq+0x4d/0x80
[<ffffffff81446b4e>] ? netif_rx_ni+0x1e/0x30
[<ffffffff8139541a>] ? tun_chr_aio_write+0x36a/0x510
[<ffffffff813950b0>] ? tun_chr_aio_write+0x0/0x510
[<ffffffff81102859>] ? do_sync_readv_writev+0xa9/0xf0
[<ffffffff810973fb>] ? ktime_get+0x5b/0xe0
[<ffffffff8104f958>] ? lapic_next_event+0x18/0x20
[<ffffffff8109be18>] ? tick_dev_program_event+0x38/0x100
[<ffffffff81102697>] ? rw_copy_check_uvector+0x77/0x130
[<ffffffff81102f0c>] ? do_readv_writev+0xdc/0x200
[<ffffffff8108dfec>] ? sys_timer_settime+0x13c/0x2e0
[<ffffffff8110317e>] ? sys_writev+0x4e/0x90
[<ffffffff81034d6b>] ? system_call_fastpath+0x16/0x1b
ffff8801e7621500: redzone 1:0xbf05bd0100000006, redzone 2:0x9f911029d74e35b

----------

Code: 40 01 00 00 4c 8b a4 24 48 01 00 00 4c 8b ac 24 50 01 00 00 4c 8b
b4 24 5
RIP [<ffffffff81652c67>] icmp_send+0x297/0x650
RSP <ffff880001c036b8>
---[ end trace 9d3f7be7684ac91e ]---
Kernel panic - not syncing: Fatal exception in interrupt
Pid: 0, comm: swapper Tainted: G D
2.6.35.9-local.20110314-144920 #2
Call Trace:
<IRQ> [<ffffffff8170eada>] ? panic+0x94/0x116
[<ffffffff81711326>] ? _raw_spin_lock_irqsave+0x26/0x40
[<ffffffff8103a05f>] ? oops_end+0xef/0xf0
[<ffffffff81711a15>] ? general_protection+0x25/0x30
[<ffffffff81652c2f>] ? icmp_send+0x25f/0x650
[<ffffffff81652c67>] ? icmp_send+0x297/0x650
[<ffffffff815fd8e6>] ? nf_iterate+0x66/0xb0
[<ffffffff816dbfa0>] ? br_nf_forward_finish+0x0/0x170
[<ffffffff815fd994>] ? nf_hook_slow+0x64/0xf0
[<ffffffff816dbfa0>] ? br_nf_forward_finish+0x0/0x170
[<ffffffff816dc461>] ? br_nf_forward_ip+0x201/0x3e0
[<ffffffff815fd8e6>] ? nf_iterate+0x66/0xb0
[<ffffffff816d6620>] ? br_forward_finish+0x0/0x60
[<ffffffff815fd994>] ? nf_hook_slow+0x64/0xf0
[<ffffffff816d6620>] ? br_forward_finish+0x0/0x60
[<ffffffff816d66e9>] ? __br_forward+0x69/0xb0
[<ffffffff816d741a>] ? br_handle_frame_finish+0x12a/0x280
[<ffffffff816dcac8>] ? br_nf_pre_routing_finish+0x208/0x370
[<ffffffff815fd994>] ? nf_hook_slow+0x64/0xf0
[<ffffffff816dc8c0>] ? br_nf_pre_routing_finish+0x0/0x370
[<ffffffff816dc538>] ? br_nf_forward_ip+0x2d8/0x3e0
[<ffffffff816dd3b5>] ? br_nf_pre_routing+0x785/0x980
[<ffffffff815fd8e6>] ? nf_iterate+0x66/0xb0
[<ffffffff815fd994>] ? nf_hook_slow+0x64/0xf0
[<ffffffff816d72f0>] ? br_handle_frame_finish+0x0/0x280
[<ffffffff815fd994>] ? nf_hook_slow+0x64/0xf0
[<ffffffff816d72f0>] ? br_handle_frame_finish+0x0/0x280
[<ffffffff816d76fc>] ? br_handle_frame+0x18c/0x250
[<ffffffff815dec5b>] ? __netif_receive_skb+0x1cb/0x350
[<ffffffff8103d115>] ? read_tsc+0x5/0x20
[<ffffffff815dfa18>] ? netif_receive_skb+0x78/0x80
[<ffffffff815e0217>] ? napi_gro_receive+0x27/0x40
[<ffffffff815e01d8>] ? napi_skb_finish+0x38/0x50
[<ffffffff8152586d>] ? bnx2_poll_work+0xd0d/0x13d0
[<ffffffff8160c950>] ? ctnetlink_conntrack_event+0x210/0x7d0
[<ffffffff81092029>] ? autoremove_wake_function+0x9/0x30
[<ffffffff8109a71b>] ? ktime_get+0x5b/0xe0
[<ffffffff81526051>] ? bnx2_poll+0x61/0x230
[<ffffffff81051db8>] ? lapic_next_event+0x18/0x20
[<ffffffff815dfbef>] ? net_rx_action+0x9f/0x200
[<ffffffff8109636f>] ? __hrtimer_start_range_ns+0x22f/0x410
[<ffffffff8107c35f>] ? __do_softirq+0xaf/0x1e0
[<ffffffff810ab547>] ? handle_IRQ_event+0x47/0x160
[<ffffffff81036d5c>] ? call_softirq+0x1c/0x30
[<ffffffff81038c85>] ? do_softirq+0x65/0xa0
[<ffffffff8107c235>] ? irq_exit+0x85/0x90
[<ffffffff8103820b>] ? do_IRQ+0x6b/0xe0
[<ffffffff817117d3>] ? ret_from_intr+0x0/0x11
<EOI> [<ffffffff81269340>] ? intel_idle+0xf0/0x180
[<ffffffff81269320>] ? intel_idle+0xd0/0x180
[<ffffffff815b0b0f>] ? cpuidle_idle_call+0x9f/0x140
[<ffffffff81035032>] ? cpu_idle+0x62/0xb0
[<ffffffff81a40c77>] ? start_kernel+0x2ef/0x2fa
[<ffffffff81a403e3>] ? x86_64_start_kernel+0xfb/0x10a



--
"A search of his car uncovered pornography, a homemade sex aid, women's
stockings and a Jack Russell terrier."
- http://www.dailytelegraph.com.au/news/wacky/indeed/story-e6frev20-1111118083480

2011-06-03 13:39:15

by Brad Campbell

[permalink] [raw]
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39

On 02/06/11 07:03, CaT wrote:
> On Wed, Jun 01, 2011 at 07:52:33PM +0800, Brad Campbell wrote:
>> Unfortunately the only interface that is mentioned by name anywhere
>> in my firewall is $DMZ (which is ppp0 and not part of any bridge).
>>
>> All of the nat/dnat and other horrible hacks are based on IP addresses.
>
> Damn. Not referencing the bridge interfaces at all stopped our host from
> going down in flames when we passed it a few packets. These are two
> of the oopses we got from it. Whilst the kernel here is .35 we got the
> same issue from a range of kernels. Seems related.

Well, I tried sending an explanatory message to netdev, netfilter & cc'd
to kvm, but it appears not to have made it to kvm or netfilter, and the
cc to netdev has not elicited a response. My resend to netfilter seems
to have dropped into the bit bucket also.

Is there anyone who can point me at the appropriate cage to rattle? I
know it appears to be a netfilter issue, but I don't seem to be able to
get a message to the list (and I am subscribed to it and have been
getting mail for months) and I'm not sure who to pester. The other
alternative is I just stop doing "that" and wait for it to bite someone
else.

Cheers.
Brad

2011-06-03 16:10:08

by Bernhard Held

[permalink] [raw]
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39

Am 03.06.2011 15:38, schrieb Brad Campbell:
> On 02/06/11 07:03, CaT wrote:
>> On Wed, Jun 01, 2011 at 07:52:33PM +0800, Brad Campbell wrote:
>>> Unfortunately the only interface that is mentioned by name anywhere
>>> in my firewall is $DMZ (which is ppp0 and not part of any bridge).
>>>
>>> All of the nat/dnat and other horrible hacks are based on IP addresses.
>>
>> Damn. Not referencing the bridge interfaces at all stopped our host from
>> going down in flames when we passed it a few packets. These are two
>> of the oopses we got from it. Whilst the kernel here is .35 we got the
>> same issue from a range of kernels. Seems related.
>
> Well, I tried sending an explanatory message to netdev, netfilter & cc'd to kvm,
> but it appears not to have made it to kvm or netfilter, and the cc to netdev has
> not elicited a response. My resend to netfilter seems to have dropped into the
> bit bucket also.
Just another reference 3.5 months ago:
http://www.spinics.net/lists/netfilter-devel/msg17239.html

Bernhard

2011-06-03 16:07:34

by Brad Campbell

[permalink] [raw]
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39

On 03/06/11 23:50, Bernhard Held wrote:
> Am 03.06.2011 15:38, schrieb Brad Campbell:
>> On 02/06/11 07:03, CaT wrote:
>>> On Wed, Jun 01, 2011 at 07:52:33PM +0800, Brad Campbell wrote:
>>>> Unfortunately the only interface that is mentioned by name anywhere
>>>> in my firewall is $DMZ (which is ppp0 and not part of any bridge).
>>>>
>>>> All of the nat/dnat and other horrible hacks are based on IP addresses.
>>>
>>> Damn. Not referencing the bridge interfaces at all stopped our host from
>>> going down in flames when we passed it a few packets. These are two
>>> of the oopses we got from it. Whilst the kernel here is .35 we got the
>>> same issue from a range of kernels. Seems related.
>>
>> Well, I tried sending an explanatory message to netdev, netfilter &
>> cc'd to kvm,
>> but it appears not to have made it to kvm or netfilter, and the cc to
>> netdev has
>> not elicited a response. My resend to netfilter seems to have dropped
>> into the
>> bit bucket also.
> Just another reference 3.5 months ago:
> http://www.spinics.net/lists/netfilter-devel/msg17239.html

<waves hands around shouting "I have a reproducible test case for this
and don't mind patching and crashing the machine to get it fixed">

Attempted to add netfilter-devel to the cc this time.

2011-06-05 08:15:04

by Avi Kivity

[permalink] [raw]
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39

On 06/03/2011 04:38 PM, Brad Campbell wrote:
>
> Is there anyone who can point me at the appropriate cage to rattle? I
> know it appears to be a netfilter issue, but I don't seem to be able
> to get a message to the list (and I am subscribed to it and have been
> getting mail for months) and I'm not sure who to pester. The other
> alternative is I just stop doing "that" and wait for it to bite
> someone else.

The mailing list might be set not to send your own mails back to you.
Check the list archive.

--
error compiling committee.c: too many arguments to function

2011-06-05 13:45:51

by Brad Campbell

[permalink] [raw]
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39

On 05/06/11 16:14, Avi Kivity wrote:
> On 06/03/2011 04:38 PM, Brad Campbell wrote:
>>
>> Is there anyone who can point me at the appropriate cage to rattle? I
>> know it appears to be a netfilter issue, but I don't seem to be able
>> to get a message to the list (and I am subscribed to it and have been
>> getting mail for months) and I'm not sure who to pester. The other
>> alternative is I just stop doing "that" and wait for it to bite
>> someone else.
>
> The mailing list might be set not to send your own mails back to you.
> Check the list archive.

Yep, I did that first..

Given the response to previous issues along the same line, it looks a
bit like I just remember not to actually use the system in the way that
triggers the bug and be happy that 99% of the time the kernel does not
panic, but have that lovely feeling in the back of the skull that says
"any time now, and without obvious reason the whole machine might just
come crashing down"..

I guess it's still better than running Xen or Windows..

2011-06-05 13:59:12

by Avi Kivity

[permalink] [raw]
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39

On 06/05/2011 04:45 PM, Brad Campbell wrote:
>> The mailing list might be set not to send your own mails back to you.
>> Check the list archive.
>
>
> Yep, I did that first..
>
> Given the response to previous issues along the same line, it looks a
> bit like I just remember not to actually use the system in the way
> that triggers the bug and be happy that 99% of the time the kernel
> does not panic, but have that lovely feeling in the back of the skull
> that says "any time now, and without obvious reason the whole machine
> might just come crashing down"..
>
> I guess it's still better than running Xen or Windows..

Not at all. Can some networking/netfilter expert look at this?

Please file a bug with all the relevant information in this thread. If
you can look for a previous version that worked, that might increase the
chances of the bug being resolved faster.

--
error compiling committee.c: too many arguments to function

2011-06-06 20:10:59

by Bart De Schuymer

[permalink] [raw]
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39

Hi Brad,

This has probably nothing to do with ebtables, so please rmmod in case
it's loaded.
A few questions I didn't directly see an answer to in the threads I
scanned...
I'm assuming you actually use the bridging firewall functionality. So,
what iptables modules do you use? Can you reduce your iptables rules to
a core that triggers the bug?
Or does it get triggered even with an empty set of firewall rules?
Are you using a stock .35 kernel or is it patched?
Is this something I can trigger on a poor guy's laptop or does it
require specialized hardware (I'm catching up on qemu/kvm...)?

cheers,
Bart

PS: I'm not sure if we should keep CC-ing everybody, netfilter-devel
together with kvm should probably do fine.

Op 3/06/2011 18:07, Brad Campbell schreef:
> On 03/06/11 23:50, Bernhard Held wrote:
>> Am 03.06.2011 15:38, schrieb Brad Campbell:
>>> On 02/06/11 07:03, CaT wrote:
>>>> On Wed, Jun 01, 2011 at 07:52:33PM +0800, Brad Campbell wrote:
>>>>> Unfortunately the only interface that is mentioned by name anywhere
>>>>> in my firewall is $DMZ (which is ppp0 and not part of any bridge).
>>>>>
>>>>> All of the nat/dnat and other horrible hacks are based on IP
>>>>> addresses.
>>>>
>>>> Damn. Not referencing the bridge interfaces at all stopped our host
>>>> from
>>>> going down in flames when we passed it a few packets. These are two
>>>> of the oopses we got from it. Whilst the kernel here is .35 we got the
>>>> same issue from a range of kernels. Seems related.
>>>
>>> Well, I tried sending an explanatory message to netdev, netfilter &
>>> cc'd to kvm,
>>> but it appears not to have made it to kvm or netfilter, and the cc to
>>> netdev has
>>> not elicited a response. My resend to netfilter seems to have dropped
>>> into the
>>> bit bucket also.
>> Just another reference 3.5 months ago:
>> http://www.spinics.net/lists/netfilter-devel/msg17239.html
>
> <waves hands around shouting "I have a reproducible test case for this
> and don't mind patching and crashing the machine to get it fixed">
>
> Attempted to add netfilter-devel to the cc this time.
> --
> To unsubscribe from this list: send the line "unsubscribe
> netfilter-devel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>


--
Bart De Schuymer
http://www.artinalgorithms.be

2011-06-06 20:22:34

by Eric Dumazet

[permalink] [raw]
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39

Le dimanche 05 juin 2011 à 21:45 +0800, Brad Campbell a écrit :
> On 05/06/11 16:14, Avi Kivity wrote:
> > On 06/03/2011 04:38 PM, Brad Campbell wrote:
> >>
> >> Is there anyone who can point me at the appropriate cage to rattle? I
> >> know it appears to be a netfilter issue, but I don't seem to be able
> >> to get a message to the list (and I am subscribed to it and have been
> >> getting mail for months) and I'm not sure who to pester. The other
> >> alternative is I just stop doing "that" and wait for it to bite
> >> someone else.
> >
> > The mailing list might be set not to send your own mails back to you.
> > Check the list archive.
>
> Yep, I did that first..
>
> Given the response to previous issues along the same line, it looks a
> bit like I just remember not to actually use the system in the way that
> triggers the bug and be happy that 99% of the time the kernel does not
> panic, but have that lovely feeling in the back of the skull that says
> "any time now, and without obvious reason the whole machine might just
> come crashing down"..
>
> I guess it's still better than running Xen or Windows..

Could you please try latest linux-2.6 tree ?

We fixed many networking bugs that could explain your crash.


2011-06-06 20:23:25

by Eric Dumazet

[permalink] [raw]
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39

Le lundi 06 juin 2011 à 22:10 +0200, Bart De Schuymer a écrit :
> Hi Brad,
>
> This has probably nothing to do with ebtables, so please rmmod in case
> it's loaded.
> A few questions I didn't directly see an answer to in the threads I
> scanned...
> I'm assuming you actually use the bridging firewall functionality. So,
> what iptables modules do you use? Can you reduce your iptables rules to
> a core that triggers the bug?
> Or does it get triggered even with an empty set of firewall rules?
> Are you using a stock .35 kernel or is it patched?
> Is this something I can trigger on a poor guy's laptop or does it
> require specialized hardware (I'm catching up on qemu/kvm...)?
>

Keep netdev, as this most probably is a networking bug.


2011-06-07 03:34:09

by Brad Campbell

[permalink] [raw]
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39

On 07/06/11 04:10, Bart De Schuymer wrote:
> Hi Brad,
>
> This has probably nothing to do with ebtables, so please rmmod in case
> it's loaded.
> A few questions I didn't directly see an answer to in the threads I
> scanned...
> I'm assuming you actually use the bridging firewall functionality. So,
> what iptables modules do you use? Can you reduce your iptables rules to
> a core that triggers the bug?
> Or does it get triggered even with an empty set of firewall rules?
> Are you using a stock .35 kernel or is it patched?
> Is this something I can trigger on a poor guy's laptop or does it
> require specialized hardware (I'm catching up on qemu/kvm...)?

Not specialised hardware as such, I've just not been able to reproduce
it outside of this specific operating scenario.

I can't trigger it with empty firewall rules as it relies on a DNAT to
occur. If I try it directly to the internal IP address (as I have to
without netfilter loaded) then of course nothing fails.

It's a pain in the bum as a fault, but it's one I can easily reproduce
as long as I use the same set of circumstances.

I'll try using 3.0-rc2 (current git) tonight, and if I can reproduce it
on that then I'll attempt to pare down the IPTABLES rules to a bare minimum.

It is nothing to do with ebtables as I don't compile it. I'm not really
sure about "bridging firewall" functionality. I just use a couple of
hand coded bash scripts to set the tables up.

brad@srv:~$ lsmod
Module Size Used by
xt_iprange 1637 1
xt_DSCP 2077 2
xt_length 1216 1
xt_CLASSIFY 1091 26
sch_sfq 6681 4
xt_CHECKSUM 1229 2 brad@srv:~$ lsmod
Module Size Used by
xt_iprange 1637 1
xt_DSCP 2077 2
xt_length 1216 1
xt_CLASSIFY 1091 26
sch_sfq 6681 4
xt_CHECKSUM 1229 2
ipt_REJECT 2277 1
ipt_MASQUERADE 1759 7
ipt_REDIRECT 1133 1
xt_recent 8223 2
xt_state 1226 5
iptable_nat 3993 1
nf_nat 16773 3 ipt_MASQUERADE,ipt_REDIRECT,iptable_nat
nf_conntrack_ipv4 11868 8 iptable_nat,nf_nat
nf_conntrack 60962 5
ipt_MASQUERADE,xt_state,iptable_nat,nf_nat,nf_conntrack_ipv4
nf_defrag_ipv4 1417 1 nf_conntrack_ipv4
xt_TCPMSS 2567 2
xt_tcpmss 1469 0
xt_tcpudp 2467 56
iptable_mangle 1487 1
pppoe 9574 2
pppox 2188 1 pppoe
iptable_filter 1442 1
ip_tables 16762 3 iptable_nat,iptable_mangle,iptable_filter
x_tables 20462 17
xt_iprange,xt_DSCP,xt_length,xt_CLASSIFY,xt_CHECKSUM,ipt_REJECT,ipt_MASQUERADE,ipt_REDIRECT,xt_recent,xt_state,iptable_nat,xt_TCPMSS,xt_tcpmss,xt_tcpudp,iptable_mangle,iptable_filter,ip_tables
ppp_generic 24243 6 pppoe,pppox
slhc 5293 1 ppp_generic
cls_u32 6468 6
sch_htb 14432 2
deflate 1937 0
zlib_deflate 21228 1 deflate
des_generic 16135 0
cbc 2721 0
ecb 1975 0
crypto_blkcipher 13645 2 cbc,ecb
sha1_generic 2095 0
md5 4001 0
hmac 2977 0
crypto_hash 14519 3 sha1_generic,md5,hmac
cryptomgr 2636 0
aead 6137 1 cryptomgr
crypto_algapi 15289 9
deflate,des_generic,cbc,ecb,crypto_blkcipher,hmac,crypto_hash,cryptomgr,aead
af_key 27372 0
fuse 66747 1
w83627ehf 32052 0
hwmon_vid 2867 1 w83627ehf
vhost_net 16802 6
powernow_k8 12932 0
mperf 1263 1 powernow_k8
kvm_amd 53431 24
kvm 235155 1 kvm_amd
pl2303 12732 1
xhci_hcd 62865 0
i2c_piix4 8391 0
k10temp 3183 0
usbserial 34452 3 pl2303
usb_storage 37887 1
usb_libusual 10999 1 usb_storage
ohci_hcd 18105 0
ehci_hcd 33641 0
ahci 20748 4
usbcore 130936 7
pl2303,xhci_hcd,usbserial,usb_storage,usb_libusual,ohci_hcd,ehci_hcd
libahci 21202 1 ahci
sata_mv 26939 0
megaraid_sas 71659 14

Nat Table (external ip substituted for xxx.xxx.xxx.xxx)

Chain PREROUTING (policy ACCEPT 1761K packets, 152M bytes)
pkts bytes target prot opt in out source
destination
5 210 DNAT udp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 udp dpt:1195 to:192.168.253.199
6 252 DNAT udp -- !ppp0 * 0.0.0.0/0
xxx.xxx.xxx.xxx udp dpt:1195 to:192.168.253.199
0 0 DNAT tcp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 tcp dpt:25001 to:192.168.253.199:465
0 0 DNAT tcp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 tcp dpt:25000 to:192.168.253.199:993
0 0 DNAT tcp -- !ppp0 * 0.0.0.0/0
xxx.xxx.xxx.xxx tcp dpt:25001 to:192.168.253.199:465
0 0 DNAT tcp -- !ppp0 * 0.0.0.0/0
xxx.xxx.xxx.xxx tcp dpt:25000 to:192.168.253.199:993
2 142 DNAT 47 -- ppp0 * 0.0.0.0/0
0.0.0.0/0 to:192.168.253.199
18 880 DNAT tcp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 tcp dpt:1723 to:192.168.253.199
0 0 DNAT 47 -- !ppp0 * 0.0.0.0/0
xxx.xxx.xxx.xxx to:192.168.253.199
0 0 DNAT tcp -- !ppp0 * 0.0.0.0/0
xxx.xxx.xxx.xxx tcp dpt:1723 to:192.168.253.199
2969 149K DNAT tcp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 tcp dpt:443 to:192.168.253.198
20 1280 DNAT tcp -- !ppp0 * 0.0.0.0/0
xxx.xxx.xxx.xxx tcp dpt:443 to:192.168.253.198
0 0 DNAT tcp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 tcp dpt:3101 to:192.168.253.197
0 0 DNAT tcp -- !ppp0 * 0.0.0.0/0
xxx.xxx.xxx.xxx tcp dpt:3101 to:192.168.253.197
0 0 DNAT tcp -- !ppp0 * 0.0.0.0/0
xxx.xxx.xxx.xxx tcp dpt:4101 to:192.168.253.197
44528 2718K REDIRECT tcp -- !ppp0 * 0.0.0.0/0
!192.168.0.0/16 tcp dpt:80 redir ports 8080
0 0 DNAT tcp -- * * 0.0.0.0/0
0.0.0.0/0 tcp dpt:3724 to:192.168.2.107
596K 33M DNAT tcp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 tcp dpts:2001:2030 to:10.99.99.2
1420K 119M DNAT udp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 udp dpts:2001:2030 to:10.99.99.2
7483 449K DNAT all -- !ppp0 * 0.0.0.0/0
xxx.xxx.xxx.xxx to:192.168.2.1


Mangle Table

Chain INPUT (policy ACCEPT 270K packets, 17M bytes)
pkts bytes target prot opt in out source
destination

Chain OUTPUT (policy ACCEPT 170K packets, 12M bytes)
pkts bytes target prot opt in out source
destination

Chain POSTROUTING (policy ACCEPT 2205K packets, 166M bytes)
pkts bytes target prot opt in out source
destination
0 0 MASQUERADE all -- * * 0.0.0.0/0
192.168.254.3
6 360 ACCEPT all -- * * 0.0.0.0/0
xxx.xxx.xxx.xxx
20424 2120K MASQUERADE all -- * ppp0 192.168.0.0/16
!192.168.0.0/16
0 0 MASQUERADE all -- * ppp0 10.0.0.0/24
0.0.0.0/0
3 204 MASQUERADE all -- * * 192.168.2.0/24
10.8.0.0/24
1418K 128M MASQUERADE all -- * * 10.99.99.0/24
0.0.0.0/0
68248 4095K MASQUERADE all -- * * 192.168.253.0/24
10.8.0.0/16
13305 2405K MASQUERADE all -- * * 192.168.253.0/24
!192.168.0.0/16

Chain PREROUTING (policy ACCEPT 278M packets, 293G bytes)
pkts bytes target prot opt in out source
destination
169 55528 CHECKSUM udp -- br1 * 0.0.0.0/0
0.0.0.0/0 udp dpt:67 CHECKSUM fill

Chain INPUT (policy ACCEPT 180M packets, 250G bytes)
pkts bytes target prot opt in out source
destination

Chain FORWARD (policy ACCEPT 98M packets, 44G bytes)
pkts bytes target prot opt in out source
destination

Chain OUTPUT (policy ACCEPT 155M packets, 180G bytes)
pkts bytes target prot opt in out source
destination

Chain POSTROUTING (policy ACCEPT 253M packets, 223G bytes)
pkts bytes target prot opt in out source
destination
165 54182 CHECKSUM udp -- * br1 0.0.0.0/0
0.0.0.0/0 udp spt:67 CHECKSUM fill
51 3712 CLASSIFY tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp dpt:53 CLASSIFY set 1:20
85274 6454K CLASSIFY udp -- * ppp0 0.0.0.0/0
0.0.0.0/0 udp dpt:53 CLASSIFY set 1:20
187 257K CLASSIFY tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp spt:81 CLASSIFY set 1:20
25M 1180M CLASSIFY tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp flags:0x3F/0x10 state ESTABLISHED length 40:100
CLASSIFY set 1:15
728K 67M CLASSIFY icmp -- * ppp0 0.0.0.0/0
0.0.0.0/0 CLASSIFY set 1:15
231 23484 CLASSIFY tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp dpt:2401 CLASSIFY set 1:15
65636 5610K CLASSIFY tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp dpt:22 CLASSIFY set 1:10
2018 315K CLASSIFY tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp spt:22 CLASSIFY set 1:10
80 10092 CLASSIFY tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp dpt:3389 CLASSIFY set 1:10
26063 8910K CLASSIFY tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp dpt:8080 CLASSIFY set 1:15
932K 131M CLASSIFY tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp dpt:80 CLASSIFY set 1:15
3511 267K CLASSIFY udp -- * ppp0 0.0.0.0/0
0.0.0.0/0 udp dpt:123 CLASSIFY set 1:10
0 0 CLASSIFY tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp spt:20 CLASSIFY set 1:15
3 180 CLASSIFY tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp dpt:20 CLASSIFY set 1:15
94058 38M CLASSIFY 47 -- * ppp0 0.0.0.0/0
0.0.0.0/0 CLASSIFY set 1:10
1086K 183M CLASSIFY udp -- * ppp0 0.0.0.0/0
0.0.0.0/0 udp spt:1194 CLASSIFY set 1:10
1086K 183M TOS udp -- * ppp0 0.0.0.0/0
0.0.0.0/0 udp spt:1194 TOS set 0x10/0x3f
48817 10M CLASSIFY udp -- * ppp0 0.0.0.0/0
0.0.0.0/0 udp spt:1195 CLASSIFY set 1:10
48817 10M TOS udp -- * ppp0 0.0.0.0/0
0.0.0.0/0 udp spt:1195 TOS set 0x10/0x3f
94058 38M CLASSIFY 47 -- * ppp0 0.0.0.0/0
0.0.0.0/0 CLASSIFY set 1:15
106 7207 CLASSIFY tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp dpt:1863 CLASSIFY set 1:15
188K 34M CLASSIFY tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp dpt:443 CLASSIFY set 1:15
51541 3327K CLASSIFY tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp dpts:6660:6669 CLASSIFY set 1:15
0 0 CLASSIFY tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp spts:2021:2030 CLASSIFY set 1:15
85 4944 CLASSIFY tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp dpt:19999 CLASSIFY set 1:15
208K 86M CLASSIFY udp -- * * 0.0.0.0/0
0.0.0.0/0 source IP range 192.168.2.80-192.168.2.120 CLASSIFY
set 1:10
0 0 CLASSIFY tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp spt:12345 CLASSIFY set 1:15
1 80 CLASSIFY udp -- * ppp0 0.0.0.0/0
0.0.0.0/0 udp spt:12345 CLASSIFY set 1:15


Default table

Chain INPUT (policy ACCEPT 176M packets, 247G bytes)
pkts bytes target prot opt in out source
destination
0 0 ACCEPT udp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 udp dpt:4569
1187K 582M ACCEPT udp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 udp dpt:1194
2 577 ACCEPT udp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 udp dpt:1195
28 1224 ACCEPT tcp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 tcp dpt:3389
230 12372 tcp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 tcp dpt:22 state NEW recent: SET name: DEFAULT side:
source
3 180 DROP tcp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 tcp dpt:22 state NEW recent: UPDATE seconds: 300
hit_count: 4 name: DEFAULT side: source
1750 143K ACCEPT tcp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 tcp dpt:22
3 144 ACCEPT tcp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 tcp dpt:113
120 6090 ACCEPT tcp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 tcp dpt:81
36094 29M ACCEPT tcp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 tcp dpt:25
1456K 1706M ACCEPT all -- ppp0 * 0.0.0.0/0
0.0.0.0/0 state RELATED,ESTABLISHED
31047 2334K REJECT tcp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 tcp option=!2 reject-with tcp-reset
552K 60M ACCEPT all -- !ppp0 * 0.0.0.0/0
0.0.0.0/0 state NEW
13552 1207K ACCEPT icmp -- ppp0 * 0.0.0.0/0
0.0.0.0/0
5712 392K DROP all -- ppp0 * 0.0.0.0/0
0.0.0.0/0

Chain FORWARD (policy ACCEPT 98M packets, 44G bytes)
pkts bytes target prot opt in out source
destination
1207K 68M TCPMSS tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp flags:0x06/0x02 TCPMSS clamp to PMTU

Chain OUTPUT (policy ACCEPT 155M packets, 180G bytes)
pkts bytes target prot opt in out source
destination
31675 1895K TCPMSS tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp flags:0x06/0x02 TCPMSS clamp to PMTU

lsmod

ipt_REJECT 2277 1
ipt_MASQUERADE 1759 7
ipt_REDIRECT 1133 1
xt_recent 8223 2
xt_state 1226 5
iptable_nat 3993 1
nf_nat 16773 3 ipt_MASQUERADE,ipt_REDIRECT,iptable_nat
nf_conntrack_ipv4 11868 8 iptable_nat,nf_nat
nf_conntrack 60962 5
ipt_MASQUERADE,xt_state,iptable_nat,nf_nat,nf_conntrack_ipv4
nf_defrag_ipv4 1417 1 nf_conntrack_ipv4
xt_TCPMSS 2567 2
xt_tcpmss 1469 0
xt_tcpudp 2467 56
iptable_mangle 1487 1
pppoe 9574 2
pppox 2188 1 pppoe
iptable_filter 1442 1
ip_tables 16762 3 iptable_nat,iptable_mangle,iptable_filter
x_tables 20462 17
xt_iprange,xt_DSCP,xt_length,xt_CLASSIFY,xt_CHECKSUM,ipt_REJECT,ipt_MASQUERADE,ipt_REDIRECT,xt_recent,xt_state,iptable_nat,xt_TCPMSS,xt_tcpmss,xt_tcpudp,iptable_mangle,iptable_filter,ip_tables
ppp_generic 24243 6 pppoe,pppox
slhc 5293 1 ppp_generic
cls_u32 6468 6
sch_htb 14432 2
deflate 1937 0
zlib_deflate 21228 1 deflate
des_generic 16135 0
cbc 2721 0
ecb 1975 0
crypto_blkcipher 13645 2 cbc,ecb
sha1_generic 2095 0
md5 4001 0
hmac 2977 0
crypto_hash 14519 3 sha1_generic,md5,hmac
cryptomgr 2636 0
aead 6137 1 cryptomgr
crypto_algapi 15289 9
deflate,des_generic,cbc,ecb,crypto_blkcipher,hmac,crypto_hash,cryptomgr,aead
af_key 27372 0
fuse 66747 1
w83627ehf 32052 0
hwmon_vid 2867 1 w83627ehf
vhost_net 16802 6
powernow_k8 12932 0
mperf 1263 1 powernow_k8
kvm_amd 53431 24
kvm 235155 1 kvm_amd
pl2303 12732 1
xhci_hcd 62865 0
i2c_piix4 8391 0
k10temp 3183 0
usbserial 34452 3 pl2303
usb_storage 37887 1
usb_libusual 10999 1 usb_storage
ohci_hcd 18105 0
ehci_hcd 33641 0
ahci 20748 4
usbcore 130936 7
pl2303,xhci_hcd,usbserial,usb_storage,usb_libusual,ohci_hcd,ehci_hcd
libahci 21202 1 ahci
sata_mv 26939 0
megaraid_sas 71659 14

2011-06-07 13:27:46

by Brad Campbell

[permalink] [raw]
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39

On 07/06/11 04:22, Eric Dumazet wrote:

> Could you please try latest linux-2.6 tree ?
>
> We fixed many networking bugs that could explain your crash.
>
>
>
>

No good I'm afraid.

[ 543.040056]
=============================================================================
[ 543.040136] BUG ip_dst_cache: Padding overwritten.
0xffff8803e4217ffe-0xffff8803e4217fff
[ 543.040194]
-----------------------------------------------------------------------------
[ 543.040198]
[ 543.040298] INFO: Slab 0xffffea000d9e74d0 objects=25 used=25 fp=0x
(null) flags=0x8000000000004081
[ 543.040364] Pid: 4576, comm: kworker/1:2 Not tainted 3.0.0-rc2 #1
[ 543.040415] Call Trace:
[ 543.040472] [<ffffffff810b9c1d>] ? slab_err+0xad/0xd0
[ 543.040528] [<ffffffff8102e034>] ? check_preempt_wakeup+0xa4/0x160
[ 543.040595] [<ffffffff810ba206>] ? slab_pad_check+0x126/0x170
[ 543.040650] [<ffffffff8133045b>] ? dst_destroy+0x8b/0x110
[ 543.040701] [<ffffffff810ba29a>] ? check_slab+0x4a/0xc0
[ 543.040753] [<ffffffff810baf2d>] ? free_debug_processing+0x2d/0x250
[ 543.040808] [<ffffffff810bb27b>] ? __slab_free+0x12b/0x140
[ 543.040862] [<ffffffff810bbe99>] ? kmem_cache_free+0x99/0xa0
[ 543.040915] [<ffffffff8133045b>] ? dst_destroy+0x8b/0x110
[ 543.040967] [<ffffffff813307f6>] ? dst_gc_task+0x196/0x1f0
[ 543.041021] [<ffffffff8104e954>] ? queue_delayed_work_on+0x154/0x160
[ 543.041081] [<ffffffff813066fe>] ? do_dbs_timer+0x20e/0x3d0
[ 543.041133] [<ffffffff81330660>] ? dst_alloc+0x180/0x180
[ 543.041187] [<ffffffff8104f28b>] ? process_one_work+0xfb/0x3b0
[ 543.041242] [<ffffffff8104f964>] ? worker_thread+0x144/0x3d0
[ 543.041296] [<ffffffff8102cc10>] ? __wake_up_common+0x50/0x80
[ 543.041678] [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
[ 543.041729] [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
[ 543.041782] [<ffffffff81053436>] ? kthread+0x96/0xa0
[ 543.041835] [<ffffffff813e1d14>] ? kernel_thread_helper+0x4/0x10
[ 543.041890] [<ffffffff810533a0>] ? kthread_worker_fn+0x120/0x120
[ 543.041944] [<ffffffff813e1d10>] ? gs_change+0xb/0xb
[ 543.041993] Padding 0xffff8803e4217f40: 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 543.042718] Padding 0xffff8803e4217f50: 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 543.043433] Padding 0xffff8803e4217f60: 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 543.044155] Padding 0xffff8803e4217f70: 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 543.044866] Padding 0xffff8803e4217f80: 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 543.045590] Padding 0xffff8803e4217f90: 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 543.046311] Padding 0xffff8803e4217fa0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 543.047034] Padding 0xffff8803e4217fb0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 543.047755] Padding 0xffff8803e4217fc0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 543.048474] Padding 0xffff8803e4217fd0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 543.049203] Padding 0xffff8803e4217fe0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 543.049909] Padding 0xffff8803e4217ff0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a 5a 00 00 ZZZZZZZZZZZZZZ..
[ 543.050021] FIX ip_dst_cache: Restoring
0xffff8803e4217f40-0xffff8803e4217fff=0x5a
[ 543.050021]

Dropped -mm, Hugh and Andrea from CC as this does not appear to be mm or
ksm related.

I'll pare down the firewall and see if I can make it break easier with a
smaller test set.

2011-06-07 13:30:17

by Patrick McHardy

[permalink] [raw]
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39

On 07.06.2011 05:33, Brad Campbell wrote:
> On 07/06/11 04:10, Bart De Schuymer wrote:
>> Hi Brad,
>>
>> This has probably nothing to do with ebtables, so please rmmod in case
>> it's loaded.
>> A few questions I didn't directly see an answer to in the threads I
>> scanned...
>> I'm assuming you actually use the bridging firewall functionality. So,
>> what iptables modules do you use? Can you reduce your iptables rules to
>> a core that triggers the bug?
>> Or does it get triggered even with an empty set of firewall rules?
>> Are you using a stock .35 kernel or is it patched?
>> Is this something I can trigger on a poor guy's laptop or does it
>> require specialized hardware (I'm catching up on qemu/kvm...)?
>
> Not specialised hardware as such, I've just not been able to reproduce
> it outside of this specific operating scenario.

The last similar problem we've had was related to the 32/64 bit compat
code. Are you running 32 bit userspace on a 64 bit kernel?

> I can't trigger it with empty firewall rules as it relies on a DNAT to
> occur. If I try it directly to the internal IP address (as I have to
> without netfilter loaded) then of course nothing fails.
>
> It's a pain in the bum as a fault, but it's one I can easily reproduce
> as long as I use the same set of circumstances.
>
> I'll try using 3.0-rc2 (current git) tonight, and if I can reproduce it
> on that then I'll attempt to pare down the IPTABLES rules to a bare
> minimum.
>
> It is nothing to do with ebtables as I don't compile it. I'm not really
> sure about "bridging firewall" functionality. I just use a couple of
> hand coded bash scripts to set the tables up.

>From one of your previous mails:

> # CONFIG_BRIDGE_NF_EBTABLES is not set

How about CONFIG_BRIDGE_NETFILTER?

2011-06-07 13:38:03

by Eric Dumazet

[permalink] [raw]
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39

Le mardi 07 juin 2011 à 21:27 +0800, Brad Campbell a écrit :
> On 07/06/11 04:22, Eric Dumazet wrote:
>
> > Could you please try latest linux-2.6 tree ?
> >
> > We fixed many networking bugs that could explain your crash.
> >
> >
> >
> >
>
> No good I'm afraid.
>
> [ 543.040056]
> =============================================================================
> [ 543.040136] BUG ip_dst_cache: Padding overwritten.
> 0xffff8803e4217ffe-0xffff8803e4217fff
> [ 543.040194]

Thats pretty strange : These are the last two bytes of a page, set to
0x0000 (a 16 bit value)

There is no way a dst field could actually sit on this location (its a
padding), since a dst is a bit less than 256 bytes (0xe8), and each
entry is aligned on a 64byte address.

grep dst /proc/slabinfo

ip_dst_cache 32823 62944 256 32 2 : tunables 0 0
0 : slabdata 1967 1967 0

sizeof(struct rtable)=0xe8


> -----------------------------------------------------------------------------
> [ 543.040198]
> [ 543.040298] INFO: Slab 0xffffea000d9e74d0 objects=25 used=25 fp=0x
> (null) flags=0x8000000000004081
> [ 543.040364] Pid: 4576, comm: kworker/1:2 Not tainted 3.0.0-rc2 #1
> [ 543.040415] Call Trace:
> [ 543.040472] [<ffffffff810b9c1d>] ? slab_err+0xad/0xd0
> [ 543.040528] [<ffffffff8102e034>] ? check_preempt_wakeup+0xa4/0x160
> [ 543.040595] [<ffffffff810ba206>] ? slab_pad_check+0x126/0x170
> [ 543.040650] [<ffffffff8133045b>] ? dst_destroy+0x8b/0x110
> [ 543.040701] [<ffffffff810ba29a>] ? check_slab+0x4a/0xc0
> [ 543.040753] [<ffffffff810baf2d>] ? free_debug_processing+0x2d/0x250
> [ 543.040808] [<ffffffff810bb27b>] ? __slab_free+0x12b/0x140
> [ 543.040862] [<ffffffff810bbe99>] ? kmem_cache_free+0x99/0xa0
> [ 543.040915] [<ffffffff8133045b>] ? dst_destroy+0x8b/0x110
> [ 543.040967] [<ffffffff813307f6>] ? dst_gc_task+0x196/0x1f0
> [ 543.041021] [<ffffffff8104e954>] ? queue_delayed_work_on+0x154/0x160
> [ 543.041081] [<ffffffff813066fe>] ? do_dbs_timer+0x20e/0x3d0
> [ 543.041133] [<ffffffff81330660>] ? dst_alloc+0x180/0x180
> [ 543.041187] [<ffffffff8104f28b>] ? process_one_work+0xfb/0x3b0
> [ 543.041242] [<ffffffff8104f964>] ? worker_thread+0x144/0x3d0
> [ 543.041296] [<ffffffff8102cc10>] ? __wake_up_common+0x50/0x80
> [ 543.041678] [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
> [ 543.041729] [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
> [ 543.041782] [<ffffffff81053436>] ? kthread+0x96/0xa0
> [ 543.041835] [<ffffffff813e1d14>] ? kernel_thread_helper+0x4/0x10
> [ 543.041890] [<ffffffff810533a0>] ? kthread_worker_fn+0x120/0x120
> [ 543.041944] [<ffffffff813e1d10>] ? gs_change+0xb/0xb
> [ 543.041993] Padding 0xffff8803e4217f40: 5a 5a 5a 5a 5a 5a 5a 5a 5a
> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [ 543.042718] Padding 0xffff8803e4217f50: 5a 5a 5a 5a 5a 5a 5a 5a 5a
> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [ 543.043433] Padding 0xffff8803e4217f60: 5a 5a 5a 5a 5a 5a 5a 5a 5a
> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [ 543.044155] Padding 0xffff8803e4217f70: 5a 5a 5a 5a 5a 5a 5a 5a 5a
> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [ 543.044866] Padding 0xffff8803e4217f80: 5a 5a 5a 5a 5a 5a 5a 5a 5a
> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [ 543.045590] Padding 0xffff8803e4217f90: 5a 5a 5a 5a 5a 5a 5a 5a 5a
> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [ 543.046311] Padding 0xffff8803e4217fa0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [ 543.047034] Padding 0xffff8803e4217fb0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [ 543.047755] Padding 0xffff8803e4217fc0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [ 543.048474] Padding 0xffff8803e4217fd0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [ 543.049203] Padding 0xffff8803e4217fe0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [ 543.049909] Padding 0xffff8803e4217ff0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
> 5a 5a 5a 5a 5a 00 00 ZZZZZZZZZZZZZZ..
> [ 543.050021] FIX ip_dst_cache: Restoring
> 0xffff8803e4217f40-0xffff8803e4217fff=0x5a
> [ 543.050021]
>
> Dropped -mm, Hugh and Andrea from CC as this does not appear to be mm or
> ksm related.
>
> I'll pare down the firewall and see if I can make it break easier with a
> smaller test set.

Hmm, not sure now :(

Could you reproduce another bug please ?


2011-06-07 14:40:41

by Brad Campbell

[permalink] [raw]
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39

On 07/06/11 21:30, Patrick McHardy wrote:
> On 07.06.2011 05:33, Brad Campbell wrote:
>> On 07/06/11 04:10, Bart De Schuymer wrote:
>>> Hi Brad,
>>>
>>> This has probably nothing to do with ebtables, so please rmmod in case
>>> it's loaded.
>>> A few questions I didn't directly see an answer to in the threads I
>>> scanned...
>>> I'm assuming you actually use the bridging firewall functionality. So,
>>> what iptables modules do you use? Can you reduce your iptables rules to
>>> a core that triggers the bug?
>>> Or does it get triggered even with an empty set of firewall rules?
>>> Are you using a stock .35 kernel or is it patched?
>>> Is this something I can trigger on a poor guy's laptop or does it
>>> require specialized hardware (I'm catching up on qemu/kvm...)?
>>
>> Not specialised hardware as such, I've just not been able to reproduce
>> it outside of this specific operating scenario.
>
> The last similar problem we've had was related to the 32/64 bit compat
> code. Are you running 32 bit userspace on a 64 bit kernel?

No, 32 bit Guest OS, but a completely 64 bit userspace on a 64 bit kernel.

Userspace is current Debian Stable. Kernel is Vanilla and qemu-kvm is
current git


>> I can't trigger it with empty firewall rules as it relies on a DNAT to
>> occur. If I try it directly to the internal IP address (as I have to
>> without netfilter loaded) then of course nothing fails.
>>
>> It's a pain in the bum as a fault, but it's one I can easily reproduce
>> as long as I use the same set of circumstances.
>>
>> I'll try using 3.0-rc2 (current git) tonight, and if I can reproduce it
>> on that then I'll attempt to pare down the IPTABLES rules to a bare
>> minimum.
>>
>> It is nothing to do with ebtables as I don't compile it. I'm not really
>> sure about "bridging firewall" functionality. I just use a couple of
>> hand coded bash scripts to set the tables up.
>
> From one of your previous mails:
>
>> # CONFIG_BRIDGE_NF_EBTABLES is not set
>
> How about CONFIG_BRIDGE_NETFILTER?
>

It was compiled in.

With the following table set I was able to reproduce the problem on
3.0-rc2. Replaced my IP with xxx.xxx.xxx.xxx, but otherwise unmodified

root@srv:~# iptables-save
# Generated by iptables-save v1.4.10 on Tue Jun 7 22:11:30 2011
*filter
:INPUT ACCEPT [978:107619]
:FORWARD ACCEPT [142:7068]
:OUTPUT ACCEPT [1659:291870]
-A INPUT -i ppp0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT ! -i ppp0 -m state --state NEW -j ACCEPT
-A INPUT -i ppp0 -j DROP
COMMIT
# Completed on Tue Jun 7 22:11:30 2011
# Generated by iptables-save v1.4.10 on Tue Jun 7 22:11:30 2011
*nat
:PREROUTING ACCEPT [813:49170]
:INPUT ACCEPT [91:7090]
:OUTPUT ACCEPT [267:20731]
:POSTROUTING ACCEPT [296:22281]
-A PREROUTING -d xxx.xxx.xxx.xxx/32 ! -i ppp0 -p tcp -m tcp --dport 443
-j DNAT --to-destination 192.168.253.198
COMMIT
# Completed on Tue Jun 7 22:11:30 2011
# Generated by iptables-save v1.4.10 on Tue Jun 7 22:11:30 2011
*mangle
:PREROUTING ACCEPT [2729:274392]
:INPUT ACCEPT [2508:262976]
:FORWARD ACCEPT [142:7068]
:OUTPUT ACCEPT [1674:293701]
:POSTROUTING ACCEPT [2131:346411]
-A FORWARD -o ppp0 -p tcp -m tcp --tcp-flags SYN,RST SYN -m tcpmss --mss
1400:1536 -j TCPMSS --clamp-mss-to-pmtu
COMMIT
# Completed on Tue Jun 7 22:11:30 2011

I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
the address the way I was doing it, so that's a no-go for me.

2011-06-07 15:15:58

by Brad Campbell

[permalink] [raw]
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39

On 07/06/11 21:37, Eric Dumazet wrote:
> Le mardi 07 juin 2011 à 21:27 +0800, Brad Campbell a écrit :
>> On 07/06/11 04:22, Eric Dumazet wrote:
>>
>>> Could you please try latest linux-2.6 tree ?
>>>
>>> We fixed many networking bugs that could explain your crash.
>>>
>>>
>>>
>>>
>>
>> No good I'm afraid.
>>
>> [ 543.040056]
>> =============================================================================
>> [ 543.040136] BUG ip_dst_cache: Padding overwritten.
>> 0xffff8803e4217ffe-0xffff8803e4217fff
>> [ 543.040194]
>
> Thats pretty strange : These are the last two bytes of a page, set to
> 0x0000 (a 16 bit value)
>
> There is no way a dst field could actually sit on this location (its a
> padding), since a dst is a bit less than 256 bytes (0xe8), and each
> entry is aligned on a 64byte address.
>
> grep dst /proc/slabinfo
>
> ip_dst_cache 32823 62944 256 32 2 : tunables 0 0
> 0 : slabdata 1967 1967 0
>
> sizeof(struct rtable)=0xe8
>
>
>> -----------------------------------------------------------------------------
>> [ 543.040198]
>> [ 543.040298] INFO: Slab 0xffffea000d9e74d0 objects=25 used=25 fp=0x
>> (null) flags=0x8000000000004081
>> [ 543.040364] Pid: 4576, comm: kworker/1:2 Not tainted 3.0.0-rc2 #1
>> [ 543.040415] Call Trace:
>> [ 543.040472] [<ffffffff810b9c1d>] ? slab_err+0xad/0xd0
>> [ 543.040528] [<ffffffff8102e034>] ? check_preempt_wakeup+0xa4/0x160
>> [ 543.040595] [<ffffffff810ba206>] ? slab_pad_check+0x126/0x170
>> [ 543.040650] [<ffffffff8133045b>] ? dst_destroy+0x8b/0x110
>> [ 543.040701] [<ffffffff810ba29a>] ? check_slab+0x4a/0xc0
>> [ 543.040753] [<ffffffff810baf2d>] ? free_debug_processing+0x2d/0x250
>> [ 543.040808] [<ffffffff810bb27b>] ? __slab_free+0x12b/0x140
>> [ 543.040862] [<ffffffff810bbe99>] ? kmem_cache_free+0x99/0xa0
>> [ 543.040915] [<ffffffff8133045b>] ? dst_destroy+0x8b/0x110
>> [ 543.040967] [<ffffffff813307f6>] ? dst_gc_task+0x196/0x1f0
>> [ 543.041021] [<ffffffff8104e954>] ? queue_delayed_work_on+0x154/0x160
>> [ 543.041081] [<ffffffff813066fe>] ? do_dbs_timer+0x20e/0x3d0
>> [ 543.041133] [<ffffffff81330660>] ? dst_alloc+0x180/0x180
>> [ 543.041187] [<ffffffff8104f28b>] ? process_one_work+0xfb/0x3b0
>> [ 543.041242] [<ffffffff8104f964>] ? worker_thread+0x144/0x3d0
>> [ 543.041296] [<ffffffff8102cc10>] ? __wake_up_common+0x50/0x80
>> [ 543.041678] [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
>> [ 543.041729] [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
>> [ 543.041782] [<ffffffff81053436>] ? kthread+0x96/0xa0
>> [ 543.041835] [<ffffffff813e1d14>] ? kernel_thread_helper+0x4/0x10
>> [ 543.041890] [<ffffffff810533a0>] ? kthread_worker_fn+0x120/0x120
>> [ 543.041944] [<ffffffff813e1d10>] ? gs_change+0xb/0xb
>> [ 543.041993] Padding 0xffff8803e4217f40: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.042718] Padding 0xffff8803e4217f50: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.043433] Padding 0xffff8803e4217f60: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.044155] Padding 0xffff8803e4217f70: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.044866] Padding 0xffff8803e4217f80: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.045590] Padding 0xffff8803e4217f90: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.046311] Padding 0xffff8803e4217fa0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.047034] Padding 0xffff8803e4217fb0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.047755] Padding 0xffff8803e4217fc0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.048474] Padding 0xffff8803e4217fd0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.049203] Padding 0xffff8803e4217fe0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.049909] Padding 0xffff8803e4217ff0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 00 00 ZZZZZZZZZZZZZZ..
>> [ 543.050021] FIX ip_dst_cache: Restoring
>> 0xffff8803e4217f40-0xffff8803e4217fff=0x5a
>> [ 543.050021]
>>
>> Dropped -mm, Hugh and Andrea from CC as this does not appear to be mm or
>> ksm related.
>>
>> I'll pare down the firewall and see if I can make it break easier with a
>> smaller test set.
>
> Hmm, not sure now :(
>
> Could you reproduce another bug please ?

Here is the remainder of the log from that bug that I managed to scrape
off the disk. Odd as usually the netconsole dies and so does the disk
subsystem. In this case it managed to get the log to disk.

[ 543.040056]
=============================================================================
[ 543.040136] BUG ip_dst_cache: Padding overwritten.
0xffff8803e4217ffe-0xffff8803e4217fff
[ 543.040194]
-----------------------------------------------------------------------------
[ 543.040198]
[ 543.040298] INFO: Slab 0xffffea000d9e74d0 objects=25 used=25 fp=0x
(null) flags=0x8000000000004081
[ 543.040364] Pid: 4576, comm: kworker/1:2 Not tainted 3.0.0-rc2 #1
[ 543.040415] Call Trace:
[ 543.040472] [<ffffffff810b9c1d>] ? slab_err+0xad/0xd0
[ 543.040528] [<ffffffff8102e034>] ? check_preempt_wakeup+0xa4/0x160
[ 543.040595] [<ffffffff810ba206>] ? slab_pad_check+0x126/0x170
[ 543.040650] [<ffffffff8133045b>] ? dst_destroy+0x8b/0x110
[ 543.040701] [<ffffffff810ba29a>] ? check_slab+0x4a/0xc0
[ 543.040753] [<ffffffff810baf2d>] ? free_debug_processing+0x2d/0x250
[ 543.040808] [<ffffffff810bb27b>] ? __slab_free+0x12b/0x140
[ 543.040862] [<ffffffff810bbe99>] ? kmem_cache_free+0x99/0xa0
[ 543.040915] [<ffffffff8133045b>] ? dst_destroy+0x8b/0x110
[ 543.040967] [<ffffffff813307f6>] ? dst_gc_task+0x196/0x1f0
[ 543.041021] [<ffffffff8104e954>] ? queue_delayed_work_on+0x154/0x160
[ 543.041081] [<ffffffff813066fe>] ? do_dbs_timer+0x20e/0x3d0
[ 543.041133] [<ffffffff81330660>] ? dst_alloc+0x180/0x180
[ 543.041187] [<ffffffff8104f28b>] ? process_one_work+0xfb/0x3b0
[ 543.041242] [<ffffffff8104f964>] ? worker_thread+0x144/0x3d0
[ 543.041296] [<ffffffff8102cc10>] ? __wake_up_common+0x50/0x80
[ 543.041678] [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
[ 543.041729] [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
[ 543.041782] [<ffffffff81053436>] ? kthread+0x96/0xa0
[ 543.041835] [<ffffffff813e1d14>] ? kernel_thread_helper+0x4/0x10
[ 543.041890] [<ffffffff810533a0>] ? kthread_worker_fn+0x120/0x120
[ 543.041944] [<ffffffff813e1d10>] ? gs_change+0xb/0xb
[ 543.041993] Padding 0xffff8803e4217f40: 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 543.042718] Padding 0xffff8803e4217f50: 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 543.043433] Padding 0xffff8803e4217f60: 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 543.044155] Padding 0xffff8803e4217f70: 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 543.044866] Padding 0xffff8803e4217f80: 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 543.045590] Padding 0xffff8803e4217f90: 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 543.046311] Padding 0xffff8803e4217fa0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 543.047034] Padding 0xffff8803e4217fb0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 543.047755] Padding 0xffff8803e4217fc0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 543.048474] Padding 0xffff8803e4217fd0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 543.049203] Padding 0xffff8803e4217fe0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 543.049909] Padding 0xffff8803e4217ff0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a 5a 00 00 ZZZZZZZZZZZZZZ..
[ 543.050021] FIX ip_dst_cache: Restoring
0xffff8803e4217f40-0xffff8803e4217fff=0x5a
[ 543.050021]
[ 885.504318] br1: port 4(vnet2) entering forwarding state
[ 885.504797] device vnet2 left promiscuous mode
[ 885.504868] br1: port 4(vnet2) entering disabled state
[ 885.847186] br1: port 3(vnet1) entering forwarding state
[ 885.847667] device vnet1 left promiscuous mode
[ 885.847740] br1: port 3(vnet1) entering disabled state
[ 885.940042]
=============================================================================
[ 885.940122] BUG kmalloc-512: Object padding overwritten
[ 885.940171]
-----------------------------------------------------------------------------
[ 885.940175]
[ 885.940273] INFO: 0xffff8803dc09322e-0xffff8803dc09322f. First byte
0x0 instead of 0x5a
[ 885.940344] INFO: Allocated in qdisc_alloc+0x59/0x110 age=66496 cpu=2
pid=4948
[ 885.940404] INFO: Freed in qdisc_alloc+0x49/0x110 age=66496 cpu=2
pid=4948
[ 885.940457] INFO: Slab 0xffffea000d821f80 objects=28 used=21
fp=0xffff8803dc090b68 flags=0x8000000000004081
[ 885.940519] INFO: Object 0xffff8803dc092fe8 @offset=12264
fp=0xffff8803dc092238
[ 885.940523]
[ 885.940618] Bytes b4 0xffff8803dc092fd8: d3 e0 ff ff 00 00 00 00 5a
5a 5a 5a 5a 5a 5a 5a Óàÿÿ....ZZZZZZZZ
[ 885.941373] Object 0xffff8803dc092fe8: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 885.942120] Object 0xffff8803dc092ff8: 00 00 00 00 00 00 00 00 70
0a 34 81 ff ff ff ff ........p.4.ÿÿÿÿ
[ 885.942847] Object 0xffff8803dc093008: b0 08 34 81 ff ff ff ff 04
00 00 00 18 00 00 00 °.4.ÿÿÿÿ........
[ 885.943585] Object 0xffff8803dc093018: 60 ac 5d 81 ff ff ff ff 00
00 00 00 00 00 00 00 `¬].ÿÿÿÿ........
[ 885.944324] Object 0xffff8803dc093028: 28 30 09 dc 03 88 ff ff 28
30 09 dc 03 88 ff ff (0.Ü..ÿÿ(0.Ü..ÿÿ
[ 885.945071] Object 0xffff8803dc093038: 00 00 00 00 ff ff ff ff 00
00 00 00 00 00 00 00 ....ÿÿÿÿ........
[ 885.945794] Object 0xffff8803dc093048: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 885.946530] Object 0xffff8803dc093058: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 885.947281] Object 0xffff8803dc093068: 98 f3 f9 0e 04 88 ff ff 00
00 00 00 00 00 00 00 .óù...ÿÿ........
[ 885.948011] Object 0xffff8803dc093078: 00 00 00 00 00 00 00 00 02
00 00 00 00 00 00 00 ................
[ 885.948735] Object 0xffff8803dc093088: 88 30 09 dc 03 88 ff ff 88
30 09 dc 03 88 ff ff .0.Ü..ÿÿ.0.Ü..ÿÿ
[ 885.949483] Object 0xffff8803dc093098: 00 00 00 00 bf bf 00 00 ca
29 01 00 00 00 00 00 ....¿¿..Ê)......
[ 885.950007] Object 0xffff8803dc0930a8: de 01 00 00 00 00 00 00 00
00 00 00 00 00 00 00 Þ...............
[ 885.950007] Object 0xffff8803dc0930b8: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 885.950007] Object 0xffff8803dc0930c8: 10 cd 13 e7 03 88 ff ff a0
13 34 81 ff ff ff ff .Í.ç..ÿÿ..4.ÿÿÿÿ
[ 885.950007] Object 0xffff8803dc0930d8: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 885.950007] Object 0xffff8803dc0930e8: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 885.950007] Object 0xffff8803dc0930f8: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 885.950007] Object 0xffff8803dc093108: 08 31 09 dc 03 88 ff ff 08
31 09 dc 03 88 ff ff .1.Ü..ÿÿ.1.Ü..ÿÿ
[ 885.950007] Object 0xffff8803dc093118: 00 00 00 00 00 00 00 00 20
31 09 dc 03 88 ff ff .........1.Ü..ÿÿ
[ 885.950007] Object 0xffff8803dc093128: 20 31 09 dc 03 88 ff ff 00
00 00 00 00 00 00 00 .1.Ü..ÿÿ........
[ 885.950007] Object 0xffff8803dc093138: 38 31 09 dc 03 88 ff ff 38
31 09 dc 03 88 ff ff 81.Ü..ÿÿ81.Ü..ÿÿ
[ 885.950007] Object 0xffff8803dc093148: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 885.950007] Object 0xffff8803dc093158: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 885.950007] Object 0xffff8803dc093168: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 885.950007] Object 0xffff8803dc093178: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 885.950007] Object 0xffff8803dc093188: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 885.950007] Object 0xffff8803dc093198: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 885.950007] Object 0xffff8803dc0931a8: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 885.950007] Object 0xffff8803dc0931b8: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 885.950007] Object 0xffff8803dc0931c8: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 885.950007] Object 0xffff8803dc0931d8: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 885.950007] Redzone 0xffff8803dc0931e8: cc cc cc cc cc cc cc cc
ÌÌÌÌÌÌÌÌ
[ 885.950007] Padding 0xffff8803dc093228: 5a 5a 5a 5a 5a 5a 00 00
ZZZZZZ..
[ 885.950007] Pid: 11, comm: rcuc1 Not tainted 3.0.0-rc2 #1
[ 885.950007] Call Trace:
[ 885.950007] [<ffffffff810ba41d>] ? check_bytes_and_report+0x10d/0x150
[ 885.950007] [<ffffffff810ba509>] ? check_object+0xa9/0x260
[ 885.950007] [<ffffffff81075a49>] ? __rcu_process_callbacks+0x119/0x340
[ 885.950007] [<ffffffff810bb033>] ? free_debug_processing+0x133/0x250
[ 885.950007] [<ffffffff810bb27b>] ? __slab_free+0x12b/0x140
[ 885.950007] [<ffffffff81075a49>] ? __rcu_process_callbacks+0x119/0x340
[ 885.950007] [<ffffffff8107666b>] ? rcu_cpu_kthread+0x27b/0x3a0
[ 885.950007] [<ffffffff8102d5b9>] ? ttwu_do_wakeup+0x19/0xa0
[ 885.950007] [<ffffffff81074a80>] ? invoke_rcu_cpu_kthread+0x30/0x30
[ 885.950007] [<ffffffff810763f0>] ? rcu_check_callbacks+0x100/0x100
[ 885.950007] [<ffffffff810763f0>] ? rcu_check_callbacks+0x100/0x100
[ 885.950007] [<ffffffff81053436>] ? kthread+0x96/0xa0
[ 885.950007] [<ffffffff813e1d14>] ? kernel_thread_helper+0x4/0x10
[ 885.950007] [<ffffffff810533a0>] ? kthread_worker_fn+0x120/0x120
[ 885.950007] [<ffffffff813e1d10>] ? gs_change+0xb/0xb
[ 885.950007] FIX kmalloc-512: Restoring
0xffff8803dc09322e-0xffff8803dc09322f=0x5a
[ 885.950007]
[ 886.050119]
=============================================================================
[ 886.050198] BUG kmalloc-512: Object padding overwritten
[ 886.050247]
-----------------------------------------------------------------------------
[ 886.050251]
[ 886.050353] INFO: 0xffff8803dc091916-0xffff8803dc091917. First byte
0x0 instead of 0x5a
[ 886.050451] INFO: Allocated in kvm_ioapic_init+0x3c/0xd0 [kvm]
age=66256 cpu=0 pid=5122
[ 886.050517] INFO: Freed in __kfree_skb+0x11/0x90 age=66311 cpu=4 pid=5054
[ 886.050568] INFO: Slab 0xffffea000d821f80 objects=28 used=17
fp=0xffff8803dc091488 flags=0x8000000000004081
[ 886.050630] INFO: Object 0xffff8803dc0916d0 @offset=5840 fp=0x
(null)
[ 886.050634]
[ 886.050726] Bytes b4 0xffff8803dc0916c0: ea e4 00 00 01 00 00 00 5a
5a 5a 5a 5a 5a 5a 5a êä......ZZZZZZZZ
[ 886.051451] Object 0xffff8803dc0916d0: 00 00 c0 fe 00 00 00 00 3f
00 00 00 00 00 00 00 ..Àþ....?.......
[ 886.052183] Object 0xffff8803dc0916e0: 00 08 00 00 00 00 00 00 00
00 01 00 00 00 00 00 ................
[ 886.052898] Object 0xffff8803dc0916f0: 00 00 01 00 00 00 00 00 00
00 01 00 00 00 00 00 ................
[ 886.053613] Object 0xffff8803dc091700: 00 00 01 00 00 00 00 00 00
00 01 00 00 00 00 00 ................
[ 886.054325] Object 0xffff8803dc091710: 00 00 01 00 00 00 00 00 00
00 01 00 00 00 00 00 ................
[ 886.055032] Object 0xffff8803dc091720: 00 00 01 00 00 00 00 00 00
00 01 00 00 00 00 00 ................
[ 886.055743] Object 0xffff8803dc091730: 00 00 01 00 00 00 00 00 00
00 01 00 00 00 00 00 ................
[ 886.056462] Object 0xffff8803dc091740: 00 00 01 00 00 00 00 00 00
00 01 00 00 00 00 00 ................
[ 886.057177] Object 0xffff8803dc091750: 00 00 01 00 00 00 00 00 00
00 01 00 00 00 00 00 ................
[ 886.058210] Object 0xffff8803dc091760: 00 00 01 00 00 00 00 00 00
00 01 00 00 00 00 00 ................
[ 886.058957] Object 0xffff8803dc091770: 00 00 01 00 00 00 00 00 00
00 01 00 00 00 00 00 ................
[ 886.059667] Object 0xffff8803dc091780: 00 00 01 00 00 00 00 00 00
00 01 00 00 00 00 00 ................
[ 886.060104] Object 0xffff8803dc091790: 00 00 01 00 00 00 00 00 00
00 01 00 00 00 00 00 ................
[ 886.060104] Object 0xffff8803dc0917a0: 00 00 01 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 886.060104] Object 0xffff8803dc0917b0: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 886.060104] Object 0xffff8803dc0917c0: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 886.060104] Object 0xffff8803dc0917d0: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 886.060104] Object 0xffff8803dc0917e0: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 886.060104] Object 0xffff8803dc0917f0: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 886.060104] Object 0xffff8803dc091800: 01 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 886.060104] Object 0xffff8803dc091810: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 886.060104] Object 0xffff8803dc091820: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 886.060104] Object 0xffff8803dc091830: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 886.060104] Object 0xffff8803dc091840: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 886.060104] Object 0xffff8803dc091850: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 886.060104] Object 0xffff8803dc091860: 00 00 00 00 00 00 00 00 b0
80 10 a0 ff ff ff ff ........°...ÿÿÿÿ
[ 886.060104] Object 0xffff8803dc091870: 00 00 df f5 03 88 ff ff 00
00 00 00 00 00 00 00 ..ßõ..ÿÿ........
[ 886.060104] Object 0xffff8803dc091880: 43 43 00 00 00 00 00 00 01
00 00 00 00 00 00 00 CC..............
[ 886.060104] Object 0xffff8803dc091890: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 886.060104] Object 0xffff8803dc0918a0: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 886.060104] Object 0xffff8803dc0918b0: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 886.060104] Object 0xffff8803dc0918c0: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 886.060104] Redzone 0xffff8803dc0918d0: cc cc cc cc cc cc cc cc
ÌÌÌÌÌÌÌÌ
[ 886.060104] Padding 0xffff8803dc091910: 5a 5a 5a 5a 5a 5a 00 00
ZZZZZZ..
[ 886.060104] Pid: 7359, comm: qemu Not tainted 3.0.0-rc2 #1
[ 886.060104] Call Trace:
[ 886.060104] [<ffffffff810ba41d>] ? check_bytes_and_report+0x10d/0x150
[ 886.060104] [<ffffffff810ba509>] ? check_object+0xa9/0x260
[ 886.060104] [<ffffffffa00eb8bf>] ? kvm_arch_destroy_vm+0x2f/0x150 [kvm]
[ 886.060104] [<ffffffff810bb033>] ? free_debug_processing+0x133/0x250
[ 886.060104] [<ffffffff810bb27b>] ? __slab_free+0x12b/0x140
[ 886.060104] [<ffffffffa00eb8bf>] ? kvm_arch_destroy_vm+0x2f/0x150 [kvm]
[ 886.060104] [<ffffffffa00e2dd9>] ? kvm_put_kvm+0xe9/0x130 [kvm]
[ 886.060104] [<ffffffffa00e3763>] ? kvm_vcpu_release+0x13/0x20 [kvm]
[ 886.060104] [<ffffffff810c65fd>] ? fput+0xdd/0x270
[ 886.060104] [<ffffffff810c2e1c>] ? filp_close+0x5c/0x90
[ 886.060104] [<ffffffff8103ae1d>] ? put_files_struct+0xcd/0x100
[ 886.060104] [<ffffffff8103ccc7>] ? do_exit+0x6d7/0x7e0
[ 886.060104] [<ffffffff8103ce20>] ? do_group_exit+0x50/0xc0
[ 886.060104] [<ffffffff81048914>] ? get_signal_to_deliver+0x1a4/0x3a0
[ 886.060104] [<ffffffff8100188a>] ? do_signal+0x8a/0x7d0
[ 886.060104] [<ffffffff81049148>] ? kill_pid_info+0x38/0x60
[ 886.060104] [<ffffffff810493e4>] ? sys_kill+0x94/0x1b0
[ 886.060104] [<ffffffff8100201d>] ? do_notify_resume+0x4d/0x70
[ 886.060104] [<ffffffff810c59c4>] ? sys_pread64+0x74/0xa0
[ 886.060104] [<ffffffff813e11db>] ? int_signal+0x12/0x17
[ 886.060104] FIX kmalloc-512: Restoring
0xffff8803dc091916-0xffff8803dc091917=0x5a
[ 886.060104]
[ 926.148014] br1: port 6(vnet4) entering forwarding state
[ 926.148385] device vnet4 left promiscuous mode
[ 926.148457] br1: port 6(vnet4) entering disabled state
[ 945.659523] br1: port 2(vnet0) entering forwarding state
[ 945.660041] device vnet0 left promiscuous mode
[ 945.660116] br1: port 2(vnet0) entering disabled state
[ 1121.085267] br1: port 5(vnet3) entering forwarding state
[ 1121.085783] device vnet3 left promiscuous mode
[ 1121.085860] br1: port 5(vnet3) entering disabled state
[ 1176.307099] nfsd: last server has exited, flushing export cache
[ 1176.430113]
=============================================================================
[ 1176.430194] BUG kmalloc-512: Object padding overwritten
[ 1176.430199]
-----------------------------------------------------------------------------
[ 1176.430204]
[ 1176.430210] INFO: 0xffff8803f5df5b5e-0xffff8803f5df5b5f. First byte
0x0 instead of 0x5a
[ 1176.430225] INFO: Allocated in sock_alloc_send_pskb+0x1d0/0x320
age=93370 cpu=0 pid=5126
[ 1176.430234] INFO: Freed in __kfree_skb+0x11/0x90 age=93370 cpu=0 pid=4495
[ 1176.430240] INFO: Slab 0xffffea000ddc8d60 objects=28 used=0
fp=0xffff8803f5df76c0 flags=0x8000000000004080
[ 1176.430246] INFO: Object 0xffff8803f5df5918 @offset=6424
fp=0xffff8803f5df7908
[ 1176.430249]
[ 1176.430253] Bytes b4 0xffff8803f5df5908: ab f0 ff ff 00 00 00 00 5a
5a 5a 5a 5a 5a 5a 5a «ðÿÿ....ZZZZZZZZ
[ 1176.430270] Object 0xffff8803f5df5918: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430285] Object 0xffff8803f5df5928: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430300] Object 0xffff8803f5df5938: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430314] Object 0xffff8803f5df5948: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430328] Object 0xffff8803f5df5958: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430343] Object 0xffff8803f5df5968: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430357] Object 0xffff8803f5df5978: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430371] Object 0xffff8803f5df5988: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430386] Object 0xffff8803f5df5998: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430400] Object 0xffff8803f5df59a8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430414] Object 0xffff8803f5df59b8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430429] Object 0xffff8803f5df59c8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430443] Object 0xffff8803f5df59d8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430457] Object 0xffff8803f5df59e8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430471] Object 0xffff8803f5df59f8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430485] Object 0xffff8803f5df5a08: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430500] Object 0xffff8803f5df5a18: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430514] Object 0xffff8803f5df5a28: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430528] Object 0xffff8803f5df5a38: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430542] Object 0xffff8803f5df5a48: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430557] Object 0xffff8803f5df5a58: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430571] Object 0xffff8803f5df5a68: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430585] Object 0xffff8803f5df5a78: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430599] Object 0xffff8803f5df5a88: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430614] Object 0xffff8803f5df5a98: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430628] Object 0xffff8803f5df5aa8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430642] Object 0xffff8803f5df5ab8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430656] Object 0xffff8803f5df5ac8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430671] Object 0xffff8803f5df5ad8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430685] Object 0xffff8803f5df5ae8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430699] Object 0xffff8803f5df5af8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430714] Object 0xffff8803f5df5b08: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[ 1176.430729] Redzone 0xffff8803f5df5b18: bb bb bb bb bb bb bb bb
»»»»»»»»
[ 1176.430742] Padding 0xffff8803f5df5b58: 5a 5a 5a 5a 5a 5a 00 00
ZZZZZZ..
[ 1176.430758] Pid: 24, comm: rcuc4 Not tainted 3.0.0-rc2 #1
[ 1176.430762] Call Trace:
[ 1176.430774] [<ffffffff810ba41d>] ? check_bytes_and_report+0x10d/0x150
[ 1176.430782] [<ffffffff810ba509>] ? check_object+0xa9/0x260
[ 1176.430789] [<ffffffff810bac15>] ? init_object+0x85/0xa0
[ 1176.430795] [<ffffffff810ba816>] ? __free_slab+0x156/0x180
[ 1176.430803] [<ffffffff81075a49>] ? __rcu_process_callbacks+0x119/0x340
[ 1176.430810] [<ffffffff8107666b>] ? rcu_cpu_kthread+0x27b/0x3a0
[ 1176.430818] [<ffffffff8102d5b9>] ? ttwu_do_wakeup+0x19/0xa0
[ 1176.430825] [<ffffffff81074a80>] ? invoke_rcu_cpu_kthread+0x30/0x30
[ 1176.430832] [<ffffffff810763f0>] ? rcu_check_callbacks+0x100/0x100
[ 1176.430838] [<ffffffff810763f0>] ? rcu_check_callbacks+0x100/0x100
[ 1176.430846] [<ffffffff81053436>] ? kthread+0x96/0xa0
[ 1176.430855] [<ffffffff813e1d14>] ? kernel_thread_helper+0x4/0x10
[ 1176.430863] [<ffffffff810533a0>] ? kthread_worker_fn+0x120/0x120
[ 1176.430870] [<ffffffff813e1d10>] ? gs_change+0xb/0xb
[ 1176.430876] FIX kmalloc-512: Restoring
0xffff8803f5df5b5e-0xffff8803f5df5b5f=0x5a
[ 1176.430879]
[ 1176.430897]
=============================================================================
[ 1176.430902] BUG kmalloc-512: Object padding overwritten
[ 1176.430905]
-----------------------------------------------------------------------------
[ 1176.430909]
[ 1176.430912] INFO: 0xffff8803f5df6fe6-0xffff8803f5df6fe7. First byte
0x0 instead of 0x5a
[ 1176.430921] INFO: Allocated in sock_alloc_send_pskb+0x1d0/0x320
age=93370 cpu=0 pid=5126
[ 1176.430928] INFO: Freed in __kfree_skb+0x11/0x90 age=93370 cpu=0 pid=4495
[ 1176.430935] INFO: Slab 0xffffea000ddc8d60 objects=28 used=0
fp=0xffff8803f5df76c0 flags=0x8000000000004080
[ 1176.430940] INFO: Object 0xffff8803f5df6da0 @offset=11680
fp=0xffff8803f5df5488
[ 1176.430943]
[ 1176.430946] Bytes b4 0xffff8803f5df6d90: cd 7a 00 00 01 00 00 00 5a
5a 5a 5a 5a 5a 5a 5a Íz......ZZZZZZZZ
[ 1176.430962] Object 0xffff8803f5df6da0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430977] Object 0xffff8803f5df6db0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430992] Object 0xffff8803f5df6dc0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431006] Object 0xffff8803f5df6dd0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431021] Object 0xffff8803f5df6de0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431036] Object 0xffff8803f5df6df0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431050] Object 0xffff8803f5df6e00: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431065] Object 0xffff8803f5df6e10: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431080] Object 0xffff8803f5df6e20: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431094] Object 0xffff8803f5df6e30: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431109] Object 0xffff8803f5df6e40: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431124] Object 0xffff8803f5df6e50: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431138] Object 0xffff8803f5df6e60: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431153] Object 0xffff8803f5df6e70: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431168] Object 0xffff8803f5df6e80: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431182] Object 0xffff8803f5df6e90: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431197] Object 0xffff8803f5df6ea0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431213] Object 0xffff8803f5df6eb0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431228] Object 0xffff8803f5df6ec0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431243] Object 0xffff8803f5df6ed0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431258] Object 0xffff8803f5df6ee0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431273] Object 0xffff8803f5df6ef0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431287] Object 0xffff8803f5df6f00: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431303] Object 0xffff8803f5df6f10: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431317] Object 0xffff8803f5df6f20: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431332] Object 0xffff8803f5df6f30: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431346] Object 0xffff8803f5df6f40: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431361] Object 0xffff8803f5df6f50: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431376] Object 0xffff8803f5df6f60: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431391] Object 0xffff8803f5df6f70: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431405] Object 0xffff8803f5df6f80: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431420] Object 0xffff8803f5df6f90: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[ 1176.431435] Redzone 0xffff8803f5df6fa0: bb bb bb bb bb bb bb bb
»»»»»»»»
[ 1176.431448] Padding 0xffff8803f5df6fe0: 5a 5a 5a 5a 5a 5a 00 00
ZZZZZZ..
[ 1176.431463] Pid: 24, comm: rcuc4 Not tainted 3.0.0-rc2 #1
[ 1176.431467] Call Trace:
[ 1176.431474] [<ffffffff810ba41d>] ? check_bytes_and_report+0x10d/0x150
[ 1176.431481] [<ffffffff810ba509>] ? check_object+0xa9/0x260
[ 1176.431487] [<ffffffff810bac15>] ? init_object+0x85/0xa0
[ 1176.431494] [<ffffffff810ba816>] ? __free_slab+0x156/0x180
[ 1176.431501] [<ffffffff81075a49>] ? __rcu_process_callbacks+0x119/0x340
[ 1176.431508] [<ffffffff8107666b>] ? rcu_cpu_kthread+0x27b/0x3a0
[ 1176.431515] [<ffffffff8102d5b9>] ? ttwu_do_wakeup+0x19/0xa0
[ 1176.431522] [<ffffffff81074a80>] ? invoke_rcu_cpu_kthread+0x30/0x30
[ 1176.431529] [<ffffffff810763f0>] ? rcu_check_callbacks+0x100/0x100
[ 1176.431535] [<ffffffff810763f0>] ? rcu_check_callbacks+0x100/0x100
[ 1176.431543] [<ffffffff81053436>] ? kthread+0x96/0xa0
[ 1176.431550] [<ffffffff813e1d14>] ? kernel_thread_helper+0x4/0x10
[ 1176.431558] [<ffffffff810533a0>] ? kthread_worker_fn+0x120/0x120
[ 1176.431565] [<ffffffff813e1d10>] ? gs_change+0xb/0xb
[ 1176.431571] FIX kmalloc-512: Restoring
0xffff8803f5df6fe6-0xffff8803f5df6fe7=0x5a
[ 1176.431574]
[ 1179.773660]
=============================================================================
[ 1179.773742] BUG kmalloc-1024: Object padding overwritten
[ 1179.773792]
-----------------------------------------------------------------------------
[ 1179.773796]
[ 1179.773901] INFO: 0xffff8803e430e6be-0xffff8803e430e6bf. First byte
0x0 instead of 0x5a
[ 1179.773972] INFO: Allocated in tty_buffer_request_room+0xcc/0x1a0
age=93277 cpu=1 pid=5109
[ 1179.774042] INFO: Freed in __blockdev_direct_IO+0xa16/0xa90 age=93277
cpu=5 pid=5146
[ 1179.774102] INFO: Slab 0xffffea000d9ea9c0 objects=29 used=14
fp=0xffff8803e430d9e8 flags=0x8000000000004081
[ 1179.774163] INFO: Object 0xffff8803e430e278 @offset=25208
fp=0xffff8803e4309df8
[ 1179.774166]
[ 1179.774260] Bytes b4 0xffff8803e430e268: a5 57 01 00 01 00 00 00 5a
5a 5a 5a 5a 5a 5a 5a ¥W......ZZZZZZZZ
[ 1179.775297] Object 0xffff8803e430e278: 08 eb 30 e4 03 88 ff ff a0
e2 30 e4 03 88 ff ff .ë0ä..ÿÿ.â0ä..ÿÿ
[ 1179.776003] Object 0xffff8803e430e288: a0 e3 30 e4 03 88 ff ff be
00 00 00 00 01 00 00 .ã0ä..ÿÿŸ.......
[ 1179.776974] Object 0xffff8803e430e298: be 00 00 00 be 00 00 00 5b
20 20 20 20 32 2e 34 Ÿ...Ÿ...[....2.4
[ 1179.777656] Object 0xffff8803e430e2a8: 31 30 32 34 36 5d 20 70 63
69 20 30 30 30 30 3a 10246].pci.0000:
[ 1179.778609] Object 0xffff8803e430e2b8: 30 30 3a 31 34 2e 34 3a 20
20 20 62 72 69 64 67 00:14.4:...bridg
[ 1179.779289] Object 0xffff8803e430e2c8: 65 20 77 69 6e 64 6f 77 20
5b 69 6f 20 20 30 78 e.window.[io..0x
[ 1179.779976] Object 0xffff8803e430e2d8: 30 64 30 30 2d 30 78 66 66
66 66 5d 20 28 73 75 0d00-0xffff].(su
[ 1179.780327] Object 0xffff8803e430e2e8: 62 74 72 61 63 74 69 76 65
20 64 65 63 6f 64 65 btractive.decode
[ 1179.780327] Object 0xffff8803e430e2f8: 29 0d 0a 5b 20 20 20 20 32
2e 34 31 30 32 34 38 )..[....2.410248
[ 1179.780327] Object 0xffff8803e430e308: 5d 20 70 63 69 20 30 30 30
30 3a 30 30 3a 31 34 ].pci.0000:00:14
[ 1179.780327] Object 0xffff8803e430e318: 2e 34 3a 20 20 20 62 72 69
64 67 65 20 77 69 6e .4:...bridge.win
[ 1179.780327] Object 0xffff8803e430e328: 64 6f 77 20 5b 6d 65 6d 20
30 78 30 30 30 61 30 dow.[mem.0x000a0
[ 1179.780327] Object 0xffff8803e430e338: 30 30 30 2d 30 78 30 30 30
62 66 66 66 66 5d 20 000-0x000bffff].
[ 1179.780327] Object 0xffff8803e430e348: 28 73 75 62 74 72 61 63 74
69 76 65 20 64 65 63 (subtractive.dec
[ 1179.780327] Object 0xffff8803e430e358: 6f 64 65 29 0d 0a 32 30 5d
20 78 48 43 49 20 78 ode)..20].xHCI.x
[ 1179.780327] Object 0xffff8803e430e368: 68 63 69 5f 63 68 65 63 6b
5f 62 61 6e 64 77 69 hci_check_bandwi
[ 1179.780327] Object 0xffff8803e430e378: 64 74 68 20 63 61 6c 6c 65
64 20 66 6f 72 20 72 dth.called.for.r
[ 1179.780327] Object 0xffff8803e430e388: 6f 6f 74 20 68 75 62 0d 0a
0d 0a 0d 0a 0d 0a 0d oot.hub.........
[ 1179.780327] Object 0xffff8803e430e398: 0a 30 30 0d 0a 0d 0a 0a 00
00 00 00 00 00 00 00 .00.............
[ 1179.780327] Object 0xffff8803e430e3a8: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1179.780327] Object 0xffff8803e430e3b8: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1179.780327] Object 0xffff8803e430e3c8: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1179.780327] Object 0xffff8803e430e3d8: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1179.780327] Object 0xffff8803e430e3e8: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1179.780327] Object 0xffff8803e430e3f8: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1179.780327] Object 0xffff8803e430e408: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1179.780327] Object 0xffff8803e430e418: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1179.780327] Object 0xffff8803e430e428: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1179.780327] Object 0xffff8803e430e438: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1179.780327] Object 0xffff8803e430e448: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1179.780327] Object 0xffff8803e430e458: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1179.780327] Object 0xffff8803e430e468: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1179.780327] Object 0xffff8803e430e478: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1179.780327] Object 0xffff8803e430e488: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1179.780327] Object 0xffff8803e430e498: 00 00 00 00 00 00 00 00 6b
6b 6b 6b 6b 6b 6b 6b ........kkkkkkkk
[ 1179.780327] Object 0xffff8803e430e4a8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e4b8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e4c8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e4d8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e4e8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e4f8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e508: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e518: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e528: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e538: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e548: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e558: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e568: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e578: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e588: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e598: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e5a8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e5b8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e5c8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e5d8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e5e8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e5f8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e608: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e618: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e628: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e638: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e648: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e658: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e668: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[ 1179.780327] Redzone 0xffff8803e430e678: cc cc cc cc cc cc cc cc
ÌÌÌÌÌÌÌÌ
[ 1179.780327] Padding 0xffff8803e430e6b8: 5a 5a 5a 5a 5a 5a 00 00
ZZZZZZ..
[ 1179.780327] Pid: 2212, comm: kworker/5:2 Not tainted 3.0.0-rc2 #1
[ 1179.780327] Call Trace:
[ 1179.780327] [<ffffffff810ba41d>] ? check_bytes_and_report+0x10d/0x150
[ 1179.780327] [<ffffffff810ba509>] ? check_object+0xa9/0x260
[ 1179.780327] [<ffffffff8125acf7>] ? tty_buffer_free_all+0x47/0x70
[ 1179.780327] [<ffffffff810bb033>] ? free_debug_processing+0x133/0x250
[ 1179.780327] [<ffffffff810bb27b>] ? __slab_free+0x12b/0x140
[ 1179.780327] [<ffffffff81254c70>] ? free_tty_struct+0x30/0x30
[ 1179.780327] [<ffffffff8125acf7>] ? tty_buffer_free_all+0x47/0x70
[ 1179.780327] [<ffffffff81254c66>] ? free_tty_struct+0x26/0x30
[ 1179.780327] [<ffffffff8104f28b>] ? process_one_work+0xfb/0x3b0
[ 1179.780327] [<ffffffff8104f964>] ? worker_thread+0x144/0x3d0
[ 1179.780327] [<ffffffff8102cc10>] ? __wake_up_common+0x50/0x80
[ 1179.780327] [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
[ 1179.780327] [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
[ 1179.780327] [<ffffffff81053436>] ? kthread+0x96/0xa0
[ 1179.780327] [<ffffffff813e1d14>] ? kernel_thread_helper+0x4/0x10
[ 1179.780327] [<ffffffff810533a0>] ? kthread_worker_fn+0x120/0x120
[ 1179.780327] [<ffffffff813e1d10>] ? gs_change+0xb/0xb
[ 1179.780327] FIX kmalloc-1024: Restoring
0xffff8803e430e6be-0xffff8803e430e6bf=0x5a
[ 1179.780327]
[ 1180.826212]
=============================================================================
[ 1180.826289] BUG kmalloc-1024: Object padding overwritten
[ 1180.826338]
-----------------------------------------------------------------------------
[ 1180.826341]
[ 1180.826439] INFO: 0xffff8804097a9566-0xffff8804097a9567. First byte
0x0 instead of 0x5a
[ 1180.826505] INFO: Allocated in tty_buffer_request_room+0xcc/0x1a0
age=69291 cpu=0 pid=5978
[ 1180.826567] INFO: Freed in __blockdev_direct_IO+0xa16/0xa90 age=80371
cpu=5 pid=5713
[ 1180.826626] INFO: Slab 0xffffea000e212cc0 objects=29 used=3
fp=0xffff8804097a8890 flags=0x8000000000004081
[ 1180.826686] INFO: Object 0xffff8804097a9120 @offset=4384
fp=0xffff8804097a8890
[ 1180.826689]
[ 1180.826781] Bytes b4 0xffff8804097a9110: 91 48 01 00 01 00 00 00 5a
5a 5a 5a 5a 5a 5a 5a .H......ZZZZZZZZ
[ 1180.827478] Object 0xffff8804097a9120: 30 de 70 fc 03 88 ff ff 48
91 7a 09 04 88 ff ff 0Þpü..ÿÿH.z...ÿÿ
[ 1180.828179] Object 0xffff8804097a9130: 48 92 7a 09 04 88 ff ff ce
00 00 00 00 01 00 00 H.z...ÿÿÎ.......
[ 1180.829149] Object 0xffff8804097a9140: ce 00 00 00 ce 00 00 00 5b
20 20 20 20 32 2e 33 Î...Î...[....2.3
[ 1180.830162] Object 0xffff8804097a9150: 32 31 31 35 36 5d 20 70 63
69 20 30 30 30 30 3a 21156].pci.0000:
[ 1180.831118] Object 0xffff8804097a9160: 30 30 3a 31 33 2e 30 3a 20
5b 31 30 30 32 3a 34 00:13.0:.[1002:4
[ 1180.831796] Object 0xffff8804097a9170: 33 39 37 5d 20 74 79 70 65
20 30 20 63 6c 61 73 397].type.0.clas
[ 1180.832492] Object 0xffff8804097a9180: 73 20 30 78 30 30 30 63 30
33 0d 0a 5b 20 20 20 s.0x000c03..[...
[ 1180.833482] Object 0xffff8804097a9190: 20 32 2e 33 32 31 31 36 36
5d 20 70 63 69 20 30 .2.321166].pci.0
[ 1180.834173] Object 0xffff8804097a91a0: 30 30 30 3a 30 30 3a 31 33
2e 30 3a 20 72 65 67 000:00:13.0:.reg
[ 1180.835134] Object 0xffff8804097a91b0: 20 31 30 3a 20 5b 6d 65 6d
20 30 78 66 65 31 66 .10:.[mem.0xfe1f
[ 1180.835240] Object 0xffff8804097a91c0: 64 30 30 30 2d 30 78 66 65
31 66 64 66 66 66 5d d000-0xfe1fdfff]
[ 1180.835240] Object 0xffff8804097a91d0: 0d 0a 5b 20 20 20 20 32 2e
33 32 31 32 31 38 5d ..[....2.321218]
[ 1180.835240] Object 0xffff8804097a91e0: 20 70 63 69 20 30 30 30 30
3a 30 30 3a 31 33 2e .pci.0000:00:13.
[ 1180.835240] Object 0xffff8804097a91f0: 32 3a 20 5b 31 30 30 32 3a
34 33 39 36 5d 20 74 2:.[1002:4396].t
[ 1180.835240] Object 0xffff8804097a9200: 79 70 65 20 30 20 63 6c 61
73 73 20 30 78 30 30 ype.0.class.0x00
[ 1180.835240] Object 0xffff8804097a9210: 30 63 30 33 0d 0a 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b 0c03..kkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a9220: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a9230: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a9240: 6b 6b 6b 6b 6b 6b 6b 6b 00
00 00 00 00 00 00 00 kkkkkkkk........
[ 1180.835240] Object 0xffff8804097a9250: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.835240] Object 0xffff8804097a9260: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.835240] Object 0xffff8804097a9270: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.835240] Object 0xffff8804097a9280: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.835240] Object 0xffff8804097a9290: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.835240] Object 0xffff8804097a92a0: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.835240] Object 0xffff8804097a92b0: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.835240] Object 0xffff8804097a92c0: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.835240] Object 0xffff8804097a92d0: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.835240] Object 0xffff8804097a92e0: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.835240] Object 0xffff8804097a92f0: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.835240] Object 0xffff8804097a9300: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.835240] Object 0xffff8804097a9310: 00 00 00 00 00 00 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b ......kkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a9320: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a9330: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a9340: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a9350: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a9360: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a9370: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a9380: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a9390: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a93a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a93b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a93c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a93d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a93e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a93f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a9400: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a9410: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a9420: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a9430: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a9440: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a9450: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a9460: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a9470: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a9480: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a9490: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a94a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a94b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a94c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a94d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a94e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a94f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a9500: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a9510: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[ 1180.835240] Redzone 0xffff8804097a9520: cc cc cc cc cc cc cc cc
ÌÌÌÌÌÌÌÌ
[ 1180.835240] Padding 0xffff8804097a9560: 5a 5a 5a 5a 5a 5a 00 00
ZZZZZZ..
[ 1180.835240] Pid: 8295, comm: kworker/5:0 Not tainted 3.0.0-rc2 #1
[ 1180.835240] Call Trace:
[ 1180.835240] [<ffffffff810ba41d>] ? check_bytes_and_report+0x10d/0x150
[ 1180.835240] [<ffffffff810ba509>] ? check_object+0xa9/0x260
[ 1180.835240] [<ffffffff8125acf7>] ? tty_buffer_free_all+0x47/0x70
[ 1180.835240] [<ffffffff810bb033>] ? free_debug_processing+0x133/0x250
[ 1180.835240] [<ffffffff810bb27b>] ? __slab_free+0x12b/0x140
[ 1180.835240] [<ffffffff81254c70>] ? free_tty_struct+0x30/0x30
[ 1180.835240] [<ffffffff8125acf7>] ? tty_buffer_free_all+0x47/0x70
[ 1180.835240] [<ffffffff81254c66>] ? free_tty_struct+0x26/0x30
[ 1180.835240] [<ffffffff8104f28b>] ? process_one_work+0xfb/0x3b0
[ 1180.835240] [<ffffffff8104f964>] ? worker_thread+0x144/0x3d0
[ 1180.835240] [<ffffffff8102cc10>] ? __wake_up_common+0x50/0x80
[ 1180.835240] [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
[ 1180.835240] [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
[ 1180.835240] [<ffffffff81053436>] ? kthread+0x96/0xa0
[ 1180.835240] [<ffffffff813e1d14>] ? kernel_thread_helper+0x4/0x10
[ 1180.835240] [<ffffffff810533a0>] ? kthread_worker_fn+0x120/0x120
[ 1180.835240] [<ffffffff813e1d10>] ? gs_change+0xb/0xb
[ 1180.835240] FIX kmalloc-1024: Restoring
0xffff8804097a9566-0xffff8804097a9567=0x5a
[ 1180.835240]
[ 1180.881652]
=============================================================================
[ 1180.881713] BUG kmalloc-1024: Object padding overwritten
[ 1180.881760]
-----------------------------------------------------------------------------
[ 1180.881763]
[ 1180.881859] INFO: 0xffff8803fc70eb06-0xffff8803fc70eb07. First byte
0x0 instead of 0x5a
[ 1180.881920] INFO: Allocated in tty_buffer_request_room+0xcc/0x1a0
age=69297 cpu=0 pid=5978
[ 1180.881981] INFO: Freed in __blockdev_direct_IO+0xa16/0xa90 age=69384
cpu=5 pid=5146
[ 1180.882040] INFO: Slab 0xffffea000df389c0 objects=29 used=3
fp=0xffff8803fc70de30 flags=0x8000000000004081
[ 1180.882099] INFO: Object 0xffff8803fc70e6c0 @offset=26304
fp=0xffff8803fc70de30
[ 1180.882102]
[ 1180.882192] Bytes b4 0xffff8803fc70e6b0: aa 40 01 00 01 00 00 00 5a
5a 5a 5a 5a 5a 5a 5a ª@......ZZZZZZZZ
[ 1180.882884] Object 0xffff8803fc70e6c0: 68 95 6c fc 03 88 ff ff e8
e6 70 fc 03 88 ff ff h.lü..ÿÿèæpü..ÿÿ
[ 1180.883583] Object 0xffff8803fc70e6d0: e8 e7 70 fc 03 88 ff ff 00
01 00 00 00 01 00 00 èçpü..ÿÿ........
[ 1180.884542] Object 0xffff8803fc70e6e0: 00 01 00 00 00 01 00 00 5b
20 20 20 20 32 2e 33 ........[....2.3
[ 1180.885223] Object 0xffff8803fc70e6f0: 32 30 38 31 38 5d 20 70 63
69 20 30 30 30 30 3a 20818].pci.0000:
[ 1180.886169] Object 0xffff8803fc70e700: 30 30 3a 30 34 2e 30 3a 20
50 4d 45 23 20 73 75 00:04.0:.PME#.su
[ 1180.886850] Object 0xffff8803fc70e710: 70 70 6f 72 74 65 64 20 66
72 6f 6d 20 44 30 20 pported.from.D0.
[ 1180.887535] Object 0xffff8803fc70e720: 44 33 68 6f 74 20 44 33 63
6f 6c 64 0d 0a 5b 20 D3hot.D3cold..[.
[ 1180.888848] Object 0xffff8803fc70e730: 20 20 20 32 2e 33 32 30 38
32 30 5d 20 70 63 69 ...2.320820].pci
[ 1180.889537] Object 0xffff8803fc70e740: 20 30 30 30 30 3a 30 30 3a
30 34 2e 30 3a 20 50 .0000:00:04.0:.P
[ 1180.890481] Object 0xffff8803fc70e750: 4d 45 23 20 64 69 73 61 62
6c 65 64 0d 0a 5b 20 ME#.disabled..[.
[ 1180.891161] Object 0xffff8803fc70e760: 20 20 20 32 2e 33 32 30 38
33 36 5d 20 70 63 69 ...2.320836].pci
[ 1180.891365] Object 0xffff8803fc70e770: 20 30 30 30 30 3a 30 30 3a
30 39 2e 30 3a 20 5b .0000:00:09.0:.[
[ 1180.891365] Object 0xffff8803fc70e780: 31 30 32 32 3a 39 36 30 38
5d 20 74 79 70 65 20 1022:9608].type.
[ 1180.891365] Object 0xffff8803fc70e790: 31 20 63 6c 61 73 73 20 30
78 30 30 30 36 30 34 1.class.0x000604
[ 1180.891365] Object 0xffff8803fc70e7a0: 0d 0a 5b 20 20 20 20 32 2e
33 32 30 38 35 35 5d ..[....2.320855]
[ 1180.891365] Object 0xffff8803fc70e7b0: 20 70 63 69 20 30 30 30 30
3a 30 30 3a 30 39 2e .pci.0000:00:09.
[ 1180.891365] Object 0xffff8803fc70e7c0: 30 3a 20 50 4d 45 23 20 73
75 70 70 6f 72 74 65 0:.PME#.supporte
[ 1180.891365] Object 0xffff8803fc70e7d0: 64 20 66 72 6f 6d 20 44 30
20 44 33 68 6f 74 20 d.from.D0.D3hot.
[ 1180.891365] Object 0xffff8803fc70e7e0: 44 33 63 6f 6c 64 0d 0a 00
00 00 00 00 00 00 00 D3cold..........
[ 1180.891365] Object 0xffff8803fc70e7f0: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.891365] Object 0xffff8803fc70e800: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.891365] Object 0xffff8803fc70e810: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.891365] Object 0xffff8803fc70e820: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.891365] Object 0xffff8803fc70e830: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.891365] Object 0xffff8803fc70e840: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.891365] Object 0xffff8803fc70e850: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.891365] Object 0xffff8803fc70e860: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.891365] Object 0xffff8803fc70e870: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.891365] Object 0xffff8803fc70e880: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.891365] Object 0xffff8803fc70e890: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.891365] Object 0xffff8803fc70e8a0: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.891365] Object 0xffff8803fc70e8b0: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.891365] Object 0xffff8803fc70e8c0: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.891365] Object 0xffff8803fc70e8d0: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.891365] Object 0xffff8803fc70e8e0: 00 00 00 00 00 00 00 00 6b
6b 6b 6b 6b 6b 6b 6b ........kkkkkkkk
[ 1180.891365] Object 0xffff8803fc70e8f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70e900: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70e910: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70e920: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70e930: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70e940: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70e950: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70e960: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70e970: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70e980: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70e990: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70e9a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70e9b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70e9c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70e9d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70e9e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70e9f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70ea00: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70ea10: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70ea20: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70ea30: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70ea40: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70ea50: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70ea60: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70ea70: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70ea80: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70ea90: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70eaa0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70eab0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[ 1180.891365] Redzone 0xffff8803fc70eac0: cc cc cc cc cc cc cc cc
ÌÌÌÌÌÌÌÌ
[ 1180.891365] Padding 0xffff8803fc70eb00: 5a 5a 5a 5a 5a 5a 00 00
ZZZZZZ..
[ 1180.891365] Pid: 8295, comm: kworker/5:0 Not tainted 3.0.0-rc2 #1
[ 1180.891365] Call Trace:
[ 1180.891365] [<ffffffff810ba41d>] ? check_bytes_and_report+0x10d/0x150
[ 1180.891365] [<ffffffff810ba509>] ? check_object+0xa9/0x260
[ 1180.891365] [<ffffffff8125acf7>] ? tty_buffer_free_all+0x47/0x70
[ 1180.891365] [<ffffffff810bb033>] ? free_debug_processing+0x133/0x250
[ 1180.891365] [<ffffffff810bb27b>] ? __slab_free+0x12b/0x140
[ 1180.891365] [<ffffffff81254c70>] ? free_tty_struct+0x30/0x30
[ 1180.891365] [<ffffffff8125acf7>] ? tty_buffer_free_all+0x47/0x70
[ 1180.891365] [<ffffffff81254c66>] ? free_tty_struct+0x26/0x30
[ 1180.891365] [<ffffffff8104f28b>] ? process_one_work+0xfb/0x3b0
[ 1180.891365] [<ffffffff8104f964>] ? worker_thread+0x144/0x3d0
[ 1180.891365] [<ffffffff8102cc10>] ? __wake_up_common+0x50/0x80
[ 1180.891365] [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
[ 1180.891365] [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
[ 1180.891365] [<ffffffff81053436>] ? kthread+0x96/0xa0
[ 1180.891365] [<ffffffff813e1d14>] ? kernel_thread_helper+0x4/0x10
[ 1180.891365] [<ffffffff810533a0>] ? kthread_worker_fn+0x120/0x120
[ 1180.891365] [<ffffffff813e1d10>] ? gs_change+0xb/0xb
[ 1180.891365] FIX kmalloc-1024: Restoring
0xffff8803fc70eb06-0xffff8803fc70eb07=0x5a
[ 1180.891365]

2011-06-07 15:35:32

by Patrick McHardy

[permalink] [raw]
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39

On 07.06.2011 16:40, Brad Campbell wrote:
> On 07/06/11 21:30, Patrick McHardy wrote:
>> On 07.06.2011 05:33, Brad Campbell wrote:
>>> On 07/06/11 04:10, Bart De Schuymer wrote:
>>>> Hi Brad,
>>>>
>>>> This has probably nothing to do with ebtables, so please rmmod in case
>>>> it's loaded.
>>>> A few questions I didn't directly see an answer to in the threads I
>>>> scanned...
>>>> I'm assuming you actually use the bridging firewall functionality. So,
>>>> what iptables modules do you use? Can you reduce your iptables rules to
>>>> a core that triggers the bug?
>>>> Or does it get triggered even with an empty set of firewall rules?
>>>> Are you using a stock .35 kernel or is it patched?
>>>> Is this something I can trigger on a poor guy's laptop or does it
>>>> require specialized hardware (I'm catching up on qemu/kvm...)?
>>>
>>> Not specialised hardware as such, I've just not been able to reproduce
>>> it outside of this specific operating scenario.
>>
>> The last similar problem we've had was related to the 32/64 bit compat
>> code. Are you running 32 bit userspace on a 64 bit kernel?
>
> No, 32 bit Guest OS, but a completely 64 bit userspace on a 64 bit kernel.
>
> Userspace is current Debian Stable. Kernel is Vanilla and qemu-kvm is
> current git
>
>
>>> I can't trigger it with empty firewall rules as it relies on a DNAT to
>>> occur. If I try it directly to the internal IP address (as I have to
>>> without netfilter loaded) then of course nothing fails.
>>>
>>> It's a pain in the bum as a fault, but it's one I can easily reproduce
>>> as long as I use the same set of circumstances.
>>>
>>> I'll try using 3.0-rc2 (current git) tonight, and if I can reproduce it
>>> on that then I'll attempt to pare down the IPTABLES rules to a bare
>>> minimum.
>>>
>>> It is nothing to do with ebtables as I don't compile it. I'm not really
>>> sure about "bridging firewall" functionality. I just use a couple of
>>> hand coded bash scripts to set the tables up.
>>
>> From one of your previous mails:
>>
>>> # CONFIG_BRIDGE_NF_EBTABLES is not set
>>
>> How about CONFIG_BRIDGE_NETFILTER?
>>
>
> It was compiled in.
>
> With the following table set I was able to reproduce the problem on
> 3.0-rc2. Replaced my IP with xxx.xxx.xxx.xxx, but otherwise unmodified

Which kernel was the last version without this problem?

> root@srv:~# iptables-save
> # Generated by iptables-save v1.4.10 on Tue Jun 7 22:11:30 2011
> *filter
> :INPUT ACCEPT [978:107619]
> :FORWARD ACCEPT [142:7068]
> :OUTPUT ACCEPT [1659:291870]
> -A INPUT -i ppp0 -m state --state RELATED,ESTABLISHED -j ACCEPT
> -A INPUT ! -i ppp0 -m state --state NEW -j ACCEPT
> -A INPUT -i ppp0 -j DROP
> COMMIT
> # Completed on Tue Jun 7 22:11:30 2011
> # Generated by iptables-save v1.4.10 on Tue Jun 7 22:11:30 2011
> *nat
> :PREROUTING ACCEPT [813:49170]
> :INPUT ACCEPT [91:7090]
> :OUTPUT ACCEPT [267:20731]
> :POSTROUTING ACCEPT [296:22281]
> -A PREROUTING -d xxx.xxx.xxx.xxx/32 ! -i ppp0 -p tcp -m tcp --dport 443
> -j DNAT --to-destination 192.168.253.198
> COMMIT
> # Completed on Tue Jun 7 22:11:30 2011
> # Generated by iptables-save v1.4.10 on Tue Jun 7 22:11:30 2011
> *mangle
> :PREROUTING ACCEPT [2729:274392]
> :INPUT ACCEPT [2508:262976]
> :FORWARD ACCEPT [142:7068]
> :OUTPUT ACCEPT [1674:293701]
> :POSTROUTING ACCEPT [2131:346411]
> -A FORWARD -o ppp0 -p tcp -m tcp --tcp-flags SYN,RST SYN -m tcpmss --mss
> 1400:1536 -j TCPMSS --clamp-mss-to-pmtu
> COMMIT
> # Completed on Tue Jun 7 22:11:30 2011

The main suspects would be NAT and TCPMSS. Did you also try whether
the crash occurs with only one of these these rules?

> I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
> the address the way I was doing it, so that's a no-go for me.

That's really weird since you're apparently not using any bridge
netfilter features. It shouldn't have any effect besides changing
at which point ip_tables is invoked. How are your network devices
configured (specifically any bridges)?

2011-06-07 18:04:11

by Bart De Schuymer

[permalink] [raw]
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39

Op 7/06/2011 16:40, Brad Campbell schreef:
> On 07/06/11 21:30, Patrick McHardy wrote:
>> On 07.06.2011 05:33, Brad Campbell wrote:
>>> On 07/06/11 04:10, Bart De Schuymer wrote:
>>>> Hi Brad,
>>>>
>>>> This has probably nothing to do with ebtables, so please rmmod in case
>>>> it's loaded.
>>>> A few questions I didn't directly see an answer to in the threads I
>>>> scanned...
>>>> I'm assuming you actually use the bridging firewall functionality. So,
>>>> what iptables modules do you use? Can you reduce your iptables
>>>> rules to
>>>> a core that triggers the bug?
>>>> Or does it get triggered even with an empty set of firewall rules?
>>>> Are you using a stock .35 kernel or is it patched?
>>>> Is this something I can trigger on a poor guy's laptop or does it
>>>> require specialized hardware (I'm catching up on qemu/kvm...)?
>>>
>>> Not specialised hardware as such, I've just not been able to reproduce
>>> it outside of this specific operating scenario.
>>
>> The last similar problem we've had was related to the 32/64 bit compat
>> code. Are you running 32 bit userspace on a 64 bit kernel?
>
> No, 32 bit Guest OS, but a completely 64 bit userspace on a 64 bit
> kernel.
>
> Userspace is current Debian Stable. Kernel is Vanilla and qemu-kvm is
> current git
>
If the bug is easily triggered with your guest os, then you could try to
capture the traffic with wireshark (or something else) in a
configuration that doesn't crash your system. Save the traffic in a pcap
file. Then you can see if resending that traffic in the vulnerable
configuration triggers the bug (I don't know if something in Windows
exists, but tcpreplay should work for Linux). Once you have such a
capture , chances are the bug is even easily reproducible by us (unless
it's hardware-specific). Success isn't guaranteed, but I think it's
worth a shot...

cheers,
Bart


--
Bart De Schuymer
http://www.artinalgorithms.be

2011-06-07 18:31:38

by Eric Dumazet

[permalink] [raw]
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39

Le mardi 07 juin 2011 à 17:35 +0200, Patrick McHardy a écrit :

> The main suspects would be NAT and TCPMSS. Did you also try whether
> the crash occurs with only one of these these rules?
>
> > I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
> > the address the way I was doing it, so that's a no-go for me.
>
> That's really weird since you're apparently not using any bridge
> netfilter features. It shouldn't have any effect besides changing
> at which point ip_tables is invoked. How are your network devices
> configured (specifically any bridges)?

Something in the kernel does

u16 *ptr = addr (given by kmalloc())

ptr[-1] = 0;

Could be an off-one error in a memmove()/memcopy() or loop...

I cant see a network issue here.

I checked arch/x86/lib/memmove_64.S and it seems fine.


2011-06-07 22:57:17

by Patrick McHardy

[permalink] [raw]
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39

On 07.06.2011 20:31, Eric Dumazet wrote:
> Le mardi 07 juin 2011 ? 17:35 +0200, Patrick McHardy a ?crit :
>
>> The main suspects would be NAT and TCPMSS. Did you also try whether
>> the crash occurs with only one of these these rules?
>>
>>> I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
>>> the address the way I was doing it, so that's a no-go for me.
>>
>> That's really weird since you're apparently not using any bridge
>> netfilter features. It shouldn't have any effect besides changing
>> at which point ip_tables is invoked. How are your network devices
>> configured (specifically any bridges)?
>
> Something in the kernel does
>
> u16 *ptr = addr (given by kmalloc())
>
> ptr[-1] = 0;
>
> Could be an off-one error in a memmove()/memcopy() or loop...
>
> I cant see a network issue here.

So far me neither, but netfilter appears to trigger the bug.

> I checked arch/x86/lib/memmove_64.S and it seems fine.

I was thinking it might be a missing skb_make_writable() combined
with vhost_net specifics in the netfilter code (TCPMSS and NAT are
both suspect), but was unable to find something. I also went
through the dst_metrics() conversion to see whether anything could
cause problems with the bridge fake_rttable, but also nothing
so far.

2011-06-08 00:24:52

by Brad Campbell

[permalink] [raw]
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39

On 07/06/11 23:35, Patrick McHardy wrote:

> The main suspects would be NAT and TCPMSS. Did you also try whether
> the crash occurs with only one of these these rules?

To be honest I'm actually having trouble finding where TCPMSS is
actually set in that ruleset. This is a production machine so I can only
take it down after about 9PM at night. I'll have another crack at it
tonight.

>> I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
>> the address the way I was doing it, so that's a no-go for me.
>
> That's really weird since you're apparently not using any bridge
> netfilter features. It shouldn't have any effect besides changing
> at which point ip_tables is invoked. How are your network devices
> configured (specifically any bridges)?
>

I have one bridge with all my virtual machines on it.

In this particular instance the packets leave VM A destined for the IP
address of ppp0 (the external interface). This is intercepted by the
DNAT PREROUTING rule above and shunted back to VM B.

The VM's are on br1 and the external address is ppp0. Without
CONFIG_BRIDGE_NETFILTER compiled in I can see the traffic entering and
leaving VM B with tcpdump, but the packets never seem to get back to VM A.

VM A is XP 32 bit, VM B is Linux. I have some other Linux VM's, so I'll
do some more testing tonight between those to see where the packets are
going without CONFIG_BRIDGE_NETFILTER set.

2011-06-08 00:15:20

by Brad Campbell

[permalink] [raw]
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39

On 08/06/11 02:04, Bart De Schuymer wrote:

> If the bug is easily triggered with your guest os, then you could try to
> capture the traffic with wireshark (or something else) in a
> configuration that doesn't crash your system. Save the traffic in a pcap
> file. Then you can see if resending that traffic in the vulnerable
> configuration triggers the bug (I don't know if something in Windows
> exists, but tcpreplay should work for Linux). Once you have such a
> capture , chances are the bug is even easily reproducible by us (unless
> it's hardware-specific). Success isn't guaranteed, but I think it's
> worth a shot...

The issue with this is I don't have a configuration that does not crash
the system. This only happens under the specific circumstance that
traffic from VM A is being DNAT'd to VM B. If I disable
CONFIG_BRIDGE_NETFILTER, or I leave out the DNAT then I can't replicate
the problem as I don't seem to be able to get the packets to go where I
want them to go.

Let me try and explain it a little more clearly with made up IP
addresses to illustrate the problem.

I have VM A (1.1.1.2) and VM B (1.1.1.3) on br1 (1.1.1.1)
I have public IP on ppp0 (2.2.2.2).

VM B can talk to VM A using its host address (1.1.1.2) and there is no
problem.

The DNAT says anything destined for PPP0 that is on port 443 and coming
from anywhere other than PPP0 (ie inside the network) is to be DNAT'd to
1.1.1.3.

So VM B (1.1.1.3) tries to connect to ppp0 (2.2.2.2) on port 443, and
this is redirected to VM B on 1.1.1.2.

Only under this specific circumstance does the problem occur. I can get
VM B (1.1.1.3) to talk directly to VM A (1.1.1.2) all day long and there
is no problem, it's only when VM B tries to talk to ppp0 that there is
an issue (and it happens within seconds of the initial connection).

All these tests have been performed with VM B being a Windows XP guest.
Tonight I'll try it with a Linux guest and see if I can make it happen.
If that works I might be able to come up with some reproducible test
case for you. I have a desktop machine that has Intel VT extensions, so
I'll work toward making a portable test case.

2011-06-08 00:18:18

by Brad Campbell

[permalink] [raw]
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39

On 08/06/11 06:57, Patrick McHardy wrote:
> On 07.06.2011 20:31, Eric Dumazet wrote:
>> Le mardi 07 juin 2011 ? 17:35 +0200, Patrick McHardy a ?crit :
>>
>>> The main suspects would be NAT and TCPMSS. Did you also try whether
>>> the crash occurs with only one of these these rules?
>>>
>>>> I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
>>>> the address the way I was doing it, so that's a no-go for me.
>>>
>>> That's really weird since you're apparently not using any bridge
>>> netfilter features. It shouldn't have any effect besides changing
>>> at which point ip_tables is invoked. How are your network devices
>>> configured (specifically any bridges)?
>>
>> Something in the kernel does
>>
>> u16 *ptr = addr (given by kmalloc())
>>
>> ptr[-1] = 0;
>>
>> Could be an off-one error in a memmove()/memcopy() or loop...
>>
>> I cant see a network issue here.
>
> So far me neither, but netfilter appears to trigger the bug.

Would it help if I tried some older kernels? This issue only surfaced
for me recently as I only installed the VM's in question about 12 weeks
ago and have only just started really using them in anger. I could try
reproducing it on progressively older kernels to see if I can find one
that works and then bisect from there.

2011-06-08 03:59:12

by Eric Dumazet

[permalink] [raw]
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39

Le mercredi 08 juin 2011 à 08:18 +0800, Brad Campbell a écrit :
> On 08/06/11 06:57, Patrick McHardy wrote:
> > On 07.06.2011 20:31, Eric Dumazet wrote:
> >> Le mardi 07 juin 2011 à 17:35 +0200, Patrick McHardy a écrit :
> >>
> >>> The main suspects would be NAT and TCPMSS. Did you also try whether
> >>> the crash occurs with only one of these these rules?
> >>>
> >>>> I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
> >>>> the address the way I was doing it, so that's a no-go for me.
> >>>
> >>> That's really weird since you're apparently not using any bridge
> >>> netfilter features. It shouldn't have any effect besides changing
> >>> at which point ip_tables is invoked. How are your network devices
> >>> configured (specifically any bridges)?
> >>
> >> Something in the kernel does
> >>
> >> u16 *ptr = addr (given by kmalloc())
> >>
> >> ptr[-1] = 0;
> >>
> >> Could be an off-one error in a memmove()/memcopy() or loop...
> >>
> >> I cant see a network issue here.
> >
> > So far me neither, but netfilter appears to trigger the bug.
>
> Would it help if I tried some older kernels? This issue only surfaced
> for me recently as I only installed the VM's in question about 12 weeks
> ago and have only just started really using them in anger. I could try
> reproducing it on progressively older kernels to see if I can find one
> that works and then bisect from there.

Well, a bisection definitely should help, but needs a lot of time in
your case.

Could you try following patch, because this is the 'usual suspect' I had
yesterday :

diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index 46cbd28..9f548f9 100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -792,6 +792,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
fastpath = atomic_read(&skb_shinfo(skb)->dataref) == delta;
}

+#if 0
if (fastpath &&
size + sizeof(struct skb_shared_info) <= ksize(skb->head)) {
memmove(skb->head + size, skb_shinfo(skb),
@@ -802,7 +803,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
off = nhead;
goto adjust_others;
}
-
+#endif
data = kmalloc(size + sizeof(struct skb_shared_info), gfp_mask);
if (!data)
goto nodata;

2011-06-08 17:40:10

by Brad Campbell

[permalink] [raw]
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39

On 08/06/11 11:59, Eric Dumazet wrote:

> Well, a bisection definitely should help, but needs a lot of time in
> your case.

Yes. compile, test, crash, walk out to the other building to press
reset, lather, rinse, repeat.

I need a reset button on the end of a 50M wire, or a hardware watchdog!

Actually it's not so bad. If I turn off slub debugging the kernel panics
and reboots itself.

This.. :
[ 2.913034] netconsole: remote ethernet address 00:16:cb:a7:dd:d1
[ 2.913066] netconsole: device eth0 not up yet, forcing it
[ 3.660062] Refined TSC clocksource calibration: 3213.422 MHz.
[ 3.660118] Switching to clocksource tsc
[ 63.200273] r8169 0000:03:00.0: eth0: unable to load firmware patch
rtl_nic/rtl8168e-1.fw (-2)
[ 63.223513] r8169 0000:03:00.0: eth0: link down
[ 63.223556] r8169 0000:03:00.0: eth0: link down

..is slowing down reboots considerably. 3.0-rc does _not_ like some
timing hardware in my machine. Having said that, at least it does not
randomly panic on SCSI like 2.6.39 does.

Ok, I've ruled out TCPMSS. Found out where it was being set and neutered
it. I've replicated it with only the single DNAT rule.


> Could you try following patch, because this is the 'usual suspect' I had
> yesterday :
>
> diff --git a/net/core/skbuff.c b/net/core/skbuff.c
> index 46cbd28..9f548f9 100644
> --- a/net/core/skbuff.c
> +++ b/net/core/skbuff.c
> @@ -792,6 +792,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
> fastpath = atomic_read(&skb_shinfo(skb)->dataref) == delta;
> }
>
> +#if 0
> if (fastpath&&
> size + sizeof(struct skb_shared_info)<= ksize(skb->head)) {
> memmove(skb->head + size, skb_shinfo(skb),
> @@ -802,7 +803,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
> off = nhead;
> goto adjust_others;
> }
> -
> +#endif
> data = kmalloc(size + sizeof(struct skb_shared_info), gfp_mask);
> if (!data)
> goto nodata;
>
>
>

Nope.. that's not it. <sigh> That might have changed the characteristic
of the fault slightly, but unfortunately I got caught with a couple of
fsck's, so I only got to test it 3 times tonight.

It's unfortunate that this is a production system, so I can only take it
down between about 9pm and 1am. That would normally be pretty
productive, except that an fsck of a 14TB ext4 can take 30 minutes if it
panics at the wrong time.

I'm out of time tonight, but I'll have a crack at some bisection
tomorrow night. Now I just have to go back far enough that it works, and
be near enough not to have to futz around with /proc /sys or drivers.

I really, really, really appreciate you guys helping me with this. It
has been driving me absolutely bonkers. If I'm ever in the same town as
any of you, dinner and drinks are on me.

2011-06-08 21:22:38

by Eric Dumazet

[permalink] [raw]
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39

Le jeudi 09 juin 2011 à 01:02 +0800, Brad Campbell a écrit :
> On 08/06/11 11:59, Eric Dumazet wrote:
>
> > Well, a bisection definitely should help, but needs a lot of time in
> > your case.
>
> Yes. compile, test, crash, walk out to the other building to press
> reset, lather, rinse, repeat.
>
> I need a reset button on the end of a 50M wire, or a hardware watchdog!
>
> Actually it's not so bad. If I turn off slub debugging the kernel panics
> and reboots itself.
>
> This.. :
> [ 2.913034] netconsole: remote ethernet address 00:16:cb:a7:dd:d1
> [ 2.913066] netconsole: device eth0 not up yet, forcing it
> [ 3.660062] Refined TSC clocksource calibration: 3213.422 MHz.
> [ 3.660118] Switching to clocksource tsc
> [ 63.200273] r8169 0000:03:00.0: eth0: unable to load firmware patch
> rtl_nic/rtl8168e-1.fw (-2)
> [ 63.223513] r8169 0000:03:00.0: eth0: link down
> [ 63.223556] r8169 0000:03:00.0: eth0: link down
>
> ..is slowing down reboots considerably. 3.0-rc does _not_ like some
> timing hardware in my machine. Having said that, at least it does not
> randomly panic on SCSI like 2.6.39 does.
>
> Ok, I've ruled out TCPMSS. Found out where it was being set and neutered
> it. I've replicated it with only the single DNAT rule.
>
>
> > Could you try following patch, because this is the 'usual suspect' I had
> > yesterday :
> >
> > diff --git a/net/core/skbuff.c b/net/core/skbuff.c
> > index 46cbd28..9f548f9 100644
> > --- a/net/core/skbuff.c
> > +++ b/net/core/skbuff.c
> > @@ -792,6 +792,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
> > fastpath = atomic_read(&skb_shinfo(skb)->dataref) == delta;
> > }
> >
> > +#if 0
> > if (fastpath&&
> > size + sizeof(struct skb_shared_info)<= ksize(skb->head)) {
> > memmove(skb->head + size, skb_shinfo(skb),
> > @@ -802,7 +803,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
> > off = nhead;
> > goto adjust_others;
> > }
> > -
> > +#endif
> > data = kmalloc(size + sizeof(struct skb_shared_info), gfp_mask);
> > if (!data)
> > goto nodata;
> >
> >
> >
>
> Nope.. that's not it. <sigh> That might have changed the characteristic
> of the fault slightly, but unfortunately I got caught with a couple of
> fsck's, so I only got to test it 3 times tonight.
>
> It's unfortunate that this is a production system, so I can only take it
> down between about 9pm and 1am. That would normally be pretty
> productive, except that an fsck of a 14TB ext4 can take 30 minutes if it
> panics at the wrong time.
>
> I'm out of time tonight, but I'll have a crack at some bisection
> tomorrow night. Now I just have to go back far enough that it works, and
> be near enough not to have to futz around with /proc /sys or drivers.
>
> I really, really, really appreciate you guys helping me with this. It
> has been driving me absolutely bonkers. If I'm ever in the same town as
> any of you, dinner and drinks are on me.

Hmm, I wonder if kmemcheck could help you, but its slow as hell, so not
appropriate for production :(


2011-06-10 02:52:57

by Simon Horman

[permalink] [raw]
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39

On Thu, Jun 09, 2011 at 01:02:13AM +0800, Brad Campbell wrote:
> On 08/06/11 11:59, Eric Dumazet wrote:
>
> >Well, a bisection definitely should help, but needs a lot of time in
> >your case.
>
> Yes. compile, test, crash, walk out to the other building to press
> reset, lather, rinse, repeat.
>
> I need a reset button on the end of a 50M wire, or a hardware watchdog!

Not strictly on-topic, but in situations where I have machines
that either don't have lights-out facilities or have broken ones
I find that network controlled power switches to be very useful.

At one point I would have need an 8000km long wire to the reset switch :-)

2011-06-10 12:37:29

by Mark Lord

[permalink] [raw]
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39

On 11-06-09 10:52 PM, Simon Horman wrote:
> On Thu, Jun 09, 2011 at 01:02:13AM +0800, Brad Campbell wrote:
>> On 08/06/11 11:59, Eric Dumazet wrote:
>>
>>> Well, a bisection definitely should help, but needs a lot of time in
>>> your case.
>>
>> Yes. compile, test, crash, walk out to the other building to press
>> reset, lather, rinse, repeat.
>>
>> I need a reset button on the end of a 50M wire, or a hardware watchdog!


Something many of us don't realize is that nearly all Intel chipsets
have a built-in hardware watchdog timer. This includes chipset for
consumer desktop boards as well as the big iron server stuff.

It's the "i8xx_tco" driver in the kernel enables use of them:

modprobe i8xx_tco

Cheers

Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39

On Fri, 10 Jun 2011, Mark Lord wrote:
> Something many of us don't realize is that nearly all Intel chipsets
> have a built-in hardware watchdog timer. This includes chipset for
> consumer desktop boards as well as the big iron server stuff.
>
> It's the "i8xx_tco" driver in the kernel enables use of them:

That's the old module name, but yes, it is very useful in desktops and
laptops (when it works). Server-class hardware will have a baseboard
management unit that can really power-cycle the system instead of just
rebooting.

And test it first before you depend on it triggering at a remote location,
as the firmware might cause the Intel chipset watchdog to actually hang the
box instead of causing a proper reboot (happens on the IBM thinkpad T43, for
example).

--
"One disk to rule them all, One disk to find them. One disk to bring
them all and in the darkness grind them. In the Land of Redmond
where the shadows lie." -- The Silicon Valley Tarot
Henrique Holschuh

2011-06-12 15:38:38

by Avi Kivity

[permalink] [raw]
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39

On 06/10/2011 05:52 AM, Simon Horman wrote:
> At one point I would have need an 8000km long wire to the reset switch :-)

Even more off-topic, there has been a case when a 200,000,000 km long
wire to the reset button was needed. IIRC they got away with a watchdog.

--
error compiling committee.c: too many arguments to function