I started seeing this very soon after creating interfaces.
Maybe caused by a write-after-free?
This is on the ath9k system, and I enabled SLUB debugging,
etc.
Oct 5 09:50:18 localhost kernel: =============================================================================
Oct 5 09:50:18 localhost kernel: BUG kmalloc-8192: Poison overwritten
Oct 5 09:50:18 localhost kernel: -----------------------------------------------------------------------------
Oct 5 09:50:18 localhost kernel:
Oct 5 09:50:18 localhost kernel: INFO: 0xf5b18040-0xf5b1812f. First byte 0x80 instead of 0x6b
Oct 5 09:50:18 localhost kernel: INFO: Allocated in ath_rxbuf_alloc+0x1d/0x74 [ath] age=54091 cpu=0 pid=613
Oct 5 09:50:18 localhost kernel: INFO: Freed in skb_release_data+0x8c/0x90 age=89 cpu=0 pid=4029
Oct 5 09:50:18 localhost kernel: INFO: Slab 0xc1fef300 objects=3 used=2 fp=0xf5b18000 flags=0x400040c1
Oct 5 09:50:18 localhost kernel: INFO: Object 0xf5b18000 @offset=0 fp=0x(null)
Oct 5 09:50:18 localhost kernel:
Oct 5 09:50:18 localhost kernel: Object 0xf5b18000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18020: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18030: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18040: 80 00 00 00 ff ff ff ff ff ff c8 4c 75 20 e8 fd ....???????Lu.??
Oct 5 09:50:18 localhost kernel: Object 0xf5b18050: c8 4c 75 20 e8 fd d0 88 80 c1 e1 03 00 00 00 00 ?Lu.???..??.....
Oct 5 09:50:18 localhost kernel: Object 0xf5b18060: 90 01 21 04 00 09 63 69 73 63 6f 73 62 2d 31 01 ..!...ciscosb-1.
Oct 5 09:50:18 localhost kernel: Object 0xf5b18070: 08 82 84 8b 96 0c 12 18 24 03 01 0b 05 04 00 01 ........$.......
Oct 5 09:50:18 localhost kernel: Object 0xf5b18080: 00 00 2a 01 00 32 04 30 48 60 6c dd 18 00 50 f2 ..*..2.0H`l?..P?
Oct 5 09:50:18 localhost kernel: Object 0xf5b18090: 02 01 01 84 00 03 a4 00 00 27 a4 00 00 42 43 5e ......?..'?..BC^
Oct 5 09:50:18 localhost kernel: Object 0xf5b180a0: 00 62 32 2f 00 dd 1e 00 90 4c 33 4c 10 1b ff ff .b2/.?...L3L..??
Oct 5 09:50:18 localhost kernel: Object 0xf5b180b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Oct 5 09:50:18 localhost kernel: Object 0xf5b180c0: 00 00 00 00 00 2d 1a 4c 10 1b ff ff 00 00 00 00 .....-.L..??....
Oct 5 09:50:18 localhost kernel: Object 0xf5b180d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Oct 5 09:50:18 localhost kernel: Object 0xf5b180e0: 00 dd 1a 00 90 4c 34 0b 08 08 00 00 00 00 00 00 .?...L4.........
Oct 5 09:50:18 localhost kernel: Object 0xf5b180f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 3d 16 0b .............=..
Oct 5 09:50:18 localhost kernel: Object 0xf5b18100: 08 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Oct 5 09:50:18 localhost kernel: Object 0xf5b18110: 00 00 00 00 00 dd 09 00 03 7f 01 01 00 00 ff 7f .....?........?.
Oct 5 09:50:18 localhost kernel: Object 0xf5b18120: dd 0a 00 03 7f 04 01 00 00 00 00 00 c9 74 23 19 ?...........?t#.
Oct 5 09:50:18 localhost kernel: Object 0xf5b18130: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18140: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18150: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18160: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18170: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18180: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18190: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b181a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b181b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b181c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b181d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b181e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b181f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18200: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18210: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18220: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18230: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18240: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18250: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18260: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18270: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18280: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18290: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b182a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b182b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b182c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b182d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b182e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b182f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18300: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18310: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18320: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18330: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18340: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18350: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18360: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18370: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18380: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18390: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b183a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b183b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b183c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b183d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b183e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b183f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18400: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18410: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18420: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18430: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18440: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18450: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18460: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18470: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18480: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18490: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b184a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b184b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b184c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b184d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b184e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b184f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18500: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18510: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18520: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18530: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18540: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18550: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18560: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18570: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18580: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18590: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b185a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b185b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b185c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b185d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b185e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b185f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18600: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18610: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18620: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18630: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18640: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18650: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18660: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18670: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18680: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18690: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b186a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b186b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b186c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b186d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b186e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b186f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18700: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18710: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18720: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18730: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18740: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18750: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18760: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18770: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18780: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18790: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b187a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b187b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b187c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b187d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b187e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b187f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18800: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18810: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18820: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18830: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18840: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18850: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18860: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18870: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18880: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18890: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b188a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b188b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b188c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b188d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b188e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b188f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18900: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18910: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18920: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18930: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18940: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18950: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18960: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18970: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18980: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18990: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b189a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b189b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b189c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b189d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b189e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b189f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18a00: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18a10: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18a20: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18a30: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18a40: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18a50: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18a60: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18a70: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18a80: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18a90: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18aa0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18ab0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18ac0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18ad0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18ae0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18af0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18b00: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18b10: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18b20: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18b30: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18b40: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18b50: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18b60: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18b70: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18b80: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18b90: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18ba0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18bb0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18bc0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18bd0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18be0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18bf0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18c00: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18c10: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18c20: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18c30: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18c40: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18c50: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18c60: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18c70: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18c80: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18c90: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18ca0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18cb0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18cc0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18cd0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18ce0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18cf0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18d00: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18d10: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18d20: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18d30: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18d40: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18d50: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18d60: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18d70: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18d80: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18d90: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18da0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18db0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18dc0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18dd0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18de0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18df0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18e00: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18e10: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18e20: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18e30: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18e40: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18e50: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18e60: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18e70: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18e80: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18e90: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18ea0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18eb0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18ec0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18ed0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18ee0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18ef0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18f00: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18f10: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18f20: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18f30: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18f40: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18f50: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18f60: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18f70: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18f80: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18f90: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18fa0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18fb0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18fc0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18fd0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18fe0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Object 0xf5b18ff0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 09:50:18 localhost kernel: Redzone 0xf5b1a000: bb bb bb bb ????
Oct 5 09:50:18 localhost kernel: Padding 0xf5b1a028: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ
Oct 5 09:50:18 localhost kernel: Pid: 4049, comm: wpa_supplicant Not tainted 2.6.36-rc6-wl+ #5
Oct 5 09:50:18 localhost kernel: Call Trace:
Oct 5 09:50:18 localhost kernel: [<c04ab513>] print_trailer+0xdc/0xe4
Oct 5 09:50:18 localhost kernel: [<c04ab8e9>] check_bytes_and_report+0x96/0xcc
Oct 5 09:50:18 localhost kernel: [<c04ab9c6>] check_object+0xa7/0x15d
Oct 5 09:50:18 localhost kernel: [<c04acc37>] __slab_alloc+0x339/0x3dd
Oct 5 09:50:18 localhost kernel: [<c0455df5>] ? mark_held_locks+0x47/0x5f
Oct 5 09:50:18 localhost kernel: [<c04ad0d2>] ? kmem_cache_alloc+0xa0/0xc5
Oct 5 09:50:18 localhost kernel: [<c04ad6fd>] __kmalloc_track_caller+0xa1/0xf2
Oct 5 09:50:18 localhost kernel: [<c06c7331>] ? skb_copy+0x2e/0x83
Oct 5 09:50:18 localhost kernel: [<c06c7331>] ? skb_copy+0x2e/0x83
Oct 5 09:50:18 localhost kernel: [<c06c6d6a>] __alloc_skb+0x58/0xf4
Oct 5 09:50:18 localhost kernel: [<c06c7331>] skb_copy+0x2e/0x83
Oct 5 09:50:18 localhost kernel: [<f8d706dd>] ieee80211_prepare_and_rx_handle+0x3b1/0x86d [mac80211]
Oct 5 09:50:18 localhost kernel: [<c0455bee>] ? mark_lock+0x1e/0x1de
Oct 5 09:50:18 localhost kernel: [<f8d5f8cf>] ? rcu_read_lock_held+0x1d/0x1f [mac80211]
Oct 5 09:50:18 localhost kernel: [<f8d5facf>] ? sta_info_get_bss+0xcb/0x10c [mac80211]
Oct 5 09:50:18 localhost kernel: [<f8d71352>] ieee80211_rx+0x70f/0x7b5 [mac80211]
Oct 5 09:50:18 localhost kernel: [<c04ad729>] ? __kmalloc_track_caller+0xcd/0xf2
Oct 5 09:50:18 localhost kernel: [<c0456038>] ? trace_hardirqs_on_caller+0xeb/0x125
Oct 5 09:50:18 localhost kernel: [<f8c55836>] ath_rx_send_to_mac80211+0x5a/0x60 [ath9k]
Oct 5 09:50:18 localhost kernel: [<c045607d>] ? trace_hardirqs_on+0xb/0xd
Oct 5 09:50:18 localhost kernel: [<f8c57066>] ath_rx_tasklet+0x1253/0x130e [ath9k]
Oct 5 09:50:18 localhost kernel: [<f8c55232>] ath9k_tasklet+0x9a/0x12a [ath9k]
Oct 5 09:50:18 localhost kernel: [<c0438fd1>] tasklet_action+0x73/0xc6
Oct 5 09:50:18 localhost kernel: [<c043945f>] __do_softirq+0x86/0x111
Oct 5 09:50:18 localhost kernel: [<c0439520>] do_softirq+0x36/0x5a
Oct 5 09:50:18 localhost kernel: [<c0439659>] irq_exit+0x35/0x69
Oct 5 09:50:18 localhost kernel: [<c0403fb9>] do_IRQ+0x86/0x9a
Oct 5 09:50:18 localhost kernel: [<c04034ee>] common_interrupt+0x2e/0x40
Oct 5 09:50:18 localhost kernel: [<c045007b>] ? do_adjtimex+0x217/0x55e
Oct 5 09:50:18 localhost kernel: [<c04ad0d7>] ? kmem_cache_alloc+0xa5/0xc5
Oct 5 09:50:18 localhost kernel: [<c04bfe3a>] ? getname+0x1b/0xb8
Oct 5 09:50:18 localhost kernel: [<c04bfe3a>] getname+0x1b/0xb8
Oct 5 09:50:18 localhost kernel: [<c04b5f3e>] do_sys_open+0x14/0xd0
Oct 5 09:50:18 localhost kernel: [<c04b603c>] sys_open+0x1e/0x26
Oct 5 09:50:18 localhost kernel: [<c0760585>] syscall_call+0x7/0xb
Oct 5 09:50:18 localhost kernel: FIX kmalloc-8192: Restoring 0xf5b18040-0xf5b1812f=0x6b
Oct 5 09:50:18 localhost kernel:
--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
On 10/05/2010 11:14 AM, Ben Greear wrote:
> On 10/05/2010 10:55 AM, Luis R. Rodriguez wrote:
>> On Tue, Oct 5, 2010 at 10:47 AM, Ben Greear<[email protected]>
>> wrote:
>
>>>> Please do, you can use the log2n approach, should take you 7 tries,
>>>> each by a power of 2, start at the middle of course.
>>>
>>> Sure...will be a few minutes.
>>>
>>> If it's any sane bug, then hopefully I can hit it with a small
>>> number.
>>>
>>> Out of curiosity, is there any particular number of STAs> 1 you guys
>>> test with?
>>
>> We don't test this :)
>
> Heh, I sort of supposed so :)
>
> Anyway, I started with one, doubled each time,
> and when I added the 4 to bring the system to
> 8 STA, it crapped itself.
>
> I ran 64kbps UDP data streams on pairs of STA interfaces
> for 5 minutes between STA increments. I didn't get a chance to
> start any traffic for the 4 that I had just added.
>
> Earlier, I wasn't running any traffic at all, but of course
> there is still noise on the network.
>
> Note I'm running 'iwconfig foo' and 'cat /proc/net/wireless' in
> the background as well, and that may have something to do with
> things.
I tried with the wmb() patch, and the problem still exists, so
it must be something else.
Thanks,
Ben
--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
Hi Ben,
First of all keep up the good work. :)
On Wed, Oct 13, 2010 at 6:39 PM, Ben Greear <[email protected]> wrote:
[snip]
> Either way, it seems safer to null out the bf_ampdu field after
> the memory is consumed..it could prevent some tricky bugs later.
I think this is a good idea. But it probably wont be enough to null
out bf_mpdu. You also need to look at bf_buf_addr (which if I
understand correctly is the physical address the DMA engine will
actually write RXed frames to) and bf_dmacontext (which seems in most
cases to hold an identical address and may in fact be where the DMA
engine will really write the frame).
/Bj?rn
On 10/11/2010 11:10 PM, Luis R. Rodriguez wrote:
> On Mon, Oct 11, 2010 at 8:27 PM, Ben Greear<[email protected]> wrote:
>> Another thing I was thinking about: Maybe the queue of skbs and dma
>> addresses
>> in ath9k is getting corrupted by multiple VIFs trying to write at once?
>> Maybe
>> some locking is needed in the xmit path?
>
> That was my second hunch. My first shot was to use spin_lock_irqsave()
> over the the uses of the rxbuf list and that seemed to help but I
> still managed to get a poison eventually. My next item to check for is
> of the permissibility of creating too much pressure to the point we
> end up looping over the rxbuf list and race against mac80211 free'ing
> a buffer. Will test that tomorrow if nothing else comes up creeping my
> priority queue.
This code looks weird to me. One of the paprd branches
deletes the skb, the other doesn't appear to. Neither
null out bf->bf_mpdu, which would appear to leave a dangling
pointer in at least the dev_kfree_skb_any() branch.
ath_tx_complete frees it's skb in all cases, so another
bf->bf_mpdu dangling pointer issue.
Maybe at the least we should null out bf->bf_mpdu when
skb is consumed?
static void ath_tx_complete_buf(struct ath_softc *sc, struct ath_buf *bf,
struct ath_txq *txq, struct list_head *bf_q,
struct ath_tx_status *ts, int txok, int sendbar)
{
struct sk_buff *skb = bf->bf_mpdu;
unsigned long flags;
int tx_flags = 0;
if (sendbar)
tx_flags = ATH_TX_BAR;
if (!txok) {
tx_flags |= ATH_TX_ERROR;
if (bf_isxretried(bf))
tx_flags |= ATH_TX_XRETRY;
}
dma_unmap_single(sc->dev, bf->bf_dmacontext, skb->len, DMA_TO_DEVICE);
if (bf->bf_state.bfs_paprd) {
if (time_after(jiffies,
bf->bf_state.bfs_paprd_timestamp +
msecs_to_jiffies(ATH_PAPRD_TIMEOUT)))
dev_kfree_skb_any(skb);
else
complete(&sc->paprd_complete);
} else {
ath_tx_complete(sc, skb, bf->aphy, tx_flags);
ath_debug_stat_tx(sc, txq, bf, ts);
}
>
> Luis
--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
On Thu, Oct 14, 2010 at 03:44:49PM -0700, Ben Greear wrote:
> On 10/14/2010 03:35 PM, Luis R. Rodriguez wrote:
> > On Thu, Oct 14, 2010 at 3:29 PM, Luis R. Rodriguez<[email protected]> wrote:
> >> Fun enough if I just create one monitor interface and loop quickly
> >> over some 2 GHz channels where I know I have traffic nearby I don't
> >> see the poison. So channel changes don't seem to do much because this
> >> is changing channels as fast as possible from userspace. I also can
> >> confirm that I see frames from the different channels as I move along.
> >
> > Even forcing a band change doesn't help trigger it with just one mon0
> > and one regular device scanning in a loop;
> >
> > while true; do for i in 2412 5745 2417 5745 2422 5745 2427 5745 2432
> > 5745 2442; do echo $i iw dev mon0 set freq $i; done; done
> > while true; do iw dev wlan0 scan; done
>
> What if you just make a bunch of skb copies in ath9k before it
> sends them up the stack, and then delete them? (That's basically
> what a bunch of monitor devices would be doing, eh?)
Sure, as you can see from my patch I at least do it all the time
now on RX and TX. The TX poison never shows though so currently I'm
more suspicious about RX. The other idea I had was to just run
check_bytes_and_report() often around the code, but haven't tried that
yet.
I'm also poking harware folks about some of the registers we use
to better understand how DMA works here.
Luis
On 10/05/2010 10:43 AM, Luis R. Rodriguez wrote:
> On Tue, Oct 5, 2010 at 10:38 AM, Ben Greear<[email protected]> wrote:
>> On 10/05/2010 10:36 AM, Luis R. Rodriguez wrote:
>>>
>>> On Tue, Oct 5, 2010 at 10:24 AM, Ben Greear<[email protected]>
>>> wrote:
>>>>
>>>> On 10/05/2010 10:16 AM, Luis R. Rodriguez wrote:
>>>>>
>>>>> On Tue, Oct 5, 2010 at 10:00 AM, Ben Greear<[email protected]>
>>>>> wrote:
>>>>>>
>>>>>> I started seeing this very soon after creating interfaces.
>>>>>
>>>>> Can you be more specific how one can reproduce?
>>>>
>>>> Enable SLUB debugging, DEBUG_PAGEALLOC (Debug page memory allocations),
>>>> lockdep, pre-empt.
>>>
>>> OK I just enabled PAGE_ALLOC thingy, it wasn't enabled on my kernel.
>>>
>>>> Please try creating a bunch of stations on one ath9k phy,
>>>
>>> How many?
>>
>> 130 is a nice round number :)
>
> Let me get this straight. You are creating 130 STAs with ath9k using
> iw, then running 150 instances of wpa_supplicant to connect to the
> same AP?
Well, 130 instances, yes :)
>> I'll try with a smaller number and see if I can hit it.
>
> Please do, you can use the log2n approach, should take you 7 tries,
> each by a power of 2, start at the middle of course.
Sure...will be a few minutes.
If it's any sane bug, then hopefully I can hit it with a small
number.
Out of curiosity, is there any particular number of STAs > 1 you guys
test with?
Thanks,
Ben
>
> Luis
--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
On Thu, Oct 7, 2010 at 2:36 PM, Luis R. Rodriguez <[email protected]> wrote:
> On Thu, Oct 7, 2010 at 2:31 PM, Luis R. Rodriguez <[email protected]> wrote:
>> On Thu, Oct 7, 2010 at 12:27 PM, Johannes Berg
>> <[email protected]> wrote:
>>> On Thu, 2010-10-07 at 12:22 -0700, Ben Greear wrote:
>>>
>>>> After reboot, and re-run of the script,
>>>> I saw this in the logs, and shortly after,
>>>> the SLUB poison warning dumped to screen.
>>>>
>>>> Maybe those DMA errors are serious?
>>>
>>>> ath: Failed to stop TX DMA in 100 msec after killing last frame
>>>> ath: Failed to stop TX DMA. Resetting hardware!
>>>
>>> That's TX DMA, it can hardly result in invalid memory writes like the
>>> ones you've been seeing.
>>>
>>> I'm still convinced something is wrong with ath9k RX DMA, as you've seen
>>> the contents of frames written to already freed memory regions. Since I
>>> don't know anything about ath9k, you should probably not rely on me
>>> though :-)
>>
>> I'm on this now. Lets play.
>>
>> I had to remove /lib/udev/rules.d/75-persistent-net-generator.rules
>> to avoid Ubuntu trying to remember the device names and it creating
>> stax_rename names.
>> I just ran your script with some modifications. You can find it here:
>>
>> http://www.kernel.org/pub/linux/kernel/people/mcgrof/scripts/poo.pl
>>
>> I then ran:
>>
>> for i in $(seq 0 31) ; do sudo dhclient seq$i; done
>>
>> It took about 10 minutes to get IP addresses for all interfaces but it
>> got there eventually. Odd enough I was unable to ping the AP from any
>> interface though. Not sure what that was about. But I got no oops, no
>> slub dump. All I got was some more delba warnings which seems to
>> indicate we haven't caught all the cases for its use:
>>
>> [ 3622.660344] addBA response timer expired on tid 0
>> [ 3622.660373] Tx BA session stop requested for 68:7f:74:3b:b1:0f tid 0
>> [ 3622.680133] addBA response timer expired on tid 0
>> [ 3622.687196] Tx BA session stop requested for 68:7f:74:3b:b1:0f tid 0
>> [ 3623.110077] addBA response timer expired on tid 0
>> [ 3623.110123] Tx BA session stop requested for 68:7f:74:3b:b1:0f tid 0
>> [ 3628.935895] sta10: authenticate with 68:7f:74:3b:b1:10 (try 1)
>> [ 3628.937194] switched off addBA timer for tid 0
>> [ 3628.937196] Aggregation is on for tid 0
>> [ 3628.937239] Stopping Tx BA session for 68:7f:74:3b:b1:0f tid 0
>> [ 3628.937243] ------------[ cut here ]------------
>> [ 3628.937263] WARNING: at include/net/mac80211.h:2694
>> rate_control_send_low+0xd3/0x140 [mac80211]()
>> [ 3628.937265] Hardware name: 6460DWU
>> [ 3628.937266] Modules linked in: binfmt_misc ppdev
>> snd_hda_codec_analog rfcomm sco bridge joydev stp bnep l2cap nouveau
>> ath9k snd_hda_intel mac80211 snd_hda_codec snd_hwdep snd_pcm ttm btusb
>> ath9k_common thinkpad_acpi ath9k_hw bluetooth drm_kms_helper
>> snd_seq_midi snd_rawmidi pcmcia snd_seq_midi_event drm snd_seq ath
>> snd_timer snd_seq_device tpm_tis i2c_algo_bit cfg80211 snd nvram tpm
>> tpm_bios yenta_socket pcmcia_rsrc video psmouse output pcmcia_core
>> serio_raw soundcore snd_page_alloc intel_agp lp parport ohci1394
>> e1000e ieee1394 ahci libahci
>> [ 3628.937307] Pid: 49, comm: kworker/u:3 Tainted: G W
>> 2.6.36-rc6-wl+ #263
>> [ 3628.937310] Call Trace:
>> [ 3628.937317] [<ffffffff8105ffcf>] warn_slowpath_common+0x7f/0xc0
>> [ 3628.937320] [<ffffffff8106002a>] warn_slowpath_null+0x1a/0x20
>> [ 3628.937329] [<ffffffffa032f603>] rate_control_send_low+0xd3/0x140 [mac80211]
>> [ 3628.937336] [<ffffffffa038bfd8>] ath_get_rate+0x48/0x570 [ath9k]
>> [ 3628.937340] [<ffffffff812b9f39>] ? put_dec+0x59/0x60
>> [ 3628.937349] [<ffffffffa032f42e>] rate_control_get_rate+0x8e/0x190 [mac80211]
>> [ 3628.937360] [<ffffffffa0339928>]
>> ieee80211_tx_h_rate_ctrl+0x1a8/0x4e0 [mac80211]
>> [ 3628.937370] [<ffffffffa033a000>] invoke_tx_handlers+0x100/0x140 [mac80211]
>> [ 3628.937379] [<ffffffffa033a0c5>] ieee80211_tx+0x85/0x240 [mac80211]
>> [ 3628.937384] [<ffffffff8147b890>] ? skb_release_data+0xd0/0xe0
>> [ 3628.937386] [<ffffffff8147d72f>] ? pskb_expand_head+0x10f/0x1a0
>> [ 3628.937397] [<ffffffffa033a336>] ieee80211_xmit+0xb6/0x1d0 [mac80211]
>> [ 3628.937399] [<ffffffff8147b9d3>] ? __alloc_skb+0x83/0x170
>> [ 3628.937409] [<ffffffffa033a4a4>] ieee80211_tx_skb+0x54/0x70 [mac80211]
>> [ 3628.937418] [<ffffffffa03230ad>] ieee80211_send_delba+0x11d/0x190 [mac80211]
>> [ 3628.937427] [<ffffffffa0323a18>]
>> ieee80211_stop_tx_ba_cb+0x1b8/0x240 [mac80211]
>> [ 3628.937431] [<ffffffff81036c89>] ? default_spin_lock_flags+0x9/0x10
>> [ 3628.937440] [<ffffffffa032e031>] ieee80211_iface_work+0x271/0x340 [mac80211]
>> [ 3628.937450] [<ffffffffa032ddc0>] ? ieee80211_iface_work+0x0/0x340 [mac80211]
>> [ 3628.937453] [<ffffffff8107a203>] process_one_work+0x123/0x440
>> [ 3628.937457] [<ffffffff8107c750>] worker_thread+0x170/0x400
>> [ 3628.937460] [<ffffffff8107c5e0>] ? worker_thread+0x0/0x400
>> [ 3628.937463] [<ffffffff81080b76>] kthread+0x96/0xa0
>> [ 3628.937466] [<ffffffff8100bea4>] kernel_thread_helper+0x4/0x10
>> [ 3628.937469] [<ffffffff81080ae0>] ? kthread+0x0/0xa0
>> [ 3628.937472] [<ffffffff8100bea0>] ? kernel_thread_helper+0x0/0x10
>> [ 3628.937474] ---[ end trace 9dd0d025ccb9b75c ]---
>> [ 3628.937980] switched off addBA timer for tid 0
>> [ 3628.937982] Aggregation is on for tid 0
>>
>> But other than this I got nothing. I left the box sit there for about
>> 1 hour and came back and it was still going with no issues. Mind you,
>> I can't ping but that seems like another issue.
>>
>> You can find my logs here:
>>
>> http://www.kernel.org/pub/linux/kernel/people/mcgrof/logs/2010/10-07-stress-sta-01/
>
> Doh, I did not have CONFIG_SLUB_DEBUG_ON=y so building now with that.
Yay I can reproduce now. I'll be back, going to dig now.
Luis
On 10/05/2010 10:16 AM, Luis R. Rodriguez wrote:
> On Tue, Oct 5, 2010 at 10:00 AM, Ben Greear<[email protected]> wrote:
>> I started seeing this very soon after creating interfaces.
>
> Can you be more specific how one can reproduce?
Enable SLUB debugging, DEBUG_PAGEALLOC (Debug page memory allocations),
lockdep, pre-empt.
Please try creating a bunch of stations on one ath9k phy, and
configure them to use WPA (we're using one supplicant per STA).
If you don't hit it within 2 minutes of creating STAs and starting
WPA, do let me know and I'll try to come up with something
more specific. But, at least for us, it seems extremely easy to
hit this issue.
The same kernel boots fine with no such errors on a system with ath5k, btw.
I wish there was a way to get slub debugging to print the backtrace for the
code that deleted the memory, but so far, I haven't found anything.
Thanks,
Ben
--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
On 10/15/2010 04:41 PM, Luis R. Rodriguez wrote:
> On Fri, Oct 15, 2010 at 04:38:14PM -0700, Luis Rodriguez wrote:
>> On Fri, Oct 15, 2010 at 04:33:50PM -0700, Ben Greear wrote:
>>> On 10/15/2010 04:21 PM, Luis R. Rodriguez wrote:
>>>> Ben, please give this patch a shot. I addresses three races on the PCU:
>>>>
>>>> * When we were stopping the CPU for non-EDMA cards we never locked against
>>>> anything starting the PCU again
>>>>
>>>> * ath9k_hw_startpcureceive() was being called without locking
>>>>
>>>> * Although we lock on the rxbuf lock for contention against starting/stopping
>>>> the PCU, we also need to lock on the driver in locations where we start/stop
>>>> the PCU within the same location otherwise we end up in inconsistant states
>>>> and the hardware may end up proessing an incorrect buffer for DMA. To
>>>> protect against this we use a new PCU lock on the main part of the driver to
>>>> ensure each start/stop/reset operation is done atomically.
>>>>
>>>> And fixes one issue as a side effect:
>>>>
>>>> * No more packet loss on ping flood when you have one STA associated :)
>>>>
>>>> The only issue I see with this is I eventually run out of memory and my box
>>>> becomes useless, unless I am mistaking that for some other issue.
>>>>
>>>> Please give this a shot and if it cures your woes I'll split it up into
>>>> 3 separate patches, or maybe just two, one for the first two and one for
>>>> the last issue.
>>>
>>> Sounds good, but this lockdep splat happens almost immediately upon starting
>>> my app:
>>>
>>> =======================================================
>>> [ INFO: possible circular locking dependency detected ]
>>> 2.6.36-rc8-wl+ #32
>>> -------------------------------------------------------
>>> swapper/0 is trying to acquire lock:
>>> (&(&sc->rx.pcu_lock)->rlock){+.-...}, at: [<fa16e5c7>] ath9k_tasklet+0x7e/0x140 [ath9k]
>>>
>>> but task is already holding lock:
>>> (&(&sc->rx.rxflushlock)->rlock){+.-...}, at: [<fa16e5b9>] ath9k_tasklet+0x70/0x140 [ath9k]
>>>
>>> which lock already depends on the new lock.
>>>
>>>
>>> the existing dependency chain (in reverse order) is:
>>>
>>> -> #1 (&(&sc->rx.rxflushlock)->rlock){+.-...}:
>>> [<c0457639>] lock_acquire+0x5a/0x78
>>> [<c075f6ed>] _raw_spin_lock_bh+0x20/0x2f
>>> [<fa170513>] ath_flushrecv+0x14/0x61 [ath9k]
>>
>> Ah we just need to nuke the flush lock, one second. Also remove my
>> skb_copy() otherwise you will really run out of memory quickly,
>> unless you really want to test with it :)
>
> Try this new one, note I if (0)'d the skb_copy() set that to if (1) if you want
> to force memory clobber.
Ahh, getting better. Ran for around 10 minutes with my app, never saw the poison,
but system got into a bad state. It could be some other bug exposed by my test
that was previously hidden by the poison problem, or maybe still bugs with the
locking.
First, I saw this..then things seemed to recover for a bit.. I've seen these
before...
ieee80211 phy0: sta85: No probe response from AP 00:14:d1:c6:d2:54 after 500ms, try 0
ieee80211 phy0: sta73: No probe response from AP 00:14:d1:c6:d2:54 after 500ms, try 0
ieee80211 phy0: sta109: No probe response from AP 00:14:d1:c6:d2:54 after 500ms, try 0
ath: Failed to stop TX DMA in 100 msec after killing last frame
ath: Failed to stop TX DMA. Resetting hardware!
ath: DMA failed to stop in 10 ms AR_CR=0x00000024 AR_DIAG_SW=0x42000020
sta29: authenticate with 00:14:d1:c6:d2:54 (try 1)
sta38: authenticate with 00:14:d1:c6:d2:54 (try 1)
Then a little later, hardware seems to go bad and won't recover. Interesting that
the e1000e driver is also complaining..and I seem to have lost network connectivity
to the system...
Oct 15 16:54:38 localhost kernel: ieee80211 phy0: sta131: No probe response from AP 00:14:d1:c6:d2:54 after 500ms, try 0
Oct 15 16:54:38 localhost kernel: ieee80211 phy0: sta85: No probe response from AP 00:14:d1:c6:d2:54 after 500ms, try 0
Oct 15 16:54:38 localhost kernel: ieee80211 phy0: sta73: No probe response from AP 00:14:d1:c6:d2:54 after 500ms, try 0
Oct 15 16:54:38 localhost kernel: ieee80211 phy0: sta109: No probe response from AP 00:14:d1:c6:d2:54 after 500ms, try 0
Oct 15 16:54:38 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame
Oct 15 16:54:38 localhost kernel: ath: Failed to stop TX DMA. Resetting hardware!
Oct 15 16:54:38 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame
Oct 15 16:54:38 localhost kernel: ath: Failed to stop TX DMA. Resetting hardware!
Oct 15 16:54:38 localhost kernel: ath: DMA failed to stop in 10 ms AR_CR=0x00000024 AR_DIAG_SW=0x42000020
Oct 15 16:54:38 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame
Oct 15 16:54:38 localhost kernel: ath: Failed to stop TX DMA. Resetting hardware!
Oct 15 16:54:38 localhost kernel: ath: DMA failed to stop in 10 ms AR_CR=0x00000024 AR_DIAG_SW=0x42000020
Oct 15 16:54:38 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame
Oct 15 16:54:38 localhost kernel: ath: Failed to stop TX DMA. Resetting hardware!
Oct 15 16:54:38 localhost kernel: ath: DMA failed to stop in 10 ms AR_CR=0x00000024 AR_DIAG_SW=0x42000020
Oct 15 16:54:38 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame
Oct 15 16:54:38 localhost kernel: ath: Failed to stop TX DMA. Resetting hardware!
Oct 15 16:54:38 localhost kernel: ath: DMA failed to stop in 10 ms AR_CR=0x00000024 AR_DIAG_SW=0x42000020
Oct 15 16:54:38 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame
Oct 15 16:54:38 localhost kernel: ath: Failed to stop TX DMA. Resetting hardware!
...
Oct 15 16:54:40 localhost kernel: ath: DMA failed to stop in 10 ms AR_CR=0x00000024 AR_DIAG_SW=0x42000020
Oct 15 16:54:40 localhost kernel: e1000e 0000:06:00.0: eth0: Detected Hardware Unit Hang:
Oct 15 16:54:40 localhost kernel: TDH <3d>
Oct 15 16:54:40 localhost kernel: TDT <3f>
Oct 15 16:54:40 localhost kernel: next_to_use <3f>
Oct 15 16:54:40 localhost kernel: next_to_clean <3d>
Oct 15 16:54:40 localhost kernel: buffer_info[next_to_clean]:
Oct 15 16:54:40 localhost kernel: time_stamp <3225b>
Oct 15 16:54:40 localhost kernel: next_to_watch <3e>
Oct 15 16:54:40 localhost kernel: jiffies <32c84>
Oct 15 16:54:40 localhost kernel: next_to_watch.status <0>
Oct 15 16:54:40 localhost kernel: MAC Status <80080f83>
Oct 15 16:54:40 localhost kernel: PHY Status <796d>
Oct 15 16:54:40 localhost kernel: PHY 1000BASE-T Status <7c00>
Oct 15 16:54:40 localhost kernel: PHY Extended Status <3000>
Oct 15 16:54:40 localhost kernel: PCI Status <4010>
Oct 15 16:54:40 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame
Oct 15 16:54:40 localhost kernel: ath: Failed to stop TX DMA. Resetting hardware!
...
And then HD complains:
Oct 15 16:55:18 localhost kernel: ath: DMA failed to stop in 10 ms AR_CR=0x00000024 AR_DIAG_SW=0x42000020
Oct 15 16:55:18 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame
Oct 15 16:55:18 localhost kernel: ath: Failed to stop TX DMA. Resetting hardware!
Oct 15 16:55:18 localhost kernel: ath: DMA failed to stop in 10 ms AR_CR=0x00000024 AR_DIAG_SW=0x42000020
Oct 15 16:55:18 localhost kernel: ata1: device not ready (errno=-16), forcing hardreset
Oct 15 16:55:18 localhost kernel: ata1: soft resetting link
Oct 15 16:55:18 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame
Oct 15 16:55:18 localhost kernel: ath: Failed to stop TX DMA. Resetting hardware!
Maybe IRQs were disabled and not re-enabled somehow?
There were no lockdep splats..
Here is a dump of the locks held...
Oct 15 17:01:29 localhost kernel: SysRq : Show Locks Held
Oct 15 17:01:29 localhost kernel:
Oct 15 17:01:29 localhost kernel: Showing all locks held in the system:
Oct 15 17:01:29 localhost kernel: 4 locks held by flush-253:0/502:
Oct 15 17:01:29 localhost kernel: #0: (&type->s_umount_key#22){++++..}, at: [<c04cf1e0>] writeback_inodes_wb+0x95/0xf5
Oct 15 17:01:29 localhost kernel: #1: (jbd2_handle){+.+...}, at: [<c053814d>] start_this_handle+0x4b6/0x4ec
Oct 15 17:01:29 localhost kernel: #2: (&ei->i_data_sem){++++..}, at: [<c051549a>] ext4_map_blocks+0xab/0x164
Oct 15 17:01:29 localhost kernel: #3: (&lg->lg_mutex){+.+...}, at: [<c0529792>] ext4_mb_initialize_context+0x1db/0x1e5
Oct 15 17:01:29 localhost kernel: 1 lock held by flush-253:2/1148:
Oct 15 17:01:29 localhost kernel: #0: (&type->s_umount_key#22){++++..}, at: [<c04cf1e0>] writeback_inodes_wb+0x95/0xf5
Oct 15 17:01:29 localhost kernel: 1 lock held by mingetty/1620:
Oct 15 17:01:29 localhost kernel: #0: (&tty->atomic_read_lock){+.+.+.}, at: [<c05f384d>] n_tty_read+0x1d1/0x5ed
Oct 15 17:01:29 localhost kernel: 1 lock held by mingetty/1622:
Oct 15 17:01:29 localhost kernel: #0: (&tty->atomic_read_lock){+.+.+.}, at: [<c05f384d>] n_tty_read+0x1d1/0x5ed
Oct 15 17:01:29 localhost kernel: 1 lock held by mingetty/1624:
Oct 15 17:01:29 localhost kernel: #0: (&tty->atomic_read_lock){+.+.+.}, at: [<c05f384d>] n_tty_read+0x1d1/0x5ed
Oct 15 17:01:29 localhost kernel: 1 lock held by mingetty/1626:
Oct 15 17:01:29 localhost kernel: #0: (&tty->atomic_read_lock){+.+.+.}, at: [<c05f384d>] n_tty_read+0x1d1/0x5ed
Oct 15 17:01:29 localhost kernel: 1 lock held by mingetty/1628:
Oct 15 17:01:29 localhost kernel: #0: (&tty->atomic_read_lock){+.+.+.}, at: [<c05f384d>] n_tty_read+0x1d1/0x5ed
Oct 15 17:01:29 localhost kernel: 1 lock held by mingetty/1630:
Oct 15 17:01:29 localhost kernel: #0: (&tty->atomic_read_lock){+.+.+.}, at: [<c05f384d>] n_tty_read+0x1d1/0x5ed
Oct 15 17:01:29 localhost kernel: 3 locks held by kworker/0:2/1632:
Oct 15 17:01:29 localhost kernel: #0: (events){+.+.+.}, at: [<c0443cf6>] process_one_work+0x145/0x295
Oct 15 17:01:29 localhost kernel: #1: ((&ap->scsi_rescan_task)){+.+.+.}, at: [<c0443cf6>] process_one_work+0x145/0x295
Oct 15 17:01:29 localhost kernel: #2: (&ap->scsi_scan_mutex){+.+.+.}, at: [<c063f3c6>] ata_scsi_dev_rescan+0x1e/0xb6
Oct 15 17:01:29 localhost kernel: 1 lock held by bash/1814:
Oct 15 17:01:29 localhost kernel: #0: (&tty->atomic_read_lock){+.+.+.}, at: [<c05f384d>] n_tty_read+0x1d1/0x5ed
Oct 15 17:01:29 localhost kernel: 1 lock held by bash/1863:
Oct 15 17:01:29 localhost kernel: #0: (&tty->atomic_read_lock){+.+.+.}, at: [<c05f384d>] n_tty_read+0x1d1/0x5ed
Oct 15 17:01:29 localhost kernel: 1 lock held by bash/1935:
Oct 15 17:01:29 localhost kernel: #0: (&tty->atomic_read_lock){+.+.+.}, at: [<c05f384d>] n_tty_read+0x1d1/0x5ed
Oct 15 17:01:29 localhost kernel: 2 locks held by btserver/2422:
Oct 15 17:01:29 localhost kernel: #0: (&sb->s_type->i_mutex_key#12){+.+.+.}, at: [<c04871bf>] generic_file_aio_write+0x4d/0xa6
Oct 15 17:01:29 localhost kernel: #1: (jbd2_handle){+.+...}, at: [<c053814d>] start_this_handle+0x4b6/0x4ec
Oct 15 17:01:29 localhost kernel: 2 locks held by ip/23410:
Oct 15 17:01:29 localhost kernel: #0: (&sb->s_type->i_mutex_key#12){+.+.+.}, at: [<c04871bf>] generic_file_aio_write+0x4d/0xa6
Oct 15 17:01:29 localhost kernel: #1: (jbd2_handle){+.+...}, at: [<c053814d>] start_this_handle+0x4b6/0x4ec
Oct 15 17:01:29 localhost kernel: 4 locks held by sh/23416:
Oct 15 17:01:29 localhost kernel: #0: (&sb->s_type->i_mutex_key#12){+.+.+.}, at: [<c04b6c36>] do_truncate+0x5a/0x7d
Oct 15 17:01:29 localhost kernel: #1: (&sb->s_type->i_alloc_sem_key#4){+.+...}, at: [<c04c82e2>] notify_change+0x12a/0x218
Oct 15 17:01:29 localhost kernel: #2: (jbd2_handle){+.+...}, at: [<c053814d>] start_this_handle+0x4b6/0x4ec
Oct 15 17:01:29 localhost kernel: #3: (&sbi->s_orphan_lock){+.+...}, at: [<c051975f>] ext4_orphan_add+0x1d8/0x1f7
Oct 15 17:01:29 localhost kernel:
Oct 15 17:01:29 localhost kernel: =============================================
Can't get processor info from sysrq..don't know why:
Oct 15 17:04:04 localhost kernel: ath: DMA failed to stop in 10 ms AR_CR=0x00000024 AR_DIAG_SW=0x42000020
SysRq : Show backtrace of all active CPUs
Oct 15 17:04:14 localhost kernel: SysRq : Show backtrace of all active CPUs
Oct 15 17:04:14 localhost kernel: sending NMI to all CPUs:
Oct 15 17:04:14 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame
Thanks,
Ben
--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
2010/10/13 Björn Smedman <[email protected]>:
> Hi Ben,
>
> First of all keep up the good work. :)
>
> On Wed, Oct 13, 2010 at 6:39 PM, Ben Greear <[email protected]> wrote:
> [snip]
>> Either way, it seems safer to null out the bf_ampdu field after
>> the memory is consumed..it could prevent some tricky bugs later.
>
> I think this is a good idea. But it probably wont be enough to null
> out bf_mpdu. You also need to look at bf_buf_addr (which if I
> understand correctly is the physical address the DMA engine will
> actually write RXed frames to) and bf_dmacontext (which seems in most
> cases to hold an identical address and may in fact be where the DMA
> engine will really write the frame).
See the TODO list for ath9k:
* Remove this redundancy crap: bf->bf_buf_addr = bf->bf_dmacontext;
http://wireless.kernel.org/en/users/Drivers/ath9k/todo
Luis
Ben, please give this patch a shot. I addresses three races on the PCU:
* When we were stopping the CPU for non-EDMA cards we never locked against
anything starting the PCU again
* ath9k_hw_startpcureceive() was being called without locking
* Although we lock on the rxbuf lock for contention against starting/stopping
the PCU, we also need to lock on the driver in locations where we start/stop
the PCU within the same location otherwise we end up in inconsistant states
and the hardware may end up proessing an incorrect buffer for DMA. To
protect against this we use a new PCU lock on the main part of the driver to
ensure each start/stop/reset operation is done atomically.
And fixes one issue as a side effect:
* No more packet loss on ping flood when you have one STA associated :)
The only issue I see with this is I eventually run out of memory and my box
becomes useless, unless I am mistaking that for some other issue.
Please give this a shot and if it cures your woes I'll split it up into
3 separate patches, or maybe just two, one for the first two and one for
the last issue.
diff --git a/drivers/net/wireless/ath/ath9k/ath9k.h b/drivers/net/wireless/ath/ath9k/ath9k.h
index 0c917e5..efee63c 100644
--- a/drivers/net/wireless/ath/ath9k/ath9k.h
+++ b/drivers/net/wireless/ath/ath9k/ath9k.h
@@ -312,6 +312,7 @@ struct ath_rx {
unsigned int rxfilter;
spinlock_t rxflushlock;
spinlock_t rxbuflock;
+ spinlock_t pcu_lock;
struct list_head rxbuf;
struct ath_descdma rxdma;
struct ath_buf *rx_bufptr;
diff --git a/drivers/net/wireless/ath/ath9k/main.c b/drivers/net/wireless/ath/ath9k/main.c
index 62294da..6fbfe28 100644
--- a/drivers/net/wireless/ath/ath9k/main.c
+++ b/drivers/net/wireless/ath/ath9k/main.c
@@ -251,6 +251,9 @@ int ath_set_channel(struct ath_softc *sc, struct ieee80211_hw *hw,
*/
ath9k_hw_set_interrupts(ah, 0);
ath_drain_all_txq(sc, false);
+
+ spin_lock_bh(&sc->rx.pcu_lock);
+
stopped = ath_stoprecv(sc);
/* XXX: do not flush receive queue here. We don't want
@@ -278,6 +281,7 @@ int ath_set_channel(struct ath_softc *sc, struct ieee80211_hw *hw,
"reset status %d\n",
channel->center_freq, r);
spin_unlock_bh(&sc->sc_resetlock);
+ spin_unlock_bh(&sc->rx.pcu_lock);
goto ps_restore;
}
spin_unlock_bh(&sc->sc_resetlock);
@@ -286,9 +290,12 @@ int ath_set_channel(struct ath_softc *sc, struct ieee80211_hw *hw,
ath_print(common, ATH_DBG_FATAL,
"Unable to restart recv logic\n");
r = -EIO;
+ spin_unlock_bh(&sc->rx.pcu_lock);
goto ps_restore;
}
+ spin_unlock_bh(&sc->rx.pcu_lock);
+
ath_cache_conf_rate(sc, &hw->conf);
ath_update_txpow(sc);
ath9k_hw_set_interrupts(ah, ah->imask);
@@ -626,12 +633,16 @@ void ath9k_tasklet(unsigned long data)
if (status & rxmask) {
spin_lock_bh(&sc->rx.rxflushlock);
+ spin_lock_bh(&sc->rx.pcu_lock);
+
/* Check for high priority Rx first */
if ((ah->caps.hw_caps & ATH9K_HW_CAP_EDMA) &&
(status & ATH9K_INT_RXHP))
ath_rx_tasklet(sc, 0, true);
ath_rx_tasklet(sc, 0, false);
+ spin_unlock_bh(&sc->rx.pcu_lock);
+
spin_unlock_bh(&sc->rx.rxflushlock);
}
@@ -887,6 +898,7 @@ void ath_radio_enable(struct ath_softc *sc, struct ieee80211_hw *hw)
if (!ah->curchan)
ah->curchan = ath_get_curchannel(sc, sc->hw);
+ spin_lock_bh(&sc->rx.pcu_lock);
spin_lock_bh(&sc->sc_resetlock);
r = ath9k_hw_reset(ah, ah->curchan, ah->caldata, false);
if (r) {
@@ -901,8 +913,10 @@ void ath_radio_enable(struct ath_softc *sc, struct ieee80211_hw *hw)
if (ath_startrecv(sc) != 0) {
ath_print(common, ATH_DBG_FATAL,
"Unable to restart recv logic\n");
+ spin_unlock_bh(&sc->rx.pcu_lock);
return;
}
+ spin_unlock_bh(&sc->rx.pcu_lock);
if (sc->sc_flags & SC_OP_BEACONS)
ath_beacon_config(sc, NULL); /* restart beacons */
@@ -941,6 +955,9 @@ void ath_radio_disable(struct ath_softc *sc, struct ieee80211_hw *hw)
ath9k_hw_set_interrupts(ah, 0);
ath_drain_all_txq(sc, false); /* clear pending tx frames */
+
+ spin_lock_bh(&sc->rx.pcu_lock);
+
ath_stoprecv(sc); /* turn off frame recv */
ath_flushrecv(sc); /* flush recv queue */
@@ -958,6 +975,9 @@ void ath_radio_disable(struct ath_softc *sc, struct ieee80211_hw *hw)
spin_unlock_bh(&sc->sc_resetlock);
ath9k_hw_phy_disable(ah);
+
+ spin_unlock_bh(&sc->rx.pcu_lock);
+
ath9k_hw_configpcipowersave(ah, 1, 1);
ath9k_ps_restore(sc);
ath9k_setpower(sc, ATH9K_PM_FULL_SLEEP);
@@ -977,6 +997,9 @@ int ath_reset(struct ath_softc *sc, bool retry_tx)
ath9k_hw_set_interrupts(ah, 0);
ath_drain_all_txq(sc, retry_tx);
+
+ spin_lock_bh(&sc->rx.pcu_lock);
+
ath_stoprecv(sc);
ath_flushrecv(sc);
@@ -991,6 +1014,8 @@ int ath_reset(struct ath_softc *sc, bool retry_tx)
ath_print(common, ATH_DBG_FATAL,
"Unable to start recv logic\n");
+ spin_unlock_bh(&sc->rx.pcu_lock);
+
/*
* We may be doing a reset in response to a request
* that changes the channel so update any state that
@@ -1155,6 +1180,7 @@ static int ath9k_start(struct ieee80211_hw *hw)
* be followed by initialization of the appropriate bits
* and then setup of the interrupt mask.
*/
+ spin_lock_bh(&sc->rx.pcu_lock);
spin_lock_bh(&sc->sc_resetlock);
r = ath9k_hw_reset(ah, init_channel, ah->caldata, false);
if (r) {
@@ -1163,6 +1189,7 @@ static int ath9k_start(struct ieee80211_hw *hw)
"(freq %u MHz)\n", r,
curchan->center_freq);
spin_unlock_bh(&sc->sc_resetlock);
+ spin_unlock_bh(&sc->rx.pcu_lock);
goto mutex_unlock;
}
spin_unlock_bh(&sc->sc_resetlock);
@@ -1184,8 +1211,10 @@ static int ath9k_start(struct ieee80211_hw *hw)
ath_print(common, ATH_DBG_FATAL,
"Unable to start recv logic\n");
r = -EIO;
+ spin_unlock_bh(&sc->rx.pcu_lock);
goto mutex_unlock;
}
+ spin_unlock_bh(&sc->rx.pcu_lock);
/* Setup our intr mask. */
ah->imask = ATH9K_INT_TX | ATH9K_INT_RXEOL |
@@ -1386,12 +1415,14 @@ static void ath9k_stop(struct ieee80211_hw *hw)
* before setting the invalid flag. */
ath9k_hw_set_interrupts(ah, 0);
+ spin_lock_bh(&sc->rx.pcu_lock);
if (!(sc->sc_flags & SC_OP_INVALID)) {
ath_drain_all_txq(sc, false);
ath_stoprecv(sc);
ath9k_hw_phy_disable(ah);
} else
sc->rx.rxlink = NULL;
+ spin_unlock_bh(&sc->rx.pcu_lock);
/* disable HAL and put h/w to sleep */
ath9k_hw_disable(ah);
diff --git a/drivers/net/wireless/ath/ath9k/recv.c b/drivers/net/wireless/ath/ath9k/recv.c
index fe73fc5..20025a5 100644
--- a/drivers/net/wireless/ath/ath9k/recv.c
+++ b/drivers/net/wireless/ath/ath9k/recv.c
@@ -297,19 +297,17 @@ static void ath_edma_start_recv(struct ath_softc *sc)
ath_rx_addbuffer_edma(sc, ATH9K_RX_QUEUE_LP,
sc->rx.rx_edma[ATH9K_RX_QUEUE_LP].rx_fifo_hwsize);
- spin_unlock_bh(&sc->rx.rxbuflock);
-
ath_opmode_init(sc);
ath9k_hw_startpcureceive(sc->sc_ah, (sc->sc_flags & SC_OP_OFFCHANNEL));
+
+ spin_unlock_bh(&sc->rx.rxbuflock);
}
static void ath_edma_stop_recv(struct ath_softc *sc)
{
- spin_lock_bh(&sc->rx.rxbuflock);
ath_rx_remove_buffer(sc, ATH9K_RX_QUEUE_HP);
ath_rx_remove_buffer(sc, ATH9K_RX_QUEUE_LP);
- spin_unlock_bh(&sc->rx.rxbuflock);
}
int ath_rx_init(struct ath_softc *sc, int nbufs)
@@ -321,6 +319,7 @@ int ath_rx_init(struct ath_softc *sc, int nbufs)
spin_lock_init(&sc->rx.rxflushlock);
sc->sc_flags &= ~SC_OP_RXFLUSH;
+ spin_lock_init(&sc->rx.pcu_lock);
spin_lock_init(&sc->rx.rxbuflock);
if (sc->sc_ah->caps.hw_caps & ATH9K_HW_CAP_EDMA) {
@@ -506,9 +505,9 @@ int ath_startrecv(struct ath_softc *sc)
ath9k_hw_rxena(ah);
start_recv:
- spin_unlock_bh(&sc->rx.rxbuflock);
ath_opmode_init(sc);
ath9k_hw_startpcureceive(ah, (sc->sc_flags & SC_OP_OFFCHANNEL));
+ spin_unlock_bh(&sc->rx.rxbuflock);
return 0;
}
@@ -518,6 +517,7 @@ bool ath_stoprecv(struct ath_softc *sc)
struct ath_hw *ah = sc->sc_ah;
bool stopped;
+ spin_lock_bh(&sc->rx.rxbuflock);
ath9k_hw_stoppcurecv(ah);
ath9k_hw_setrxfilter(ah, 0);
stopped = ath9k_hw_stopdmarecv(ah);
@@ -526,7 +526,9 @@ bool ath_stoprecv(struct ath_softc *sc)
ath_edma_stop_recv(sc);
else
sc->rx.rxlink = NULL;
+ spin_unlock_bh(&sc->rx.rxbuflock);
+ WARN_ON(!stopped);
return stopped;
}
@@ -663,6 +665,13 @@ static void ath_rx_send_to_mac80211(struct ieee80211_hw *hw,
struct ieee80211_rx_status *rxs)
{
struct ieee80211_hdr *hdr;
+ struct sk_buff *tmp_skb;
+ unsigned int j;
+
+ for (j=0; j < 5; j++) {
+ tmp_skb = skb_copy(skb, GFP_ATOMIC);
+ dev_kfree_skb_any(tmp_skb);
+ }
hdr = (struct ieee80211_hdr *)skb->data;
On 10/14/2010 02:32 PM, Luis R. Rodriguez wrote:
> On Thu, Oct 14, 2010 at 2:31 PM, Ben Greear<[email protected]> wrote:
>>> arch/x86/built-in.o: In function `store_stack':
>>> /home/mcgrof/wireless-testing/arch/x86/kernel/dumpstack.c:259:
>>> undefined reference to `store_trace'
>>
>> You are compiling on 32-bit system? I see the method in
>> the patch, but probably only for 32-bit x86...
>
> Ah no I'm on 64-bit.
Probably you could cut-n-paste dump_trace in dumpstack_64.c,
rename it store_trace, and hack it a bit as I did the one in
dumpstack_32.c
I don't currently have any ath9k systems running 64-bit, but
I could put one together if needed.
By the way, if you have any extra ideas of what to try next,
I'll consider trying it. I am very low on ideas personally :P
Thanks,
Ben
--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
On Wed, Oct 13, 2010 at 10:12 AM, Ben Greear <[email protected]> wrote:
> On 10/12/2010 11:40 AM, Luis R. Rodriguez wrote:
>>
>> On Tue, Oct 12, 2010 at 11:35 AM, Ben Greear<[email protected]>
>> wrote:
>>>
>>> On 10/11/2010 11:10 PM, Luis R. Rodriguez wrote:
>>>>
>>>> On Mon, Oct 11, 2010 at 8:27 PM, Ben Greear<[email protected]>
>>>> wrote:
>>>
>>>>> Another thing I was thinking about: Maybe the queue of skbs and dma
>>>>> addresses
>>>>> in ath9k is getting corrupted by multiple VIFs trying to write at once?
>>>>> Maybe
>>>>> some locking is needed in the xmit path?
>>>>
>>>> That was my second hunch. My first shot was to use spin_lock_irqsave()
>>>> over the the uses of the rxbuf list and that seemed to help but I
>>>> still managed to get a poison eventually. My next item to check for is
>>>> of the permissibility of creating too much pressure to the point we
>>>> end up looping over the rxbuf list and race against mac80211 free'ing
>>>> a buffer. Will test that tomorrow if nothing else comes up creeping my
>>>> priority queue.
>>>
>>> This code looks weird to me. One of the paprd branches
>>> deletes the skb, the other doesn't appear to. Neither
>>> null out bf->bf_mpdu, which would appear to leave a dangling
>>> pointer in at least the dev_kfree_skb_any() branch.
>>>
>>> ath_tx_complete frees it's skb in all cases, so another
>>> bf->bf_mpdu dangling pointer issue.
>>>
>>> Maybe at the least we should null out bf->bf_mpdu when
>>> skb is consumed?
>>
>> You're reading my mind, that was what I was going to test today. Still
>> doing e-mail sweep though.
>
> At least in the xmit path, it seems cards that have EDMA support do
> things a bit different. Out of curiosity, on the system(s), you reproduce
> this, are any of yours supporting EDMA? Mine appear to not support EDMA.
EDMA is used on >= AR9003 families by Atheros. And no, I am not
testing with an EDMA card, I am testing with an AR9002 family card,
the AR9280 card. I am going to disregard the TX stuff as the bug is an
RX issue :) I was able to more easily reproduce by doing an skb_copy()
and free'ing the buffer right afterwards on the ath_send_to_mac80211()
thingy, So it does appear that the poison check just happens more
often when we do an skb_copy(). One reason this is easy to reproduce
with multiple STAs is mac80211 uses skb_copy() to process each
received skb for each STA.
In my tests so far, protecting the rxbuf list with spin_lock_irqsave()
did not help, and the wmb(); didn't either, something else is going on
here. It would be nice to hack slab to keep an entire trace of the
place the buffer was last free'd at instead of just the caller that
freed it.
I haven't yet found a pattern on how this happens yet.
Luis
On 10/12/2010 11:40 AM, Luis R. Rodriguez wrote:
> On Tue, Oct 12, 2010 at 11:35 AM, Ben Greear<[email protected]> wrote:
>> On 10/11/2010 11:10 PM, Luis R. Rodriguez wrote:
>>>
>>> On Mon, Oct 11, 2010 at 8:27 PM, Ben Greear<[email protected]>
>>> wrote:
>>
>>>> Another thing I was thinking about: Maybe the queue of skbs and dma
>>>> addresses
>>>> in ath9k is getting corrupted by multiple VIFs trying to write at once?
>>>> Maybe
>>>> some locking is needed in the xmit path?
>>>
>>> That was my second hunch. My first shot was to use spin_lock_irqsave()
>>> over the the uses of the rxbuf list and that seemed to help but I
>>> still managed to get a poison eventually. My next item to check for is
>>> of the permissibility of creating too much pressure to the point we
>>> end up looping over the rxbuf list and race against mac80211 free'ing
>>> a buffer. Will test that tomorrow if nothing else comes up creeping my
>>> priority queue.
>>
>> This code looks weird to me. One of the paprd branches
>> deletes the skb, the other doesn't appear to. Neither
>> null out bf->bf_mpdu, which would appear to leave a dangling
>> pointer in at least the dev_kfree_skb_any() branch.
>>
>> ath_tx_complete frees it's skb in all cases, so another
>> bf->bf_mpdu dangling pointer issue.
>>
>> Maybe at the least we should null out bf->bf_mpdu when
>> skb is consumed?
>
> You're reading my mind, that was what I was going to test today. Still
> doing e-mail sweep though.
At least in the xmit path, it seems cards that have EDMA support do
things a bit different. Out of curiosity, on the system(s), you reproduce
this, are any of yours supporting EDMA? Mine appear to not support EDMA.
Thanks,
Ben
--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
On 10/15/2010 04:38 PM, Luis R. Rodriguez wrote:
> On Fri, Oct 15, 2010 at 04:33:50PM -0700, Ben Greear wrote:
>> On 10/15/2010 04:21 PM, Luis R. Rodriguez wrote:
>>> Ben, please give this patch a shot. I addresses three races on the PCU:
>>>
>>> * When we were stopping the CPU for non-EDMA cards we never locked against
>>> anything starting the PCU again
>>>
>>> * ath9k_hw_startpcureceive() was being called without locking
>>>
>>> * Although we lock on the rxbuf lock for contention against starting/stopping
>>> the PCU, we also need to lock on the driver in locations where we start/stop
>>> the PCU within the same location otherwise we end up in inconsistant states
>>> and the hardware may end up proessing an incorrect buffer for DMA. To
>>> protect against this we use a new PCU lock on the main part of the driver to
>>> ensure each start/stop/reset operation is done atomically.
>>>
>>> And fixes one issue as a side effect:
>>>
>>> * No more packet loss on ping flood when you have one STA associated :)
>>>
>>> The only issue I see with this is I eventually run out of memory and my box
>>> becomes useless, unless I am mistaking that for some other issue.
>>>
>>> Please give this a shot and if it cures your woes I'll split it up into
>>> 3 separate patches, or maybe just two, one for the first two and one for
>>> the last issue.
>>
>> Sounds good, but this lockdep splat happens almost immediately upon starting
>> my app:
>>
>> =======================================================
>> [ INFO: possible circular locking dependency detected ]
>> 2.6.36-rc8-wl+ #32
>> -------------------------------------------------------
>> swapper/0 is trying to acquire lock:
>> (&(&sc->rx.pcu_lock)->rlock){+.-...}, at: [<fa16e5c7>] ath9k_tasklet+0x7e/0x140 [ath9k]
>>
>> but task is already holding lock:
>> (&(&sc->rx.rxflushlock)->rlock){+.-...}, at: [<fa16e5b9>] ath9k_tasklet+0x70/0x140 [ath9k]
>>
>> which lock already depends on the new lock.
>>
>>
>> the existing dependency chain (in reverse order) is:
>>
>> -> #1 (&(&sc->rx.rxflushlock)->rlock){+.-...}:
>> [<c0457639>] lock_acquire+0x5a/0x78
>> [<c075f6ed>] _raw_spin_lock_bh+0x20/0x2f
>> [<fa170513>] ath_flushrecv+0x14/0x61 [ath9k]
>
> Ah we just need to nuke the flush lock, one second. Also remove my
> skb_copy() otherwise you will really run out of memory quickly,
> unless you really want to test with it :)
Since you free the pkt, it shouldn't really consume, eh?
Seems like we should be able to handle an extra 5 skbs floating
around the system..
I'll try your new patch now...
Thanks,
Ben
>
> Luis
--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
On Thu, Oct 14, 2010 at 3:16 PM, Luis R. Rodriguez <[email protected]> wrote:
> 2010/10/14 Ben Greear <[email protected]>:
>> On 10/14/2010 02:52 PM, Björn Smedman wrote:
>>>
>>> 2010/10/13 Björn Smedman<[email protected]>:
>>>>
>>>> Hi Ben,
>>>>
>>>> First of all keep up the good work. :)
>>>>
>>>> On Wed, Oct 13, 2010 at 6:39 PM, Ben Greear<[email protected]>
>>>> wrote:
>>>> [snip]
>>>>>
>>>>> Either way, it seems safer to null out the bf_ampdu field after
>>>>> the memory is consumed..it could prevent some tricky bugs later.
>>>>
>>>> I think this is a good idea. But it probably wont be enough to null
>>>> out bf_mpdu. You also need to look at bf_buf_addr (which if I
>>>> understand correctly is the physical address the DMA engine will
>>>> actually write RXed frames to) and bf_dmacontext (which seems in most
>>>> cases to hold an identical address and may in fact be where the DMA
>>>> engine will really write the frame).
>>>
>>> I took another look at the code. It turns out both bf_buf_addr and
>>> bf_dmacontext are in fact meaningless to the DMA. Instead each bf
>>> holds a pointer (bf_desc) to the real DMA descriptor which in turn
>>> holds the address (ds_data) where the DMA will really (really this
>>> time) write the frame. There is also a field to hold the virtual
>>> address of the same place (ds_vdata).
>>>
>>> It's a little too much work for me to set up the testbed you have Ben
>>> but would be interesting to see what happens if you set
>>> bf->bf_desc->ds_{data,vdata} = 0 as well. No?
>>
>> I'll investigate those suggestions.
>>
>> But setting up a test-bed is as easy
>> as getting an ath9k NIC in a system, with a few APs around, and run the
>> script below.
>>
>> You do not need any traffic generation, dhcp, etc...seems just beacons and
>> whatever
>> wpa_supplicant is doing is enough to hit the problem fast. (Make sure
>> you are compiled to detect memory poisoning, of course).
>>
>> You'll need to fix the paths to the executables most likely.
>>
>
> You don't need such complicated scripts, I've managed to reproduce now
> by creating a lot of monitor interfaces and then looping with a
> regular interface issuing a scan command over and over. I suspect
> I'll be able to do this as well by changing channels instead of doing
> a scan. I believe the issue may be due to races in hardware on resets
> and enabling RX on an already freed buffer.
Fun enough if I just create one monitor interface and loop quickly
over some 2 GHz channels where I know I have traffic nearby I don't
see the poison. So channel changes don't seem to do much because this
is changing channels as fast as possible from userspace. I also can
confirm that I see frames from the different channels as I move along.
Luis
On Tue, Oct 5, 2010 at 10:47 AM, Ben Greear <[email protected]> wrote:
> On 10/05/2010 10:43 AM, Luis R. Rodriguez wrote:
>>
>> On Tue, Oct 5, 2010 at 10:38 AM, Ben Greear<[email protected]>
>> wrote:
>>>
>>> On 10/05/2010 10:36 AM, Luis R. Rodriguez wrote:
>>>>
>>>> On Tue, Oct 5, 2010 at 10:24 AM, Ben Greear<[email protected]>
>>>> wrote:
>>>>>
>>>>> On 10/05/2010 10:16 AM, Luis R. Rodriguez wrote:
>>>>>>
>>>>>> On Tue, Oct 5, 2010 at 10:00 AM, Ben Greear<[email protected]>
>>>>>> wrote:
>>>>>>>
>>>>>>> I started seeing this very soon after creating interfaces.
>>>>>>
>>>>>> Can you be more specific how one can reproduce?
>>>>>
>>>>> Enable SLUB debugging, DEBUG_PAGEALLOC (Debug page memory allocations),
>>>>> lockdep, pre-empt.
>>>>
>>>> OK I just enabled PAGE_ALLOC thingy, it wasn't enabled on my kernel.
>>>>
>>>>> Please try creating a bunch of stations on one ath9k phy,
>>>>
>>>> How many?
>>>
>>> 130 is a nice round number :)
>>
>> Let me get this straight. You are creating 130 STAs with ath9k using
>> iw, then running 150 instances of wpa_supplicant to connect to the
>> same AP?
>
> Well, 130 instances, yes :)
>
>>> I'll try with a smaller number and see if I can hit it.
>>
>> Please do, you can use the log2n approach, should take you 7 tries,
>> each by a power of 2, start at the middle of course.
>
> Sure...will be a few minutes.
>
> If it's any sane bug, then hopefully I can hit it with a small
> number.
>
> Out of curiosity, is there any particular number of STAs > 1 you guys
> test with?
We don't test this :)
Luis
On Thu, Oct 14, 2010 at 2:31 PM, Ben Greear <[email protected]> wrote:
> On 10/14/2010 02:25 PM, Luis R. Rodriguez wrote:
>>
>> On Wed, Oct 13, 2010 at 10:48 AM, Ben Greear<[email protected]>
>> wrote:
>>>
>>> On 10/13/2010 10:29 AM, Luis R. Rodriguez wrote:
>>>>
>>>> On Wed, Oct 13, 2010 at 10:12 AM, Ben Greear<[email protected]>
>>>> wrote:
>>>>>
>>>>> On 10/12/2010 11:40 AM, Luis R. Rodriguez wrote:
>>>>>>
>>>>>> On Tue, Oct 12, 2010 at 11:35 AM, Ben Greear<[email protected]>
>>>>>> wrote:
>>>>>>>
>>>>>>> On 10/11/2010 11:10 PM, Luis R. Rodriguez wrote:
>>>>>>>>
>>>>>>>> On Mon, Oct 11, 2010 at 8:27 PM, Ben Greear<[email protected]>
>>>>>>>> wrote:
>>>>>>>
>>>>>>>>> Another thing I was thinking about: Maybe the queue of skbs and
>>>>>>>>> dma
>>>>>>>>> addresses
>>>>>>>>> in ath9k is getting corrupted by multiple VIFs trying to write at
>>>>>>>>> once?
>>>>>>>>> Maybe
>>>>>>>>> some locking is needed in the xmit path?
>>>>>>>>
>>>>>>>> That was my second hunch. My first shot was to use
>>>>>>>> spin_lock_irqsave()
>>>>>>>> over the the uses of the rxbuf list and that seemed to help but I
>>>>>>>> still managed to get a poison eventually. My next item to check for
>>>>>>>> is
>>>>>>>> of the permissibility of creating too much pressure to the point we
>>>>>>>> end up looping over the rxbuf list and race against mac80211
>>>>>>>> free'ing
>>>>>>>> a buffer. Will test that tomorrow if nothing else comes up creeping
>>>>>>>> my
>>>>>>>> priority queue.
>>>>>>>
>>>>>>> This code looks weird to me. One of the paprd branches
>>>>>>> deletes the skb, the other doesn't appear to. Neither
>>>>>>> null out bf->bf_mpdu, which would appear to leave a dangling
>>>>>>> pointer in at least the dev_kfree_skb_any() branch.
>>>>>>>
>>>>>>> ath_tx_complete frees it's skb in all cases, so another
>>>>>>> bf->bf_mpdu dangling pointer issue.
>>>>>>>
>>>>>>> Maybe at the least we should null out bf->bf_mpdu when
>>>>>>> skb is consumed?
>>>>>>
>>>>>> You're reading my mind, that was what I was going to test today. Still
>>>>>> doing e-mail sweep though.
>>>>>
>>>>> At least in the xmit path, it seems cards that have EDMA support do
>>>>> things a bit different. Out of curiosity, on the system(s), you
>>>>> reproduce
>>>>> this, are any of yours supporting EDMA? Mine appear to not support
>>>>> EDMA.
>>>>
>>>> EDMA is used on>= AR9003 families by Atheros. And no, I am not
>>>> testing with an EDMA card, I am testing with an AR9002 family card,
>>>> the AR9280 card. I am going to disregard the TX stuff as the bug is an
>>>> RX issue :) I was able to more easily reproduce by doing an skb_copy()
>>>> and free'ing the buffer right afterwards on the ath_send_to_mac80211()
>>>> thingy, So it does appear that the poison check just happens more
>>>> often when we do an skb_copy(). One reason this is easy to reproduce
>>>> with multiple STAs is mac80211 uses skb_copy() to process each
>>>> received skb for each STA.
>>>>
>>>> In my tests so far, protecting the rxbuf list with spin_lock_irqsave()
>>>> did not help, and the wmb(); didn't either, something else is going on
>>>> here. It would be nice to hack slab to keep an entire trace of the
>>>> place the buffer was last free'd at instead of just the caller that
>>>> freed it.
>>>
>>> I instrumented slub a while back and got the backtrace. It
>>> was always in the same place for my testing.
>>>
>>> Here's the slub patch if you are interested in using it yourself:
>>> https://patchwork.kernel.org/patch/236921/
>>
>> when compiling this patch I get:
>>
>> arch/x86/built-in.o: In function `store_stack':
>> /home/mcgrof/wireless-testing/arch/x86/kernel/dumpstack.c:259:
>> undefined reference to `store_trace'
>
> You are compiling on 32-bit system? I see the method in
> the patch, but probably only for 32-bit x86...
Ah no I'm on 64-bit.
Luis
On 10/07/2010 11:45 AM, Ben Greear wrote:
> On 10/07/2010 11:42 AM, Luis R. Rodriguez wrote:
>> On Thu, Oct 7, 2010 at 11:39 AM, Ben Greear<[email protected]>
>> wrote:
>>> On 10/07/2010 11:29 AM, Luis R. Rodriguez wrote:
>>>>
>>>> On Thu, Oct 7, 2010 at 11:14 AM, Johannes Berg
>>>> <[email protected]> wrote:
>>>>>
>>>>> On Thu, 2010-10-07 at 10:33 -0700, Ben Greear wrote:
>>>>>>
>>>>>> In case it helps, here is a dump of where the corrupted SKB was
>>>>>> deleted.
>>>>>
>>>>> I wonder, do you have a machine with a decent IOMMU? Adding IOMMU
>>>>> debugging into the mix could help you figure out if it's a DMA
>>>>> problem.
>>>>
>>>> Ben, how much traffic are you RX'ing on these virtual interfaces?
>>>
>>> I disabled my user-space application, and this script alone can
>>> reproduce
>>> the problem fairly quickly on my system. You will need to change some
>>> of those first variables. Just start it and wait a few minutes and
>>> watch the splats show on the console :)
>>>
>>> Note that I am not generating any traffic, but the wpa_supplicants are
>>> doing their thing of course...
>>>
>>> I'm using the kernel found here:
>>> http://dmz2.candelatech.com/git/gitweb.cgi?p=linux.wireless-testing.ct/.git;a=summary
>>>
>>>
>>> It's latest wireless-testing with some of my own patches, and some
>>> I've gathered from here an there. I doubt I'm causing this problem,
>>> but if you can't reproduce it with this script on your kernels,
>>> I can try with base wireless-testing or whatever you are using.
>>
>> I'll run this now, but can you try a vanilla wireless-testing? I hear
>> the latest wireless-testing is borked so maybe try (git reset --hard
>> master-2010-09-29), its what I'm on.
>
> You are liable to hit a bunch of those crashes I've been reporting
> before you hit the DMA thing if you don't use latest (with Johanne's scan
> locking patch).
>
> I'm going to poke at IOMMU debugging and see what I find.
>
> I'll start a compile of vanilla wireless-testing + scan fix as well.
Well, vanilla + scan patch locked pretty hard when I started the script.
I was able to get sysrq to dump the locks, but it didn't seem to complete
that and couldn't even dump more sysrq info after that.
Might be something entirely different, of course, and no idea if
this lock dump shows any real problem.
Oct 7 12:08:43 localhost kernel: SysRq : Show Locks Held
Oct 7 12:08:43 localhost kernel:
Oct 7 12:08:43 localhost kernel: Showing all locks held in the system:
Oct 7 12:08:43 localhost kernel: 3 locks held by kworker/0:0/4:
Oct 7 12:08:43 localhost kernel: #0: (events){+.+.+.}, at: [<c0443cea>] process_one_work+0x145/0x295
Oct 7 12:08:43 localhost kernel: #1: ((linkwatch_work).work){+.+.+.}, at: [<c0443cea>] process_one_work+0x145/0x295
Oct 7 12:08:43 localhost kernel: #2: (rtnl_mutex){+.+.+.}, at: [<c06da190>] rtnl_lock+0xf/0x11
Oct 7 12:08:43 localhost kernel: 3 locks held by kworker/u:1/38:
Oct 7 12:08:43 localhost kernel: #0: ((wiphy_name(local->hw.wiphy))){+.+.+.}, at: [<c0443cea>] process_one_work+0x145/0x295
Oct 7 12:08:43 localhost kernel: #1: ((&sta->ampdu_mlme.work)){+.+...}, at: [<c0443cea>] process_one_work+0x145/0x295
Oct 7 12:08:43 localhost kernel: #2: (&sta->ampdu_mlme.mtx){+.+...}, at: [<f8887180>] ieee80211_ba_session_work+0x52/0xbe [mac80211]
Oct 7 12:08:43 localhost kernel: 1 lock held by mingetty/1584:
Oct 7 12:08:43 localhost kernel: #0: (&tty->atomic_read_lock){+.+...}, at: [<c05f5e91>] n_tty_read+0x1d1/0x5ed
Oct 7 12:08:43 localhost kernel: 1 lock held by mingetty/1586:
Oct 7 12:08:43 localhost kernel: #0: (&tty->atomic_read_lock){+.+...}, at: [<c05f5e91>] n_tty_read+0x1d1/0x5ed
Oct 7 12:08:43 localhost kernel: 1 lock held by mingetty/1589:
Oct 7 12:08:43 localhost kernel: #0: (&tty->atomic_read_lock){+.+...}, at: [<c05f5e91>] n_tty_read+0x1d1/0x5ed
Oct 7 12:08:43 localhost kernel: 1 lock held by mingetty/1593:
Oct 7 12:08:43 localhost kernel: #0: (&tty->atomic_read_lock){+.+...}, at: [<c05f5e91>] n_tty_read+0x1d1/0x5ed
Oct 7 12:08:43 localhost kernel: 1 lock held by mingetty/1596:
Oct 7 12:08:43 localhost kernel: #0: (&tty->atomic_read_lock){+.+...}, at: [<c05f5e91>] n_tty_read+0x1d1/0x5ed
Oct 7 12:08:43 localhost kernel: 1 lock held by mingetty/1598:
Oct 7 12:08:43 localhost kernel: #0: (&tty->atomic_read_lock){+.+...}, at: [<c05f5e91>] n_tty_read+0x1d1/0x5ed
Oct 7 12:08:43 localhost kernel: 1 lock held by bash/1683:
Oct 7 12:08:43 localhost kernel: #0: (&tty->atomic_read_lock){+.+...}, at: [<c05f5e91>] n_tty_read+0x1d1/0x5ed
Oct 7 12:08:43 localhost kernel: 1 lock held by bash/1728:
Oct 7 12:08:43 localhost kernel: #0: (&tty->atomic_read_lock){+.+...}, at: [<c05f5e91>] n_tty_read+0x1d1/0x5ed
Oct 7 12:08:43 localhost kernel: 1 lock held by bash/1752:
Oct 7 12:08:43 localhost kernel: #0: (&tty->atomic_read_lock){+.+...}, at: [<c05f5e91>] n_tty_read+0x1d1/0x5ed
Oct 7 12:08:43 localhost kernel: 1 lock held by wpa_supplicant/2840:
Oct 7 12:08:43 localhost kernel: #0: (rtnl_mutex){+.+.+.}, at: [<c06da190>] rtnl_lock+0xf/0x11
Oct 7 12:08:43 localhost kernel: 1 lock held by wpa_supplicant/2842:
Oct 7 12:08:43 localhost kernel: #0: (rtnl_mutex){+.+.+.}, at: [<c06da190>] rtnl_lock+0xf/0x11
Oct 7 12:08:43 localhost kernel: 1 lock held by wpa_supplicant/2844:
Oct 7 12:08:43 localhost kernel: #0: (rtnl_mutex){+.+.+.}, at: [<c06da190>] rtnl_lock+0xf/0x11
Oct 7 12:08:43 localhost kernel: 1 lock held by wpa_supplicant/2846:
Oct 7 12:08:43 localhost kernel: #0: (rtnl_mutex){+.+.+.}, at: [<c06da190>] rtnl_lock+0xf/0x11
Oct 7 12:08:43 localhost kernel: 1 lock held by wpa_supplicant/2848:
Oct 7 12:08:43 localhost kernel: #0: (rtnl_mutex){+.+.+.}, at: [<c06da190>] rtnl_lock+0xf/0x11
Oct 7 12:08:43 localhost kernel: 1 lock held by wpa_supplicant/2850:
Oct 7 12:08:43 localhost kernel: #0: (rtnl_mutex){+.+.+.}, at: [<c06da190>] rtnl_lock+0xf/0x11
Oct 7 12:08:43 localhost kernel: 1 lock held by wpa_supplicant/2852:
Oct 7 12:08:43 localhost kernel: #0: (rtnl_mutex){+.+.+.}, at: [<c06da190>] rtnl_lock+0xf/0x11
Oct 7 12:08:43 localhost kernel: 1 lock held by wpa_supplicant/2854:
Oct 7 12:08:43 localhost kernel: #0: (rtnl_mutex){+.+.+.}, at: [<c06da190>] rtnl_lock+0xf/0x11
Oct 7 12:08:43 localhost kernel: 1 lock held by wpa_supplicant/2856:
Oct 7 12:08:43 localhost kernel: #0: (rtnl_mutex){+.+.+.}, at: [<c06da190>] rtnl_lock+0xf/0x11
OcSysRq : Show State
Process wpa_supplicant (pid: 2904, ti=f3946000 task=f4124a60 task.ti=f3946000)
Stack:
Call Trace:
Code: 73 18 8b 75 08 01 73 14 88 4c 02 1c 5b 5e 5d c3 55 89 e5 57 56 53 8b 5d 08 6b 72 04 3c ff 84 30 70 11 00 00 8b 79 10 6b 72 04 3c <8b> 7f 50 01 bc 30
SysRq : Show Locks Held
Process wpa_supplicant (pid: 2904, ti=f3946000 task=f4124a60 task.ti=f3946000)
Stack:
Call Trace:
Code: 73 18 8b 75 08 01 73 14 88 4c 02 1c 5b 5e 5d c3 55 89 e5 57 56 53 8b 5d 08 6b 72 04 3c ff 84 30 70 11 00 00 8b 79 10 6b 72 04 3c <8b> 7f 50 01 bc 30
Thanks,
Ben
--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
On 10/07/2010 05:42 PM, Bruno Randolf wrote:
> On Fri October 8 2010 04:27:22 Johannes Berg wrote:
>> I'm still convinced something is wrong with ath9k RX DMA, as you've seen
>> the contents of frames written to already freed memory regions. Since I
>> don't know anything about ath9k, you should probably not rely on me
>> though :-)
>
> not sure if this is related or not, but it reminds me of:
>
> "ath9k: BUG kmalloc-8192: Poison overwritten"
> http://marc.info/?l=linux-kernel&m=127379077422354&w=2
This looks identical...some sort of scan results in poisoned
buffer.
>
> and:
>
> "ath5k in monitor mode: Poison overwritten on buffers allocated from
> ath_rxbuf_alloc"
> https://bugzilla.kernel.org/show_bug.cgi?id=15861
We've been doing similar (to the ath9k issue)
tests with ath5k and it's been much more stable. But, our kernel is
currently totally larded down with debugging so we can't the NIC as
hard as we'd like.
I'll make sure we do some hard-core testing on ath5k soon.
Thanks,
Ben
>
> bruno
--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
On Thu, 2010-10-07 at 12:22 -0700, Ben Greear wrote:
> After reboot, and re-run of the script,
> I saw this in the logs, and shortly after,
> the SLUB poison warning dumped to screen.
>
> Maybe those DMA errors are serious?
> ath: Failed to stop TX DMA in 100 msec after killing last frame
> ath: Failed to stop TX DMA. Resetting hardware!
That's TX DMA, it can hardly result in invalid memory writes like the
ones you've been seeing.
I'm still convinced something is wrong with ath9k RX DMA, as you've seen
the contents of frames written to already freed memory regions. Since I
don't know anything about ath9k, you should probably not rely on me
though :-)
johannes
On 10/07/2010 11:29 AM, Luis R. Rodriguez wrote:
> On Thu, Oct 7, 2010 at 11:14 AM, Johannes Berg
> <[email protected]> wrote:
>> On Thu, 2010-10-07 at 10:33 -0700, Ben Greear wrote:
>>> In case it helps, here is a dump of where the corrupted SKB was deleted.
>>
>> I wonder, do you have a machine with a decent IOMMU? Adding IOMMU
>> debugging into the mix could help you figure out if it's a DMA problem.
>
> Ben, how much traffic are you RX'ing on these virtual interfaces?
I disabled my user-space application, and this script alone can reproduce
the problem fairly quickly on my system. You will need to change some
of those first variables. Just start it and wait a few minutes and
watch the splats show on the console :)
Note that I am not generating any traffic, but the wpa_supplicants are
doing their thing of course...
I'm using the kernel found here:
http://dmz2.candelatech.com/git/gitweb.cgi?p=linux.wireless-testing.ct/.git;a=summary
It's latest wireless-testing with some of my own patches, and some
I've gathered from here an there. I doubt I'm causing this problem,
but if you can't reproduce it with this script on your kernels,
I can try with base wireless-testing or whatever you are using.
#!/usr/bin/perl
use strict;
my $iw = "./local/sbin/iw";
my $ip = "./local/sbin/ip";
my $wpa_s = "./local/bin/wpa_supplicant";
my $ssid = "candela-n";
my $key = "wpadmz123";
my $phy = "phy0";
my $max = 32;
my $i;
my $bmac = "00:01:02:03:04:";
my $cmd;
# Create stations
for ($i = 0; $i<$max; $i++) {
runCmd("$iw phy $phy interface add sta$i type station");
my $mc5 = "$i";
if (length($mc5) == 1) {
$mc5 = "0$mc5"; # pad mac octet
}
my $mac = "$bmac$mc5";
runCmd("$ip link set sta$i address $mac");
runCmd("$iw dev sta$i set power_save off");
}
# Bring them up with WPA
for ($i = 0; $i<$max; $i++) {
open(FD, ">sta$i" . "_wpa.conf") || die("Couldn't open file: $!\n");
print FD "
ctrl_interface=/var/run/wpa_supplicant
fast_reauth=1
#can_scan_one=1
network={
ssid=\"$ssid\"
proto=WPA
key_mgmt=WPA-PSK
psk=\"$key\"
pairwise=TKIP CCMP
group=TKIP CCMP
}
";
runCmd("$wpa_s -B -i sta$i -c sta$i" . "_wpa.conf -P sta$i" . "_wpa.pid -t -f sta$i" . "_wpa.log");
}
sub runCmd {
my $cmd = shift;
print "$cmd\n";
`$cmd`;
}
--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
On Thu, 2010-10-07 at 12:14 -0700, Ben Greear wrote:
> > I'll start a compile of vanilla wireless-testing + scan fix as well.
>
> Well, vanilla + scan patch locked pretty hard when I started the script.
>
> I was able to get sysrq to dump the locks, but it didn't seem to complete
> that and couldn't even dump more sysrq info after that.
>
> Might be something entirely different, of course, and no idea if
> this lock dump shows any real problem.
Don't see any issues in this part of the dump -- heavy contention on the
RTNL but that was expected with your usage.
johannes
I had a chance to try your latest patch on a different machine and AP. This time,
I was using 130 or so stations, but no encryption (no supplicant).
I saw these exceptions below. The warn-on hits the !stopped check in ath_stoprecv.
The system didn't crash, but all of the STAs soon dis-associated because of "inactivity".
I haven't checked if that is some issue with my AP or what..
bool ath_stoprecv(struct ath_softc *sc)
{
struct ath_hw *ah = sc->sc_ah;
bool stopped;
spin_lock_bh(&sc->rx.rxbuflock);
ath9k_hw_stoppcurecv(ah);
ath9k_hw_setrxfilter(ah, 0);
stopped = ath9k_hw_stopdmarecv(ah);
if (sc->sc_ah->caps.hw_caps & ATH9K_HW_CAP_EDMA)
ath_edma_stop_recv(sc);
else
sc->rx.rxlink = NULL;
spin_unlock_bh(&sc->rx.rxbuflock);
WARN_ON(!stopped);
return stopped;
}
ADDRCONF(NETDEV_UP): sta130: link is not ready
sta90: no IPv6 routers present
ath: DMA failed to stop in 10 ms AR_CR=0x00000024 AR_DIAG_SW=0x40000020
------------[ cut here ]------------
WARNING: at /home/greearb/git/linux.wireless-testing/drivers/net/wireless/ath/ath9k/recv.c:532 ath_stoprecv+0x80/0x87 [ath9k]()
Hardware name: 945GM
Modules linked in: 8021q garp stp llc michael_mic macvlan pktgen iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi nfs lockd fscache nfs_acl auth_rpcgss
sunrpc p4_clockmod ipv6 uinput arc4 ecb ath9k snd_intel8x0 mac80211 snd_ac97_codec ac97_bus snd_seq snd_seq_device ath9k_common snd_pcm ath9k_hw ath snd_timer
microcode pcspkr cfg80211 i2c_i801 snd soundcore snd_page_alloc serio_raw e1000e iTCO_wdt iTCO_vendor_support yenta_socket floppy i915 drm_kms_helper drm
i2c_algo_bit i2c_core video output [last unloaded: ipt_addrtype]
Pid: 5, comm: kworker/u:0 Tainted: G W 2.6.36-rc8-wl+ #12
Call Trace:
[<c0434647>] warn_slowpath_common+0x65/0x7a
[<f9450e38>] ? ath_stoprecv+0x80/0x87 [ath9k]
[<c043466b>] warn_slowpath_null+0xf/0x13
[<f9450e38>] ath_stoprecv+0x80/0x87 [ath9k]
[<f944f580>] ath_set_channel+0x99/0x1ff [ath9k]
[<f94502b2>] ath9k_config+0x305/0x3d8 [ath9k]
[<f9246a2e>] ieee80211_hw_config+0x11b/0x125 [mac80211]
[<f924aa51>] ieee80211_scan_work+0x29e/0x3ed [mac80211]
[<c0443d72>] ? process_one_work+0x145/0x295
[<c0443dbc>] process_one_work+0x18f/0x295
[<c0443d72>] ? process_one_work+0x145/0x295
[<f924a7b3>] ? ieee80211_scan_work+0x0/0x3ed [mac80211]
[<c04453e5>] worker_thread+0xf9/0x1b8
[<c04452ec>] ? worker_thread+0x0/0x1b8
[<c0447d2f>] kthread+0x62/0x67
[<c0447ccd>] ? kthread+0x0/0x67
[<c0403506>] kernel_thread_helper+0x6/0x1a
---[ end trace bc53fa727ee2ae42 ]---
ath: DMA failed to stop in 10 ms AR_CR=0x00000024 AR_DIAG_SW=0x42000020
ath: DMA failed to stop in 10 ms AR_CR=0x00000024 AR_DIAG_SW=0x40000020
------------[ cut here ]------------
WARNING: at /home/greearb/git/linux.wireless-testing/drivers/net/wireless/ath/ath9k/recv.c:532 ath_stoprecv+0x80/0x87 [ath9k]()
Hardware name: 945GM
Modules linked in: 8021q garp stp llc michael_mic macvlan pktgen iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi nfs lockd fscache nfs_acl auth_rpcgss
sunrpc p4_clockmod ipv6 uinput arc4 ecb ath9k snd_intel8x0 mac80211 snd_ac97_codec ac97_bus snd_seq snd_seq_device ath9k_common snd_pcm ath9k_hw ath snd_timer
microcode pcspkr cfg80211 i2c_i801 snd soundcore snd_page_alloc serio_raw e1000e iTCO_wdt iTCO_vendor_support yenta_socket floppy i915 drm_kms_helper drm
i2c_algo_bit i2c_core video output [last unloaded: ipt_addrtype]
Pid: 41, comm: kworker/u:2 Tainted: G W 2.6.36-rc8-wl+ #12
Call Trace:
[<c0434647>] warn_slowpath_common+0x65/0x7a
[<f9450e38>] ? ath_stoprecv+0x80/0x87 [ath9k]
[<c043466b>] warn_slowpath_null+0xf/0x13
[<f9450e38>] ath_stoprecv+0x80/0x87 [ath9k]
[<f944f580>] ath_set_channel+0x99/0x1ff [ath9k]
[<f94502b2>] ath9k_config+0x305/0x3d8 [ath9k]
[<f9246a2e>] ieee80211_hw_config+0x11b/0x125 [mac80211]
[<f924aa51>] ieee80211_scan_work+0x29e/0x3ed [mac80211]
[<c0443d72>] ? process_one_work+0x145/0x295
[<c0443dbc>] process_one_work+0x18f/0x295
[<c0443d72>] ? process_one_work+0x145/0x295
[<f924a7b3>] ? ieee80211_scan_work+0x0/0x3ed [mac80211]
[<c04453e5>] worker_thread+0xf9/0x1b8
[<c04452ec>] ? worker_thread+0x0/0x1b8
[<c0447d2f>] kthread+0x62/0x67
[<c0447ccd>] ? kthread+0x0/0x67
[<c0403506>] kernel_thread_helper+0x6/0x1a
---[ end trace bc53fa727ee2ae43 ]---
ath: DMA failed to stop in 10 ms AR_CR=0x00000024 AR_DIAG_SW=0x42000020
sta126: direct probe to 00:18:e7:cb:ad:6e (try 1)
sta130: direct probe to 00:18:e7:cb:ad:6e (try 1)
sta91: no IPv6 routers present
sta126: direct probe to 00:18:e7:cb:ad:6e (try 2)
sta130: direct probe to 00:18:e7:cb:ad:6e (try 2)
sta126: direct probe to 00:18:e7:cb:ad:6e (try 3)
sta130: direct probe to 00:18:e7:cb:ad:6e (try 3)
sta126: direct probe to 00:18:e7:cb:ad:6e timed out
sta130: direct probe to 00:18:e7:cb:ad:6e timed out
sta92: no IPv6 routers present
...
Thanks,
Ben
--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
On 10/14/2010 04:48 PM, Luis R. Rodriguez wrote:
> On Thu, Oct 14, 2010 at 04:39:45PM -0700, Luis R. Rodriguez wrote:
>> On Thu, Oct 14, 2010 at 4:30 PM, Ben Greear<[email protected]> wrote:
>>> On 10/14/2010 04:19 PM, Luis R. Rodriguez wrote:
>>>>
>>>> Ok please try this patch, it cures it for me.
>>>
>>> Well, it got a lot further than normal, but it still
>>> hit the poison check after a few minutes.
>>>
>>> Current test case is my app loading 130 or so stations, each running
>>> wpa_supplicant. All were created, and quite a few had associated
>>> when the poison check hit.
>>>
>>> So, it definitely looks like a step in the right direction, but
>>> not fully fixed yet.
>>>
>>> I'll do some more testing with this patch applied and using just my
>>> perl script to make sure the problem is reproducible outside of my
>>> application.
>>
>> Ok, whatever userspace does it should not corrupt to kernel, unless
>> its poking /dev/mem
>
> Can also try this one instead, it will prevent any other instances
> we would not have caught on stopping and starting RX here.
It ran longer than before any of your locking patches (about 3 minutes), but
it did hit the poison check.
Before it did, I had a bunch of OOM errors trying to allocate
skbs. I have 2GB of RAM on this system, but maybe it's not tuned
properly, and not all of that can be used for networking on 32-bit kernels....
I have Felix's 3 ani patches from ~3 days ago applied, running 130 stations
in WPA mode.
I'm going to try running fewer to dodge the OOM case,
but I have a few other things to take care of first.
Thanks,
Ben
>
> diff --git a/drivers/net/wireless/ath/ath9k/recv.c b/drivers/net/wireless/ath/ath9k/recv.c
> index fe73fc5..db677c4 100644
> --- a/drivers/net/wireless/ath/ath9k/recv.c
> +++ b/drivers/net/wireless/ath/ath9k/recv.c
> @@ -306,10 +306,8 @@ static void ath_edma_start_recv(struct ath_softc *sc)
>
> static void ath_edma_stop_recv(struct ath_softc *sc)
> {
> - spin_lock_bh(&sc->rx.rxbuflock);
> ath_rx_remove_buffer(sc, ATH9K_RX_QUEUE_HP);
> ath_rx_remove_buffer(sc, ATH9K_RX_QUEUE_LP);
> - spin_unlock_bh(&sc->rx.rxbuflock);
> }
>
> int ath_rx_init(struct ath_softc *sc, int nbufs)
> @@ -482,13 +480,14 @@ int ath_startrecv(struct ath_softc *sc)
> {
> struct ath_hw *ah = sc->sc_ah;
> struct ath_buf *bf, *tbf;
> + unsigned long flags;
>
> if (ah->caps.hw_caps& ATH9K_HW_CAP_EDMA) {
> ath_edma_start_recv(sc);
> return 0;
> }
>
> - spin_lock_bh(&sc->rx.rxbuflock);
> + spin_lock_irqsave(&sc->rx.rxbuflock, flags);
> if (list_empty(&sc->rx.rxbuf))
> goto start_recv;
>
> @@ -506,7 +505,7 @@ int ath_startrecv(struct ath_softc *sc)
> ath9k_hw_rxena(ah);
>
> start_recv:
> - spin_unlock_bh(&sc->rx.rxbuflock);
> + spin_unlock_irqrestore(&sc->rx.rxbuflock, flags);
> ath_opmode_init(sc);
> ath9k_hw_startpcureceive(ah, (sc->sc_flags& SC_OP_OFFCHANNEL));
>
> @@ -517,7 +516,9 @@ bool ath_stoprecv(struct ath_softc *sc)
> {
> struct ath_hw *ah = sc->sc_ah;
> bool stopped;
> + unsigned long flags;
>
> + spin_lock_irqsave(&sc->rx.rxbuflock, flags);
> ath9k_hw_stoppcurecv(ah);
> ath9k_hw_setrxfilter(ah, 0);
> stopped = ath9k_hw_stopdmarecv(ah);
> @@ -526,6 +527,7 @@ bool ath_stoprecv(struct ath_softc *sc)
> ath_edma_stop_recv(sc);
> else
> sc->rx.rxlink = NULL;
> + spin_unlock_irqrestore(&sc->rx.rxbuflock, flags);
>
> return stopped;
> }
--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
2010/10/18 Björn Smedman <[email protected]>:
> 2010/10/18 Luis R. Rodriguez <[email protected]>:
>> 2010/10/18 Björn Smedman <[email protected]>:
>>> 2010/10/15 Björn Smedman <[email protected]>
>>>>
>>>> 2010/10/15 Ben Greear <[email protected]>:
>>>> > I tried the patch below, and it didn't seem to help. Might even
>>>> > have hurt..as it died on divide-by-zero error:
>>>>
>>>> Hmm, looks like the ani code got a zero listen time from the hw...
>>>> That just might mean that the DMA actually hits one of these
>>>> descriptors. :)
>>>
>>> Am I the only one worried about this? Leaving a DMA descriptor
>>> pointing at memory which has been passed on to somebody else... To me
>>> that's like pointing a loaded gun at someone (and it seems this
>>> particular gun can go off a little haphazardly).
>>>
>>> Luis, given how hard it seems to be to get that locking and skb
>>> shoveling right, are you sure you want to keep pointing that DMA
>>> engine on innocent people's data?
>>
>> This is why this issue is of high priority to me. I no longer get the
>> poison nor does Ben, the RX poison issue is resolved as far as I can
>> tell, I just need to split up the patches into easily reviewable
>> chunks and get them upstream.
>
> The locking issue you found looks like it could cause those
> overwritten poisons (as well as some weird problems I've seen in AP
> mode with lots of monitor interfaces). It's really great to see that
> problem go.
:)
> All I'm saying is that this stuff is difficult and the
> next time we get it wrong we should try to avoid overwriting arbitrary
> kernel memory with our RXed frames (or TXing something sensitive).
Patches welcomed.
> Will the DMA engine stop when it sees a zero ds_data? In that case I
> suggest we never keep an address there that we do not want to RX to or
> TX from.
We will always have an skb available to DMA for RX, its part of the
design on RX on ath9k. We simply do drop the frame we just got DMA'd
if we cannot allocate a new skb from the kernel. So we should always
be able to DMA over and over and over. The issue here was due to a
race on stopping and starting the PCU, it got confused on which buffer
to write to.
> Also, is there some way to verify that we are not corrupting memory
> with the DMA? I mean the poison check is great
I'm only aware of the poison checks.
> but if I understand
> correctly it cannot detect if we overwrite active memory (i.e. before
> it is freed and marked with the poison).
Right, the best you can do is to understand the code. Detecting bogus
writes from hardware to are hard to detect on already used memory
given that you'd need to ensure you understand what a writer to that
area of memory will do.
Anyway if you find actual issues instead of pure speculation please let us know.
Luis
On Thu, Oct 14, 2010 at 12:15 PM, Ben Greear <[email protected]> wrote:
> On 10/13/2010 01:03 PM, Luis R. Rodriguez wrote:
>>
>> 2010/10/13 Björn Smedman<[email protected]>:
>>>
>>> Hi Ben,
>>>
>>> First of all keep up the good work. :)
>>>
>>> On Wed, Oct 13, 2010 at 6:39 PM, Ben Greear<[email protected]>
>>> wrote:
>>> [snip]
>>>>
>>>> Either way, it seems safer to null out the bf_ampdu field after
>>>> the memory is consumed..it could prevent some tricky bugs later.
>>>
>>> I think this is a good idea. But it probably wont be enough to null
>>> out bf_mpdu. You also need to look at bf_buf_addr (which if I
>>> understand correctly is the physical address the DMA engine will
>>> actually write RXed frames to) and bf_dmacontext (which seems in most
>>> cases to hold an identical address and may in fact be where the DMA
>>> engine will really write the frame).
>>
>> See the TODO list for ath9k:
>>
>> * Remove this redundancy crap: bf->bf_buf_addr = bf->bf_dmacontext;
>>
>> http://wireless.kernel.org/en/users/Drivers/ath9k/todo
>
> I just posted a patch that attempts this. It doesn't
> fix the memory clobber issue, but maybe the code is a bit cleaner/safer
> at least...
Easier to read at least, thanks.
Luis
On 10/05/2010 10:55 AM, Luis R. Rodriguez wrote:
> On Tue, Oct 5, 2010 at 10:47 AM, Ben Greear<[email protected]> wrote:
>>> Please do, you can use the log2n approach, should take you 7 tries,
>>> each by a power of 2, start at the middle of course.
>>
>> Sure...will be a few minutes.
>>
>> If it's any sane bug, then hopefully I can hit it with a small
>> number.
>>
>> Out of curiosity, is there any particular number of STAs> 1 you guys
>> test with?
>
> We don't test this :)
Heh, I sort of supposed so :)
Anyway, I started with one, doubled each time,
and when I added the 4 to bring the system to
8 STA, it crapped itself.
I ran 64kbps UDP data streams on pairs of STA interfaces
for 5 minutes between STA increments. I didn't get a chance to
start any traffic for the 4 that I had just added.
Earlier, I wasn't running any traffic at all, but of course
there is still noise on the network.
Note I'm running 'iwconfig foo' and 'cat /proc/net/wireless' in
the background as well, and that may have something to do with
things.
Oct 5 11:05:38 localhost kernel: =============================================================================
Oct 5 11:05:38 localhost kernel: BUG kmalloc-8192: Poison overwritten
Oct 5 11:05:38 localhost kernel: -----------------------------------------------------------------------------
Oct 5 11:05:38 localhost kernel:
Oct 5 11:05:38 localhost kernel: INFO: 0xf6362100-0xf63624d3. First byte 0x96 instead of 0x6b
Oct 5 11:05:38 localhost kernel: INFO: Allocated in ath_rxbuf_alloc+0x1d/0x74 [ath] age=11216 cpu=1 pid=0
Oct 5 11:05:38 localhost kernel: INFO: Freed in skb_release_data+0x8c/0x90 age=143 cpu=1 pid=8407
Oct 5 11:05:38 localhost kernel: INFO: Slab 0xc1fffc00 objects=3 used=2 fp=0xf6362030 flags=0x400040c1
Oct 5 11:05:38 localhost kernel: INFO: Object 0xf6362030 @offset=8240 fp=0x(null)
Oct 5 11:05:38 localhost kernel:
Oct 5 11:05:38 localhost kernel: Bytes b4 0xf6362020: 2a 00 00 00 67 fd 0b 00 5a 5a 5a 5a 5a 5a 5a 5a *...g�..ZZZZZZZZ
Oct 5 11:05:38 localhost kernel: Object 0xf6362030: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362040: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362050: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362060: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362070: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362080: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362090: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf63620a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf63620b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf63620c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf63620d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf63620e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf63620f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362100: 96 72 50 c0 11 2c 8e 7e 51 78 b2 6f 4c a2 58 ce .rP�.,.~Qx�oL�X
Oct 5 11:05:38 localhost kernel: Object 0xf6362110: b3 34 36 e4 a0 b3 d7 fb bb d9 c2 a7 04 9c 62 46 �46�.�����������٧..bF
Oct 5 11:05:38 localhost kernel: Object 0xf6362120: 32 b9 8f 72 0b aa 38 13 22 9a fa 68 5d eb 11 55 2�.r.�8.".�h]�.U
Oct 5 11:05:38 localhost kernel: Object 0xf6362130: 2f a3 95 08 16 bb 7e f4 45 45 77 2d c9 c1 63 0b /�...�~�EEw-��c.
Oct 5 11:05:38 localhost kernel: Object 0xf6362140: c4 e6 cf 9b 3c cb 88 6e a0 fc 16 d7 2a eb de b3 ���.<�.n.�.�*�
Oct 5 11:05:38 localhost kernel: Object 0xf6362150: 19 67 dd d6 ea c5 8e f1 e9 a1 23 0b 91 a4 49 24 .g����.�������#..�I$
Oct 5 11:05:38 localhost kernel: Object 0xf6362160: e0 b7 07 ce 68 fd b4 77 99 1b 6f 2d e4 3d 10 2f ��.�h�෴w..o-�=./
Oct 5 11:05:38 localhost kernel: Object 0xf6362170: a5 0e 46 8b 4f c3 75 2d 63 8e dd f5 25 9e da bb �.F.O�u-c.��%.�䥻
Oct 5 11:05:38 localhost kernel: Object 0xf6362180: 64 db d3 f2 e2 3a ce bd 7d b5 2d de 75 c5 7a 48 d����:�۽}�-�u�zH
Oct 5 11:05:38 localhost kernel: Object 0xf6362190: 17 f7 81 94 c4 48 5a 7a 6c ae a1 93 e5 74 98 67 .�..�HZzlޮ�.�t.g
Oct 5 11:05:38 localhost kernel: Object 0xf63621a0: 1c a9 e3 80 35 65 24 c7 ce 7f 37 1b a7 a3 35 b5 .��.5e$��.7.婧�5�
Oct 5 11:05:38 localhost kernel: Object 0xf63621b0: ee a2 7e 9c 30 f2 a4 8d e1 f7 e6 8c 0c c7 79 71 ��~.0�.���..�yq
Oct 5 11:05:38 localhost kernel: Object 0xf63621c0: 76 99 68 e5 e1 03 34 03 21 ff a9 35 98 20 0a ec v.h��.4.!��5...
Oct 5 11:05:38 localhost kernel: Object 0xf63621d0: 0f fc 03 3d dc 88 75 b2 39 f2 65 68 fd 27 de 83 .�.=�.uᩲ9�eh�'�.
Oct 5 11:05:38 localhost kernel: Object 0xf63621e0: 99 7a 61 a6 82 6d a9 af 79 c3 fa c3 20 fd 6d 2b .za�.m�????y���.�m+
Oct 5 11:05:38 localhost kernel: Object 0xf63621f0: b9 fc 0d f7 be 4f b9 4f c3 a1 97 00 0a ad 0a e2 ù�.��O�Oá...�.
Oct 5 11:05:38 localhost kernel: Object 0xf6362200: a6 88 03 42 5b c5 c7 cf 69 a5 42 5c b8 1b d7 6d ������..B[���iťB\�.�m
Oct 5 11:05:38 localhost kernel: Object 0xf6362210: 92 7a 64 04 1e 39 77 26 7f 22 11 69 86 60 fe aa .zd..9w&.".i.`�ת
Oct 5 11:05:38 localhost kernel: Object 0xf6362220: 1c d8 49 9a 9d a8 47 dd ae eb 33 53 cc ba 21 17 .�I..بG�ݮ�3S̺!.
Oct 5 11:05:38 localhost kernel: Object 0xf6362230: 62 78 92 99 ae f6 91 f8 71 da a4 35 6b 8c fd ce bx..뺮�.�qڤ5k.�
Oct 5 11:05:38 localhost kernel: Object 0xf6362240: 95 28 a1 7f af 14 fc 7a b7 6e 52 81 bd 22 35 72 .(�.����.�z�nR.�"5r
Oct 5 11:05:38 localhost kernel: Object 0xf6362250: 33 25 be 93 65 fd 1e 18 4c 4e 85 b5 55 e9 ed 53 3%�.e�..LN.�U��S
Oct 5 11:05:38 localhost kernel: Object 0xf6362260: 52 a2 3f 5d 92 64 0e ec 45 ce 0c 3d 98 56 fe 1e R������?].d.�E�.=.V�.
Oct 5 11:05:38 localhost kernel: Object 0xf6362270: 9d 95 0d 49 b2 a7 cb c8 2a 7d 2b 8a 9a 59 36 51 ...I�첧��*}+..Y6Q
Oct 5 11:05:38 localhost kernel: Object 0xf6362280: ca 8c cf 4e c0 df 09 42 5f 61 40 a5 7f 92 d8 88 �.�N��.B_a@˥..�.
Oct 5 11:05:38 localhost kernel: Object 0xf6362290: 0d 51 75 4b c3 2e 48 e1 14 21 7f 00 23 be 42 55 .QuK�.H�.!..#ؾBU
Oct 5 11:05:38 localhost kernel: Object 0xf63622a0: 9f 6b 88 67 d2 e1 ce 1c 6c 1f 39 b6 9b 73 e9 8c .k.g���.l.9Ҷ.s�.
Oct 5 11:05:38 localhost kernel: Object 0xf63622b0: 3b 44 70 de 0d 4b 78 40 18 a3 3c fc 50 4c a6 89 ;Dp�.Kx@.�<�PL飦.
Oct 5 11:05:38 localhost kernel: Object 0xf63622c0: 78 6e d1 c8 87 81 0c c0 41 2a d9 d1 98 10 48 b1 xn��...�A*��..Hѱ
Oct 5 11:05:38 localhost kernel: Object 0xf63622d0: e2 ef 37 77 98 e0 d3 61 67 9c 0c 5b 63 72 1c d3 ��7w.��ag..[cr.
Oct 5 11:05:38 localhost kernel: Object 0xf63622e0: ee 8b cf 29 7f 4b 18 b5 46 18 2e ba 43 e1 6a bb �.�).K.�F..C�j�
Oct 5 11:05:38 localhost kernel: Object 0xf63622f0: 4d 92 86 63 05 4b e9 f0 0d 9e 36 23 11 8a 38 8f M..c.K��..6#..8.
Oct 5 11:05:38 localhost kernel: Object 0xf6362300: e4 4d 26 4b 64 51 53 05 c5 f9 01 8c 34 95 71 76 �M&KdQS.��..4.qv
Oct 5 11:05:38 localhost kernel: Object 0xf6362310: 2d 57 a8 3e ef 8c 98 5c 36 d1 04 30 91 cf 51 5f -WỨ>�..\6�.0.�Q_
Oct 5 11:05:38 localhost kernel: Object 0xf6362320: b9 f4 ab b5 3a dc a7 e6 ae 98 23 21 80 50 c6 70 ��﹫�:�ܧ��.#!.P�p
Oct 5 11:05:38 localhost kernel: Object 0xf6362330: 1c c7 14 65 af 48 54 53 76 6f bd 02 e4 b0 13 2a .�.e殯HTSvo�.��.*
Oct 5 11:05:38 localhost kernel: Object 0xf6362340: c6 37 eb 50 bc 20 35 d1 e0 6c 50 9c 1b d2 0e ed �7�P䰼.5��lP..�.
Oct 5 11:05:38 localhost kernel: Object 0xf6362350: 8f d1 b9 9f b6 7c 9c 5f ac f3 86 c1 b6 c0 31 86 .�ѹ.�|._��.���1.
Oct 5 11:05:38 localhost kernel: Object 0xf6362360: a7 e6 c2 e4 78 f6 20 41 71 16 f8 7e bf 1d 47 6e ����x�.Aq.�~????.Gn
Oct 5 11:05:38 localhost kernel: Object 0xf6362370: 82 b9 df 67 d3 b2 fe f3 08 b0 e4 b4 53 b8 23 d8 .��g�߲��.���S????#
Oct 5 11:05:38 localhost kernel: Object 0xf6362380: ad 46 3e 7b f4 6e 1d 00 de f7 c6 a9 9a 2f 4f 75 حF>{�n..��Ʃ./Ou
Oct 5 11:05:38 localhost kernel: Object 0xf6362390: a7 e3 99 8e b6 a5 f2 0d 73 ac 32 47 96 c1 cf 0b ��..������.s�2G.��.
Oct 5 11:05:38 localhost kernel: Object 0xf63623a0: 79 80 7e cd ac 3d 42 41 32 af 62 79 3d f8 98 ce y.~ͬ=BA2????by=�.
Oct 5 11:05:38 localhost kernel: Object 0xf63623b0: ab df d0 0c 24 36 8c 7c 66 78 18 c1 25 2a 06 95 ���.$6.|fx.�%*..
Oct 5 11:05:38 localhost kernel: Object 0xf63623c0: ce 9c ae 2c 84 71 3e 4c 70 4c d3 f2 f4 cc 5b b9 �.�,.q>LpL����[�
Oct 5 11:05:38 localhost kernel: Object 0xf63623d0: 42 a0 a7 73 a1 73 1b b8 5c 7d 3c 76 b9 78 bd 6b B.�����s�s.�\}<v�x�k
Oct 5 11:05:38 localhost kernel: Object 0xf63623e0: 30 ca b2 11 1d b3 30 6b 0e 5a f6 53 64 99 24 58 0�ʲ..�0k.Z�Sd.$X
Oct 5 11:05:38 localhost kernel: Object 0xf63623f0: a8 7a 47 e0 2e 1b cf b4 49 ea 3d 77 90 6f f2 87 �zG�..ϴI�=w.o�.
Oct 5 11:05:38 localhost kernel: Object 0xf6362400: e2 36 0f 51 68 9d d3 70 84 bf 7e e6 c9 12 47 29 �6.Qh.�p.����~��.G)
Oct 5 11:05:38 localhost kernel: Object 0xf6362410: 3b fb f2 32 34 58 fd 6d 2a a8 9d 47 bb b4 90 c2 ;��24X�m*�.G樻�.
Oct 5 11:05:38 localhost kernel: Object 0xf6362420: 8b a5 fc 42 d7 a7 e3 44 9c 36 37 0b b5 51 4b cc .¥�Bק�D.67.�QK
Oct 5 11:05:38 localhost kernel: Object 0xf6362430: 2a eb 55 e2 fe 98 cb 4f 34 29 b8 18 59 f9 af c1 *�U��.�O4)�.Y��
Oct 5 11:05:38 localhost kernel: Object 0xf6362440: e7 a0 3a e8 1e 01 37 80 4d 6d 9c 37 e6 8f ae 82 �.:�..7.Mm.7�.������.
Oct 5 11:05:38 localhost kernel: Object 0xf6362450: 59 5d e4 99 3f 6e f3 cf d6 26 c5 21 1b aa ce fd Y]�.?n���&�!.��
Oct 5 11:05:38 localhost kernel: Object 0xf6362460: 01 d3 59 20 38 99 5c 85 95 de d8 01 14 1a 8f 65 .�Y.8.\..��....e
Oct 5 11:05:38 localhost kernel: Object 0xf6362470: dc 54 2a f6 cd 3b 9a 54 73 0d 34 1f 73 19 ac 5c �T*��;.Ts.4.s.䪬\
Oct 5 11:05:38 localhost kernel: Object 0xf6362480: af 38 9c 4d 2a a3 7b ec b3 1a 21 c3 fe 42 4e 8f �8.M*�{��.!��BN.
Oct 5 11:05:38 localhost kernel: Object 0xf6362490: 03 a1 14 38 f2 bf a3 da 60 69 08 42 85 9e 8c 01 .쳡.8����`i.B....
Oct 5 11:05:38 localhost kernel: Object 0xf63624a0: 18 e4 b7 6e a3 f6 b9 11 bb ee 0d 07 0d 2a f2 7c .�????n���.��...*�|
Oct 5 11:05:38 localhost kernel: Object 0xf63624b0: 9c 13 2c 56 56 d5 05 8d 2a e3 45 83 26 09 50 3d ..,VV�..*�E.&.P=
Oct 5 11:05:38 localhost kernel: Object 0xf63624c0: 42 7c 49 5a bd ac 2d ca 49 90 27 2c e7 42 ef e1 B|IZ�����-�I.',�B�
Oct 5 11:05:38 localhost kernel: Object 0xf63624d0: 42 2f fa 99 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b B/�.kkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf63624e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf63624f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362500: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362510: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362520: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362530: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362540: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362550: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362560: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362570: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362580: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362590: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf63625a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf63625b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf63625c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf63625d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf63625e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf63625f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362600: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362610: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362620: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362630: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362640: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362650: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362660: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362670: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362680: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362690: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf63626a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf63626b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf63626c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf63626d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf63626e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf63626f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362700: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362710: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362720: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362730: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362740: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362750: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362760: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362770: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362780: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362790: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf63627a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf63627b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf63627c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf63627d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf63627e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf63627f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362800: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362810: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362820: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362830: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362840: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362850: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362860: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362870: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362880: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362890: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf63628a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf63628b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf63628c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf63628d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf63628e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf63628f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362900: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362910: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362920: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362930: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362940: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362950: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362960: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362970: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362980: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362990: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf63629a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf63629b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf63629c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf63629d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf63629e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf63629f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362a00: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362a10: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362a20: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362a30: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362a40: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362a50: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362a60: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362a70: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362a80: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362a90: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362aa0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362ab0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362ac0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362ad0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362ae0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362af0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362b00: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362b10: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362b20: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362b30: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362b40: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362b50: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362b60: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362b70: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362b80: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362b90: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362ba0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362bb0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362bc0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362bd0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362be0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362bf0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362c00: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362c10: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362c20: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362c30: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362c40: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362c50: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362c60: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362c70: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362c80: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362c90: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362ca0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362cb0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362cc0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362cd0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362ce0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362cf0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362d00: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362d10: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362d20: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362d30: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362d40: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362d50: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362d60: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362d70: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362d80: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362d90: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362da0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362db0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362dc0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362dd0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362de0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362df0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362e00: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362e10: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362e20: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362e30: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362e40: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362e50: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362e60: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362e70: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362e80: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362e90: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362ea0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362eb0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362ec0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362ed0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362ee0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362ef0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362f00: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362f10: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362f20: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362f30: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362f40: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362f50: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362f60: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362f70: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362f80: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362f90: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362fa0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362fb0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362fc0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362fd0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362fe0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6362ff0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6363000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6363010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Object 0xf6363020: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:38 localhost kernel: Redzone 0xf6364030: bb bb bb bb ʻ���
Oct 5 11:05:38 localhost kernel: Padding 0xf6364058: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ
Oct 5 11:05:38 localhost kernel: Pid: 8416, comm: parse-iwconfig. Not tainted 2.6.36-rc6-wl+ #5
Oct 5 11:05:38 localhost kernel: Call Trace:
Oct 5 11:05:38 localhost kernel: [<c04ab513>] print_trailer+0xdc/0xe4
Oct 5 11:05:38 localhost kernel: [<c04ab8e9>] check_bytes_and_report+0x96/0xcc
Oct 5 11:05:38 localhost kernel: [<c04ab9c6>] check_object+0xa7/0x15d
Oct 5 11:05:38 localhost kernel: [<c04acc37>] __slab_alloc+0x339/0x3dd
Oct 5 11:05:38 localhost kernel: [<c0455df5>] ? mark_held_locks+0x47/0x5f
Oct 5 11:05:38 localhost kernel: [<c04ad0d2>] ? kmem_cache_alloc+0xa0/0xc5
Oct 5 11:05:38 localhost kernel: [<c04ad6fd>] __kmalloc_track_caller+0xa1/0xf2
Oct 5 11:05:38 localhost kernel: [<c06c7331>] ? skb_copy+0x2e/0x83
Oct 5 11:05:38 localhost kernel: [<c06c7331>] ? skb_copy+0x2e/0x83
Oct 5 11:05:38 localhost kernel: [<c06c6d6a>] __alloc_skb+0x58/0xf4
Oct 5 11:05:38 localhost kernel: [<c06c7331>] skb_copy+0x2e/0x83
Oct 5 11:05:38 localhost kernel: [<f8d426dd>] ieee80211_prepare_and_rx_handle+0x3b1/0x86d [mac80211]
Oct 5 11:05:38 localhost kernel: [<c0455bee>] ? mark_lock+0x1e/0x1de
Oct 5 11:05:38 localhost kernel: [<c0455bee>] ? mark_lock+0x1e/0x1de
Oct 5 11:05:38 localhost kernel: [<f8d31ab4>] ? sta_info_get_bss+0xb0/0x10c [mac80211]
Oct 5 11:05:38 localhost kernel: [<f8d43352>] ieee80211_rx+0x70f/0x7b5 [mac80211]
Oct 5 11:05:38 localhost kernel: [<c04ad729>] ? __kmalloc_track_caller+0xcd/0xf2
Oct 5 11:05:38 localhost kernel: [<c0456038>] ? trace_hardirqs_on_caller+0xeb/0x125
Oct 5 11:05:38 localhost kernel: [<f86e3836>] ath_rx_send_to_mac80211+0x5a/0x60 [ath9k]
Oct 5 11:05:38 localhost kernel: [<c045607d>] ? trace_hardirqs_on+0xb/0xd
Oct 5 11:05:38 localhost kernel: [<f86e5066>] ath_rx_tasklet+0x1253/0x130e [ath9k]
Oct 5 11:05:38 localhost kernel: [<f86e3232>] ath9k_tasklet+0x9a/0x12a [ath9k]
Oct 5 11:05:38 localhost kernel: [<c0438fd1>] tasklet_action+0x73/0xc6
Oct 5 11:05:38 localhost kernel: [<c043945f>] __do_softirq+0x86/0x111
Oct 5 11:05:38 localhost kernel: [<c0439520>] do_softirq+0x36/0x5a
Oct 5 11:05:38 localhost kernel: [<c0439659>] irq_exit+0x35/0x69
Oct 5 11:05:38 localhost kernel: [<c0403fb9>] do_IRQ+0x86/0x9a
Oct 5 11:05:38 localhost kernel: [<c04034ee>] common_interrupt+0x2e/0x40
Oct 5 11:05:38 localhost kernel: FIX kmalloc-8192: Restoring 0xf6362100-0xf63624d3=0x6b
Oct 5 11:05:38 localhost kernel:
Oct 5 11:05:38 localhost kernel: FIX kmalloc-8192: Marking all objects used
Oct 5 11:05:39 localhost kernel: sta5: deauthenticating from 00:14:d1:c6:d2:54 by local choice (reason=3)
Oct 5 11:05:39 localhost kernel: ieee80211 phy0: Removed STA 00:14:d1:c6:d2:54
Oct 5 11:05:39 localhost kernel: ieee80211 phy0: Destroyed STA 00:14:d1:c6:d2:54
Oct 5 11:05:41 localhost kernel: sta6: no IPv6 routers present
Oct 5 11:05:41 localhost kernel: =============================================================================
Oct 5 11:05:41 localhost kernel: BUG kmalloc-8192: Poison overwritten
Oct 5 11:05:41 localhost kernel: -----------------------------------------------------------------------------
Oct 5 11:05:41 localhost kernel:
Oct 5 11:05:41 localhost kernel: INFO: 0xf34d40a0-0xf34d411f. First byte 0x80 instead of 0x6b
Oct 5 11:05:41 localhost kernel: INFO: Allocated in ath_rxbuf_alloc+0x1d/0x74 [ath] age=9852 cpu=1 pid=0
Oct 5 11:05:41 localhost kernel: INFO: Freed in skb_release_data+0x8c/0x90 age=66 cpu=1 pid=0
Oct 5 11:05:41 localhost kernel: INFO: Slab 0xc1fa2a00 objects=3 used=2 fp=0xf34d4060 flags=0x400040c1
Oct 5 11:05:41 localhost kernel: INFO: Object 0xf34d4060 @offset=16480 fp=0x(null)
Oct 5 11:05:41 localhost kernel:
Oct 5 11:05:41 localhost kernel: Bytes b4 0xf34d4050: 00 00 00 00 fe 02 0c 00 5a 5a 5a 5a 5a 5a 5a 5a ....�...ZZZZZZZZ
Oct 5 11:05:41 localhost kernel: Object 0xf34d4060: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4070: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4080: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4090: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d40a0: 80 00 00 00 ff ff ff ff ff ff 00 1d 7e 23 46 52 ....������..~#FR
Oct 5 11:05:41 localhost kernel: Object 0xf34d40b0: 00 1d 7e 23 46 52 00 ad 87 a1 f9 6a 90 06 00 00 ..~#FR.�.��j....
Oct 5 11:05:41 localhost kernel: Object 0xf34d40c0: 64 00 11 04 00 0f 46 65 72 6e 64 61 6c 65 43 68 d.....FerndaleCh
Oct 5 11:05:41 localhost kernel: Object 0xf34d40d0: 61 6d 62 65 72 01 08 82 84 8b 96 24 30 48 6c 03 amber......$0Hl.
Oct 5 11:05:41 localhost kernel: Object 0xf34d40e0: 01 06 05 04 00 01 00 00 2a 01 04 2f 01 04 32 04 ........*../..2.
Oct 5 11:05:41 localhost kernel: Object 0xf34d40f0: 0c 12 18 60 dd 09 00 10 18 02 01 f4 00 00 00 dd ...`�......�...
Oct 5 11:05:41 localhost kernel: Object 0xf34d4100: 18 00 50 f2 01 01 00 00 50 f2 02 01 00 00 50 f2 ..P�....P�....P
Oct 5 11:05:41 localhost kernel: Object 0xf34d4110: 02 01 00 00 50 f2 02 00 00 51 f0 2b 8f 51 f0 8f ....P�...Q�+.Q�.
Oct 5 11:05:41 localhost kernel: Object 0xf34d4120: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4130: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4140: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4150: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4160: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4170: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4180: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4190: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d41a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d41b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d41c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d41d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d41e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d41f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4200: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4210: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4220: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4230: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4240: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4250: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4260: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4270: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4280: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4290: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d42a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d42b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d42c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d42d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d42e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d42f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4300: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4310: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4320: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4330: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4340: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4350: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4360: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4370: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4380: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4390: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d43a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d43b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d43c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d43d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d43e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d43f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4400: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4410: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4420: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4430: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4440: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4450: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4460: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4470: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4480: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4490: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d44a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d44b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d44c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d44d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d44e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d44f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4500: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4510: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4520: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4530: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4540: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4550: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4560: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4570: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4580: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4590: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d45a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d45b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d45c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d45d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d45e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d45f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4600: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4610: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4620: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4630: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4640: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4650: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4660: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4670: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4680: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4690: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d46a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d46b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d46c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d46d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d46e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d46f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4700: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4710: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4720: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4730: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4740: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4750: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4760: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4770: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4780: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4790: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d47a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d47b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d47c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d47d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d47e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d47f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4800: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4810: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4820: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4830: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4840: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4850: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4860: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4870: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4880: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4890: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d48a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d48b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d48c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d48d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d48e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d48f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4900: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4910: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4920: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4930: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4940: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4950: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4960: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4970: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4980: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4990: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d49a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d49b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d49c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d49d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d49e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d49f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4a00: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4a10: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4a20: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4a30: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4a40: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4a50: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4a60: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4a70: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4a80: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4a90: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4aa0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4ab0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4ac0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4ad0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4ae0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4af0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4b00: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4b10: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4b20: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4b30: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4b40: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4b50: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4b60: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4b70: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4b80: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4b90: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4ba0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4bb0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4bc0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4bd0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4be0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4bf0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4c00: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4c10: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4c20: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4c30: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4c40: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4c50: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4c60: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4c70: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4c80: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4c90: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4ca0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4cb0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4cc0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4cd0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4ce0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4cf0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4d00: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4d10: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4d20: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4d30: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4d40: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4d50: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4d60: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4d70: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4d80: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4d90: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4da0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4db0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4dc0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4dd0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4de0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4df0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4e00: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4e10: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4e20: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4e30: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4e40: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4e50: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4e60: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4e70: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4e80: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4e90: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4ea0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4eb0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4ec0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4ed0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4ee0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4ef0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4f00: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4f10: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4f20: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4f30: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4f40: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4f50: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4f60: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4f70: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4f80: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4f90: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4fa0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4fb0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4fc0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4fd0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4fe0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d4ff0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d5000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d5010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d5020: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d5030: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d5040: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Object 0xf34d5050: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Oct 5 11:05:41 localhost kernel: Redzone 0xf34d6060: bb bb bb bb ��������
Oct 5 11:05:41 localhost kernel: Padding 0xf34d6088: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ
Oct 5 11:05:41 localhost kernel: Pid: 0, comm: kworker/0:0 Not tainted 2.6.36-rc6-wl+ #5
Oct 5 11:05:41 localhost kernel: Call Trace:
Oct 5 11:05:41 localhost kernel: [<c04ab513>] print_trailer+0xdc/0xe4
Oct 5 11:05:41 localhost kernel: [<c04ab8e9>] check_bytes_and_report+0x96/0xcc
Oct 5 11:05:41 localhost kernel: [<c04ab9c6>] check_object+0xa7/0x15d
Oct 5 11:05:41 localhost kernel: [<c04acc37>] __slab_alloc+0x339/0x3dd
Oct 5 11:05:41 localhost kernel: [<c0455df5>] ? mark_held_locks+0x47/0x5f
Oct 5 11:05:41 localhost kernel: [<c04ad0d2>] ? kmem_cache_alloc+0xa0/0xc5
Oct 5 11:05:41 localhost kernel: [<c04ad6fd>] __kmalloc_track_caller+0xa1/0xf2
Oct 5 11:05:41 localhost kernel: [<c06c7331>] ? skb_copy+0x2e/0x83
Oct 5 11:05:41 localhost kernel: [<c06c7331>] ? skb_copy+0x2e/0x83
Oct 5 11:05:41 localhost kernel: [<c06c6d6a>] __alloc_skb+0x58/0xf4
Oct 5 11:05:41 localhost kernel: [<c06c7331>] skb_copy+0x2e/0x83
Oct 5 11:05:41 localhost kernel: [<f8d426dd>] ieee80211_prepare_and_rx_handle+0x3b1/0x86d [mac80211]
Oct 5 11:05:41 localhost kernel: [<c0455bee>] ? mark_lock+0x1e/0x1de
Oct 5 11:05:41 localhost kernel: [<f8d31ab4>] ? sta_info_get_bss+0xb0/0x10c [mac80211]
Oct 5 11:05:41 localhost kernel: [<f8d43352>] ieee80211_rx+0x70f/0x7b5 [mac80211]
Oct 5 11:05:41 localhost kernel: [<c04ad729>] ? __kmalloc_track_caller+0xcd/0xf2
Oct 5 11:05:41 localhost kernel: [<c0456038>] ? trace_hardirqs_on_caller+0xeb/0x125
Oct 5 11:05:41 localhost kernel: [<f86e3836>] ath_rx_send_to_mac80211+0x5a/0x60 [ath9k]
Oct 5 11:05:41 localhost kernel: [<c045607d>] ? trace_hardirqs_on+0xb/0xd
Oct 5 11:05:41 localhost kernel: [<f86e5066>] ath_rx_tasklet+0x1253/0x130e [ath9k]
Oct 5 11:05:41 localhost kernel: [<f86e3232>] ath9k_tasklet+0x9a/0x12a [ath9k]
Oct 5 11:05:41 localhost kernel: [<c0438fd1>] tasklet_action+0x73/0xc6
Oct 5 11:05:41 localhost kernel: [<c043945f>] __do_softirq+0x86/0x111
Oct 5 11:05:41 localhost kernel: [<c0439520>] do_softirq+0x36/0x5a
Oct 5 11:05:41 localhost kernel: [<c0439659>] irq_exit+0x35/0x69
Oct 5 11:05:41 localhost kernel: [<c0403fb9>] do_IRQ+0x86/0x9a
Oct 5 11:05:41 localhost kernel: [<c04034ee>] common_interrupt+0x2e/0x40
Oct 5 11:05:41 localhost kernel: [<c045007b>] ? do_adjtimex+0x217/0x55e
Oct 5 11:05:41 localhost kernel: [<c0408245>] ? mwait_idle+0x5c/0x6c
Oct 5 11:05:41 localhost kernel: [<c040227f>] cpu_idle+0x4e/0x6b
Oct 5 11:05:41 localhost kernel: [<c075b591>] start_secondary+0x1a8/0x1ad
Oct 5 11:05:41 localhost kernel: FIX kmalloc-8192: Restoring 0xf34d40a0-0xf34d411f=0x6b
Thanks,
Ben
--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
On 10/05/2010 10:36 AM, Luis R. Rodriguez wrote:
> On Tue, Oct 5, 2010 at 10:24 AM, Ben Greear<[email protected]> wrote:
>> On 10/05/2010 10:16 AM, Luis R. Rodriguez wrote:
>>>
>>> On Tue, Oct 5, 2010 at 10:00 AM, Ben Greear<[email protected]>
>>> wrote:
>>>>
>>>> I started seeing this very soon after creating interfaces.
>>>
>>> Can you be more specific how one can reproduce?
>>
>> Enable SLUB debugging, DEBUG_PAGEALLOC (Debug page memory allocations),
>> lockdep, pre-empt.
>
> OK I just enabled PAGE_ALLOC thingy, it wasn't enabled on my kernel.
>
>> Please try creating a bunch of stations on one ath9k phy,
>
> How many?
130 is a nice round number :)
I'll try with a smaller number and see if I can hit it.
Thanks,
Ben
>
> Luis
--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
On Mon, Oct 11, 2010 at 8:27 PM, Ben Greear <[email protected]> wrote:
> On 10/11/2010 06:03 PM, Luis R. Rodriguez wrote:
>>
>> On Mon, Oct 11, 2010 at 1:51 PM, Ben Greear<[email protected]>
>> wrote:
>>>
>>> On 10/07/2010 02:59 PM, Luis R. Rodriguez wrote:
>>>>
>>>> On Thu, Oct 7, 2010 at 2:36 PM, Luis R. Rodriguez<[email protected]>
>>>> wrote:
>>>
>>>>>> But other than this I got nothing. I left the box sit there for about
>>>>>> 1 hour and came back and it was still going with no issues. Mind you,
>>>>>> I can't ping but that seems like another issue.
>>>>>>
>>>>>> You can find my logs here:
>>>>>>
>>>>>>
>>>>>>
>>>>>> http://www.kernel.org/pub/linux/kernel/people/mcgrof/logs/2010/10-07-stress-sta-01/
>>>>>
>>>>> Doh, I did not have CONFIG_SLUB_DEBUG_ON=y so building now with that.
>>>>
>>>> Yay I can reproduce now. I'll be back, going to dig now.
>>>
>>> Any luck tracking this down?
>>
>> No, today for example I just finished reading e-mail and its already
>> 6pm PST... But Friday I did get do do a lot of work and testing on
>> this. The only pattern I see so far is that skb_copy() is used on the
>> poison all the time. I am not sure if its because skb_copy() happens
>> to run the poison check or what. I'll work on this tomorrow.
>
> I know how that goes.
>
> Do you happen to have any magic tools that could be instrumented to show
> when DMA was happening in the chip, and to see if it somehow happens to dma
> to something after it is supposedly un-mapped?
Um, not sure, I'd have to dig. But I was looking at this as an idea to
borrow to test if its a DMA issue:
https://patchwork.kernel.org/patch/22127/
However right now I'm thinking this is simply a free and then a race
to try to use the free'd buffer.
> Another thing I was thinking about: Maybe the queue of skbs and dma
> addresses
> in ath9k is getting corrupted by multiple VIFs trying to write at once?
> Maybe
> some locking is needed in the xmit path?
That was my second hunch. My first shot was to use spin_lock_irqsave()
over the the uses of the rxbuf list and that seemed to help but I
still managed to get a poison eventually. My next item to check for is
of the permissibility of creating too much pressure to the point we
end up looping over the rxbuf list and race against mac80211 free'ing
a buffer. Will test that tomorrow if nothing else comes up creeping my
priority queue.
Luis
On 10/12/2010 11:43 AM, Ben Greear wrote:
> On 10/12/2010 11:40 AM, Luis R. Rodriguez wrote:
>> On Tue, Oct 12, 2010 at 11:35 AM, Ben Greear<[email protected]>
>> wrote:
>>> This code looks weird to me. One of the paprd branches
>>> deletes the skb, the other doesn't appear to. Neither
>>> null out bf->bf_mpdu, which would appear to leave a dangling
>>> pointer in at least the dev_kfree_skb_any() branch.
>>>
>>> ath_tx_complete frees it's skb in all cases, so another
>>> bf->bf_mpdu dangling pointer issue.
>>>
>>> Maybe at the least we should null out bf->bf_mpdu when
>>> skb is consumed?
>>
>> You're reading my mind, that was what I was going to test today. Still
>> doing e-mail sweep though.
>
> I'm trying to put together a patch now..but the paprd branch makes
> no sense at all.
>
> Shouldn't it also free the skb in the complete(&sc->paprd_complete) branch?
> I don't see anything that uses the skb, and nothing that frees it.
>
> if (bf->bf_state.bfs_paprd) {
> if (time_after(jiffies,
> bf->bf_state.bfs_paprd_timestamp +
> msecs_to_jiffies(ATH_PAPRD_TIMEOUT)))
> dev_kfree_skb_any(skb);
> else
> complete(&sc->paprd_complete);
I tried this patch, but the memory poision warnings continue.
Might still be a useful patch though...
[greearb@build-32 mac80211]$ git diff
diff --git a/drivers/net/wireless/ath/ath9k/main.c b/drivers/net/wireless/ath/ath9k/main.c
index 8656491..f43a004 100644
--- a/drivers/net/wireless/ath/ath9k/main.c
+++ b/drivers/net/wireless/ath/ath9k/main.c
@@ -398,7 +398,7 @@ void ath_paprd_calibrate(struct work_struct *work)
"Timeout waiting for paprd training on "
"TX chain %d\n",
chain);
- goto fail_paprd;
+ break;
}
if (!ar9003_paprd_is_done(ah))
@@ -416,7 +416,6 @@ void ath_paprd_calibrate(struct work_struct *work)
ath_paprd_activate(sc);
}
-fail_paprd:
ath9k_ps_restore(sc);
}
diff --git a/drivers/net/wireless/ath/ath9k/xmit.c b/drivers/net/wireless/ath/ath9k/xmit.c
index 942be55..d39b4b5 100644
--- a/drivers/net/wireless/ath/ath9k/xmit.c
+++ b/drivers/net/wireless/ath/ath9k/xmit.c
@@ -1919,17 +1919,17 @@ static void ath_tx_complete_buf(struct ath_softc *sc, struct ath_buf *bf,
dma_unmap_single(sc->dev, bf->bf_dmacontext, skb->len, DMA_TO_DEVICE);
if (bf->bf_state.bfs_paprd) {
- if (time_after(jiffies,
- bf->bf_state.bfs_paprd_timestamp +
- msecs_to_jiffies(ATH_PAPRD_TIMEOUT)))
- dev_kfree_skb_any(skb);
- else
- complete(&sc->paprd_complete);
+ /* ath_paprd_calibrate owns the skb. */
+ complete(&sc->paprd_complete);
} else {
- ath_tx_complete(sc, skb, bf->aphy, tx_flags);
+ /* stat_tx must be called first, it references skb. */
ath_debug_stat_tx(sc, txq, bf, ts);
+ ath_tx_complete(sc, skb, bf->aphy, tx_flags);
}
+ /* At this point, skb is consumed one way or another */
+ bf->bf_mpdu = NULL;
+
/*
* Return the list of ath_buf of this mpdu to free queue
*/
Thanks,
Ben
--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
On Wed, Oct 13, 2010 at 10:48 AM, Ben Greear <[email protected]> wrote:
> On 10/13/2010 10:29 AM, Luis R. Rodriguez wrote:
>>
>> On Wed, Oct 13, 2010 at 10:12 AM, Ben Greear<[email protected]>
>> wrote:
>>>
>>> On 10/12/2010 11:40 AM, Luis R. Rodriguez wrote:
>>>>
>>>> On Tue, Oct 12, 2010 at 11:35 AM, Ben Greear<[email protected]>
>>>> wrote:
>>>>>
>>>>> On 10/11/2010 11:10 PM, Luis R. Rodriguez wrote:
>>>>>>
>>>>>> On Mon, Oct 11, 2010 at 8:27 PM, Ben Greear<[email protected]>
>>>>>> wrote:
>>>>>
>>>>>>> Another thing I was thinking about: Maybe the queue of skbs and dma
>>>>>>> addresses
>>>>>>> in ath9k is getting corrupted by multiple VIFs trying to write at
>>>>>>> once?
>>>>>>> Maybe
>>>>>>> some locking is needed in the xmit path?
>>>>>>
>>>>>> That was my second hunch. My first shot was to use spin_lock_irqsave()
>>>>>> over the the uses of the rxbuf list and that seemed to help but I
>>>>>> still managed to get a poison eventually. My next item to check for is
>>>>>> of the permissibility of creating too much pressure to the point we
>>>>>> end up looping over the rxbuf list and race against mac80211 free'ing
>>>>>> a buffer. Will test that tomorrow if nothing else comes up creeping my
>>>>>> priority queue.
>>>>>
>>>>> This code looks weird to me. One of the paprd branches
>>>>> deletes the skb, the other doesn't appear to. Neither
>>>>> null out bf->bf_mpdu, which would appear to leave a dangling
>>>>> pointer in at least the dev_kfree_skb_any() branch.
>>>>>
>>>>> ath_tx_complete frees it's skb in all cases, so another
>>>>> bf->bf_mpdu dangling pointer issue.
>>>>>
>>>>> Maybe at the least we should null out bf->bf_mpdu when
>>>>> skb is consumed?
>>>>
>>>> You're reading my mind, that was what I was going to test today. Still
>>>> doing e-mail sweep though.
>>>
>>> At least in the xmit path, it seems cards that have EDMA support do
>>> things a bit different. Out of curiosity, on the system(s), you
>>> reproduce
>>> this, are any of yours supporting EDMA? Mine appear to not support EDMA.
>>
>> EDMA is used on>= AR9003 families by Atheros. And no, I am not
>> testing with an EDMA card, I am testing with an AR9002 family card,
>> the AR9280 card. I am going to disregard the TX stuff as the bug is an
>> RX issue :) I was able to more easily reproduce by doing an skb_copy()
>> and free'ing the buffer right afterwards on the ath_send_to_mac80211()
>> thingy, So it does appear that the poison check just happens more
>> often when we do an skb_copy(). One reason this is easy to reproduce
>> with multiple STAs is mac80211 uses skb_copy() to process each
>> received skb for each STA.
>>
>> In my tests so far, protecting the rxbuf list with spin_lock_irqsave()
>> did not help, and the wmb(); didn't either, something else is going on
>> here. It would be nice to hack slab to keep an entire trace of the
>> place the buffer was last free'd at instead of just the caller that
>> freed it.
>
> I instrumented slub a while back and got the backtrace. It
> was always in the same place for my testing.
>
> Here's the slub patch if you are interested in using it yourself:
> https://patchwork.kernel.org/patch/236921/
when compiling this patch I get:
arch/x86/built-in.o: In function `store_stack':
/home/mcgrof/wireless-testing/arch/x86/kernel/dumpstack.c:259:
undefined reference to `store_trace'
Luis
In case it helps, here is a dump of where the corrupted SKB was deleted.
I added debugging to slub to get this information, but it looks like
it's correct to me.
Reading symbols from /home/greearb/kernel/2.6/wireless-testing-dbg.p4s/net/mac80211/mac80211.ko...done.
(gdb) l *(ieee80211_rx+0x74d)
0x13751 is in ieee80211_rx (/home/greearb/git/linux.wireless-testing/include/linux/rcupdate.h:346).
341 *
342 * See rcu_read_lock() for more information.
343 */
344 static inline void rcu_read_unlock(void)
345 {
346 rcu_read_release();
347 __release(RCU);
348 __rcu_read_unlock();
349 }
350
(gdb)
# I don't really know what that second address means, but just in case it's useful,
# I printed it out here:
(gdb) l *(ieee80211_rx+0x7b4)
0x137b8 is in ieee80211_process_measurement_req (/home/greearb/git/linux.wireless-testing/net/mac80211/spectmgmt.c:74).
69 }
70
71 void ieee80211_process_measurement_req(struct ieee80211_sub_if_data *sdata,
72 struct ieee80211_mgmt *mgmt,
73 size_t len)
74 {
75 /*
76 * Ignoring measurement request is spec violation.
77 * Mandatory measurements must be reported optional
78 * measurements might be refused or reported incapable
INFO: Freed in skb_release_data+0x8c/0x90 age=122 cpu=1 pid=0
set_track+0x3c/0x89
__slab_free+0x17f/0x1ba
skb_release_data+0x8c/0x90
kfree+0xaf/0xdf
skb_release_data+0x8c/0x90
skb_release_data+0x8c/0x90
skb_release_data+0x8c/0x90
__kfree_skb+0x12/0x6d
consume_skb+0x2a/0x2c
ieee80211_rx+0x74d/0x7b4 [mac80211]
__kmalloc_track_caller+0xcd/0xf2
trace_hardirqs_on_caller+0xeb/0x125
ath_rx_send_to_mac80211+0x5a/0x60 [ath9k]
trace_hardirqs_on+0xb/0xd
Thanks,
Ben
--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
2010/10/15 Ben Greear <[email protected]>:
> I tried the patch below, and it didn't seem to help. ?Might even
> have hurt..as it died on divide-by-zero error:
Hmm, looks like the ani code got a zero listen time from the hw...
That just might mean that the DMA actually hits one of these
descriptors. :)
I'll send a patch for the div-by-zero in a minute, and then perhaps we
can narrow it down to which one of these descriptors is actually being
used by DMA when it really shouldn't be?
/Bj?rn
Ok please try this patch, it cures it for me.
Luis
diff --git a/drivers/net/wireless/ath/ath9k/recv.c b/drivers/net/wireless/ath/ath9k/recv.c
index fe73fc5..e581b1f 100644
--- a/drivers/net/wireless/ath/ath9k/recv.c
+++ b/drivers/net/wireless/ath/ath9k/recv.c
@@ -306,10 +306,8 @@ static void ath_edma_start_recv(struct ath_softc *sc)
static void ath_edma_stop_recv(struct ath_softc *sc)
{
- spin_lock_bh(&sc->rx.rxbuflock);
ath_rx_remove_buffer(sc, ATH9K_RX_QUEUE_HP);
ath_rx_remove_buffer(sc, ATH9K_RX_QUEUE_LP);
- spin_unlock_bh(&sc->rx.rxbuflock);
}
int ath_rx_init(struct ath_softc *sc, int nbufs)
@@ -518,6 +516,7 @@ bool ath_stoprecv(struct ath_softc *sc)
struct ath_hw *ah = sc->sc_ah;
bool stopped;
+ spin_lock_bh(&sc->rx.rxbuflock);
ath9k_hw_stoppcurecv(ah);
ath9k_hw_setrxfilter(ah, 0);
stopped = ath9k_hw_stopdmarecv(ah);
@@ -526,6 +525,7 @@ bool ath_stoprecv(struct ath_softc *sc)
ath_edma_stop_recv(sc);
else
sc->rx.rxlink = NULL;
+ spin_unlock_bh(&sc->rx.rxbuflock);
return stopped;
}
On 10/14/2010 04:39 PM, Luis R. Rodriguez wrote:
> On Thu, Oct 14, 2010 at 4:30 PM, Ben Greear<[email protected]> wrote:
>> On 10/14/2010 04:19 PM, Luis R. Rodriguez wrote:
>>>
>>> Ok please try this patch, it cures it for me.
>>
>> Well, it got a lot further than normal, but it still
>> hit the poison check after a few minutes.
>>
>> Current test case is my app loading 130 or so stations, each running
>> wpa_supplicant. All were created, and quite a few had associated
>> when the poison check hit.
>>
>> So, it definitely looks like a step in the right direction, but
>> not fully fixed yet.
>>
>> I'll do some more testing with this patch applied and using just my
>> perl script to make sure the problem is reproducible outside of my
>> application.
>
> Ok, whatever userspace does it should not corrupt to kernel, unless
> its poking /dev/mem
Sure, but it's nice to have a simpler test case.
I ran my script with 32 stations for maybe 20 minutes...seemed to be doing
OK as far as stability goes..then it crashed in the divide-by-zero ani thing.
I have to run home now, but I'll try my script with more interfaces
and/or some traffic mix, etc, to see if I can get another reasonable way
to reproduce this w/out my big binary application.
I can also work on the divide-by-zero bug if no one beats me to it.
Thanks,
Ben
>
> Luis
--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
On 10/14/2010 04:19 PM, Luis R. Rodriguez wrote:
> Ok please try this patch, it cures it for me.
Well, it got a lot further than normal, but it still
hit the poison check after a few minutes.
Current test case is my app loading 130 or so stations, each running
wpa_supplicant. All were created, and quite a few had associated
when the poison check hit.
So, it definitely looks like a step in the right direction, but
not fully fixed yet.
I'll do some more testing with this patch applied and using just my
perl script to make sure the problem is reproducible outside of my
application.
Thanks,
Ben
>
> Luis
>
> diff --git a/drivers/net/wireless/ath/ath9k/recv.c b/drivers/net/wireless/ath/ath9k/recv.c
> index fe73fc5..e581b1f 100644
> --- a/drivers/net/wireless/ath/ath9k/recv.c
> +++ b/drivers/net/wireless/ath/ath9k/recv.c
> @@ -306,10 +306,8 @@ static void ath_edma_start_recv(struct ath_softc *sc)
>
> static void ath_edma_stop_recv(struct ath_softc *sc)
> {
> - spin_lock_bh(&sc->rx.rxbuflock);
> ath_rx_remove_buffer(sc, ATH9K_RX_QUEUE_HP);
> ath_rx_remove_buffer(sc, ATH9K_RX_QUEUE_LP);
> - spin_unlock_bh(&sc->rx.rxbuflock);
> }
>
> int ath_rx_init(struct ath_softc *sc, int nbufs)
> @@ -518,6 +516,7 @@ bool ath_stoprecv(struct ath_softc *sc)
> struct ath_hw *ah = sc->sc_ah;
> bool stopped;
>
> + spin_lock_bh(&sc->rx.rxbuflock);
> ath9k_hw_stoppcurecv(ah);
> ath9k_hw_setrxfilter(ah, 0);
> stopped = ath9k_hw_stopdmarecv(ah);
> @@ -526,6 +525,7 @@ bool ath_stoprecv(struct ath_softc *sc)
> ath_edma_stop_recv(sc);
> else
> sc->rx.rxlink = NULL;
> + spin_unlock_bh(&sc->rx.rxbuflock);
>
> return stopped;
> }
--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
On Thu, Oct 7, 2010 at 12:27 PM, Johannes Berg
<[email protected]> wrote:
> On Thu, 2010-10-07 at 12:22 -0700, Ben Greear wrote:
>
>> After reboot, and re-run of the script,
>> I saw this in the logs, and shortly after,
>> the SLUB poison warning dumped to screen.
>>
>> Maybe those DMA errors are serious?
>
>> ath: Failed to stop TX DMA in 100 msec after killing last frame
>> ath: Failed to stop TX DMA. Resetting hardware!
>
> That's TX DMA, it can hardly result in invalid memory writes like the
> ones you've been seeing.
>
> I'm still convinced something is wrong with ath9k RX DMA, as you've seen
> the contents of frames written to already freed memory regions. Since I
> don't know anything about ath9k, you should probably not rely on me
> though :-)
I'm on this now. Lets play.
I had to remove /lib/udev/rules.d/75-persistent-net-generator.rules
to avoid Ubuntu trying to remember the device names and it creating
stax_rename names.
I just ran your script with some modifications. You can find it here:
http://www.kernel.org/pub/linux/kernel/people/mcgrof/scripts/poo.pl
I then ran:
for i in $(seq 0 31) ; do sudo dhclient seq$i; done
It took about 10 minutes to get IP addresses for all interfaces but it
got there eventually. Odd enough I was unable to ping the AP from any
interface though. Not sure what that was about. But I got no oops, no
slub dump. All I got was some more delba warnings which seems to
indicate we haven't caught all the cases for its use:
[ 3622.660344] addBA response timer expired on tid 0
[ 3622.660373] Tx BA session stop requested for 68:7f:74:3b:b1:0f tid 0
[ 3622.680133] addBA response timer expired on tid 0
[ 3622.687196] Tx BA session stop requested for 68:7f:74:3b:b1:0f tid 0
[ 3623.110077] addBA response timer expired on tid 0
[ 3623.110123] Tx BA session stop requested for 68:7f:74:3b:b1:0f tid 0
[ 3628.935895] sta10: authenticate with 68:7f:74:3b:b1:10 (try 1)
[ 3628.937194] switched off addBA timer for tid 0
[ 3628.937196] Aggregation is on for tid 0
[ 3628.937239] Stopping Tx BA session for 68:7f:74:3b:b1:0f tid 0
[ 3628.937243] ------------[ cut here ]------------
[ 3628.937263] WARNING: at include/net/mac80211.h:2694
rate_control_send_low+0xd3/0x140 [mac80211]()
[ 3628.937265] Hardware name: 6460DWU
[ 3628.937266] Modules linked in: binfmt_misc ppdev
snd_hda_codec_analog rfcomm sco bridge joydev stp bnep l2cap nouveau
ath9k snd_hda_intel mac80211 snd_hda_codec snd_hwdep snd_pcm ttm btusb
ath9k_common thinkpad_acpi ath9k_hw bluetooth drm_kms_helper
snd_seq_midi snd_rawmidi pcmcia snd_seq_midi_event drm snd_seq ath
snd_timer snd_seq_device tpm_tis i2c_algo_bit cfg80211 snd nvram tpm
tpm_bios yenta_socket pcmcia_rsrc video psmouse output pcmcia_core
serio_raw soundcore snd_page_alloc intel_agp lp parport ohci1394
e1000e ieee1394 ahci libahci
[ 3628.937307] Pid: 49, comm: kworker/u:3 Tainted: G W
2.6.36-rc6-wl+ #263
[ 3628.937310] Call Trace:
[ 3628.937317] [<ffffffff8105ffcf>] warn_slowpath_common+0x7f/0xc0
[ 3628.937320] [<ffffffff8106002a>] warn_slowpath_null+0x1a/0x20
[ 3628.937329] [<ffffffffa032f603>] rate_control_send_low+0xd3/0x140 [mac80211]
[ 3628.937336] [<ffffffffa038bfd8>] ath_get_rate+0x48/0x570 [ath9k]
[ 3628.937340] [<ffffffff812b9f39>] ? put_dec+0x59/0x60
[ 3628.937349] [<ffffffffa032f42e>] rate_control_get_rate+0x8e/0x190 [mac80211]
[ 3628.937360] [<ffffffffa0339928>]
ieee80211_tx_h_rate_ctrl+0x1a8/0x4e0 [mac80211]
[ 3628.937370] [<ffffffffa033a000>] invoke_tx_handlers+0x100/0x140 [mac80211]
[ 3628.937379] [<ffffffffa033a0c5>] ieee80211_tx+0x85/0x240 [mac80211]
[ 3628.937384] [<ffffffff8147b890>] ? skb_release_data+0xd0/0xe0
[ 3628.937386] [<ffffffff8147d72f>] ? pskb_expand_head+0x10f/0x1a0
[ 3628.937397] [<ffffffffa033a336>] ieee80211_xmit+0xb6/0x1d0 [mac80211]
[ 3628.937399] [<ffffffff8147b9d3>] ? __alloc_skb+0x83/0x170
[ 3628.937409] [<ffffffffa033a4a4>] ieee80211_tx_skb+0x54/0x70 [mac80211]
[ 3628.937418] [<ffffffffa03230ad>] ieee80211_send_delba+0x11d/0x190 [mac80211]
[ 3628.937427] [<ffffffffa0323a18>]
ieee80211_stop_tx_ba_cb+0x1b8/0x240 [mac80211]
[ 3628.937431] [<ffffffff81036c89>] ? default_spin_lock_flags+0x9/0x10
[ 3628.937440] [<ffffffffa032e031>] ieee80211_iface_work+0x271/0x340 [mac80211]
[ 3628.937450] [<ffffffffa032ddc0>] ? ieee80211_iface_work+0x0/0x340 [mac80211]
[ 3628.937453] [<ffffffff8107a203>] process_one_work+0x123/0x440
[ 3628.937457] [<ffffffff8107c750>] worker_thread+0x170/0x400
[ 3628.937460] [<ffffffff8107c5e0>] ? worker_thread+0x0/0x400
[ 3628.937463] [<ffffffff81080b76>] kthread+0x96/0xa0
[ 3628.937466] [<ffffffff8100bea4>] kernel_thread_helper+0x4/0x10
[ 3628.937469] [<ffffffff81080ae0>] ? kthread+0x0/0xa0
[ 3628.937472] [<ffffffff8100bea0>] ? kernel_thread_helper+0x0/0x10
[ 3628.937474] ---[ end trace 9dd0d025ccb9b75c ]---
[ 3628.937980] switched off addBA timer for tid 0
[ 3628.937982] Aggregation is on for tid 0
But other than this I got nothing. I left the box sit there for about
1 hour and came back and it was still going with no issues. Mind you,
I can't ping but that seems like another issue.
You can find my logs here:
http://www.kernel.org/pub/linux/kernel/people/mcgrof/logs/2010/10-07-stress-sta-01/
Luis
On Thu, Oct 14, 2010 at 03:35:34PM -0700, Luis R. Rodriguez wrote:
> On Thu, Oct 14, 2010 at 3:29 PM, Luis R. Rodriguez <[email protected]> wrote:
> > Fun enough if I just create one monitor interface and loop quickly
> > over some 2 GHz channels where I know I have traffic nearby I don't
> > see the poison. So channel changes don't seem to do much because this
> > is changing channels as fast as possible from userspace. I also can
> > confirm that I see frames from the different channels as I move along.
>
> Even forcing a band change doesn't help trigger it with just one mon0
> and one regular device scanning in a loop;
>
> while true; do for i in 2412 5745 2417 5745 2422 5745 2427 5745 2432
> 5745 2442; do echo $i iw dev mon0 set freq $i; done; done
> while true; do iw dev wlan0 scan; done
OK so just so you know where I'm poking, this is what I have so far. The
ath9k_hw_rxprocdesc() suggestion came from Jouni but it didn't seem to help.
I'm disabling HT as I want to rule out things step by step. I haven't yet
ruled out TX as haven't been able to trigger this poison yet just based
on monitor interfaces and no frame TX's, I needed at probe requests sent
by one STA.
So the script I used was:
#!/usr/bin/perl
use strict;
my $iw = "/usr/sbin/iw";
my $ip = "/sbin/ip";
my $phy = "phy0";
my $max = 300;
my $i;
my $cmd;
# Create stations
for ($i = 0; $i<$max; $i++) {
runCmd("$iw phy $phy interface add mon$i type monitor");
runCmd("$ip link set dev mon$i up");
}
sub runCmd {
my $cmd = shift;
print "$cmd\n";
`$cmd`;
}
And what I have on top of my tree right now, after your two new patches:
I should note I never hit the WARN_ON() nor the printks, so that rules
those out.
diff --git a/drivers/net/wireless/ath/ath9k/init.c b/drivers/net/wireless/ath/ath9k/init.c
index a4c5ed4..cd61727 100644
--- a/drivers/net/wireless/ath/ath9k/init.c
+++ b/drivers/net/wireless/ath/ath9k/init.c
@@ -192,6 +192,7 @@ static void setup_ht_cap(struct ath_softc *sc,
int i, max_streams;
ht_info->ht_supported = true;
+ ht_info->ht_supported = false;
ht_info->cap = IEEE80211_HT_CAP_SUP_WIDTH_20_40 |
IEEE80211_HT_CAP_SM_PS |
IEEE80211_HT_CAP_SGI_40 |
diff --git a/drivers/net/wireless/ath/ath9k/mac.c b/drivers/net/wireless/ath/ath9k/mac.c
index 8c13479..a96327e 100644
--- a/drivers/net/wireless/ath/ath9k/mac.c
+++ b/drivers/net/wireless/ath/ath9k/mac.c
@@ -639,6 +639,10 @@ int ath9k_hw_rxprocdesc(struct ath_hw *ah, struct ath_desc *ds,
if ((adsp->ds_rxstatus8 & AR_RxDone) == 0)
return -EINPROGRESS;
+ ds->ds_data = 0;
+ ds->ds_vdata = 0;
+ wmb();
+
ads.u.rx = adsp->u.rx;
rs->rs_status = 0;
diff --git a/drivers/net/wireless/ath/ath9k/main.c b/drivers/net/wireless/ath/ath9k/main.c
index bcd3892..b31b5fe 100644
--- a/drivers/net/wireless/ath/ath9k/main.c
+++ b/drivers/net/wireless/ath/ath9k/main.c
@@ -1243,6 +1243,10 @@ static int ath9k_tx(struct ieee80211_hw *hw,
int padpos, padsize;
struct ieee80211_hdr *hdr = (struct ieee80211_hdr *) skb->data;
int qnum;
+ struct sk_buff *tmp_skb;
+
+ tmp_skb = skb_copy(skb, GFP_ATOMIC);
+ dev_kfree_skb_any(tmp_skb);
if (aphy->state != ATH_WIPHY_ACTIVE && aphy->state != ATH_WIPHY_SCAN) {
ath_print(common, ATH_DBG_XMIT,
diff --git a/drivers/net/wireless/ath/ath9k/recv.c b/drivers/net/wireless/ath/ath9k/recv.c
index fe73fc5..8348199 100644
--- a/drivers/net/wireless/ath/ath9k/recv.c
+++ b/drivers/net/wireless/ath/ath9k/recv.c
@@ -502,6 +502,9 @@ int ath_startrecv(struct ath_softc *sc)
goto start_recv;
bf = list_first_entry(&sc->rx.rxbuf, struct ath_buf, list);
+ /* This is fishy, what if the bf->bf_daddr is not valid ? */
+ if (!bf->bf_daddr)
+ printk("= hah bf->bf_daddr is 0!\n");
ath9k_hw_putrxbuf(ah, bf->bf_daddr);
ath9k_hw_rxena(ah);
@@ -663,6 +666,12 @@ static void ath_rx_send_to_mac80211(struct ieee80211_hw *hw,
struct ieee80211_rx_status *rxs)
{
struct ieee80211_hdr *hdr;
+ struct sk_buff *tmp_skb;
+
+ if (1) {
+ tmp_skb = skb_copy(skb, GFP_ATOMIC);
+ dev_kfree_skb_any(tmp_skb);
+ }
hdr = (struct ieee80211_hdr *)skb->data;
@@ -815,11 +821,17 @@ static struct ath_buf *ath_get_next_rx_buf(struct ath_softc *sc,
ret = ath9k_hw_rxprocdesc(ah, tds, &trs, 0);
if (ret == -EINPROGRESS)
return NULL;
+ WARN_ON(1);
}
if (!bf->bf_mpdu)
return bf;
+ if (!bf->bf_buf_addr)
+ printk("bf->bf_buf_addr = 0\n");
/*
* Synchronize the DMA transfer with CPU before
* 1. accessing the frame
On Thu, 2010-10-07 at 10:33 -0700, Ben Greear wrote:
> In case it helps, here is a dump of where the corrupted SKB was deleted.
I wonder, do you have a machine with a decent IOMMU? Adding IOMMU
debugging into the mix could help you figure out if it's a DMA problem.
johannes
On Thu, Oct 14, 2010 at 4:30 PM, Ben Greear <[email protected]> wrote:
> On 10/14/2010 04:19 PM, Luis R. Rodriguez wrote:
>>
>> Ok please try this patch, it cures it for me.
>
> Well, it got a lot further than normal, but it still
> hit the poison check after a few minutes.
>
> Current test case is my app loading 130 or so stations, each running
> wpa_supplicant. All were created, and quite a few had associated
> when the poison check hit.
>
> So, it definitely looks like a step in the right direction, but
> not fully fixed yet.
>
> I'll do some more testing with this patch applied and using just my
> perl script to make sure the problem is reproducible outside of my
> application.
Ok, whatever userspace does it should not corrupt to kernel, unless
its poking /dev/mem
Luis
On Thu, Oct 14, 2010 at 3:29 PM, Luis R. Rodriguez <[email protected]> wrote:
> Fun enough if I just create one monitor interface and loop quickly
> over some 2 GHz channels where I know I have traffic nearby I don't
> see the poison. So channel changes don't seem to do much because this
> is changing channels as fast as possible from userspace. I also can
> confirm that I see frames from the different channels as I move along.
Even forcing a band change doesn't help trigger it with just one mon0
and one regular device scanning in a loop;
while true; do for i in 2412 5745 2417 5745 2422 5745 2427 5745 2432
5745 2442; do echo $i iw dev mon0 set freq $i; done; done
while true; do iw dev wlan0 scan; done
Luis
On 10/15/2010 04:33 PM, Ben Greear wrote:
> On 10/15/2010 04:21 PM, Luis R. Rodriguez wrote:
>> Ben, please give this patch a shot. I addresses three races on the PCU:
>>
>> * When we were stopping the CPU for non-EDMA cards we never locked
>> against
>> anything starting the PCU again
>>
>> * ath9k_hw_startpcureceive() was being called without locking
>>
>> * Although we lock on the rxbuf lock for contention against
>> starting/stopping
>> the PCU, we also need to lock on the driver in locations where we
>> start/stop
>> the PCU within the same location otherwise we end up in inconsistant
>> states
>> and the hardware may end up proessing an incorrect buffer for DMA. To
>> protect against this we use a new PCU lock on the main part of the
>> driver to
>> ensure each start/stop/reset operation is done atomically.
>>
>> And fixes one issue as a side effect:
>>
>> * No more packet loss on ping flood when you have one STA associated :)
>>
>> The only issue I see with this is I eventually run out of memory and
>> my box
>> becomes useless, unless I am mistaking that for some other issue.
>>
>> Please give this a shot and if it cures your woes I'll split it up into
>> 3 separate patches, or maybe just two, one for the first two and one for
>> the last issue.
>
> Sounds good, but this lockdep splat happens almost immediately upon
> starting
> my app:
>
> =======================================================
> [ INFO: possible circular locking dependency detected ]
> 2.6.36-rc8-wl+ #32
It ran for a bit..never did see the poison warning, but system hard-locked
(sysrq b on the serial console got no response) after a bit. Hopefully
just because it hit this potential deadlock.
Thanks,
Ben
--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
On 10/15/2010 04:21 PM, Luis R. Rodriguez wrote:
> Ben, please give this patch a shot. I addresses three races on the PCU:
>
> * When we were stopping the CPU for non-EDMA cards we never locked against
> anything starting the PCU again
>
> * ath9k_hw_startpcureceive() was being called without locking
>
> * Although we lock on the rxbuf lock for contention against starting/stopping
> the PCU, we also need to lock on the driver in locations where we start/stop
> the PCU within the same location otherwise we end up in inconsistant states
> and the hardware may end up proessing an incorrect buffer for DMA. To
> protect against this we use a new PCU lock on the main part of the driver to
> ensure each start/stop/reset operation is done atomically.
>
> And fixes one issue as a side effect:
>
> * No more packet loss on ping flood when you have one STA associated :)
>
> The only issue I see with this is I eventually run out of memory and my box
> becomes useless, unless I am mistaking that for some other issue.
>
> Please give this a shot and if it cures your woes I'll split it up into
> 3 separate patches, or maybe just two, one for the first two and one for
> the last issue.
Sounds good, but this lockdep splat happens almost immediately upon starting
my app:
=======================================================
[ INFO: possible circular locking dependency detected ]
2.6.36-rc8-wl+ #32
-------------------------------------------------------
swapper/0 is trying to acquire lock:
(&(&sc->rx.pcu_lock)->rlock){+.-...}, at: [<fa16e5c7>] ath9k_tasklet+0x7e/0x140 [ath9k]
but task is already holding lock:
(&(&sc->rx.rxflushlock)->rlock){+.-...}, at: [<fa16e5b9>] ath9k_tasklet+0x70/0x140 [ath9k]
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #1 (&(&sc->rx.rxflushlock)->rlock){+.-...}:
[<c0457639>] lock_acquire+0x5a/0x78
[<c075f6ed>] _raw_spin_lock_bh+0x20/0x2f
[<fa170513>] ath_flushrecv+0x14/0x61 [ath9k]
[<fa16dda2>] ath_radio_disable+0x83/0x143 [ath9k]
[<fa16e370>] ath9k_config+0x3c3/0x3d8 [ath9k]
[<fa09ca2e>] ieee80211_hw_config+0x11b/0x125 [mac80211]
[<fa0a8edf>] ieee80211_do_open+0x3c5/0x466 [mac80211]
[<fa0a8fdb>] ieee80211_open+0x5b/0x5e [mac80211]
[<c06ce76b>] __dev_open+0x80/0xae
[<c06cc99b>] __dev_change_flags+0xa0/0x115
[<c06ce6bf>] dev_change_flags+0x13/0x3f
[<c06d7e78>] do_setlink+0x23a/0x51b
[<c06d847c>] rtnl_newlink+0x269/0x431
[<c06d79e2>] rtnetlink_rcv_msg+0x182/0x198
[<c06e503c>] netlink_rcv_skb+0x30/0x77
[<c06d7859>] rtnetlink_rcv+0x1b/0x22
[<c06e4e77>] netlink_unicast+0xbe/0x119
[<c06e5a15>] netlink_sendmsg+0x234/0x24c
[<c06bf93a>] __sock_sendmsg+0x51/0x5a
[<c06bfba4>] sock_sendmsg+0x93/0xa7
[<c06bfd8c>] sys_sendmsg+0x149/0x193
[<c06c148b>] sys_socketcall+0x15e/0x1a5
[<c0402f1c>] sysenter_do_call+0x12/0x38
-> #0 (&(&sc->rx.pcu_lock)->rlock){+.-...}:
[<c0457374>] __lock_acquire+0x921/0xb8c
[<c0457639>] lock_acquire+0x5a/0x78
[<c075f6ed>] _raw_spin_lock_bh+0x20/0x2f
[<fa16e5c7>] ath9k_tasklet+0x7e/0x140 [ath9k]
[<c0438fd1>] tasklet_action+0x73/0xc6
[<c043945f>] __do_softirq+0x86/0x111
[<c0439520>] do_softirq+0x36/0x5a
[<c0439659>] irq_exit+0x35/0x69
[<c0403fb9>] do_IRQ+0x86/0x9a
[<c04034ee>] common_interrupt+0x2e/0x40
[<c040227f>] cpu_idle+0x4e/0x6b
[<c074b6e9>] rest_init+0x8d/0x92
[<c09758ea>] start_kernel+0x320/0x325
[<c09750d0>] i386_start_kernel+0xd0/0xd7
other info that might help us debug this:
1 lock held by swapper/0:
#0: (&(&sc->rx.rxflushlock)->rlock){+.-...}, at: [<fa16e5b9>] ath9k_tasklet+0x70/0x140 [ath9k]
stack backtrace:
Pid: 0, comm: swapper Not tainted 2.6.36-rc8-wl+ #32
Call Trace:
[<c075d940>] ? printk+0xf/0x17
[<c04565af>] print_circular_bug+0x91/0x9d
[<c0457374>] __lock_acquire+0x921/0xb8c
[<c0457639>] lock_acquire+0x5a/0x78
[<fa16e5c7>] ? ath9k_tasklet+0x7e/0x140 [ath9k]
[<c075f6ed>] _raw_spin_lock_bh+0x20/0x2f
[<fa16e5c7>] ? ath9k_tasklet+0x7e/0x140 [ath9k]
[<fa16e5c7>] ath9k_tasklet+0x7e/0x140 [ath9k]
[<c0438fd1>] tasklet_action+0x73/0xc6
[<c043945f>] __do_softirq+0x86/0x111
[<c0439520>] do_softirq+0x36/0x5a
[<c0439659>] irq_exit+0x35/0x69
[<c0403fb9>] do_IRQ+0x86/0x9a
[<c04034ee>] common_interrupt+0x2e/0x40
[<c045007b>] ? do_adjtimex+0x223/0x55e
[<c0408245>] ? mwait_idle+0x5c/0x6c
[<c040227f>] cpu_idle+0x4e/0x6b
[<c074b6e9>] rest_init+0x8d/0x92
[<c09758ea>] start_kernel+0x320/0x325
[<c09750d0>] i386_start_kernel+0xd0/0xd7
--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
On Tue, Oct 5, 2010 at 10:24 AM, Ben Greear <[email protected]> wrote:
> On 10/05/2010 10:16 AM, Luis R. Rodriguez wrote:
>>
>> On Tue, Oct 5, 2010 at 10:00 AM, Ben Greear<[email protected]>
>> wrote:
>>>
>>> I started seeing this very soon after creating interfaces.
>>
>> Can you be more specific how one can reproduce?
>
> Enable SLUB debugging, DEBUG_PAGEALLOC (Debug page memory allocations),
> lockdep, pre-empt.
OK I just enabled PAGE_ALLOC thingy, it wasn't enabled on my kernel.
> Please try creating a bunch of stations on one ath9k phy,
How many?
Luis
On 10/12/2010 11:40 AM, Luis R. Rodriguez wrote:
> On Tue, Oct 12, 2010 at 11:35 AM, Ben Greear<[email protected]> wrote:
>> On 10/11/2010 11:10 PM, Luis R. Rodriguez wrote:
>>>
>>> On Mon, Oct 11, 2010 at 8:27 PM, Ben Greear<[email protected]>
>>> wrote:
>>
>>>> Another thing I was thinking about: Maybe the queue of skbs and dma
>>>> addresses
>>>> in ath9k is getting corrupted by multiple VIFs trying to write at once?
>>>> Maybe
>>>> some locking is needed in the xmit path?
>>>
>>> That was my second hunch. My first shot was to use spin_lock_irqsave()
>>> over the the uses of the rxbuf list and that seemed to help but I
>>> still managed to get a poison eventually. My next item to check for is
>>> of the permissibility of creating too much pressure to the point we
>>> end up looping over the rxbuf list and race against mac80211 free'ing
>>> a buffer. Will test that tomorrow if nothing else comes up creeping my
>>> priority queue.
>>
>> This code looks weird to me. One of the paprd branches
>> deletes the skb, the other doesn't appear to. Neither
>> null out bf->bf_mpdu, which would appear to leave a dangling
>> pointer in at least the dev_kfree_skb_any() branch.
>>
>> ath_tx_complete frees it's skb in all cases, so another
>> bf->bf_mpdu dangling pointer issue.
>>
>> Maybe at the least we should null out bf->bf_mpdu when
>> skb is consumed?
>
> You're reading my mind, that was what I was going to test today. Still
> doing e-mail sweep though.
I'm trying to put together a patch now..but the paprd branch makes
no sense at all.
Shouldn't it also free the skb in the complete(&sc->paprd_complete) branch?
I don't see anything that uses the skb, and nothing that frees it.
if (bf->bf_state.bfs_paprd) {
if (time_after(jiffies,
bf->bf_state.bfs_paprd_timestamp +
msecs_to_jiffies(ATH_PAPRD_TIMEOUT)))
dev_kfree_skb_any(skb);
else
complete(&sc->paprd_complete);
Thanks,
Ben
>
> Luis
--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
2010/10/15 Bj?rn Smedman <[email protected]>
>
> 2010/10/15 Ben Greear <[email protected]>:
> > I tried the patch below, and it didn't seem to help. ?Might even
> > have hurt..as it died on divide-by-zero error:
>
> Hmm, looks like the ani code got a zero listen time from the hw...
> That just might mean that the DMA actually hits one of these
> descriptors. :)
Am I the only one worried about this? Leaving a DMA descriptor
pointing at memory which has been passed on to somebody else... To me
that's like pointing a loaded gun at someone (and it seems this
particular gun can go off a little haphazardly).
Luis, given how hard it seems to be to get that locking and skb
shoveling right, are you sure you want to keep pointing that DMA
engine on innocent people's data?
/Bj?rn
On 10/07/2010 11:42 AM, Luis R. Rodriguez wrote:
> On Thu, Oct 7, 2010 at 11:39 AM, Ben Greear<[email protected]> wrote:
>> On 10/07/2010 11:29 AM, Luis R. Rodriguez wrote:
>>>
>>> On Thu, Oct 7, 2010 at 11:14 AM, Johannes Berg
>>> <[email protected]> wrote:
>>>>
>>>> On Thu, 2010-10-07 at 10:33 -0700, Ben Greear wrote:
>>>>>
>>>>> In case it helps, here is a dump of where the corrupted SKB was deleted.
>>>>
>>>> I wonder, do you have a machine with a decent IOMMU? Adding IOMMU
>>>> debugging into the mix could help you figure out if it's a DMA problem.
>>>
>>> Ben, how much traffic are you RX'ing on these virtual interfaces?
>>
>> I disabled my user-space application, and this script alone can reproduce
>> the problem fairly quickly on my system. You will need to change some
>> of those first variables. Just start it and wait a few minutes and
>> watch the splats show on the console :)
>>
>> Note that I am not generating any traffic, but the wpa_supplicants are
>> doing their thing of course...
>>
>> I'm using the kernel found here:
>> http://dmz2.candelatech.com/git/gitweb.cgi?p=linux.wireless-testing.ct/.git;a=summary
>>
>> It's latest wireless-testing with some of my own patches, and some
>> I've gathered from here an there. I doubt I'm causing this problem,
>> but if you can't reproduce it with this script on your kernels,
>> I can try with base wireless-testing or whatever you are using.
>
> I'll run this now, but can you try a vanilla wireless-testing? I hear
> the latest wireless-testing is borked so maybe try (git reset --hard
> master-2010-09-29), its what I'm on.
After reboot, and re-run of the script,
I saw this in the logs, and shortly after,
the SLUB poison warning dumped to screen.
Maybe those DMA errors are serious?
Also, I enabled CONFIG_DMA_API_DEBUG, but saw no
indication it detected any problems.
DDRCONF(NETDEV_UP): sta29: link is not ready
ADDRCONF(NETDEV_UP): sta30: link is not ready
ADDRCONF(NETDEV_UP): sta31: link is not ready
sta0: authenticate with 00:14:d1:c6:d2:54 (try 1)
sta0: authenticated
ieee80211 phy0: device now idle
ieee80211 phy0: device no longer idle - working
sta0: associate with 00:14:d1:c6:d2:54 (try 1)
sta18: authenticate with 00:14:d1:c6:d2:54 (try 1)
sta0: associate with 00:14:d1:c6:d2:54 (try 2)
sta18: authenticate with 00:14:d1:c6:d2:54 (try 2)
sta0: associate with 00:14:d1:c6:d2:54 (try 3)
sta18: authenticate with 00:14:d1:c6:d2:54 (try 3)
sta0: association with 00:14:d1:c6:d2:54 timed out
sta18: authentication with 00:14:d1:c6:d2:54 timed out
ath: Failed to stop TX DMA in 100 msec after killing last frame
ath: Failed to stop TX DMA. Resetting hardware!
ieee80211 phy0: device now idle
ieee80211 phy0: device no longer idle - scanning
ieee80211 phy0: device now idle
ieee80211 phy0: device no longer idle - working
sta1: authenticate with 00:14:d1:c6:d2:54 (try 1)
sta1: authenticated
sta1: associate with 00:14:d1:c6:d2:54 (try 1)
sta1: RX AssocResp from 00:14:d1:c6:d2:54 (capab=0x431 status=0 aid=18)
sta1: associated
ieee80211 phy0: Allocated STA 00:14:d1:c6:d2:54
ieee80211 phy0: Inserted STA 00:14:d1:c6:d2:54
ieee80211 phy0: WMM queue=2 aci=0 acm=0 aifs=3 cWmin=15 cWmax=1023 txop=0 uapsd=0
ieee80211 phy0: WMM queue=3 aci=1 acm=0 aifs=7 cWmin=15 cWmax=1023 txop=0 uapsd=0
ieee80211 phy0: WMM queue=1 aci=2 acm=0 aifs=2 cWmin=7 cWmax=15 txop=94 uapsd=0
ieee80211 phy0: WMM queue=0 aci=3 acm=0 aifs=2 cWmin=3 cWmax=7 txop=47 uapsd=0
Thanks,
Ben
--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
On 10/13/2010 10:29 AM, Luis R. Rodriguez wrote:
> On Wed, Oct 13, 2010 at 10:12 AM, Ben Greear<[email protected]> wrote:
>> On 10/12/2010 11:40 AM, Luis R. Rodriguez wrote:
>>>
>>> On Tue, Oct 12, 2010 at 11:35 AM, Ben Greear<[email protected]>
>>> wrote:
>>>>
>>>> On 10/11/2010 11:10 PM, Luis R. Rodriguez wrote:
>>>>>
>>>>> On Mon, Oct 11, 2010 at 8:27 PM, Ben Greear<[email protected]>
>>>>> wrote:
>>>>
>>>>>> Another thing I was thinking about: Maybe the queue of skbs and dma
>>>>>> addresses
>>>>>> in ath9k is getting corrupted by multiple VIFs trying to write at once?
>>>>>> Maybe
>>>>>> some locking is needed in the xmit path?
>>>>>
>>>>> That was my second hunch. My first shot was to use spin_lock_irqsave()
>>>>> over the the uses of the rxbuf list and that seemed to help but I
>>>>> still managed to get a poison eventually. My next item to check for is
>>>>> of the permissibility of creating too much pressure to the point we
>>>>> end up looping over the rxbuf list and race against mac80211 free'ing
>>>>> a buffer. Will test that tomorrow if nothing else comes up creeping my
>>>>> priority queue.
>>>>
>>>> This code looks weird to me. One of the paprd branches
>>>> deletes the skb, the other doesn't appear to. Neither
>>>> null out bf->bf_mpdu, which would appear to leave a dangling
>>>> pointer in at least the dev_kfree_skb_any() branch.
>>>>
>>>> ath_tx_complete frees it's skb in all cases, so another
>>>> bf->bf_mpdu dangling pointer issue.
>>>>
>>>> Maybe at the least we should null out bf->bf_mpdu when
>>>> skb is consumed?
>>>
>>> You're reading my mind, that was what I was going to test today. Still
>>> doing e-mail sweep though.
>>
>> At least in the xmit path, it seems cards that have EDMA support do
>> things a bit different. Out of curiosity, on the system(s), you reproduce
>> this, are any of yours supporting EDMA? Mine appear to not support EDMA.
>
> EDMA is used on>= AR9003 families by Atheros. And no, I am not
> testing with an EDMA card, I am testing with an AR9002 family card,
> the AR9280 card. I am going to disregard the TX stuff as the bug is an
> RX issue :) I was able to more easily reproduce by doing an skb_copy()
> and free'ing the buffer right afterwards on the ath_send_to_mac80211()
> thingy, So it does appear that the poison check just happens more
> often when we do an skb_copy(). One reason this is easy to reproduce
> with multiple STAs is mac80211 uses skb_copy() to process each
> received skb for each STA.
>
> In my tests so far, protecting the rxbuf list with spin_lock_irqsave()
> did not help, and the wmb(); didn't either, something else is going on
> here. It would be nice to hack slab to keep an entire trace of the
> place the buffer was last free'd at instead of just the caller that
> freed it.
I instrumented slub a while back and got the backtrace. It
was always in the same place for my testing.
Here's the slub patch if you are interested in using it yourself:
https://patchwork.kernel.org/patch/236921/
Are you able to reproduce this with a single STA interface? If so, we
should be able to somewhat tie-break mac80211 by using another /n NIC,
hopefully with similar AMPDU support, etc.
[From a mail I sent on 10/7 in this thread]
In case it helps, here is a dump of where the corrupted SKB was deleted.
I added debugging to slub to get this information, but it looks like
it's correct to me.
Reading symbols from /home/greearb/kernel/2.6/wireless-testing-dbg.p4s/net/mac80211/mac80211.ko...done.
(gdb) l *(ieee80211_rx+0x74d)
0x13751 is in ieee80211_rx (/home/greearb/git/linux.wireless-testing/include/linux/rcupdate.h:346).
341 *
342 * See rcu_read_lock() for more information.
343 */
344 static inline void rcu_read_unlock(void)
345 {
346 rcu_read_release();
347 __release(RCU);
348 __rcu_read_unlock();
349 }
350
(gdb)
# I don't really know what that second address means, but just in case it's useful,
# I printed it out here:
(gdb) l *(ieee80211_rx+0x7b4)
0x137b8 is in ieee80211_process_measurement_req (/home/greearb/git/linux.wireless-testing/net/mac80211/spectmgmt.c:74).
69 }
70
71 void ieee80211_process_measurement_req(struct ieee80211_sub_if_data *sdata,
72 struct ieee80211_mgmt *mgmt,
73 size_t len)
74 {
75 /*
76 * Ignoring measurement request is spec violation.
77 * Mandatory measurements must be reported optional
78 * measurements might be refused or reported incapable
INFO: Freed in skb_release_data+0x8c/0x90 age=122 cpu=1 pid=0
set_track+0x3c/0x89
__slab_free+0x17f/0x1ba
skb_release_data+0x8c/0x90
kfree+0xaf/0xdf
skb_release_data+0x8c/0x90
skb_release_data+0x8c/0x90
skb_release_data+0x8c/0x90
__kfree_skb+0x12/0x6d
consume_skb+0x2a/0x2c
ieee80211_rx+0x74d/0x7b4 [mac80211]
__kmalloc_track_caller+0xcd/0xf2
trace_hardirqs_on_caller+0xeb/0x125
ath_rx_send_to_mac80211+0x5a/0x60 [ath9k]
trace_hardirqs_on+0xb/0xd
--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
On 10/14/2010 02:52 PM, Bj?rn Smedman wrote:
> 2010/10/13 Bj?rn Smedman<[email protected]>:
>> Hi Ben,
>>
>> First of all keep up the good work. :)
>>
>> On Wed, Oct 13, 2010 at 6:39 PM, Ben Greear<[email protected]> wrote:
>> [snip]
>>> Either way, it seems safer to null out the bf_ampdu field after
>>> the memory is consumed..it could prevent some tricky bugs later.
>>
>> I think this is a good idea. But it probably wont be enough to null
>> out bf_mpdu. You also need to look at bf_buf_addr (which if I
>> understand correctly is the physical address the DMA engine will
>> actually write RXed frames to) and bf_dmacontext (which seems in most
>> cases to hold an identical address and may in fact be where the DMA
>> engine will really write the frame).
>
> I took another look at the code. It turns out both bf_buf_addr and
> bf_dmacontext are in fact meaningless to the DMA. Instead each bf
> holds a pointer (bf_desc) to the real DMA descriptor which in turn
> holds the address (ds_data) where the DMA will really (really this
> time) write the frame. There is also a field to hold the virtual
> address of the same place (ds_vdata).
>
> It's a little too much work for me to set up the testbed you have Ben
> but would be interesting to see what happens if you set
> bf->bf_desc->ds_{data,vdata} = 0 as well. No?
I'll investigate those suggestions.
But setting up a test-bed is as easy
as getting an ath9k NIC in a system, with a few APs around, and run the
script below.
You do not need any traffic generation, dhcp, etc...seems just beacons and whatever
wpa_supplicant is doing is enough to hit the problem fast. (Make sure
you are compiled to detect memory poisoning, of course).
You'll need to fix the paths to the executables most likely.
#!/usr/bin/perl
use strict;
my $iw = "./local/sbin/iw";
my $ip = "./local/sbin/ip";
my $wpa_s = "./local/bin/wpa_supplicant";
my $ssid = "candela-n";
my $key = "wpadmz123";
my $phy = "phy0";
my $max = 32;
my $i;
my $bmac = "00:01:02:03:04:";
my $cmd;
# Create stations
for ($i = 0; $i<$max; $i++) {
runCmd("$iw phy $phy interface add sta$i type station");
my $mc5 = "$i";
if (length($mc5) == 1) {
$mc5 = "0$mc5"; # pad mac octet
}
my $mac = "$bmac$mc5";
runCmd("$ip link set sta$i address $mac");
runCmd("$iw dev sta$i set power_save off");
}
# Bring them up with WPA
for ($i = 0; $i<$max; $i++) {
open(FD, ">sta$i" . "_wpa.conf") || die("Couldn't open file: $!\n");
print FD "
ctrl_interface=/var/run/wpa_supplicant
fast_reauth=1
#can_scan_one=1
network={
ssid=\"$ssid\"
proto=WPA
key_mgmt=WPA-PSK
psk=\"$key\"
pairwise=TKIP CCMP
group=TKIP CCMP
}
";
runCmd("$wpa_s -B -i sta$i -c sta$i" . "_wpa.conf -P sta$i" . "_wpa.pid -t -f sta$i" . "_wpa.log");
}
sub runCmd {
my $cmd = shift;
print "$cmd\n";
`$cmd`;
}
Thanks,
Ben
--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
On Thu, Oct 14, 2010 at 04:39:45PM -0700, Luis R. Rodriguez wrote:
> On Thu, Oct 14, 2010 at 4:30 PM, Ben Greear <[email protected]> wrote:
> > On 10/14/2010 04:19 PM, Luis R. Rodriguez wrote:
> >>
> >> Ok please try this patch, it cures it for me.
> >
> > Well, it got a lot further than normal, but it still
> > hit the poison check after a few minutes.
> >
> > Current test case is my app loading 130 or so stations, each running
> > wpa_supplicant. ?All were created, and quite a few had associated
> > when the poison check hit.
> >
> > So, it definitely looks like a step in the right direction, but
> > not fully fixed yet.
> >
> > I'll do some more testing with this patch applied and using just my
> > perl script to make sure the problem is reproducible outside of my
> > application.
>
> Ok, whatever userspace does it should not corrupt to kernel, unless
> its poking /dev/mem
Can also try this one instead, it will prevent any other instances
we would not have caught on stopping and starting RX here.
diff --git a/drivers/net/wireless/ath/ath9k/recv.c b/drivers/net/wireless/ath/ath9k/recv.c
index fe73fc5..db677c4 100644
--- a/drivers/net/wireless/ath/ath9k/recv.c
+++ b/drivers/net/wireless/ath/ath9k/recv.c
@@ -306,10 +306,8 @@ static void ath_edma_start_recv(struct ath_softc *sc)
static void ath_edma_stop_recv(struct ath_softc *sc)
{
- spin_lock_bh(&sc->rx.rxbuflock);
ath_rx_remove_buffer(sc, ATH9K_RX_QUEUE_HP);
ath_rx_remove_buffer(sc, ATH9K_RX_QUEUE_LP);
- spin_unlock_bh(&sc->rx.rxbuflock);
}
int ath_rx_init(struct ath_softc *sc, int nbufs)
@@ -482,13 +480,14 @@ int ath_startrecv(struct ath_softc *sc)
{
struct ath_hw *ah = sc->sc_ah;
struct ath_buf *bf, *tbf;
+ unsigned long flags;
if (ah->caps.hw_caps & ATH9K_HW_CAP_EDMA) {
ath_edma_start_recv(sc);
return 0;
}
- spin_lock_bh(&sc->rx.rxbuflock);
+ spin_lock_irqsave(&sc->rx.rxbuflock, flags);
if (list_empty(&sc->rx.rxbuf))
goto start_recv;
@@ -506,7 +505,7 @@ int ath_startrecv(struct ath_softc *sc)
ath9k_hw_rxena(ah);
start_recv:
- spin_unlock_bh(&sc->rx.rxbuflock);
+ spin_unlock_irqrestore(&sc->rx.rxbuflock, flags);
ath_opmode_init(sc);
ath9k_hw_startpcureceive(ah, (sc->sc_flags & SC_OP_OFFCHANNEL));
@@ -517,7 +516,9 @@ bool ath_stoprecv(struct ath_softc *sc)
{
struct ath_hw *ah = sc->sc_ah;
bool stopped;
+ unsigned long flags;
+ spin_lock_irqsave(&sc->rx.rxbuflock, flags);
ath9k_hw_stoppcurecv(ah);
ath9k_hw_setrxfilter(ah, 0);
stopped = ath9k_hw_stopdmarecv(ah);
@@ -526,6 +527,7 @@ bool ath_stoprecv(struct ath_softc *sc)
ath_edma_stop_recv(sc);
else
sc->rx.rxlink = NULL;
+ spin_unlock_irqrestore(&sc->rx.rxbuflock, flags);
return stopped;
}
2010/10/13 Bj?rn Smedman <[email protected]>:
> Hi Ben,
>
> First of all keep up the good work. :)
>
> On Wed, Oct 13, 2010 at 6:39 PM, Ben Greear <[email protected]> wrote:
> [snip]
>> Either way, it seems safer to null out the bf_ampdu field after
>> the memory is consumed..it could prevent some tricky bugs later.
>
> I think this is a good idea. But it probably wont be enough to null
> out bf_mpdu. You also need to look at bf_buf_addr (which if I
> understand correctly is the physical address the DMA engine will
> actually write RXed frames to) and bf_dmacontext (which seems in most
> cases to hold an identical address and may in fact be where the DMA
> engine will really write the frame).
I took another look at the code. It turns out both bf_buf_addr and
bf_dmacontext are in fact meaningless to the DMA. Instead each bf
holds a pointer (bf_desc) to the real DMA descriptor which in turn
holds the address (ds_data) where the DMA will really (really this
time) write the frame. There is also a field to hold the virtual
address of the same place (ds_vdata).
It's a little too much work for me to set up the testbed you have Ben
but would be interesting to see what happens if you set
bf->bf_desc->ds_{data,vdata} = 0 as well. No?
/Bj?rn
On 10/11/2010 06:03 PM, Luis R. Rodriguez wrote:
> On Mon, Oct 11, 2010 at 1:51 PM, Ben Greear<[email protected]> wrote:
>> On 10/07/2010 02:59 PM, Luis R. Rodriguez wrote:
>>>
>>> On Thu, Oct 7, 2010 at 2:36 PM, Luis R. Rodriguez<[email protected]>
>>> wrote:
>>
>>>>> But other than this I got nothing. I left the box sit there for about
>>>>> 1 hour and came back and it was still going with no issues. Mind you,
>>>>> I can't ping but that seems like another issue.
>>>>>
>>>>> You can find my logs here:
>>>>>
>>>>>
>>>>> http://www.kernel.org/pub/linux/kernel/people/mcgrof/logs/2010/10-07-stress-sta-01/
>>>>
>>>> Doh, I did not have CONFIG_SLUB_DEBUG_ON=y so building now with that.
>>>
>>> Yay I can reproduce now. I'll be back, going to dig now.
>>
>> Any luck tracking this down?
>
> No, today for example I just finished reading e-mail and its already
> 6pm PST... But Friday I did get do do a lot of work and testing on
> this. The only pattern I see so far is that skb_copy() is used on the
> poison all the time. I am not sure if its because skb_copy() happens
> to run the poison check or what. I'll work on this tomorrow.
I know how that goes.
Do you happen to have any magic tools that could be instrumented to show
when DMA was happening in the chip, and to see if it somehow happens to dma
to something after it is supposedly un-mapped?
Another thing I was thinking about: Maybe the queue of skbs and dma addresses
in ath9k is getting corrupted by multiple VIFs trying to write at once? Maybe
some locking is needed in the xmit path?
Thanks,
Ben
--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
On 10/15/2010 11:47 AM, Luis R. Rodriguez wrote:
> On Fri, Oct 15, 2010 at 9:51 AM, Ben Greear<[email protected]> wrote:
>> On 10/14/2010 04:48 PM, Luis R. Rodriguez wrote:
>>>
>>> On Thu, Oct 14, 2010 at 04:39:45PM -0700, Luis R. Rodriguez wrote:
>>>>
>>>> On Thu, Oct 14, 2010 at 4:30 PM, Ben Greear<[email protected]>
>>>> wrote:
>>>>>
>>>>> On 10/14/2010 04:19 PM, Luis R. Rodriguez wrote:
>>>>>>
>>>>>> Ok please try this patch, it cures it for me.
>>>>>
>>>>> Well, it got a lot further than normal, but it still
>>>>> hit the poison check after a few minutes.
>>>>>
>>>>> Current test case is my app loading 130 or so stations, each running
>>>>> wpa_supplicant. All were created, and quite a few had associated
>>>>> when the poison check hit.
>>>>>
>>>>> So, it definitely looks like a step in the right direction, but
>>>>> not fully fixed yet.
>>>>>
>>>>> I'll do some more testing with this patch applied and using just my
>>>>> perl script to make sure the problem is reproducible outside of my
>>>>> application.
>>>>
>>>> Ok, whatever userspace does it should not corrupt to kernel, unless
>>>> its poking /dev/mem
>>>
>>> Can also try this one instead, it will prevent any other instances
>>> we would not have caught on stopping and starting RX here.
>>
>> It ran longer than before any of your locking patches (about 3 minutes), but
>> it did hit the poison check.
>>
>> Before it did, I had a bunch of OOM errors trying to allocate
>> skbs. I have 2GB of RAM on this system, but maybe it's not tuned
>> properly, and not all of that can be used for networking on 32-bit
>> kernels....
>>
>> I have Felix's 3 ani patches from ~3 days ago applied, running 130 stations
>> in WPA mode.
>>
>> I'm going to try running fewer to dodge the OOM case,
>> but I have a few other things to take care of first.
>
> Thanks for testing. So now I cannot reproduce the poison anymore, any
> other ideas what I can try? Does the perl script still give you poison
> or just with your Über proprietary application?
I was just writing that my script wouldn't reproduce it..but it did before I
was done typing. Same looking poison exception as ever.
I also notice my Trendnet AP is very un-happy with anything past around 30 STA
devices associated, and according to it's status page..NONE of my STAs are associated,
though things show up in /proc/net/wireless and I see auth/assoc messages in
/var/log/messages on my STA system, so the AP may just be funky.
On my system, most of the STAs are constantly trying to associate and being
rejected by the AP.
Here is updated script, creates 130 STAs and sleeps a bit between starting supplicants.
It assumes you have a single PHY device, and if you do, it will use it's name regardless
of how many times you reload the driver.
Please forgive the lameness in the MAC creation logic..though it does at least appear
to work :)
Thanks,
Ben
#!/usr/bin/perl
use strict;
my $iw = "./local/sbin/iw";
my $ip = "./local/sbin/ip";
my $wpa_s = "./local/bin/wpa_supplicant";
my $ssid = "candela-n";
my $key = "wpadmz123";
my $phy = `cat /sys/class/ieee80211/*/name`;
chomp($phy);
my $max = 130;
my $i;
my $j = 4;
my $k = 1;
my $bmac = "00:01:02:03:";
my $cmd;
# Create stations
for ($i = 0; $i<$max; $i++) {
runCmd("$iw phy $phy interface add sta$i type station");
if ($j > 99) {
$j = 1;
$k++;
}
my $mc5 = "$j";
if (length($mc5) == 1) {
$mc5 = "0$mc5"; # pad mac octet
}
my $mc4 = "$k";
if (length($mc4) == 1) {
$mc4 = "0$mc4"; # pad mac octet
}
$j++;
my $mac = "$bmac$mc4:$mc5";
runCmd("$ip link set sta$i address $mac");
runCmd("$iw dev sta$i set power_save off");
}
# Bring them up with WPA
for ($i = 0; $i<$max; $i++) {
open(FD, ">sta$i" . "_wpa.conf") || die("Couldn't open file: $!\n");
print FD "
ctrl_interface=/var/run/wpa_supplicant
fast_reauth=1
#can_scan_one=1
network={
ssid=\"$ssid\"
proto=WPA
key_mgmt=WPA-PSK
psk=\"$key\"
pairwise=TKIP CCMP
group=TKIP CCMP
}
";
runCmd("$wpa_s -B -i sta$i -c sta$i" . "_wpa.conf -P sta$i" . "_wpa.pid -t -f sta$i" . "_wpa.log");
sleep(2);
}
sub runCmd {
my $cmd = shift;
print "$cmd\n";
`$cmd`;
}
>
> Luis
> --
> To unsubscribe from this list: send the line "unsubscribe linux-wireless" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
On Tue, Oct 5, 2010 at 10:38 AM, Ben Greear <[email protected]> wrote:
> On 10/05/2010 10:36 AM, Luis R. Rodriguez wrote:
>>
>> On Tue, Oct 5, 2010 at 10:24 AM, Ben Greear<[email protected]>
>> wrote:
>>>
>>> On 10/05/2010 10:16 AM, Luis R. Rodriguez wrote:
>>>>
>>>> On Tue, Oct 5, 2010 at 10:00 AM, Ben Greear<[email protected]>
>>>> wrote:
>>>>>
>>>>> I started seeing this very soon after creating interfaces.
>>>>
>>>> Can you be more specific how one can reproduce?
>>>
>>> Enable SLUB debugging, DEBUG_PAGEALLOC (Debug page memory allocations),
>>> lockdep, pre-empt.
>>
>> OK I just enabled PAGE_ALLOC thingy, it wasn't enabled on my kernel.
>>
>>> Please try creating a bunch of stations on one ath9k phy,
>>
>> How many?
>
> 130 is a nice round number :)
Let me get this straight. You are creating 130 STAs with ath9k using
iw, then running 150 instances of wpa_supplicant to connect to the
same AP?
> I'll try with a smaller number and see if I can hit it.
Please do, you can use the log2n approach, should take you 7 tries,
each by a power of 2, start at the middle of course.
Luis
On 10/14/2010 02:45 PM, Johannes Berg wrote:
> On Wed, 2010-10-13 at 10:48 -0700, Ben Greear wrote:
>
>> Here's the slub patch if you are interested in using it yourself:
>> https://patchwork.kernel.org/patch/236921/
>
> That really really screams include/linux/stacktrace.h
Heh, that looks like it would have saved me a lot of pain :P
Ben
>
> johannes
--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
On Tue, 2010-10-05 at 10:00 -0700, Ben Greear wrote:
> Oct 5 09:50:18 localhost kernel: Object 0xf5b18030: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> Oct 5 09:50:18 localhost kernel: Object 0xf5b18040: 80 00 00 00 ff ff ff ff ff ff c8 4c 75 20 e8 fd ....ÿÿÿÿÿÿÈLu.èý
> Oct 5 09:50:18 localhost kernel: Object 0xf5b18050: c8 4c 75 20 e8 fd d0 88 80 c1 e1 03 00 00 00 00 ÈLu.èýÐ..Áá.....
> Oct 5 09:50:18 localhost kernel: Object 0xf5b18060: 90 01 21 04 00 09 63 69 73 63 6f 73 62 2d 31 01 ..!...ciscosb-1.
Evidently, a beacon was received into an already freed SKB.
johannes
On Sun, Oct 17, 2010 at 12:44 PM, Ben Greear <[email protected]> wrote:
> I had a chance to try your latest patch on a different machine and AP. This
> time,
> I was using 130 or so stations, but no encryption (no supplicant).
>
> I saw these exceptions below. The warn-on hits the !stopped check in
> ath_stoprecv.
>
> The system didn't crash, but all of the STAs soon dis-associated because of
> "inactivity".
> I haven't checked if that is some issue with my AP or what..
>
>
> bool ath_stoprecv(struct ath_softc *sc)
> {
> struct ath_hw *ah = sc->sc_ah;
> bool stopped;
>
> spin_lock_bh(&sc->rx.rxbuflock);
> ath9k_hw_stoppcurecv(ah);
> ath9k_hw_setrxfilter(ah, 0);
> stopped = ath9k_hw_stopdmarecv(ah);
>
> if (sc->sc_ah->caps.hw_caps & ATH9K_HW_CAP_EDMA)
> ath_edma_stop_recv(sc);
> else
> sc->rx.rxlink = NULL;
> spin_unlock_bh(&sc->rx.rxbuflock);
>
> WARN_ON(!stopped);
> return stopped;
> }
>
>
> ADDRCONF(NETDEV_UP): sta130: link is not ready
> sta90: no IPv6 routers present
> ath: DMA failed to stop in 10 ms AR_CR=0x00000024 AR_DIAG_SW=0x40000020
We should find out what happened here.
> ------------[ cut here ]------------
> WARNING: at
> /home/greearb/git/linux.wireless-testing/drivers/net/wireless/ath/ath9k/recv.c:532
> ath_stoprecv+0x80/0x87 [ath9k]()
> Hardware name: 945GM
> Modules linked in: 8021q garp stp llc michael_mic macvlan pktgen iscsi_tcp
> libiscsi_tcp libiscsi scsi_transport_iscsi nfs lockd fscache nfs_acl
> auth_rpcgss sunrpc p4_clockmod ipv6 uinput arc4 ecb ath9k snd_intel8x0
> mac80211 snd_ac97_codec ac97_bus snd_seq snd_seq_device ath9k_common snd_pcm
> ath9k_hw ath snd_timer microcode pcspkr cfg80211 i2c_i801 snd soundcore
> snd_page_alloc serio_raw e1000e iTCO_wdt iTCO_vendor_support yenta_socket
> floppy i915 drm_kms_helper drm i2c_algo_bit i2c_core video output [last
> unloaded: ipt_addrtype]
> Pid: 5, comm: kworker/u:0 Tainted: G W 2.6.36-rc8-wl+ #12
> Call Trace:
> [<c0434647>] warn_slowpath_common+0x65/0x7a
> [<f9450e38>] ? ath_stoprecv+0x80/0x87 [ath9k]
> [<c043466b>] warn_slowpath_null+0xf/0x13
> [<f9450e38>] ath_stoprecv+0x80/0x87 [ath9k]
> [<f944f580>] ath_set_channel+0x99/0x1ff [ath9k]
> [<f94502b2>] ath9k_config+0x305/0x3d8 [ath9k]
> [<f9246a2e>] ieee80211_hw_config+0x11b/0x125 [mac80211]
> [<f924aa51>] ieee80211_scan_work+0x29e/0x3ed [mac80211]
> [<c0443d72>] ? process_one_work+0x145/0x295
> [<c0443dbc>] process_one_work+0x18f/0x295
> [<c0443d72>] ? process_one_work+0x145/0x295
> [<f924a7b3>] ? ieee80211_scan_work+0x0/0x3ed [mac80211]
> [<c04453e5>] worker_thread+0xf9/0x1b8
> [<c04452ec>] ? worker_thread+0x0/0x1b8
> [<c0447d2f>] kthread+0x62/0x67
> [<c0447ccd>] ? kthread+0x0/0x67
> [<c0403506>] kernel_thread_helper+0x6/0x1a
> ---[ end trace bc53fa727ee2ae42 ]---
> ath: DMA failed to stop in 10 ms AR_CR=0x00000024 AR_DIAG_SW=0x42000020
> ath: DMA failed to stop in 10 ms AR_CR=0x00000024 AR_DIAG_SW=0x40000020
> ------------[ cut here ]------------
> WARNING: at
> /home/greearb/git/linux.wireless-testing/drivers/net/wireless/ath/ath9k/recv.c:532
> ath_stoprecv+0x80/0x87 [ath9k]()
> Hardware name: 945GM
> Modules linked in: 8021q garp stp llc michael_mic macvlan pktgen iscsi_tcp
> libiscsi_tcp libiscsi scsi_transport_iscsi nfs lockd fscache nfs_acl
> auth_rpcgss sunrpc p4_clockmod ipv6 uinput arc4 ecb ath9k snd_intel8x0
> mac80211 snd_ac97_codec ac97_bus snd_seq snd_seq_device ath9k_common snd_pcm
> ath9k_hw ath snd_timer microcode pcspkr cfg80211 i2c_i801 snd soundcore
> snd_page_alloc serio_raw e1000e iTCO_wdt iTCO_vendor_support yenta_socket
> floppy i915 drm_kms_helper drm i2c_algo_bit i2c_core video output [last
> unloaded: ipt_addrtype]
> Pid: 41, comm: kworker/u:2 Tainted: G W 2.6.36-rc8-wl+ #12
> Call Trace:
> [<c0434647>] warn_slowpath_common+0x65/0x7a
> [<f9450e38>] ? ath_stoprecv+0x80/0x87 [ath9k]
> [<c043466b>] warn_slowpath_null+0xf/0x13
> [<f9450e38>] ath_stoprecv+0x80/0x87 [ath9k]
> [<f944f580>] ath_set_channel+0x99/0x1ff [ath9k]
> [<f94502b2>] ath9k_config+0x305/0x3d8 [ath9k]
> [<f9246a2e>] ieee80211_hw_config+0x11b/0x125 [mac80211]
> [<f924aa51>] ieee80211_scan_work+0x29e/0x3ed [mac80211]
> [<c0443d72>] ? process_one_work+0x145/0x295
> [<c0443dbc>] process_one_work+0x18f/0x295
> [<c0443d72>] ? process_one_work+0x145/0x295
> [<f924a7b3>] ? ieee80211_scan_work+0x0/0x3ed [mac80211]
> [<c04453e5>] worker_thread+0xf9/0x1b8
> [<c04452ec>] ? worker_thread+0x0/0x1b8
> [<c0447d2f>] kthread+0x62/0x67
> [<c0447ccd>] ? kthread+0x0/0x67
> [<c0403506>] kernel_thread_helper+0x6/0x1a
> ---[ end trace bc53fa727ee2ae43 ]---
> ath: DMA failed to stop in 10 ms AR_CR=0x00000024 AR_DIAG_SW=0x42000020
> sta126: direct probe to 00:18:e7:cb:ad:6e (try 1)
> sta130: direct probe to 00:18:e7:cb:ad:6e (try 1)
> sta91: no IPv6 routers present
> sta126: direct probe to 00:18:e7:cb:ad:6e (try 2)
> sta130: direct probe to 00:18:e7:cb:ad:6e (try 2)
> sta126: direct probe to 00:18:e7:cb:ad:6e (try 3)
> sta130: direct probe to 00:18:e7:cb:ad:6e (try 3)
> sta126: direct probe to 00:18:e7:cb:ad:6e timed out
> sta130: direct probe to 00:18:e7:cb:ad:6e timed out
> sta92: no IPv6 routers present
> ...
So I put the warning there for debugging purposes. The reason I put it
is we have no gaurantee we've told hardware to stop writing to that
area of memory so if we then race and start RX I think we can likely
run into a situation where we may not know which buffer hardware will
be writing to next.
I think we should add the warning on the next kernel development cycle
and address all of its causes, if hardware cannot be stopped I believe
we run the potential to also run into this same poison issue.
The RX poison issue is resolved then but the other issues are separate
issues which need to be debugged further. For example the stop TX dma
issue might be resolved by also preventing to start TX and stop TX
atomically. Something like what I did could likely also be done for
TX.
When I get some time (I have other uber higher priority issues now as
usual) I'll break this patch into a 3 or 2 patches and submit
upstream.
Luis
On Fri, Oct 15, 2010 at 12:36:31PM -0700, Ben Greear wrote:
> I was just writing that my script wouldn't reproduce it..but it did before I
> was done typing. Same looking poison exception as ever.
OK I was able to reproduce with the latest patch.
> I also notice my Trendnet AP is very un-happy with anything past around 30 STA
> devices associated, and according to it's status page..NONE of my STAs are associated,
> though things show up in /proc/net/wireless and I see auth/assoc messages in
> /var/log/messages on my STA system, so the AP may just be funky.
Well we are stress testing the hell out of the AP too!
> On my system, most of the STAs are constantly trying to associate and being
> rejected by the AP.
>
> Here is updated script, creates 130 STAs and sleeps a bit between starting supplicants.
> It assumes you have a single PHY device, and if you do, it will use it's name regardless
> of how many times you reload the driver.
>
> Please forgive the lameness in the MAC creation logic..though it does at least appear
> to work :)
I'm now using this patch to force pressure:
diff --git a/drivers/net/wireless/ath/ath9k/recv.c b/drivers/net/wireless/ath/ath9k/recv.c
index db677c4..2834c41 100644
--- a/drivers/net/wireless/ath/ath9k/recv.c
+++ b/drivers/net/wireless/ath/ath9k/recv.c
@@ -665,6 +665,13 @@ static void ath_rx_send_to_mac80211(struct ieee80211_hw *hw,
struct ieee80211_rx_status *rxs)
{
struct ieee80211_hdr *hdr;
+ struct sk_buff *tmp_skb;
+ unsigned int i;
+
+ for (i=0; i < 5; i++) {
+ tmp_skb = skb_copy(skb, GFP_ATOMIC);
+ dev_kfree_skb_any(tmp_skb);
+ }
hdr = (struct ieee80211_hdr *)skb->data;
And this script, slightly simplified mac address configuration.
#!/usr/bin/perl
use strict;
my $iw = "/usr/sbin/iw";
my $ip = "/sbin/ip";
my $wpa_s = "/usr/local/sbin/wpa_supplicant";
my $ssid = "tesla-2g-bcm";
my $key = "stuff";
my $phy = "phy0";
my $max = 130;
my $i;
my $bmac = "00:01:02:03:04";
my $cmd;
# Create stations
for ($i = 0; $i<$max; $i++) {
runCmd("$iw phy $phy interface add sta$i type station");
my $mc5 = sprintf("%02x", $i);
my $mac = "$bmac:$mc5";
runCmd("$ip link set sta$i address $mac");
runCmd("$iw dev sta$i set power_save off");
}
# Bring them up with WPA
for ($i = 0; $i<$max; $i++) {
open(FD, ">sta$i" . "_wpa.conf") || die("Couldn't open file: $!\n");
print FD "
ctrl_interface=/var/run/wpa_supplicant
fast_reauth=1
#can_scan_one=1
network={
ssid=\"$ssid\"
proto=RSN
key_mgmt=WPA-PSK
psk=\"$key\"
pairwise=CCMP
group=CCMP
}
";
runCmd("$wpa_s -B -i sta$i -c sta$i" . "_wpa.conf -P sta$i" . "_wpa.pid -t ");
sleep(2);
}
sub runCmd {
my $cmd = shift;
print "$cmd\n";
`$cmd`;
}
On Fri, Oct 15, 2010 at 04:33:50PM -0700, Ben Greear wrote:
> On 10/15/2010 04:21 PM, Luis R. Rodriguez wrote:
> > Ben, please give this patch a shot. I addresses three races on the PCU:
> >
> > * When we were stopping the CPU for non-EDMA cards we never locked against
> > anything starting the PCU again
> >
> > * ath9k_hw_startpcureceive() was being called without locking
> >
> > * Although we lock on the rxbuf lock for contention against starting/stopping
> > the PCU, we also need to lock on the driver in locations where we start/stop
> > the PCU within the same location otherwise we end up in inconsistant states
> > and the hardware may end up proessing an incorrect buffer for DMA. To
> > protect against this we use a new PCU lock on the main part of the driver to
> > ensure each start/stop/reset operation is done atomically.
> >
> > And fixes one issue as a side effect:
> >
> > * No more packet loss on ping flood when you have one STA associated :)
> >
> > The only issue I see with this is I eventually run out of memory and my box
> > becomes useless, unless I am mistaking that for some other issue.
> >
> > Please give this a shot and if it cures your woes I'll split it up into
> > 3 separate patches, or maybe just two, one for the first two and one for
> > the last issue.
>
> Sounds good, but this lockdep splat happens almost immediately upon starting
> my app:
>
> =======================================================
> [ INFO: possible circular locking dependency detected ]
> 2.6.36-rc8-wl+ #32
> -------------------------------------------------------
> swapper/0 is trying to acquire lock:
> (&(&sc->rx.pcu_lock)->rlock){+.-...}, at: [<fa16e5c7>] ath9k_tasklet+0x7e/0x140 [ath9k]
>
> but task is already holding lock:
> (&(&sc->rx.rxflushlock)->rlock){+.-...}, at: [<fa16e5b9>] ath9k_tasklet+0x70/0x140 [ath9k]
>
> which lock already depends on the new lock.
>
>
> the existing dependency chain (in reverse order) is:
>
> -> #1 (&(&sc->rx.rxflushlock)->rlock){+.-...}:
> [<c0457639>] lock_acquire+0x5a/0x78
> [<c075f6ed>] _raw_spin_lock_bh+0x20/0x2f
> [<fa170513>] ath_flushrecv+0x14/0x61 [ath9k]
Ah we just need to nuke the flush lock, one second. Also remove my
skb_copy() otherwise you will really run out of memory quickly,
unless you really want to test with it :)
Luis
On Fri, Oct 15, 2010 at 04:38:14PM -0700, Luis Rodriguez wrote:
> On Fri, Oct 15, 2010 at 04:33:50PM -0700, Ben Greear wrote:
> > On 10/15/2010 04:21 PM, Luis R. Rodriguez wrote:
> > > Ben, please give this patch a shot. I addresses three races on the PCU:
> > >
> > > * When we were stopping the CPU for non-EDMA cards we never locked against
> > > anything starting the PCU again
> > >
> > > * ath9k_hw_startpcureceive() was being called without locking
> > >
> > > * Although we lock on the rxbuf lock for contention against starting/stopping
> > > the PCU, we also need to lock on the driver in locations where we start/stop
> > > the PCU within the same location otherwise we end up in inconsistant states
> > > and the hardware may end up proessing an incorrect buffer for DMA. To
> > > protect against this we use a new PCU lock on the main part of the driver to
> > > ensure each start/stop/reset operation is done atomically.
> > >
> > > And fixes one issue as a side effect:
> > >
> > > * No more packet loss on ping flood when you have one STA associated :)
> > >
> > > The only issue I see with this is I eventually run out of memory and my box
> > > becomes useless, unless I am mistaking that for some other issue.
> > >
> > > Please give this a shot and if it cures your woes I'll split it up into
> > > 3 separate patches, or maybe just two, one for the first two and one for
> > > the last issue.
> >
> > Sounds good, but this lockdep splat happens almost immediately upon starting
> > my app:
> >
> > =======================================================
> > [ INFO: possible circular locking dependency detected ]
> > 2.6.36-rc8-wl+ #32
> > -------------------------------------------------------
> > swapper/0 is trying to acquire lock:
> > (&(&sc->rx.pcu_lock)->rlock){+.-...}, at: [<fa16e5c7>] ath9k_tasklet+0x7e/0x140 [ath9k]
> >
> > but task is already holding lock:
> > (&(&sc->rx.rxflushlock)->rlock){+.-...}, at: [<fa16e5b9>] ath9k_tasklet+0x70/0x140 [ath9k]
> >
> > which lock already depends on the new lock.
> >
> >
> > the existing dependency chain (in reverse order) is:
> >
> > -> #1 (&(&sc->rx.rxflushlock)->rlock){+.-...}:
> > [<c0457639>] lock_acquire+0x5a/0x78
> > [<c075f6ed>] _raw_spin_lock_bh+0x20/0x2f
> > [<fa170513>] ath_flushrecv+0x14/0x61 [ath9k]
>
> Ah we just need to nuke the flush lock, one second. Also remove my
> skb_copy() otherwise you will really run out of memory quickly,
> unless you really want to test with it :)
Try this new one, note I if (0)'d the skb_copy() set that to if (1) if you want
to force memory clobber.
diff --git a/drivers/net/wireless/ath/ath9k/ath9k.h b/drivers/net/wireless/ath/ath9k/ath9k.h
index 0c917e5..5dc5421 100644
--- a/drivers/net/wireless/ath/ath9k/ath9k.h
+++ b/drivers/net/wireless/ath/ath9k/ath9k.h
@@ -310,8 +310,8 @@ struct ath_rx {
u8 rxotherant;
u32 *rxlink;
unsigned int rxfilter;
- spinlock_t rxflushlock;
spinlock_t rxbuflock;
+ spinlock_t pcu_lock;
struct list_head rxbuf;
struct ath_descdma rxdma;
struct ath_buf *rx_bufptr;
diff --git a/drivers/net/wireless/ath/ath9k/main.c b/drivers/net/wireless/ath/ath9k/main.c
index 62294da..b06f074 100644
--- a/drivers/net/wireless/ath/ath9k/main.c
+++ b/drivers/net/wireless/ath/ath9k/main.c
@@ -251,6 +251,9 @@ int ath_set_channel(struct ath_softc *sc, struct ieee80211_hw *hw,
*/
ath9k_hw_set_interrupts(ah, 0);
ath_drain_all_txq(sc, false);
+
+ spin_lock_bh(&sc->rx.pcu_lock);
+
stopped = ath_stoprecv(sc);
/* XXX: do not flush receive queue here. We don't want
@@ -278,6 +281,7 @@ int ath_set_channel(struct ath_softc *sc, struct ieee80211_hw *hw,
"reset status %d\n",
channel->center_freq, r);
spin_unlock_bh(&sc->sc_resetlock);
+ spin_unlock_bh(&sc->rx.pcu_lock);
goto ps_restore;
}
spin_unlock_bh(&sc->sc_resetlock);
@@ -286,9 +290,12 @@ int ath_set_channel(struct ath_softc *sc, struct ieee80211_hw *hw,
ath_print(common, ATH_DBG_FATAL,
"Unable to restart recv logic\n");
r = -EIO;
+ spin_unlock_bh(&sc->rx.pcu_lock);
goto ps_restore;
}
+ spin_unlock_bh(&sc->rx.pcu_lock);
+
ath_cache_conf_rate(sc, &hw->conf);
ath_update_txpow(sc);
ath9k_hw_set_interrupts(ah, ah->imask);
@@ -624,7 +631,7 @@ void ath9k_tasklet(unsigned long data)
rxmask = (ATH9K_INT_RX | ATH9K_INT_RXEOL | ATH9K_INT_RXORN);
if (status & rxmask) {
- spin_lock_bh(&sc->rx.rxflushlock);
+ spin_lock_bh(&sc->rx.pcu_lock);
/* Check for high priority Rx first */
if ((ah->caps.hw_caps & ATH9K_HW_CAP_EDMA) &&
@@ -632,7 +639,7 @@ void ath9k_tasklet(unsigned long data)
ath_rx_tasklet(sc, 0, true);
ath_rx_tasklet(sc, 0, false);
- spin_unlock_bh(&sc->rx.rxflushlock);
+ spin_unlock_bh(&sc->rx.pcu_lock);
}
if (status & ATH9K_INT_TX) {
@@ -887,6 +894,7 @@ void ath_radio_enable(struct ath_softc *sc, struct ieee80211_hw *hw)
if (!ah->curchan)
ah->curchan = ath_get_curchannel(sc, sc->hw);
+ spin_lock_bh(&sc->rx.pcu_lock);
spin_lock_bh(&sc->sc_resetlock);
r = ath9k_hw_reset(ah, ah->curchan, ah->caldata, false);
if (r) {
@@ -901,8 +909,10 @@ void ath_radio_enable(struct ath_softc *sc, struct ieee80211_hw *hw)
if (ath_startrecv(sc) != 0) {
ath_print(common, ATH_DBG_FATAL,
"Unable to restart recv logic\n");
+ spin_unlock_bh(&sc->rx.pcu_lock);
return;
}
+ spin_unlock_bh(&sc->rx.pcu_lock);
if (sc->sc_flags & SC_OP_BEACONS)
ath_beacon_config(sc, NULL); /* restart beacons */
@@ -941,6 +951,9 @@ void ath_radio_disable(struct ath_softc *sc, struct ieee80211_hw *hw)
ath9k_hw_set_interrupts(ah, 0);
ath_drain_all_txq(sc, false); /* clear pending tx frames */
+
+ spin_lock_bh(&sc->rx.pcu_lock);
+
ath_stoprecv(sc); /* turn off frame recv */
ath_flushrecv(sc); /* flush recv queue */
@@ -958,6 +971,9 @@ void ath_radio_disable(struct ath_softc *sc, struct ieee80211_hw *hw)
spin_unlock_bh(&sc->sc_resetlock);
ath9k_hw_phy_disable(ah);
+
+ spin_unlock_bh(&sc->rx.pcu_lock);
+
ath9k_hw_configpcipowersave(ah, 1, 1);
ath9k_ps_restore(sc);
ath9k_setpower(sc, ATH9K_PM_FULL_SLEEP);
@@ -977,6 +993,9 @@ int ath_reset(struct ath_softc *sc, bool retry_tx)
ath9k_hw_set_interrupts(ah, 0);
ath_drain_all_txq(sc, retry_tx);
+
+ spin_lock_bh(&sc->rx.pcu_lock);
+
ath_stoprecv(sc);
ath_flushrecv(sc);
@@ -991,6 +1010,8 @@ int ath_reset(struct ath_softc *sc, bool retry_tx)
ath_print(common, ATH_DBG_FATAL,
"Unable to start recv logic\n");
+ spin_unlock_bh(&sc->rx.pcu_lock);
+
/*
* We may be doing a reset in response to a request
* that changes the channel so update any state that
@@ -1155,6 +1176,7 @@ static int ath9k_start(struct ieee80211_hw *hw)
* be followed by initialization of the appropriate bits
* and then setup of the interrupt mask.
*/
+ spin_lock_bh(&sc->rx.pcu_lock);
spin_lock_bh(&sc->sc_resetlock);
r = ath9k_hw_reset(ah, init_channel, ah->caldata, false);
if (r) {
@@ -1163,6 +1185,7 @@ static int ath9k_start(struct ieee80211_hw *hw)
"(freq %u MHz)\n", r,
curchan->center_freq);
spin_unlock_bh(&sc->sc_resetlock);
+ spin_unlock_bh(&sc->rx.pcu_lock);
goto mutex_unlock;
}
spin_unlock_bh(&sc->sc_resetlock);
@@ -1184,8 +1207,10 @@ static int ath9k_start(struct ieee80211_hw *hw)
ath_print(common, ATH_DBG_FATAL,
"Unable to start recv logic\n");
r = -EIO;
+ spin_unlock_bh(&sc->rx.pcu_lock);
goto mutex_unlock;
}
+ spin_unlock_bh(&sc->rx.pcu_lock);
/* Setup our intr mask. */
ah->imask = ATH9K_INT_TX | ATH9K_INT_RXEOL |
@@ -1386,12 +1411,14 @@ static void ath9k_stop(struct ieee80211_hw *hw)
* before setting the invalid flag. */
ath9k_hw_set_interrupts(ah, 0);
+ spin_lock_bh(&sc->rx.pcu_lock);
if (!(sc->sc_flags & SC_OP_INVALID)) {
ath_drain_all_txq(sc, false);
ath_stoprecv(sc);
ath9k_hw_phy_disable(ah);
} else
sc->rx.rxlink = NULL;
+ spin_unlock_bh(&sc->rx.pcu_lock);
/* disable HAL and put h/w to sleep */
ath9k_hw_disable(ah);
diff --git a/drivers/net/wireless/ath/ath9k/recv.c b/drivers/net/wireless/ath/ath9k/recv.c
index fe73fc5..128035c 100644
--- a/drivers/net/wireless/ath/ath9k/recv.c
+++ b/drivers/net/wireless/ath/ath9k/recv.c
@@ -297,19 +297,17 @@ static void ath_edma_start_recv(struct ath_softc *sc)
ath_rx_addbuffer_edma(sc, ATH9K_RX_QUEUE_LP,
sc->rx.rx_edma[ATH9K_RX_QUEUE_LP].rx_fifo_hwsize);
- spin_unlock_bh(&sc->rx.rxbuflock);
-
ath_opmode_init(sc);
ath9k_hw_startpcureceive(sc->sc_ah, (sc->sc_flags & SC_OP_OFFCHANNEL));
+
+ spin_unlock_bh(&sc->rx.rxbuflock);
}
static void ath_edma_stop_recv(struct ath_softc *sc)
{
- spin_lock_bh(&sc->rx.rxbuflock);
ath_rx_remove_buffer(sc, ATH9K_RX_QUEUE_HP);
ath_rx_remove_buffer(sc, ATH9K_RX_QUEUE_LP);
- spin_unlock_bh(&sc->rx.rxbuflock);
}
int ath_rx_init(struct ath_softc *sc, int nbufs)
@@ -319,8 +317,8 @@ int ath_rx_init(struct ath_softc *sc, int nbufs)
struct ath_buf *bf;
int error = 0;
- spin_lock_init(&sc->rx.rxflushlock);
sc->sc_flags &= ~SC_OP_RXFLUSH;
+ spin_lock_init(&sc->rx.pcu_lock);
spin_lock_init(&sc->rx.rxbuflock);
if (sc->sc_ah->caps.hw_caps & ATH9K_HW_CAP_EDMA) {
@@ -506,9 +504,9 @@ int ath_startrecv(struct ath_softc *sc)
ath9k_hw_rxena(ah);
start_recv:
- spin_unlock_bh(&sc->rx.rxbuflock);
ath_opmode_init(sc);
ath9k_hw_startpcureceive(ah, (sc->sc_flags & SC_OP_OFFCHANNEL));
+ spin_unlock_bh(&sc->rx.rxbuflock);
return 0;
}
@@ -518,6 +516,7 @@ bool ath_stoprecv(struct ath_softc *sc)
struct ath_hw *ah = sc->sc_ah;
bool stopped;
+ spin_lock_bh(&sc->rx.rxbuflock);
ath9k_hw_stoppcurecv(ah);
ath9k_hw_setrxfilter(ah, 0);
stopped = ath9k_hw_stopdmarecv(ah);
@@ -526,19 +525,19 @@ bool ath_stoprecv(struct ath_softc *sc)
ath_edma_stop_recv(sc);
else
sc->rx.rxlink = NULL;
+ spin_unlock_bh(&sc->rx.rxbuflock);
+ WARN_ON(!stopped);
return stopped;
}
void ath_flushrecv(struct ath_softc *sc)
{
- spin_lock_bh(&sc->rx.rxflushlock);
sc->sc_flags |= SC_OP_RXFLUSH;
if (sc->sc_ah->caps.hw_caps & ATH9K_HW_CAP_EDMA)
ath_rx_tasklet(sc, 1, true);
ath_rx_tasklet(sc, 1, false);
sc->sc_flags &= ~SC_OP_RXFLUSH;
- spin_unlock_bh(&sc->rx.rxflushlock);
}
static bool ath_beacon_dtim_pending_cab(struct sk_buff *skb)
@@ -663,6 +662,15 @@ static void ath_rx_send_to_mac80211(struct ieee80211_hw *hw,
struct ieee80211_rx_status *rxs)
{
struct ieee80211_hdr *hdr;
+ struct sk_buff *tmp_skb;
+ unsigned int j;
+
+ if (0) {
+ for (j=0; j < 5; j++) {
+ tmp_skb = skb_copy(skb, GFP_ATOMIC);
+ dev_kfree_skb_any(tmp_skb);
+ }
+ }
hdr = (struct ieee80211_hdr *)skb->data;
On Thu, Oct 7, 2010 at 2:31 PM, Luis R. Rodriguez <[email protected]> wrote:
> On Thu, Oct 7, 2010 at 12:27 PM, Johannes Berg
> <[email protected]> wrote:
>> On Thu, 2010-10-07 at 12:22 -0700, Ben Greear wrote:
>>
>>> After reboot, and re-run of the script,
>>> I saw this in the logs, and shortly after,
>>> the SLUB poison warning dumped to screen.
>>>
>>> Maybe those DMA errors are serious?
>>
>>> ath: Failed to stop TX DMA in 100 msec after killing last frame
>>> ath: Failed to stop TX DMA. Resetting hardware!
>>
>> That's TX DMA, it can hardly result in invalid memory writes like the
>> ones you've been seeing.
>>
>> I'm still convinced something is wrong with ath9k RX DMA, as you've seen
>> the contents of frames written to already freed memory regions. Since I
>> don't know anything about ath9k, you should probably not rely on me
>> though :-)
>
> I'm on this now. Lets play.
>
> I had to remove /lib/udev/rules.d/75-persistent-net-generator.rules
> to avoid Ubuntu trying to remember the device names and it creating
> stax_rename names.
> I just ran your script with some modifications. You can find it here:
>
> http://www.kernel.org/pub/linux/kernel/people/mcgrof/scripts/poo.pl
>
> I then ran:
>
> for i in $(seq 0 31) ; do sudo dhclient seq$i; done
>
> It took about 10 minutes to get IP addresses for all interfaces but it
> got there eventually. Odd enough I was unable to ping the AP from any
> interface though. Not sure what that was about. But I got no oops, no
> slub dump. All I got was some more delba warnings which seems to
> indicate we haven't caught all the cases for its use:
>
> [ 3622.660344] addBA response timer expired on tid 0
> [ 3622.660373] Tx BA session stop requested for 68:7f:74:3b:b1:0f tid 0
> [ 3622.680133] addBA response timer expired on tid 0
> [ 3622.687196] Tx BA session stop requested for 68:7f:74:3b:b1:0f tid 0
> [ 3623.110077] addBA response timer expired on tid 0
> [ 3623.110123] Tx BA session stop requested for 68:7f:74:3b:b1:0f tid 0
> [ 3628.935895] sta10: authenticate with 68:7f:74:3b:b1:10 (try 1)
> [ 3628.937194] switched off addBA timer for tid 0
> [ 3628.937196] Aggregation is on for tid 0
> [ 3628.937239] Stopping Tx BA session for 68:7f:74:3b:b1:0f tid 0
> [ 3628.937243] ------------[ cut here ]------------
> [ 3628.937263] WARNING: at include/net/mac80211.h:2694
> rate_control_send_low+0xd3/0x140 [mac80211]()
> [ 3628.937265] Hardware name: 6460DWU
> [ 3628.937266] Modules linked in: binfmt_misc ppdev
> snd_hda_codec_analog rfcomm sco bridge joydev stp bnep l2cap nouveau
> ath9k snd_hda_intel mac80211 snd_hda_codec snd_hwdep snd_pcm ttm btusb
> ath9k_common thinkpad_acpi ath9k_hw bluetooth drm_kms_helper
> snd_seq_midi snd_rawmidi pcmcia snd_seq_midi_event drm snd_seq ath
> snd_timer snd_seq_device tpm_tis i2c_algo_bit cfg80211 snd nvram tpm
> tpm_bios yenta_socket pcmcia_rsrc video psmouse output pcmcia_core
> serio_raw soundcore snd_page_alloc intel_agp lp parport ohci1394
> e1000e ieee1394 ahci libahci
> [ 3628.937307] Pid: 49, comm: kworker/u:3 Tainted: G W
> 2.6.36-rc6-wl+ #263
> [ 3628.937310] Call Trace:
> [ 3628.937317] [<ffffffff8105ffcf>] warn_slowpath_common+0x7f/0xc0
> [ 3628.937320] [<ffffffff8106002a>] warn_slowpath_null+0x1a/0x20
> [ 3628.937329] [<ffffffffa032f603>] rate_control_send_low+0xd3/0x140 [mac80211]
> [ 3628.937336] [<ffffffffa038bfd8>] ath_get_rate+0x48/0x570 [ath9k]
> [ 3628.937340] [<ffffffff812b9f39>] ? put_dec+0x59/0x60
> [ 3628.937349] [<ffffffffa032f42e>] rate_control_get_rate+0x8e/0x190 [mac80211]
> [ 3628.937360] [<ffffffffa0339928>]
> ieee80211_tx_h_rate_ctrl+0x1a8/0x4e0 [mac80211]
> [ 3628.937370] [<ffffffffa033a000>] invoke_tx_handlers+0x100/0x140 [mac80211]
> [ 3628.937379] [<ffffffffa033a0c5>] ieee80211_tx+0x85/0x240 [mac80211]
> [ 3628.937384] [<ffffffff8147b890>] ? skb_release_data+0xd0/0xe0
> [ 3628.937386] [<ffffffff8147d72f>] ? pskb_expand_head+0x10f/0x1a0
> [ 3628.937397] [<ffffffffa033a336>] ieee80211_xmit+0xb6/0x1d0 [mac80211]
> [ 3628.937399] [<ffffffff8147b9d3>] ? __alloc_skb+0x83/0x170
> [ 3628.937409] [<ffffffffa033a4a4>] ieee80211_tx_skb+0x54/0x70 [mac80211]
> [ 3628.937418] [<ffffffffa03230ad>] ieee80211_send_delba+0x11d/0x190 [mac80211]
> [ 3628.937427] [<ffffffffa0323a18>]
> ieee80211_stop_tx_ba_cb+0x1b8/0x240 [mac80211]
> [ 3628.937431] [<ffffffff81036c89>] ? default_spin_lock_flags+0x9/0x10
> [ 3628.937440] [<ffffffffa032e031>] ieee80211_iface_work+0x271/0x340 [mac80211]
> [ 3628.937450] [<ffffffffa032ddc0>] ? ieee80211_iface_work+0x0/0x340 [mac80211]
> [ 3628.937453] [<ffffffff8107a203>] process_one_work+0x123/0x440
> [ 3628.937457] [<ffffffff8107c750>] worker_thread+0x170/0x400
> [ 3628.937460] [<ffffffff8107c5e0>] ? worker_thread+0x0/0x400
> [ 3628.937463] [<ffffffff81080b76>] kthread+0x96/0xa0
> [ 3628.937466] [<ffffffff8100bea4>] kernel_thread_helper+0x4/0x10
> [ 3628.937469] [<ffffffff81080ae0>] ? kthread+0x0/0xa0
> [ 3628.937472] [<ffffffff8100bea0>] ? kernel_thread_helper+0x0/0x10
> [ 3628.937474] ---[ end trace 9dd0d025ccb9b75c ]---
> [ 3628.937980] switched off addBA timer for tid 0
> [ 3628.937982] Aggregation is on for tid 0
>
> But other than this I got nothing. I left the box sit there for about
> 1 hour and came back and it was still going with no issues. Mind you,
> I can't ping but that seems like another issue.
>
> You can find my logs here:
>
> http://www.kernel.org/pub/linux/kernel/people/mcgrof/logs/2010/10-07-stress-sta-01/
Doh, I did not have CONFIG_SLUB_DEBUG_ON=y so building now with that.
Luis
On 10/07/2010 02:31 PM, Luis R. Rodriguez wrote:
> On Thu, Oct 7, 2010 at 12:27 PM, Johannes Berg
> <[email protected]> wrote:
>> On Thu, 2010-10-07 at 12:22 -0700, Ben Greear wrote:
>>
>>> After reboot, and re-run of the script,
>>> I saw this in the logs, and shortly after,
>>> the SLUB poison warning dumped to screen.
>>>
>>> Maybe those DMA errors are serious?
>>
>>> ath: Failed to stop TX DMA in 100 msec after killing last frame
>>> ath: Failed to stop TX DMA. Resetting hardware!
>>
>> That's TX DMA, it can hardly result in invalid memory writes like the
>> ones you've been seeing.
>>
>> I'm still convinced something is wrong with ath9k RX DMA, as you've seen
>> the contents of frames written to already freed memory regions. Since I
>> don't know anything about ath9k, you should probably not rely on me
>> though :-)
>
> I'm on this now. Lets play.
>
> I had to remove /lib/udev/rules.d/75-persistent-net-generator.rules
> to avoid Ubuntu trying to remember the device names and it creating
> stax_rename names.
Right, we disable udev for 'sta*' devices.
> I just ran your script with some modifications. You can find it here:
>
> http://www.kernel.org/pub/linux/kernel/people/mcgrof/scripts/poo.pl
Can you post your kernel .config somewhere, and confirm which
kernel you are using?
Also, what ath9k NIC, platform, etc?
We see the problem on two different systems (haven't tried more). I can
figure out the brands of the NICs if that helps, and have included
lspci information below.
I've uploaded my kernel config to here:
http://www.candelatech.com/~greearb/ctwl_kernel.cfg
* Dual core Intel Pentium-D 32-bit
2GB RAM
Fedora 13 (but with custom compiled top-of-tree iw, hostap, libnl
Atheros NIC: from lspci -vv:
08:01.0 Network controller: Atheros Communications Inc. AR922X Wireless Network Adapter (rev 01)
Subsystem: Device 0777:4002
Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR+ FastB2B+ DisINTx-
Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Interrupt: pin A routed to IRQ 17
Region 0: Memory at d0300000 (32-bit, non-prefetchable) [size=64K]
Capabilities: [44] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=100mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Kernel modules: ath9k
* Dual-core Intel Atom, 32-bit
1GB RAM
Fedora 13 (custom compiled top-of-tree iw, hostap, libnl)
Atheros mini-pcie NIC, from lspci -vv:
03:00.0 Network controller: Atheros Communications Inc. AR928X Wireless Network Adapter (PCI-Express) (rev 01)
Subsystem: Device 0777:4e05
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 19
Region 0: Memory at febf0000 (64-bit, non-prefetchable) [size=64K]
Capabilities: [40] Power Management version 2
Flags: PMEClk- DSI- D1+ D2- AuxCurrent=375mA PME(D0+,D1+,D2-,D3hot+,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit-
Address: 00000000 Data: 0000
Capabilities: [60] Express (v1) Legacy Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM unknown, Latency L0 <512ns, L1 <64us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 128 bytes Disabled- Retrain- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
Capabilities: [90] MSI-X: Enable- Count=1 Masked-
Vector table: BAR=0 offset=00000000
PBA: BAR=0 offset=00000000
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
AERCap: First Error Pointer: 14, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [140 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntrySize=0
Arb: Fixed- WRR32- WRR64- WRR128- 100ns- onfig- - - TableOffset=0
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- Fixed- RR32-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=01
Status: NegoPending- InProgress-
Capabilities: [160 v1] Device Serial Number 00-00-00-00-00-00-00-00
Kernel driver in use: ath9k
Kernel modules: ath9k
>
> I then ran:
>
> for i in $(seq 0 31) ; do sudo dhclient seq$i; done
>
> It took about 10 minutes to get IP addresses for all interfaces but it
> got there eventually. Odd enough I was unable to ping the AP from any
> interface though. Not sure what that was about. But I got no oops, no
> slub dump. All I got was some more delba warnings which seems to
> indicate we haven't caught all the cases for its use:
If you just create one or two interfaces, can you ping as expected?
Thanks,
Ben
--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
On Mon, Oct 11, 2010 at 1:51 PM, Ben Greear <[email protected]> wrote:
> On 10/07/2010 02:59 PM, Luis R. Rodriguez wrote:
>>
>> On Thu, Oct 7, 2010 at 2:36 PM, Luis R. Rodriguez<[email protected]>
>> wrote:
>
>>>> But other than this I got nothing. I left the box sit there for about
>>>> 1 hour and came back and it was still going with no issues. Mind you,
>>>> I can't ping but that seems like another issue.
>>>>
>>>> You can find my logs here:
>>>>
>>>>
>>>> http://www.kernel.org/pub/linux/kernel/people/mcgrof/logs/2010/10-07-stress-sta-01/
>>>
>>> Doh, I did not have CONFIG_SLUB_DEBUG_ON=y so building now with that.
>>
>> Yay I can reproduce now. I'll be back, going to dig now.
>
> Any luck tracking this down?
No, today for example I just finished reading e-mail and its already
6pm PST... But Friday I did get do do a lot of work and testing on
this. The only pattern I see so far is that skb_copy() is used on the
poison all the time. I am not sure if its because skb_copy() happens
to run the poison check or what. I'll work on this tomorrow.
Luis
On 10/13/2010 01:03 PM, Luis R. Rodriguez wrote:
> 2010/10/13 Björn Smedman<[email protected]>:
>> Hi Ben,
>>
>> First of all keep up the good work. :)
>>
>> On Wed, Oct 13, 2010 at 6:39 PM, Ben Greear<[email protected]> wrote:
>> [snip]
>>> Either way, it seems safer to null out the bf_ampdu field after
>>> the memory is consumed..it could prevent some tricky bugs later.
>>
>> I think this is a good idea. But it probably wont be enough to null
>> out bf_mpdu. You also need to look at bf_buf_addr (which if I
>> understand correctly is the physical address the DMA engine will
>> actually write RXed frames to) and bf_dmacontext (which seems in most
>> cases to hold an identical address and may in fact be where the DMA
>> engine will really write the frame).
>
> See the TODO list for ath9k:
>
> * Remove this redundancy crap: bf->bf_buf_addr = bf->bf_dmacontext;
>
> http://wireless.kernel.org/en/users/Drivers/ath9k/todo
I just posted a patch that attempts this. It doesn't
fix the memory clobber issue, but maybe the code is a bit cleaner/safer
at least...
Thanks,
Ben
--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
On 10/12/2010 10:31 PM, Vasanthakumar Thiagarajan wrote:
> On Wed, Oct 13, 2010 at 12:05:53AM +0530, Ben Greear wrote:
>> On 10/11/2010 11:10 PM, Luis R. Rodriguez wrote:
>>> On Mon, Oct 11, 2010 at 8:27 PM, Ben Greear<[email protected]> wrote:
>>
>>>> Another thing I was thinking about: Maybe the queue of skbs and dma
>>>> addresses
>>>> in ath9k is getting corrupted by multiple VIFs trying to write at once?
>>>> Maybe
>>>> some locking is needed in the xmit path?
>>>
>>> That was my second hunch. My first shot was to use spin_lock_irqsave()
>>> over the the uses of the rxbuf list and that seemed to help but I
>>> still managed to get a poison eventually. My next item to check for is
>>> of the permissibility of creating too much pressure to the point we
>>> end up looping over the rxbuf list and race against mac80211 free'ing
>>> a buffer. Will test that tomorrow if nothing else comes up creeping my
>>> priority queue.
>>
>> This code looks weird to me. One of the paprd branches
>> deletes the skb, the other doesn't appear to. Neither
>> null out bf->bf_mpdu, which would appear to leave a dangling
>> pointer in at least the dev_kfree_skb_any() branch.
>
> Single skb is (re)used for sending paprd training frames on more
> than one chains. This skb needs to be freed only when paprd fails on
> any of the chains or it succeeded on all the chains. The failure
> case is handled in ath_tx_complete_buf() and success case is in
> ath_paprd_calibrate().
>>
>> ath_tx_complete frees it's skb in all cases, so another
>> bf->bf_mpdu dangling pointer issue.
>>
>> Maybe at the least we should null out bf->bf_mpdu when
>> skb is consumed?
>
> I dont see any point in NULLing out bf->bf_mpdu. bf is
> reclaimed onto a free tx buf pool as soon as it is done
> with the skb. bf_mpdu of any of the bf's is never accessed
> without any initialization (bf_ampdu = skb).
The code can use skb after its deleted currently, because
ath_debug_stat_tx(sc, txq, bf, ts); references the bf_ampdu
object (I think I added that reference lately..so it's really
a bug that I caused). At the least, we should move the ath_debug_stat_tx
logic before the ath_tx_complete() call.
As for the paprd path, it looks racy to me: What if the paprd timer
expires while the ath_tx_complete_buf logic is running?
Either way, it seems safer to null out the bf_ampdu field after
the memory is consumed..it could prevent some tricky bugs later.
Thanks,
Ben
--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
On 10/14/2010 02:52 PM, Bj?rn Smedman wrote:
> 2010/10/13 Bj?rn Smedman<[email protected]>:
>> Hi Ben,
>>
>> First of all keep up the good work. :)
>>
>> On Wed, Oct 13, 2010 at 6:39 PM, Ben Greear<[email protected]> wrote:
>> [snip]
>>> Either way, it seems safer to null out the bf_ampdu field after
>>> the memory is consumed..it could prevent some tricky bugs later.
>>
>> I think this is a good idea. But it probably wont be enough to null
>> out bf_mpdu. You also need to look at bf_buf_addr (which if I
>> understand correctly is the physical address the DMA engine will
>> actually write RXed frames to) and bf_dmacontext (which seems in most
>> cases to hold an identical address and may in fact be where the DMA
>> engine will really write the frame).
>
> I took another look at the code. It turns out both bf_buf_addr and
> bf_dmacontext are in fact meaningless to the DMA. Instead each bf
> holds a pointer (bf_desc) to the real DMA descriptor which in turn
> holds the address (ds_data) where the DMA will really (really this
> time) write the frame. There is also a field to hold the virtual
> address of the same place (ds_vdata).
>
> It's a little too much work for me to set up the testbed you have Ben
> but would be interesting to see what happens if you set
> bf->bf_desc->ds_{data,vdata} = 0 as well. No?
I tried the patch below, and it didn't seem to help. Might even
have hurt..as it died on divide-by-zero error:
Call Trace:
[<c075e490>] ? printk+0xf/0x17
[<c075e37e>] panic+0x50/0x153
[<c07619db>] oops_end+0x92/0xa1
[<c04051cc>] die+0x53/0x59
[<c07612a3>] do_trap+0x89/0xa2
[<c0403b12>] ? do_divide_error+0x0/0x78
[<c0403b80>] do_divide_error+0x6e/0x78
[<faef811e>] ? ath9k_hw_ani_monitor+0x37/0x40e [ath9k_hw]
[<fb1f77d9>] ? ath9k_ioread32+0x25/0x5b [ath9k]
[<c045553a>] ? trace_hardirqs_off+0xb/0xd
[<c0581210>] ? trace_hardirqs_off_thunk+0xc/0x10
[<c076103f>] error_code+0x5f/0x70
[<c0403b12>] ? do_divide_error+0x0/0x78
[<faef811e>] ? ath9k_hw_ani_monitor+0x37/0x40e [ath9k_hw]
[<fb1fa783>] ath_ani_calibrate+0x143/0x20b [ath9k]
[<c043d58f>] run_timer_softirq+0x14f/0x1e7
That might be an existing bug, however...
Thanks,
Ben
diff --git a/drivers/net/wireless/ath/ath9k/ath9k.h b/drivers/net/wireless/ath/ath9k/ath9k.h
index 0c917e5..8fba13d 100644
--- a/drivers/net/wireless/ath/ath9k/ath9k.h
+++ b/drivers/net/wireless/ath/ath9k/ath9k.h
@@ -737,4 +737,6 @@ bool ath_mac80211_start_queue(struct ath_softc *sc, u16 skb_queue);
void ath_start_rfkill_poll(struct ath_softc *sc);
extern void ath9k_rfkill_poll_state(struct ieee80211_hw *hw);
+void ath_clear_dma_ptrs(struct ath_buf *bf);
+
#endif /* ATH9K_H */
diff --git a/drivers/net/wireless/ath/ath9k/beacon.c b/drivers/net/wireless/ath/ath9k/beacon.c
index 97d471f..cc406f9 100644
--- a/drivers/net/wireless/ath/ath9k/beacon.c
+++ b/drivers/net/wireless/ath/ath9k/beacon.c
@@ -139,7 +139,7 @@ static struct ath_buf *ath_beacon_generate(struct ieee80211_hw *hw,
dma_unmap_single(sc->dev, bf->bf_buf_addr,
skb->len, DMA_TO_DEVICE);
dev_kfree_skb_any(skb);
- bf->bf_buf_addr = 0;
+ ath_clear_dma_ptrs(bf);
}
/* Get a new beacon from mac80211 */
@@ -167,8 +167,7 @@ static struct ath_buf *ath_beacon_generate(struct ieee80211_hw *hw,
skb->len, DMA_TO_DEVICE);
if (unlikely(dma_mapping_error(sc->dev, bf->bf_buf_addr))) {
dev_kfree_skb_any(skb);
- bf->bf_mpdu = NULL;
- bf->bf_buf_addr = 0;
+ ath_clear_dma_ptrs(bf);
ath_print(common, ATH_DBG_FATAL,
"dma_mapping_error on beaconing\n");
return NULL;
@@ -256,8 +255,7 @@ int ath_beacon_alloc(struct ath_wiphy *aphy, struct ieee80211_vif *vif)
dma_unmap_single(sc->dev, bf->bf_buf_addr,
skb->len, DMA_TO_DEVICE);
dev_kfree_skb_any(skb);
- bf->bf_mpdu = NULL;
- bf->bf_buf_addr = 0;
+ ath_clear_dma_ptrs(bf);
}
/* NB: the beacon data buffer must be 32-bit aligned. */
@@ -302,8 +300,7 @@ int ath_beacon_alloc(struct ath_wiphy *aphy, struct ieee80211_vif *vif)
skb->len, DMA_TO_DEVICE);
if (unlikely(dma_mapping_error(sc->dev, bf->bf_buf_addr))) {
dev_kfree_skb_any(skb);
- bf->bf_mpdu = NULL;
- bf->bf_buf_addr = 0;
+ ath_clear_dma_ptrs(bf);
ath_print(common, ATH_DBG_FATAL,
"dma_mapping_error on beacon alloc\n");
return -ENOMEM;
@@ -329,8 +326,7 @@ void ath_beacon_return(struct ath_softc *sc, struct ath_vif *avp)
dma_unmap_single(sc->dev, bf->bf_buf_addr,
skb->len, DMA_TO_DEVICE);
dev_kfree_skb_any(skb);
- bf->bf_mpdu = NULL;
- bf->bf_buf_addr = 0;
+ ath_clear_dma_ptrs(bf);
}
list_add_tail(&bf->list, &sc->beacon.bbuf);
diff --git a/drivers/net/wireless/ath/ath9k/main.c b/drivers/net/wireless/ath/ath9k/main.c
index bcd3892..1722a21 100644
--- a/drivers/net/wireless/ath/ath9k/main.c
+++ b/drivers/net/wireless/ath/ath9k/main.c
@@ -213,6 +213,17 @@ static void ath_update_survey_stats(struct ath_softc *sc)
ath_update_survey_nf(sc, pos);
}
+void ath_clear_dma_ptrs(struct ath_buf *bf)
+{
+ struct ath_desc *ds = bf->bf_desc;
+ bf->bf_buf_addr = 0;
+ bf->bf_mpdu = NULL;
+ if (ds) {
+ ds->ds_data = 0;
+ ds->ds_vdata = NULL;
+ }
+}
+
/*
* Set/change channels. If the channel is really being changed, it's done
* by reseting the chip. To accomplish this we must first cleanup any pending
diff --git a/drivers/net/wireless/ath/ath9k/recv.c b/drivers/net/wireless/ath/ath9k/recv.c
index fd778d2..5afb46f 100644
--- a/drivers/net/wireless/ath/ath9k/recv.c
+++ b/drivers/net/wireless/ath/ath9k/recv.c
@@ -269,8 +269,7 @@ static int ath_rx_edma_init(struct ath_softc *sc, int nbufs)
if (unlikely(dma_mapping_error(sc->dev,
bf->bf_buf_addr))) {
dev_kfree_skb_any(skb);
- bf->bf_mpdu = NULL;
- bf->bf_buf_addr = 0;
+ ath_clear_dma_ptrs(bf);
ath_print(common, ATH_DBG_FATAL,
"dma_mapping_error() on RX init\n");
error = -ENOMEM;
@@ -360,8 +359,7 @@ int ath_rx_init(struct ath_softc *sc, int nbufs)
if (unlikely(dma_mapping_error(sc->dev,
bf->bf_buf_addr))) {
dev_kfree_skb_any(skb);
- bf->bf_mpdu = NULL;
- bf->bf_buf_addr = 0;
+ ath_clear_dma_ptrs(bf);
ath_print(common, ATH_DBG_FATAL,
"dma_mapping_error() on RX init\n");
error = -ENOMEM;
@@ -396,8 +394,7 @@ void ath_rx_cleanup(struct ath_softc *sc)
common->rx_bufsize,
DMA_FROM_DEVICE);
dev_kfree_skb(skb);
- bf->bf_buf_addr = 0;
- bf->bf_mpdu = NULL;
+ ath_clear_dma_ptrs(bf);
}
}
@@ -1741,8 +1738,7 @@ int ath_rx_tasklet(struct ath_softc *sc, int flush, bool hp)
if (unlikely(dma_mapping_error(sc->dev,
bf->bf_buf_addr))) {
dev_kfree_skb_any(requeue_skb);
- bf->bf_mpdu = NULL;
- bf->bf_buf_addr = 0;
+ ath_clear_dma_ptrs(bf);
ath_print(common, ATH_DBG_FATAL,
"dma_mapping_error() on RX\n");
ath_rx_send_to_mac80211(hw, sc, skb, rxs);
diff --git a/drivers/net/wireless/ath/ath9k/xmit.c b/drivers/net/wireless/ath/ath9k/xmit.c
index a5e5f27..e86f59c 100644
--- a/drivers/net/wireless/ath/ath9k/xmit.c
+++ b/drivers/net/wireless/ath/ath9k/xmit.c
@@ -1644,8 +1644,7 @@ static int ath_tx_setup_buffer(struct ieee80211_hw *hw, struct ath_buf *bf,
bf->bf_buf_addr = dma_map_single(sc->dev, skb->data,
skb->len, DMA_TO_DEVICE);
if (unlikely(dma_mapping_error(sc->dev, bf->bf_buf_addr))) {
- bf->bf_mpdu = NULL;
- bf->bf_buf_addr = 0;
+ ath_clear_dma_ptrs(bf);
ath_print(ath9k_hw_common(sc->sc_ah), ATH_DBG_FATAL,
"dma_mapping_error() on TX\n");
return -ENOMEM;
@@ -1915,7 +1914,6 @@ static void ath_tx_complete_buf(struct ath_softc *sc, struct ath_buf *bf,
}
dma_unmap_single(sc->dev, bf->bf_buf_addr, skb->len, DMA_TO_DEVICE);
- bf->bf_buf_addr = 0;
if (bf->bf_state.bfs_paprd) {
if (time_after(jiffies,
@@ -1931,7 +1929,7 @@ static void ath_tx_complete_buf(struct ath_softc *sc, struct ath_buf *bf,
/* At this point, skb (bf->bf_mpdu) is consumed...make sure we don't
* accidentally reference it later.
*/
- bf->bf_mpdu = NULL;
+ ath_clear_dma_ptrs(bf);
/*
* Return the list of ath_buf of this mpdu to free queue
>
> /Bj?rn
> --
> To unsubscribe from this list: send the line "unsubscribe linux-wireless" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
On Tue, Oct 5, 2010 at 10:00 AM, Ben Greear <[email protected]> wrote:
> I started seeing this very soon after creating interfaces.
Can you be more specific how one can reproduce?
Luis
On Wed, Oct 13, 2010 at 12:05:53AM +0530, Ben Greear wrote:
> On 10/11/2010 11:10 PM, Luis R. Rodriguez wrote:
> > On Mon, Oct 11, 2010 at 8:27 PM, Ben Greear<[email protected]> wrote:
>
> >> Another thing I was thinking about: Maybe the queue of skbs and dma
> >> addresses
> >> in ath9k is getting corrupted by multiple VIFs trying to write at once?
> >> Maybe
> >> some locking is needed in the xmit path?
> >
> > That was my second hunch. My first shot was to use spin_lock_irqsave()
> > over the the uses of the rxbuf list and that seemed to help but I
> > still managed to get a poison eventually. My next item to check for is
> > of the permissibility of creating too much pressure to the point we
> > end up looping over the rxbuf list and race against mac80211 free'ing
> > a buffer. Will test that tomorrow if nothing else comes up creeping my
> > priority queue.
>
> This code looks weird to me. One of the paprd branches
> deletes the skb, the other doesn't appear to. Neither
> null out bf->bf_mpdu, which would appear to leave a dangling
> pointer in at least the dev_kfree_skb_any() branch.
Single skb is (re)used for sending paprd training frames on more
than one chains. This skb needs to be freed only when paprd fails on
any of the chains or it succeeded on all the chains. The failure
case is handled in ath_tx_complete_buf() and success case is in
ath_paprd_calibrate().
>
> ath_tx_complete frees it's skb in all cases, so another
> bf->bf_mpdu dangling pointer issue.
>
> Maybe at the least we should null out bf->bf_mpdu when
> skb is consumed?
I dont see any point in NULLing out bf->bf_mpdu. bf is
reclaimed onto a free tx buf pool as soon as it is done
with the skb. bf_mpdu of any of the bf's is never accessed
without any initialization (bf_ampdu = skb).
Vasanth
On Thu, Oct 7, 2010 at 11:39 AM, Ben Greear <[email protected]> wrote:
> On 10/07/2010 11:29 AM, Luis R. Rodriguez wrote:
>>
>> On Thu, Oct 7, 2010 at 11:14 AM, Johannes Berg
>> <[email protected]> wrote:
>>>
>>> On Thu, 2010-10-07 at 10:33 -0700, Ben Greear wrote:
>>>>
>>>> In case it helps, here is a dump of where the corrupted SKB was deleted.
>>>
>>> I wonder, do you have a machine with a decent IOMMU? Adding IOMMU
>>> debugging into the mix could help you figure out if it's a DMA problem.
>>
>> Ben, how much traffic are you RX'ing on these virtual interfaces?
>
> I disabled my user-space application, and this script alone can reproduce
> the problem fairly quickly on my system. You will need to change some
> of those first variables. Just start it and wait a few minutes and
> watch the splats show on the console :)
>
> Note that I am not generating any traffic, but the wpa_supplicants are
> doing their thing of course...
>
> I'm using the kernel found here:
> http://dmz2.candelatech.com/git/gitweb.cgi?p=linux.wireless-testing.ct/.git;a=summary
>
> It's latest wireless-testing with some of my own patches, and some
> I've gathered from here an there. I doubt I'm causing this problem,
> but if you can't reproduce it with this script on your kernels,
> I can try with base wireless-testing or whatever you are using.
I'll run this now, but can you try a vanilla wireless-testing? I hear
the latest wireless-testing is borked so maybe try (git reset --hard
master-2010-09-29), its what I'm on.
Luis
On 10/07/2010 02:59 PM, Luis R. Rodriguez wrote:
> On Thu, Oct 7, 2010 at 2:36 PM, Luis R. Rodriguez<[email protected]> wrote:
>>> But other than this I got nothing. I left the box sit there for about
>>> 1 hour and came back and it was still going with no issues. Mind you,
>>> I can't ping but that seems like another issue.
>>>
>>> You can find my logs here:
>>>
>>> http://www.kernel.org/pub/linux/kernel/people/mcgrof/logs/2010/10-07-stress-sta-01/
>>
>> Doh, I did not have CONFIG_SLUB_DEBUG_ON=y so building now with that.
>
> Yay I can reproduce now. I'll be back, going to dig now.
Any luck tracking this down?
Thanks,
Ben
--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
2010/10/18 Björn Smedman <[email protected]>:
> 2010/10/15 Björn Smedman <[email protected]>
>>
>> 2010/10/15 Ben Greear <[email protected]>:
>> > I tried the patch below, and it didn't seem to help. Might even
>> > have hurt..as it died on divide-by-zero error:
>>
>> Hmm, looks like the ani code got a zero listen time from the hw...
>> That just might mean that the DMA actually hits one of these
>> descriptors. :)
>
> Am I the only one worried about this? Leaving a DMA descriptor
> pointing at memory which has been passed on to somebody else... To me
> that's like pointing a loaded gun at someone (and it seems this
> particular gun can go off a little haphazardly).
>
> Luis, given how hard it seems to be to get that locking and skb
> shoveling right, are you sure you want to keep pointing that DMA
> engine on innocent people's data?
This is why this issue is of high priority to me. I no longer get the
poison nor does Ben, the RX poison issue is resolved as far as I can
tell, I just need to split up the patches into easily reviewable
chunks and get them upstream.
Luis
2010/10/14 Ben Greear <[email protected]>:
> On 10/14/2010 02:52 PM, Björn Smedman wrote:
>>
>> 2010/10/13 Björn Smedman<[email protected]>:
>>>
>>> Hi Ben,
>>>
>>> First of all keep up the good work. :)
>>>
>>> On Wed, Oct 13, 2010 at 6:39 PM, Ben Greear<[email protected]>
>>> wrote:
>>> [snip]
>>>>
>>>> Either way, it seems safer to null out the bf_ampdu field after
>>>> the memory is consumed..it could prevent some tricky bugs later.
>>>
>>> I think this is a good idea. But it probably wont be enough to null
>>> out bf_mpdu. You also need to look at bf_buf_addr (which if I
>>> understand correctly is the physical address the DMA engine will
>>> actually write RXed frames to) and bf_dmacontext (which seems in most
>>> cases to hold an identical address and may in fact be where the DMA
>>> engine will really write the frame).
>>
>> I took another look at the code. It turns out both bf_buf_addr and
>> bf_dmacontext are in fact meaningless to the DMA. Instead each bf
>> holds a pointer (bf_desc) to the real DMA descriptor which in turn
>> holds the address (ds_data) where the DMA will really (really this
>> time) write the frame. There is also a field to hold the virtual
>> address of the same place (ds_vdata).
>>
>> It's a little too much work for me to set up the testbed you have Ben
>> but would be interesting to see what happens if you set
>> bf->bf_desc->ds_{data,vdata} = 0 as well. No?
>
> I'll investigate those suggestions.
>
> But setting up a test-bed is as easy
> as getting an ath9k NIC in a system, with a few APs around, and run the
> script below.
>
> You do not need any traffic generation, dhcp, etc...seems just beacons and
> whatever
> wpa_supplicant is doing is enough to hit the problem fast. (Make sure
> you are compiled to detect memory poisoning, of course).
>
> You'll need to fix the paths to the executables most likely.
>
You don't need such complicated scripts, I've managed to reproduce now
by creating a lot of monitor interfaces and then looping with a
regular interface issuing a scan command over and over. I suspect
I'll be able to do this as well by changing channels instead of doing
a scan. I believe the issue may be due to races in hardware on resets
and enabling RX on an already freed buffer.
Luis
On Tue, Oct 12, 2010 at 11:35 AM, Ben Greear <[email protected]> wrote:
> On 10/11/2010 11:10 PM, Luis R. Rodriguez wrote:
>>
>> On Mon, Oct 11, 2010 at 8:27 PM, Ben Greear<[email protected]>
>> wrote:
>
>>> Another thing I was thinking about: Maybe the queue of skbs and dma
>>> addresses
>>> in ath9k is getting corrupted by multiple VIFs trying to write at once?
>>> Maybe
>>> some locking is needed in the xmit path?
>>
>> That was my second hunch. My first shot was to use spin_lock_irqsave()
>> over the the uses of the rxbuf list and that seemed to help but I
>> still managed to get a poison eventually. My next item to check for is
>> of the permissibility of creating too much pressure to the point we
>> end up looping over the rxbuf list and race against mac80211 free'ing
>> a buffer. Will test that tomorrow if nothing else comes up creeping my
>> priority queue.
>
> This code looks weird to me. One of the paprd branches
> deletes the skb, the other doesn't appear to. Neither
> null out bf->bf_mpdu, which would appear to leave a dangling
> pointer in at least the dev_kfree_skb_any() branch.
>
> ath_tx_complete frees it's skb in all cases, so another
> bf->bf_mpdu dangling pointer issue.
>
> Maybe at the least we should null out bf->bf_mpdu when
> skb is consumed?
You're reading my mind, that was what I was going to test today. Still
doing e-mail sweep though.
Luis
On 10/07/2010 11:42 AM, Luis R. Rodriguez wrote:
> On Thu, Oct 7, 2010 at 11:39 AM, Ben Greear<[email protected]> wrote:
>> On 10/07/2010 11:29 AM, Luis R. Rodriguez wrote:
>>>
>>> On Thu, Oct 7, 2010 at 11:14 AM, Johannes Berg
>>> <[email protected]> wrote:
>>>>
>>>> On Thu, 2010-10-07 at 10:33 -0700, Ben Greear wrote:
>>>>>
>>>>> In case it helps, here is a dump of where the corrupted SKB was deleted.
>>>>
>>>> I wonder, do you have a machine with a decent IOMMU? Adding IOMMU
>>>> debugging into the mix could help you figure out if it's a DMA problem.
>>>
>>> Ben, how much traffic are you RX'ing on these virtual interfaces?
>>
>> I disabled my user-space application, and this script alone can reproduce
>> the problem fairly quickly on my system. You will need to change some
>> of those first variables. Just start it and wait a few minutes and
>> watch the splats show on the console :)
>>
>> Note that I am not generating any traffic, but the wpa_supplicants are
>> doing their thing of course...
>>
>> I'm using the kernel found here:
>> http://dmz2.candelatech.com/git/gitweb.cgi?p=linux.wireless-testing.ct/.git;a=summary
>>
>> It's latest wireless-testing with some of my own patches, and some
>> I've gathered from here an there. I doubt I'm causing this problem,
>> but if you can't reproduce it with this script on your kernels,
>> I can try with base wireless-testing or whatever you are using.
>
> I'll run this now, but can you try a vanilla wireless-testing? I hear
> the latest wireless-testing is borked so maybe try (git reset --hard
> master-2010-09-29), its what I'm on.
You are liable to hit a bunch of those crashes I've been reporting
before you hit the DMA thing if you don't use latest (with Johanne's scan
locking patch).
I'm going to poke at IOMMU debugging and see what I find.
I'll start a compile of vanilla wireless-testing + scan fix as well.
Thanks,
Ben
>
> Luis
--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
On Fri, Oct 15, 2010 at 9:51 AM, Ben Greear <[email protected]> wrote:
> On 10/14/2010 04:48 PM, Luis R. Rodriguez wrote:
>>
>> On Thu, Oct 14, 2010 at 04:39:45PM -0700, Luis R. Rodriguez wrote:
>>>
>>> On Thu, Oct 14, 2010 at 4:30 PM, Ben Greear<[email protected]>
>>> wrote:
>>>>
>>>> On 10/14/2010 04:19 PM, Luis R. Rodriguez wrote:
>>>>>
>>>>> Ok please try this patch, it cures it for me.
>>>>
>>>> Well, it got a lot further than normal, but it still
>>>> hit the poison check after a few minutes.
>>>>
>>>> Current test case is my app loading 130 or so stations, each running
>>>> wpa_supplicant. All were created, and quite a few had associated
>>>> when the poison check hit.
>>>>
>>>> So, it definitely looks like a step in the right direction, but
>>>> not fully fixed yet.
>>>>
>>>> I'll do some more testing with this patch applied and using just my
>>>> perl script to make sure the problem is reproducible outside of my
>>>> application.
>>>
>>> Ok, whatever userspace does it should not corrupt to kernel, unless
>>> its poking /dev/mem
>>
>> Can also try this one instead, it will prevent any other instances
>> we would not have caught on stopping and starting RX here.
>
> It ran longer than before any of your locking patches (about 3 minutes), but
> it did hit the poison check.
>
> Before it did, I had a bunch of OOM errors trying to allocate
> skbs. I have 2GB of RAM on this system, but maybe it's not tuned
> properly, and not all of that can be used for networking on 32-bit
> kernels....
>
> I have Felix's 3 ani patches from ~3 days ago applied, running 130 stations
> in WPA mode.
>
> I'm going to try running fewer to dodge the OOM case,
> but I have a few other things to take care of first.
Thanks for testing. So now I cannot reproduce the poison anymore, any
other ideas what I can try? Does the perl script still give you poison
or just with your Über proprietary application?
Luis
On Fri October 8 2010 04:27:22 Johannes Berg wrote:
> I'm still convinced something is wrong with ath9k RX DMA, as you've seen
> the contents of frames written to already freed memory regions. Since I
> don't know anything about ath9k, you should probably not rely on me
> though :-)
not sure if this is related or not, but it reminds me of:
"ath9k: BUG kmalloc-8192: Poison overwritten"
http://marc.info/?l=linux-kernel&m=127379077422354&w=2
and:
"ath5k in monitor mode: Poison overwritten on buffers allocated from
ath_rxbuf_alloc"
https://bugzilla.kernel.org/show_bug.cgi?id=15861
bruno
2010/10/18 Luis R. Rodriguez <[email protected]>:
> 2010/10/18 Bj?rn Smedman <[email protected]>:
>> 2010/10/15 Bj?rn Smedman <[email protected]>
>>>
>>> 2010/10/15 Ben Greear <[email protected]>:
>>> > I tried the patch below, and it didn't seem to help. ?Might even
>>> > have hurt..as it died on divide-by-zero error:
>>>
>>> Hmm, looks like the ani code got a zero listen time from the hw...
>>> That just might mean that the DMA actually hits one of these
>>> descriptors. :)
>>
>> Am I the only one worried about this? Leaving a DMA descriptor
>> pointing at memory which has been passed on to somebody else... To me
>> that's like pointing a loaded gun at someone (and it seems this
>> particular gun can go off a little haphazardly).
>>
>> Luis, given how hard it seems to be to get that locking and skb
>> shoveling right, are you sure you want to keep pointing that DMA
>> engine on innocent people's data?
>
> This is why this issue is of high priority to me. I no longer get the
> poison nor does Ben, the RX poison issue is resolved as far as I can
> tell, I just need to split up the patches into easily reviewable
> chunks and get them upstream.
The locking issue you found looks like it could cause those
overwritten poisons (as well as some weird problems I've seen in AP
mode with lots of monitor interfaces). It's really great to see that
problem go. All I'm saying is that this stuff is difficult and the
next time we get it wrong we should try to avoid overwriting arbitrary
kernel memory with our RXed frames (or TXing something sensitive).
Will the DMA engine stop when it sees a zero ds_data? In that case I
suggest we never keep an address there that we do not want to RX to or
TX from.
Also, is there some way to verify that we are not corrupting memory
with the DMA? I mean the poison check is great but if I understand
correctly it cannot detect if we overwrite active memory (i.e. before
it is freed and marked with the poison).
/Bj?rn
On 10/14/2010 03:35 PM, Luis R. Rodriguez wrote:
> On Thu, Oct 14, 2010 at 3:29 PM, Luis R. Rodriguez<[email protected]> wrote:
>> Fun enough if I just create one monitor interface and loop quickly
>> over some 2 GHz channels where I know I have traffic nearby I don't
>> see the poison. So channel changes don't seem to do much because this
>> is changing channels as fast as possible from userspace. I also can
>> confirm that I see frames from the different channels as I move along.
>
> Even forcing a band change doesn't help trigger it with just one mon0
> and one regular device scanning in a loop;
>
> while true; do for i in 2412 5745 2417 5745 2422 5745 2427 5745 2432
> 5745 2442; do echo $i iw dev mon0 set freq $i; done; done
> while true; do iw dev wlan0 scan; done
What if you just make a bunch of skb copies in ath9k before it
sends them up the stack, and then delete them? (That's basically
what a bunch of monitor devices would be doing, eh?)
Thanks,
Ben
>
> Luis
--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
On Fri, Oct 15, 2010 at 4:42 PM, Ben Greear <[email protected]> wrote:
> On 10/15/2010 04:38 PM, Luis R. Rodriguez wrote:
>>
>> On Fri, Oct 15, 2010 at 04:33:50PM -0700, Ben Greear wrote:
>>>
>>> On 10/15/2010 04:21 PM, Luis R. Rodriguez wrote:
>>>>
>>>> Ben, please give this patch a shot. I addresses three races on the PCU:
>>>>
>>>> * When we were stopping the CPU for non-EDMA cards we never locked
>>>> against
>>>> anything starting the PCU again
>>>>
>>>> * ath9k_hw_startpcureceive() was being called without locking
>>>>
>>>> * Although we lock on the rxbuf lock for contention against
>>>> starting/stopping
>>>> the PCU, we also need to lock on the driver in locations where we
>>>> start/stop
>>>> the PCU within the same location otherwise we end up in
>>>> inconsistant states
>>>> and the hardware may end up proessing an incorrect buffer for DMA.
>>>> To
>>>> protect against this we use a new PCU lock on the main part of the
>>>> driver to
>>>> ensure each start/stop/reset operation is done atomically.
>>>>
>>>> And fixes one issue as a side effect:
>>>>
>>>> * No more packet loss on ping flood when you have one STA associated
>>>> :)
>>>>
>>>> The only issue I see with this is I eventually run out of memory and my
>>>> box
>>>> becomes useless, unless I am mistaking that for some other issue.
>>>>
>>>> Please give this a shot and if it cures your woes I'll split it up into
>>>> 3 separate patches, or maybe just two, one for the first two and one for
>>>> the last issue.
>>>
>>> Sounds good, but this lockdep splat happens almost immediately upon
>>> starting
>>> my app:
>>>
>>> =======================================================
>>> [ INFO: possible circular locking dependency detected ]
>>> 2.6.36-rc8-wl+ #32
>>> -------------------------------------------------------
>>> swapper/0 is trying to acquire lock:
>>> (&(&sc->rx.pcu_lock)->rlock){+.-...}, at: [<fa16e5c7>]
>>> ath9k_tasklet+0x7e/0x140 [ath9k]
>>>
>>> but task is already holding lock:
>>> (&(&sc->rx.rxflushlock)->rlock){+.-...}, at: [<fa16e5b9>]
>>> ath9k_tasklet+0x70/0x140 [ath9k]
>>>
>>> which lock already depends on the new lock.
>>>
>>>
>>> the existing dependency chain (in reverse order) is:
>>>
>>> -> #1 (&(&sc->rx.rxflushlock)->rlock){+.-...}:
>>> [<c0457639>] lock_acquire+0x5a/0x78
>>> [<c075f6ed>] _raw_spin_lock_bh+0x20/0x2f
>>> [<fa170513>] ath_flushrecv+0x14/0x61 [ath9k]
>>
>> Ah we just need to nuke the flush lock, one second. Also remove my
>> skb_copy() otherwise you will really run out of memory quickly,
>> unless you really want to test with it :)
>
> Since you free the pkt, it shouldn't really consume, eh?
What we want to do is to use a kernel routine that gets free memory
*and* does a poison check against the memory area, so it does not
matter if we free it immediately.
> Seems like we should be able to handle an extra 5 skbs floating
> around the system..
It does consume 5 times more for each RX's buffer, so for 130 STAs it
would consume about 650 skbs instead of 130 if each RX'd an skb.
> I'll try your new patch now...
Thanks,
Luis
On Thu, Oct 7, 2010 at 11:14 AM, Johannes Berg
<[email protected]> wrote:
> On Thu, 2010-10-07 at 10:33 -0700, Ben Greear wrote:
>> In case it helps, here is a dump of where the corrupted SKB was deleted.
>
> I wonder, do you have a machine with a decent IOMMU? Adding IOMMU
> debugging into the mix could help you figure out if it's a DMA problem.
Ben, how much traffic are you RX'ing on these virtual interfaces?
Luis
On Wed, 2010-10-13 at 10:48 -0700, Ben Greear wrote:
> Here's the slub patch if you are interested in using it yourself:
> https://patchwork.kernel.org/patch/236921/
That really really screams include/linux/stacktrace.h
johannes
> >
> > I dont see any point in NULLing out bf->bf_mpdu. bf is
> > reclaimed onto a free tx buf pool as soon as it is done
> > with the skb. bf_mpdu of any of the bf's is never accessed
> > without any initialization (bf_ampdu = skb).
>
> The code can use skb after its deleted currently, because
> ath_debug_stat_tx(sc, txq, bf, ts); references the bf_ampdu
> object (I think I added that reference lately..so it's really
> a bug that I caused). At the least, we should move the ath_debug_stat_tx
> logic before the ath_tx_complete() call.
Yes, this is a serious issue irrespective of initializing bf_mpdu to
NULL.
>
> As for the paprd path, it looks racy to me: What if the paprd timer
> expires while the ath_tx_complete_buf logic is running?
That is the goal here, if we timed out on paprd training
, ath_tx_complete_buf() has to free the skb. At least I dont
see any race here, can you elaborate on your finding?, remember
ath_tx_complete_buf() is in tasklet context.
Vasanth
On 10/14/2010 02:25 PM, Luis R. Rodriguez wrote:
> On Wed, Oct 13, 2010 at 10:48 AM, Ben Greear<[email protected]> wrote:
>> On 10/13/2010 10:29 AM, Luis R. Rodriguez wrote:
>>>
>>> On Wed, Oct 13, 2010 at 10:12 AM, Ben Greear<[email protected]>
>>> wrote:
>>>>
>>>> On 10/12/2010 11:40 AM, Luis R. Rodriguez wrote:
>>>>>
>>>>> On Tue, Oct 12, 2010 at 11:35 AM, Ben Greear<[email protected]>
>>>>> wrote:
>>>>>>
>>>>>> On 10/11/2010 11:10 PM, Luis R. Rodriguez wrote:
>>>>>>>
>>>>>>> On Mon, Oct 11, 2010 at 8:27 PM, Ben Greear<[email protected]>
>>>>>>> wrote:
>>>>>>
>>>>>>>> Another thing I was thinking about: Maybe the queue of skbs and dma
>>>>>>>> addresses
>>>>>>>> in ath9k is getting corrupted by multiple VIFs trying to write at
>>>>>>>> once?
>>>>>>>> Maybe
>>>>>>>> some locking is needed in the xmit path?
>>>>>>>
>>>>>>> That was my second hunch. My first shot was to use spin_lock_irqsave()
>>>>>>> over the the uses of the rxbuf list and that seemed to help but I
>>>>>>> still managed to get a poison eventually. My next item to check for is
>>>>>>> of the permissibility of creating too much pressure to the point we
>>>>>>> end up looping over the rxbuf list and race against mac80211 free'ing
>>>>>>> a buffer. Will test that tomorrow if nothing else comes up creeping my
>>>>>>> priority queue.
>>>>>>
>>>>>> This code looks weird to me. One of the paprd branches
>>>>>> deletes the skb, the other doesn't appear to. Neither
>>>>>> null out bf->bf_mpdu, which would appear to leave a dangling
>>>>>> pointer in at least the dev_kfree_skb_any() branch.
>>>>>>
>>>>>> ath_tx_complete frees it's skb in all cases, so another
>>>>>> bf->bf_mpdu dangling pointer issue.
>>>>>>
>>>>>> Maybe at the least we should null out bf->bf_mpdu when
>>>>>> skb is consumed?
>>>>>
>>>>> You're reading my mind, that was what I was going to test today. Still
>>>>> doing e-mail sweep though.
>>>>
>>>> At least in the xmit path, it seems cards that have EDMA support do
>>>> things a bit different. Out of curiosity, on the system(s), you
>>>> reproduce
>>>> this, are any of yours supporting EDMA? Mine appear to not support EDMA.
>>>
>>> EDMA is used on>= AR9003 families by Atheros. And no, I am not
>>> testing with an EDMA card, I am testing with an AR9002 family card,
>>> the AR9280 card. I am going to disregard the TX stuff as the bug is an
>>> RX issue :) I was able to more easily reproduce by doing an skb_copy()
>>> and free'ing the buffer right afterwards on the ath_send_to_mac80211()
>>> thingy, So it does appear that the poison check just happens more
>>> often when we do an skb_copy(). One reason this is easy to reproduce
>>> with multiple STAs is mac80211 uses skb_copy() to process each
>>> received skb for each STA.
>>>
>>> In my tests so far, protecting the rxbuf list with spin_lock_irqsave()
>>> did not help, and the wmb(); didn't either, something else is going on
>>> here. It would be nice to hack slab to keep an entire trace of the
>>> place the buffer was last free'd at instead of just the caller that
>>> freed it.
>>
>> I instrumented slub a while back and got the backtrace. It
>> was always in the same place for my testing.
>>
>> Here's the slub patch if you are interested in using it yourself:
>> https://patchwork.kernel.org/patch/236921/
>
> when compiling this patch I get:
>
> arch/x86/built-in.o: In function `store_stack':
> /home/mcgrof/wireless-testing/arch/x86/kernel/dumpstack.c:259:
> undefined reference to `store_trace'
You are compiling on 32-bit system? I see the method in
the patch, but probably only for 32-bit x86...
Ben
>
> Luis
--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com