Return-path: Received: from mail2.candelatech.com ([208.74.158.173]:51211 "EHLO mail2.candelatech.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751933AbcCaV5U (ORCPT ); Thu, 31 Mar 2016 17:57:20 -0400 Subject: Re: Ath10k seems to be using stale mac80211 txq references. To: ath10k , "linux-wireless@vger.kernel.org" References: <56FD96EB.1090208@candelatech.com> From: Ben Greear Message-ID: <56FD9D3F.5060009@candelatech.com> (sfid-20160331_235723_282434_B5258ED1) Date: Thu, 31 Mar 2016 14:57:19 -0700 MIME-Version: 1.0 In-Reply-To: <56FD96EB.1090208@candelatech.com> Content-Type: text/plain; charset=utf-8; format=flowed Sender: linux-wireless-owner@vger.kernel.org List-ID: On 03/31/2016 02:30 PM, Ben Greear wrote: > hacked 4.4.6 (with most of linux.ath ath10k patches backported), hacked 10.4.3 firmware. > > I enabled kasan to help track down various bugs. This one has me a bit > perplexed. It seems ath10k is referencing some logic in mac80211 that has > already been deleted. > > Possibly this is because ath10k_flush doesn't actually drop all > skb references immediately? > > [root@ath10k ~]# wlan3: Failed to send nullfunc to AP 04:f0:21:f6:85:1c after 1000ms, disconnecting > ath10k_pci 0000:05:00.0: firmware crashed! (uuid 288562f1-b7e8-4810-ba0a-c154927476ae) > ath10k_pci 0000:05:00.0: firmware register dump: > ath10k_pci 0000:05:00.0: [00]: 0x00000009 0x000015B3 0x0099E4B6 0x00955B31 > ath10k_pci 0000:05:00.0: [04]: 0x0099E4B6 0x00060130 0x00000005 0x00000016 > ath10k_pci 0000:05:00.0: [08]: 0x00455030 0x00440C70 0x004060F0 0x00000044 > ath10k_pci 0000:05:00.0: [12]: 0x00000009 0x00000000 0x009533D0 0x009533DF > ath10k_pci 0000:05:00.0: [16]: 0x00953438 0x009C17A2 0x00940E1C 0x00000000 > ath10k_pci 0000:05:00.0: [20]: 0x4099E4B6 0x00405FEC 0x000000BE 0x00955A00 > ath10k_pci 0000:05:00.0: [24]: 0x8099E680 0x0040604C 0x00000000 0xC099E4B6 > ath10k_pci 0000:05:00.0: [28]: 0x80986D5F 0x004060AC 0x00423A14 0x004060F0 > ath10k_pci 0000:05:00.0: [32]: 0x80984E51 0x004060CC 0x00423A14 0x004060F0 > ath10k_pci 0000:05:00.0: [36]: 0x80985CBF 0x004060EC 0x00424B04 0x00440C70 > ath10k_pci 0000:05:00.0: [40]: 0x809CC91A 0x0040615C 0x00440C70 0x00424B04 > ath10k_pci 0000:05:00.0: [44]: 0x80984EBC 0x0040618C 0x00440C70 0x0040623C > ath10k_pci 0000:05:00.0: [48]: 0x809C63AC 0x0040623C 0x00440C70 0x00411988 > ath10k_pci 0000:05:00.0: [52]: 0x80984DE0 0x0040626C 0x00424B04 0x00440C70 > ath10k_pci 0000:05:00.0: [56]: 0x809CD08C 0x0040635C 0x00424B04 0x00422F34 > ath10k_pci 0000:05:00.0: ath10k_pci ATH10K_DBG_BUFFER: > ath10k: [0000]: 0001581E 17FC4C01 0F00851C 0000000A 06003007 0000FFAA FFFFFFFF 0001581E > ath10k: [0008]: 17FC4C01 71108880 00000000 00C400BF 00000000 00000FF0 0001581E 17FC4C01 > ath10k: [0016]: 71108880 00010000 00C400BF 00000000 FFFFFFFF 0001581E 17FC4C01 71108880 > ath10k: [0024]: 00020000 00C400BF 00000000 FFFFFFFF 0001581E 17FC4C01 71108880 00030000 > ath10k: [0032]: 00C400BF 000000FF FFFFFFFF 0001581E 17FC4C01 71108880 00040000 00C400BF > ath10k: [0040]: 000000FF FFFFFFFF 0001581E 17FC4C01 71108880 00050000 00C400BF 000000FF > ath10k: [0048]: FBFFFFFF 0001582D 0058581D 0001582D 0858581B 0000851C 00000000 0001582D > ath10k: [0056]: 0058581D 00015841 07FC4C02 00000004 00015846 0058581D 00015846 17FC4C01 > ath10k: [0064]: 0F00851C 0000000A 06003007 0000FFAA FFFFFFFF 00015846 17FC4C01 71108880 > ath10k: [0072]: 00000000 00C400BF 00000000 00000FF0 00015846 17FC4C01 71108880 00010000 > ath10k: [0080]: 00C400BF 00000000 FFFFFFFF 00015846 17FC4C01 71108880 00020000 00C400BF > ath10k: [0088]: 00000000 FFFFFFFF 00015846 17FC4C01 71108880 00030000 00C400BF 000000FF > ath10k: [0096]: FFFFFFFF 00015846 17FC4C01 71108880 00040000 00C400BF 000000FF FFFFFFFF > ath10k: [0104]: 00015846 17FC4C01 71108880 00050000 00C400BF 000000FF FBFFFFFF 0001584D > ath10k: [0112]: 14585853 51100001 000F118C 00000400 00000049 00440D40 0001584D 0058581D > ath10k: [0120]: 0001584D 0458581C 00000002 0001584D 0058581D 00015850 07FC4C02 00000004 > ath10k: [0128]: 00015854 0058581D 00015854 17FC4C01 0F00851C 0000000A 06003007 0000FFAA > ath10k: [0136]: FFFFFFFF 00015855 17FC4C01 71108880 00000000 00C400BF 00000000 00000FF0 > ath10k: [0144]: 00015855 17FC4C01 71108880 00010000 00C400BF 00000000 FFFFFFFF 00015855 > ath10k: [0152]: 17FC4C01 71108880 00020000 00C400BF 00000000 FFFFFFFF 00015855 17FC4C01 > ath10k: [0160]: 71108880 00030000 00C400BF 000000FF FFFFFFFF 00015855 17FC4C01 71108880 > ath10k: [0168]: 00040000 00C400BF 000000FF FFFFFFFF 00015855 17FC4C01 71108880 00050000 > ath10k: [0176]: 00C400BF 000000FF FBFFFFFF 00015861 07FC4C02 00000001 00015864 07FC4C02 > ath10k: [0184]: 00000001 00015868 085C3812 000F4CCC 00424B04 00015868 105C3809 0000143C > ath10k: [0192]: 00000001 00000000 00000000 0001586E 145C5853 51100001 000F1144 000003FC > ath10k: [0200]: 0000004A 00440C70 0001586E 145C5853 51100001 000F10FC 000003FE 0000004B > ath10k: [0208]: 00440C70 0001586E 07FC5830 00000008 0001586E 145C5854 51100002 000F10FC > ath10k: [0216]: 00000061 0000004A 00440C70 0001586E 145C5851 91107001 00424B04 00440C70 > ath10k: [0224]: 00000008 00000006 0001586E 17FC5855 91108001 00000000 00000000 00000044 > ath10k: [0232]: 000000BE 0001586E 0FFC5855 91108002 00440C70 00000010 0001586E 17FC0001 > ath10k: [0240]: 0099E4B6 000015B3 000015B3 00405EDC 00000009 > ath10k_pci 0000:05:00.0: ATH10K_END > sta22: drv-set-bitrate-mask had error return: -108 > rdev-set-bitrate-mask failed: -108 > sta21: Failed to send nullfunc to AP 04:f0:21:f6:85:1c after 1000ms, disconnecting > sta5: Failed to send nullfunc to AP 04:f0:21:f6:85:1c after 1000ms, disconnecting > sta7: Failed to send nullfunc to AP 04:f0:21:f6:85:1c after 1000ms, disconnecting > ath10k_pci 0000:05:00.0: Looped 2000 times in tx_push_pending, bailing out. > ath10k_pci 0000:05:00.0: Looped 2000 times in tx_push_pending, bailing out. > ath10k_pci 0000:05:00.0: Looped 2000 times in tx_push_pending, bailing out. > ath10k_pci 0000:05:00.0: Looped 2000 times in tx_push_pending, bailing out. > ath10k_pci 0000:05:00.0: Looped 2000 times in tx_push_pending, bailing out. > wlan3: Failed to send nullfunc to AP 04:f0:21:f6:85:1c after 1000ms, disconnecting > sta1: Failed to send nullfunc to AP 04:f0:21:f6:85:1c after 1000ms, disconnecting > sta2: Failed to send nullfunc to AP 04:f0:21:f6:85:1c after 1000ms, disconnecting > ================================================================== > BUG: KASAN: use-after-free in ath10k_mac_tx_push_txq+0x3e/0x17d [ath10k_core] at addr ffff8801bd136810 > > (gdb) l *(ath10k_mac_tx_push_txq+0x3e) > 0x10aa9 is in ath10k_mac_tx_push_txq (/home/greearb/git/linux-4.4.dev.y/drivers/net/wireless/ath/ath10k/mac.c:4241). > 4236 { > 4237 struct ath10k *ar = hw->priv; > 4238 struct ath10k_htt *htt = &ar->htt; > 4239 struct ath10k_txq *artxq = (void *)txq->drv_priv; > 4240 struct ieee80211_vif *vif = txq->vif; > 4241 struct ieee80211_sta *sta = txq->sta; > 4242 enum ath10k_hw_txrx_mode txmode; > 4243 enum ath10k_mac_tx_path txpath; > 4244 struct sk_buff *skb; > 4245 size_t skb_len; > (gdb) quit > > > Read of size 8 by task ksoftirqd/0/3 > ============================================================================= > BUG kmalloc-4096 (Tainted: G W O ): kasan: bad access detected > ----------------------------------------------------------------------------- > > INFO: Allocated in sta_info_alloc+0x42f/0x6d1 [mac80211] age=21463 cpu=2 pid=3409 > ___slab_alloc+0x2b7/0x44e > __slab_alloc.isra.64+0x44/0x74 > __kmalloc+0xae/0x13d > sta_info_alloc+0x42f/0x6d1 [mac80211] > ieee80211_prep_connection+0x16a/0xc55 [mac80211] > ieee80211_mgd_auth+0x49f/0x5cc [mac80211] > ieee80211_auth+0x13/0x15 [mac80211] > cfg80211_mlme_auth+0x2c8/0x3b0 [cfg80211] > nl80211_authenticate+0x4ba/0x513 [cfg80211] > genl_family_rcv_msg+0x497/0x543 > genl_rcv_msg+0x59/0x7d > netlink_rcv_skb+0x8d/0xeb > genl_rcv+0x23/0x32 > netlink_unicast+0x1b4/0x264 > netlink_sendmsg+0x80a/0x842 > sock_sendmsg+0x66/0x80 > INFO: Freed in sta_info_free+0xbb/0x104 [mac80211] age=223 cpu=0 pid=574 > __slab_free+0x4f/0x2a8 > kfree+0x17e/0x203 > sta_info_free+0xbb/0x104 [mac80211] > __sta_info_destroy_part2+0x2fe/0x32f [mac80211] > __sta_info_flush+0x27e/0x2d4 [mac80211] > ieee80211_set_disassoc+0x1c9/0x44c [mac80211] > ieee80211_sta_connection_lost+0x8b/0xcf [mac80211] > ieee80211_sta_work+0xb17/0x18ba [mac80211] > ieee80211_iface_work+0x43e/0x457 [mac80211] > process_one_work+0x3ed/0x77c > worker_thread+0x2ba/0x3c2 > kthread+0x162/0x171 > ret_from_fork+0x3f/0x70 > INFO: Slab 0xffffea0006f44c00 objects=7 used=4 fp=0xffff8801bd1333d8 flags=0x5fff8000004080 > INFO: Object 0xffff8801bd1367b0 @offset=26544 fp=0xffff8801bd135668 > > Thanks, > Ben > After some more poking around, I've just more questions. What cleans up this txq_data memory in mac80211/sta_info.c so that it is not leaked? if (local->ops->wake_tx_queue) { void *txq_data; int size = sizeof(struct txq_info) + ALIGN(hw->txq_data_size, sizeof(void *)); txq_data = kcalloc(ARRAY_SIZE(sta->sta.txq), size, gfp); if (!txq_data) goto free; for (i = 0; i < ARRAY_SIZE(sta->sta.txq); i++) { struct txq_info *txq = txq_data + i * size; ieee80211_init_tx_queue(sdata, sta, txq, i); } } Maybe the sta_info destroy logic needs to go clean out the txq references to sta so that ath10k cannot try to access it? Thanks, Ben -- Ben Greear Candela Technologies Inc http://www.candelatech.com