Dear Maintainers of the drivers/net/wireless/broadcom/brcm80211/brcmfmac driver,
I found the following UBSAN error in kernel 5.10.58 compiled with CLang 12.0.1
with integrated assembler (make LLVM=1 LLVM_IAS=1).
It always happens when iwd starts an access point, where /etc/iwd/main.conf
looks as follows:
[General]
UseDefaultInterface=true
DisableANQP=false
I do not observe the following error if
* kernel is compiled with gcc 10, or
* kernel version is 5.13.9 or 5.14rc5.
The reported UBSAN error is only seen with 5.10 series compiled with CLang 12.
UBSAN looks as follows. The hardware is Raspberry Pi 4B with 8GB RAM.
Aug 16 08:11:21 raspi4b-router systemd[1]: systemd-rfkill.service: Succeeded.
Aug 16 08:11:21 raspi4b-router kernel: IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
Aug 16 08:11:21 raspi4b-router systemd[1]: iwd_start_ap@Yamashita_guest.service: Succeeded.
Aug 16 08:11:21 raspi4b-router systemd[1]: Finished iwd starting Yamashita_guest access point.
Aug 16 08:11:21 raspi4b-router kernel: ================================================================================
Aug 16 08:11:21 raspi4b-router kernel: UBSAN: object-size-mismatch in ./include/linux/skbuff.h:2016:28
Aug 16 08:11:21 raspi4b-router kernel: member access within address 000000002d0b610c with insufficient space
Aug 16 08:11:21 raspi4b-router kernel: for an object of type 'struct sk_buff'
Aug 16 08:11:21 raspi4b-router kernel: CPU: 1 PID: 295 Comm: kworker/u8:3 Tainted: G C 5.10.58-clang12a #1
Aug 16 08:11:21 raspi4b-router kernel: Hardware name: Raspberry Pi 4 Model B Rev 1.4 (DT)
Aug 16 08:11:21 raspi4b-router kernel: Workqueue: brcmf_wq/mmc0:0001:1 brcmf_sdio_dataworker [brcmfmac]
Aug 16 08:11:21 raspi4b-router kernel: Call trace:
Aug 16 08:11:21 raspi4b-router kernel: dump_backtrace+0x0/0x1e4
Aug 16 08:11:21 raspi4b-router kernel: show_stack+0x18/0x24
Aug 16 08:11:21 raspi4b-router kernel: dump_stack+0xac/0x104
Aug 16 08:11:21 raspi4b-router kernel: ubsan_type_mismatch_common+0x198/0x298
Aug 16 08:11:21 raspi4b-router kernel: __ubsan_handle_type_mismatch_v1+0x40/0x50
Aug 16 08:11:21 raspi4b-router kernel: brcmf_sdio_dpc+0xa50/0x128c [brcmfmac]
Aug 16 08:11:21 raspi4b-router kernel: brcmf_sdio_dataworker+0x48/0x80 [brcmfmac]
Aug 16 08:11:21 raspi4b-router kernel: process_one_work+0x1c4/0x2e0
Aug 16 08:11:21 raspi4b-router kernel: worker_thread+0x1f4/0x2e0
Aug 16 08:11:21 raspi4b-router kernel: kthread+0x140/0x168
Aug 16 08:11:21 raspi4b-router kernel: ret_from_fork+0x10/0x18
Aug 16 08:11:21 raspi4b-router kernel: ================================================================================
Aug 16 08:11:21 raspi4b-router systemd[1]: Starting Network Service...
Aug 16 08:11:21 raspi4b-router systemd-networkd[643]: Enumeration completed
I attach dmesg by journalctl and .config of the kernel.
Best regards, Ryutaroh Matsumoto
On 8/16/2021 1:42 AM, Ryutaroh Matsumoto wrote:
> Dear Maintainers of the drivers/net/wireless/broadcom/brcm80211/brcmfmac driver,
>
> I found the following UBSAN error in kernel 5.10.58 compiled with CLang 12.0.1
> with integrated assembler (make LLVM=1 LLVM_IAS=1).
> It always happens when iwd starts an access point, where /etc/iwd/main.conf
> looks as follows:
> [General]
> UseDefaultInterface=true
> DisableANQP=false
>
> I do not observe the following error if
> * kernel is compiled with gcc 10, or
> * kernel version is 5.13.9 or 5.14rc5.
>
> The reported UBSAN error is only seen with 5.10 series compiled with CLang 12.
> UBSAN looks as follows. The hardware is Raspberry Pi 4B with 8GB RAM.
>
> Aug 16 08:11:21 raspi4b-router systemd[1]: systemd-rfkill.service: Succeeded.
> Aug 16 08:11:21 raspi4b-router kernel: IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
> Aug 16 08:11:21 raspi4b-router systemd[1]: iwd_start_ap@Yamashita_guest.service: Succeeded.
> Aug 16 08:11:21 raspi4b-router systemd[1]: Finished iwd starting Yamashita_guest access point.
> Aug 16 08:11:21 raspi4b-router kernel: ================================================================================
> Aug 16 08:11:21 raspi4b-router kernel: UBSAN: object-size-mismatch in ./include/linux/skbuff.h:2016:28
Line 2016 in skbuff.h is inline function __skb_queue_before() and as far
as I can tell brcmfmac is not using that direct or indirect. Maybe I am
reading the line info incorrectly?
> Aug 16 08:11:21 raspi4b-router kernel: member access within address 000000002d0b610c with insufficient space
> Aug 16 08:11:21 raspi4b-router kernel: for an object of type 'struct sk_buff'
> Aug 16 08:11:21 raspi4b-router kernel: CPU: 1 PID: 295 Comm: kworker/u8:3 Tainted: G C 5.10.58-clang12a #1
> Aug 16 08:11:21 raspi4b-router kernel: Hardware name: Raspberry Pi 4 Model B Rev 1.4 (DT)
> Aug 16 08:11:21 raspi4b-router kernel: Workqueue: brcmf_wq/mmc0:0001:1 brcmf_sdio_dataworker [brcmfmac]
> Aug 16 08:11:21 raspi4b-router kernel: Call trace:
> Aug 16 08:11:21 raspi4b-router kernel: dump_backtrace+0x0/0x1e4
> Aug 16 08:11:21 raspi4b-router kernel: show_stack+0x18/0x24
> Aug 16 08:11:21 raspi4b-router kernel: dump_stack+0xac/0x104
> Aug 16 08:11:21 raspi4b-router kernel: ubsan_type_mismatch_common+0x198/0x298
> Aug 16 08:11:21 raspi4b-router kernel: __ubsan_handle_type_mismatch_v1+0x40/0x50
> Aug 16 08:11:21 raspi4b-router kernel: brcmf_sdio_dpc+0xa50/0x128c [brcmfmac]
Would you be able to provide information as to what line
brcmf_sdio_dpc+0xa50 refers to.
Regards,
Arend
Hi Arend, thank you for paying attention to this.
> Line 2016 in skbuff.h is inline function __skb_queue_before() and as
> far as I can tell brcmfmac is not using that direct or indirect. Maybe
> I am reading the line info incorrectly?
I am unsure of it. On the other hand, I have also seen somewhat similar
UBSAN from a header file "include/net/flow.h" as reported at
https://lore.kernel.org/netdev/[email protected]/
All UBSANs that I have seen come from *.h compiled with clang...
> Would you be able to provide information as to what line
> brcmf_sdio_dpc+0xa50 refers to.
I'd like to do, but I do not know how to let kernel UBSAN include a line number,
though I know it with user-space applications...
Best regards, Ryutaroh
From: Arend van Spriel <[email protected]>
Subject: Re: 5.10.58 UBSAN from brcmf_sdio_dpc+0xa50/0x128c [brcmfmac]
Date: Mon, 16 Aug 2021 11:54:31 +0200
> On 8/16/2021 1:42 AM, Ryutaroh Matsumoto wrote:
>> Dear Maintainers of the
>> drivers/net/wireless/broadcom/brcm80211/brcmfmac driver,
>> I found the following UBSAN error in kernel 5.10.58 compiled with
>> CLang 12.0.1
>> with integrated assembler (make LLVM=1 LLVM_IAS=1).
>> It always happens when iwd starts an access point, where
>> /etc/iwd/main.conf
>> looks as follows:
>> [General]
>> UseDefaultInterface=true
>> DisableANQP=false
>> I do not observe the following error if
>> * kernel is compiled with gcc 10, or
>> * kernel version is 5.13.9 or 5.14rc5.
>> The reported UBSAN error is only seen with 5.10 series compiled with
>> CLang 12.
>> UBSAN looks as follows. The hardware is Raspberry Pi 4B with 8GB RAM.
>> Aug 16 08:11:21 raspi4b-router systemd[1]: systemd-rfkill.service:
>> Succeeded.
>> Aug 16 08:11:21 raspi4b-router kernel: IPv6: ADDRCONF(NETDEV_CHANGE):
>> wlan0: link becomes ready
>> Aug 16 08:11:21 raspi4b-router systemd[1]:
>> iwd_start_ap@Yamashita_guest.service: Succeeded.
>> Aug 16 08:11:21 raspi4b-router systemd[1]: Finished iwd starting
>> Yamashita_guest access point.
>> Aug 16 08:11:21 raspi4b-router kernel:
>> ================================================================================
>> Aug 16 08:11:21 raspi4b-router kernel: UBSAN: object-size-mismatch in
>> ./include/linux/skbuff.h:2016:28
>
> Line 2016 in skbuff.h is inline function __skb_queue_before() and as
> far as I can tell brcmfmac is not using that direct or indirect. Maybe
> I am reading the line info incorrectly?
>
>> Aug 16 08:11:21 raspi4b-router kernel: member access within address
>> 000000002d0b610c with insufficient space
>> Aug 16 08:11:21 raspi4b-router kernel: for an object of type 'struct
>> sk_buff'
>> Aug 16 08:11:21 raspi4b-router kernel: CPU: 1 PID: 295 Comm:
>> kworker/u8:3 Tainted: G C 5.10.58-clang12a #1
>> Aug 16 08:11:21 raspi4b-router kernel: Hardware name: Raspberry Pi 4
>> Model B Rev 1.4 (DT)
>> Aug 16 08:11:21 raspi4b-router kernel: Workqueue: brcmf_wq/mmc0:0001:1
>> brcmf_sdio_dataworker [brcmfmac]
>> Aug 16 08:11:21 raspi4b-router kernel: Call trace:
>> Aug 16 08:11:21 raspi4b-router kernel: dump_backtrace+0x0/0x1e4
>> Aug 16 08:11:21 raspi4b-router kernel: show_stack+0x18/0x24
>> Aug 16 08:11:21 raspi4b-router kernel: dump_stack+0xac/0x104
>> Aug 16 08:11:21 raspi4b-router kernel:
>> ubsan_type_mismatch_common+0x198/0x298
>> Aug 16 08:11:21 raspi4b-router kernel:
>> __ubsan_handle_type_mismatch_v1+0x40/0x50
>> Aug 16 08:11:21 raspi4b-router kernel: brcmf_sdio_dpc+0xa50/0x128c
>> [brcmfmac]
>
> Would you be able to provide information as to what line
> brcmf_sdio_dpc+0xa50 refers to.
>
> Regards,
> Arend
>>> I do not observe the following error if
>>> * kernel is compiled with gcc 10, or
>>> * kernel version is 5.13.9 or 5.14rc5.
>>> The reported UBSAN error is only seen with 5.10 series compiled with
>>> CLang 12.
Sorry, what I originally reported is partially wrong.
I also observe UBSAN with 5.13.11 compiled with CLang 11 as below,
which has the same pattern as 5.10:
Aug 17 10:22:19 raspi4b-router kernel: IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
Aug 17 10:22:19 raspi4b-router systemd[1]: iwd_start_ap@Yamashita_guest.service: Succeeded.
Aug 17 10:22:19 raspi4b-router systemd[1]: Finished iwd starting Yamashita_guest access point.
Aug 17 10:22:19 raspi4b-router kernel: ================================================================================
Aug 17 10:22:19 raspi4b-router kernel: UBSAN: object-size-mismatch in ./include/linux/skbuff.h:2038:28
Aug 17 10:22:19 raspi4b-router kernel: member access within address 00000000eea757fb with insufficient space
Aug 17 10:22:19 raspi4b-router kernel: for an object of type 'struct sk_buff'
Aug 17 10:22:19 raspi4b-router kernel: CPU: 0 PID: 7 Comm: kworker/u8:0 Tainted: G C 5.13.11-clang11 #1
Aug 17 10:22:19 raspi4b-router kernel: Hardware name: Raspberry Pi 4 Model B Rev 1.4 (DT)
Aug 17 10:22:19 raspi4b-router kernel: Workqueue: brcmf_wq/mmc0:0001:1 brcmf_sdio_dataworker [brcmfmac]
Aug 17 10:22:19 raspi4b-router kernel: Call trace:
Aug 17 10:22:19 raspi4b-router kernel: dump_backtrace+0x0/0x2a4
Aug 17 10:22:19 raspi4b-router kernel: dump_stack+0xc8/0x150
Aug 17 10:22:19 raspi4b-router kernel: ubsan_type_mismatch_common+0x198/0x2a4
Aug 17 10:22:19 raspi4b-router kernel: __ubsan_handle_type_mismatch_v1+0x40/0x50
Aug 17 10:22:19 raspi4b-router kernel: brcmf_sdio_dpc+0x2078/0x28f0 [brcmfmac]
Aug 17 10:22:19 raspi4b-router kernel: brcmf_sdio_dataworker+0x50/0x8c [brcmfmac]
Aug 17 10:22:19 raspi4b-router kernel: process_one_work+0x318/0x4fc
Aug 17 10:22:19 raspi4b-router kernel: worker_thread+0x2f0/0x660
Aug 17 10:22:19 raspi4b-router kernel: kthread+0x304/0x340
Aug 17 10:22:19 raspi4b-router kernel: ret_from_fork+0x10/0x18
Aug 17 10:22:19 raspi4b-router kernel: ================================================================================
Aug 17 10:22:19 raspi4b-router systemd[1]: Starting Network Service...
I attach full journalctl -b on kernel 5.13.11.
Best regards, Ryutaroh
Using different email to avoid disclaimers...
On August 17, 2021 2:39:56 AM Ryutaroh Matsumoto
<[email protected]> wrote:
> Hi Arend, thank you for paying attention to this.
>
>> Line 2016 in skbuff.h is inline function __skb_queue_before() and as
>> far as I can tell brcmfmac is not using that direct or indirect. Maybe
>> I am reading the line info incorrectly?
>
> I am unsure of it. On the other hand, I have also seen somewhat similar
> UBSAN from a header file "include/net/flow.h" as reported at
> https://lore.kernel.org/netdev/[email protected]/
>
> All UBSANs that I have seen come from *.h compiled with clang...
>
>> Would you be able to provide information as to what line
>> brcmf_sdio_dpc+0xa50 refers to.
>
> I'd like to do, but I do not know how to let kernel UBSAN include a line
> number,
> though I know it with user-space applications...
If you enable CONFIG_DEBUG_INFO in your kernel .config and recompile
brcmfmac you can load the module in gdb:
gdb> add-symbol-file brcmfmac.ko [address]
gdb> l *brcmf_sdio_dpc+0xa50
The [address] is not very important so just fill in a nice value. The 'l'
command should provide the line number.
Regards,
Arend
+netdev, +Dave
On 8/17/2021 7:42 AM, Arend van Spriel wrote:
> Using different email to avoid disclaimers...
>
>
> On August 17, 2021 2:39:56 AM Ryutaroh Matsumoto
> <[email protected]> wrote:
>
>> Hi Arend, thank you for paying attention to this.
>>
>>> Line 2016 in skbuff.h is inline function __skb_queue_before() and as
>>> far as I can tell brcmfmac is not using that direct or indirect. Maybe
>>> I am reading the line info incorrectly?
>>
>> I am unsure of it. On the other hand, I have also seen somewhat similar
>> UBSAN from a header file "include/net/flow.h" as reported at
>> https://lore.kernel.org/netdev/[email protected]/
>>
>>
>> All UBSANs that I have seen come from *.h compiled with clang...
>>
>>> Would you be able to provide information as to what line
>>> brcmf_sdio_dpc+0xa50 refers to.
>>
>> I'd like to do, but I do not know how to let kernel UBSAN include a
>> line number,
>> though I know it with user-space applications...
>
> If you enable CONFIG_DEBUG_INFO in your kernel .config and recompile
> brcmfmac you can load the module in gdb:
>
> gdb> add-symbol-file brcmfmac.ko [address]
> gdb> l *brcmf_sdio_dpc+0xa50
>
> The [address] is not very important so just fill in a nice value. The
> 'l' command should provide the line number.
Hi Ryutaroh,
Meanwhile I did some digging in the brcmfmac driver and I think I found
the location in brcmf_sdio_sendfromq() where we do a __skb_queue_tail().
So I looked at that and it does following:
static inline void __skb_queue_tail(struct sk_buff_head *list,
struct sk_buff *newsk)
{
__skb_queue_before(list, (struct sk_buff *)list, newsk);
}
Your report seems to be coming from the cast that is done here, which is
fine as long as sk_buff and sk_buff_head have the same members 'next'
and 'prev' at the start, which is true today and hopefully forever ;-) I
am inclined to say this is a false report.
Can you please confirm the stack trace indeed points to
brcmf_sdio_sendfromq() in your report.
Regards,
Arend
Hi Arend, sorry for my late response, partly because compilation takes 10 hours
on RPi4B...
I did the same thing with version 5.10.59 and CLang 11 and got the same
UBSAN.
>> If you enable CONFIG_DEBUG_INFO in your kernel .config and recompile
>> brcmfmac you can load the module in gdb:
>> gdb> add-symbol-file brcmfmac.ko [address]
>> gdb> l *brcmf_sdio_dpc+0xa50
>> The [address] is not very important so just fill in a nice value. The
>> 'l' command should provide the line number.
>
> Hi Ryutaroh,
>
> Meanwhile I did some digging in the brcmfmac driver and I think I
> found the location in brcmf_sdio_sendfromq() where we do a
> __skb_queue_tail(). So I looked at that and it does following:
>
> static inline void __skb_queue_tail(struct sk_buff_head *list,
> struct sk_buff *newsk)
> {
> __skb_queue_before(list, (struct sk_buff *)list, newsk);
> }
>
> Your report seems to be coming from the cast that is done here, which
> is fine as long as sk_buff and sk_buff_head have the same members
> 'next' and 'prev' at the start, which is true today and hopefully
> forever ;-) I am inclined to say this is a false report.
>
> Can you please confirm the stack trace indeed points to
> brcmf_sdio_sendfromq() in your report.
Summary: I confirm that the stack trace indeed points to
brcmf_sdio_sendfromq(). The detail follows (you don't have to read it if
you believe in me :-)
If kernel version 5.10.x or 5.13.x is compiled with gcc 10, I have never seen
UBSAN. I wonder if CLang 11/12 tends to generate falsely positive UBSAN.
(gdb) add-symbol-file drivers/net/wireless/broadcom/brcm80211/brcmfmac/brcmfmac.ko 0x0
add symbol table from file "drivers/net/wireless/broadcom/brcm80211/brcmfmac/brcmfmac.ko" at
.text_addr = 0x0
(y or n) y
(gdb) l *brcmf_sdio_dpc+0xa50
warning: Could not find DWO CU drivers/net/wireless/broadcom/brcm80211/brcmfmac/sdio.dwo(0x79060145b4b45807) referenced by CU at offset 0x394 [in module /usr/lib/debug/lib/modules/5.10.59-clang11debug/kernel/drivers/net/wireless/broadcom/brcm80211/brcmfmac/brcmfmac.ko]
0x277a4 is at ./include/linux/skbuff.h:2016.
2011 ./include/linux/skbuff.h: No such file or directory.
Line 2016 of skbuff.h is __skb_insert in the next inline function:
static inline void __skb_queue_before(struct sk_buff_head *list,
struct sk_buff *next,
struct sk_buff *newsk)
{
__skb_insert(newsk, next->prev, next, list);
}
Then:
(gdb) l *brcmf_sdio_dpc+0xa43
0x27797 is at drivers/net/wireless/broadcom/brcm80211/brcmfmac/sdio.c:2346.
2341 drivers/net/wireless/broadcom/brcm80211/brcmfmac/sdio.c: No such file or directory.
The line 2346 is in the function brcmf_sdio_sendfromq().
Best regards, Ryutaroh
On August 18, 2021 11:41:14 AM Ryutaroh Matsumoto
<[email protected]> wrote:
> Hi Arend, sorry for my late response, partly because compilation takes 10 hours
> on RPi4B...
> I did the same thing with version 5.10.59 and CLang 11 and got the same
> UBSAN.
>
>>> If you enable CONFIG_DEBUG_INFO in your kernel .config and recompile
>>> brcmfmac you can load the module in gdb:
>>> gdb> add-symbol-file brcmfmac.ko [address]
>>> gdb> l *brcmf_sdio_dpc+0xa50
>>> The [address] is not very important so just fill in a nice value. The
>>> 'l' command should provide the line number.
>>
>> Hi Ryutaroh,
>>
>> Meanwhile I did some digging in the brcmfmac driver and I think I
>> found the location in brcmf_sdio_sendfromq() where we do a
>> __skb_queue_tail(). So I looked at that and it does following:
>>
>> static inline void __skb_queue_tail(struct sk_buff_head *list,
>> struct sk_buff *newsk)
>> {
>> __skb_queue_before(list, (struct sk_buff *)list, newsk);
>> }
>>
>> Your report seems to be coming from the cast that is done here, which
>> is fine as long as sk_buff and sk_buff_head have the same members
>> 'next' and 'prev' at the start, which is true today and hopefully
>> forever ;-) I am inclined to say this is a false report.
>>
>> Can you please confirm the stack trace indeed points to
>> brcmf_sdio_sendfromq() in your report.
>
> Summary: I confirm that the stack trace indeed points to
> brcmf_sdio_sendfromq(). The detail follows (you don't have to read it if
> you believe in me :-)
> If kernel version 5.10.x or 5.13.x is compiled with gcc 10, I have never seen
> UBSAN. I wonder if CLang 11/12 tends to generate falsely positive UBSAN.
UBSAN seems to be wrong in this case...
>
> (gdb) add-symbol-file
> drivers/net/wireless/broadcom/brcm80211/brcmfmac/brcmfmac.ko 0x0
> add symbol table from file
> "drivers/net/wireless/broadcom/brcm80211/brcmfmac/brcmfmac.ko" at
> .text_addr = 0x0
> (y or n) y
>
> (gdb) l *brcmf_sdio_dpc+0xa50
> warning: Could not find DWO CU
> drivers/net/wireless/broadcom/brcm80211/brcmfmac/sdio.dwo(0x79060145b4b45807)
> referenced by CU at offset 0x394 [in module
> /usr/lib/debug/lib/modules/5.10.59-clang11debug/kernel/drivers/net/wireless/broadcom/brcm80211/brcmfmac/brcmfmac.ko]
> 0x277a4 is at ./include/linux/skbuff.h:2016.
> 2011 ./include/linux/skbuff.h: No such file or directory.
>
> Line 2016 of skbuff.h is __skb_insert in the next inline function:
> static inline void __skb_queue_before(struct sk_buff_head *list,
> struct sk_buff *next,
> struct sk_buff *newsk)
> {
> __skb_insert(newsk, next->prev, next, list);
> }
... because the next pointer is same as list so it is sk_buff_head instance
which is smaller than sk_buff, but using next->prev is ok given the
structure definitions.
Thanks for the confirmation regarding the line number in brcmfmac source.
Regards,
Arend
--
This electronic communication and the information and any files transmitted
with it, or attached to it, are confidential and are intended solely for
the use of the individual or entity to whom it is addressed and may contain
information that is confidential, legally privileged, protected by privacy
laws, or otherwise restricted from disclosure to anyone else. If you are
not the intended recipient or the person responsible for delivering the
e-mail to the intended recipient, you are hereby notified that any use,
copying, distributing, dissemination, forwarding, printing, or copying of
this e-mail is strictly prohibited. If you received this e-mail in error,
please return the e-mail to the sender, delete it from your computer, and
destroy any printed copy of it.