LinuxLists.cc - wifi breakage due to commit "wifi: brcmfmac: add support for vendor-specific firmware api"

[permalink] [raw]

Subject: Re: wifi breakage due to commit "wifi: brcmfmac: add support for vendor-specific firmware api"

On March 14, 2023 8:39:05 PM Michael Stapelberg
<[email protected]> wrote:

> Hey Arend
>
> I recently bisected a user-reported WiFi breakage on the Raspberry Pi
> 3B+ to your commit
> https://github.com/torvalds/linux/commit/d6a5c562214f26e442c8ec3ff1e28e16675d1bcf
> https://lore.kernel.org/r/[email protected]
>
> On our end, the issue was reported here:
> https://github.com/gokrazy/wifi/issues/3
>
> As of Linux 6.2.6, this seems to still be unfixed, so I wanted to
> check in and see if you could take a look please?

I have a suspicion, but can you provide the kernel .config file so I can
see if I am on the right track.

> + brcmfmac: brcmf_fwvid_request_module: mod=wcc: failed -2
> + ieee80211 phy0: brcmf_attach: brcmf_fwvid_attach failed
> + brcmfmac: brcmf_sdio_firmware_callback: brcmf_attach failed

Regards,
Arend

Attachments:

2023-03-14 22:49:10

[permalink] [raw]

Subject: Re: wifi breakage due to commit "wifi: brcmfmac: add support for vendor-specific firmware api"

Of course, find it attached. Thanks.

On Tue, 14 Mar 2023 at 23:26, Arend Van Spriel
<[email protected]> wrote:
>
> On March 14, 2023 8:39:05 PM Michael Stapelberg
> <[email protected]> wrote:
>
> > Hey Arend
> >
> > I recently bisected a user-reported WiFi breakage on the Raspberry Pi
> > 3B+ to your commit
> > https://github.com/torvalds/linux/commit/d6a5c562214f26e442c8ec3ff1e28e16675d1bcf
> > https://lore.kernel.org/r/[email protected]
> >
> > On our end, the issue was reported here:
> > https://github.com/gokrazy/wifi/issues/3
> >
> > As of Linux 6.2.6, this seems to still be unfixed, so I wanted to
> > check in and see if you could take a look please?
>
> I have a suspicion, but can you provide the kernel .config file so I can
> see if I am on the right track.
>
> > + brcmfmac: brcmf_fwvid_request_module: mod=wcc: failed -2
> > + ieee80211 phy0: brcmf_attach: brcmf_fwvid_attach failed
> > + brcmfmac: brcmf_sdio_firmware_callback: brcmf_attach failed
>
> Regards,
> Arend
>
>

Attachments:

config (241.37 kB)

2023-03-14 23:21:43

[permalink] [raw]

Subject: Re: wifi breakage due to commit "wifi: brcmfmac: add support for vendor-specific firmware api"

On March 14, 2023 11:49:05 PM Michael Stapelberg
<[email protected]> wrote:

> Of course, find it attached. Thanks.

Strange. Could it be there is some build config needed to install the
additional kernel modules (brcmfmac-wcc.ko, etc.) on RPi image?

Regards,
Arend

>
> On Tue, 14 Mar 2023 at 23:26, Arend Van Spriel
> <[email protected]> wrote:
>>
>> On March 14, 2023 8:39:05 PM Michael Stapelberg
>> <[email protected]> wrote:
>>
>>> Hey Arend
>>>
>>> I recently bisected a user-reported WiFi breakage on the Raspberry Pi
>>> 3B+ to your commit
>>> https://github.com/torvalds/linux/commit/d6a5c562214f26e442c8ec3ff1e28e16675d1bcf
>>> https://lore.kernel.org/r/[email protected]
>>>
>>> On our end, the issue was reported here:
>>> https://github.com/gokrazy/wifi/issues/3
>>>
>>> As of Linux 6.2.6, this seems to still be unfixed, so I wanted to
>>> check in and see if you could take a look please?
>>
>> I have a suspicion, but can you provide the kernel .config file so I can
>> see if I am on the right track.
>>
>>> + brcmfmac: brcmf_fwvid_request_module: mod=wcc: failed -2
>>> + ieee80211 phy0: brcmf_attach: brcmf_fwvid_attach failed
>>> + brcmfmac: brcmf_sdio_firmware_callback: brcmf_attach failed
>>
>> Regards,
>> Arend

Attachments:

2023-03-15 07:58:51

[permalink] [raw]

Subject: Re: wifi breakage due to commit "wifi: brcmfmac: add support for vendor-specific firmware api"

Hey Arend

Thanks for taking a look.

The brcmfmac-wcc module is present on my installation, but I’m not
currently loading it.
(I only load brcmutil, then brcmfmac.)

But, even if I load it, the wlan0 interface still doesn’t show up.

In the modules.dep, I see that brcmfmac-wcc should be loaded after
brcmutil and brcmfmac.
Is that accurate?

My installation is somewhat unusual in that I use a custom userland
(see https://gokrazy.org/).

Thanks
Best regards
Michael

On Wed, 15 Mar 2023 at 00:21, Arend Van Spriel
<[email protected]> wrote:
>
> On March 14, 2023 11:49:05 PM Michael Stapelberg
> <[email protected]> wrote:
>
> > Of course, find it attached. Thanks.
>
> Strange. Could it be there is some build config needed to install the
> additional kernel modules (brcmfmac-wcc.ko, etc.) on RPi image?
>
> Regards,
> Arend
>
> >
> > On Tue, 14 Mar 2023 at 23:26, Arend Van Spriel
> > <[email protected]> wrote:
> >>
> >> On March 14, 2023 8:39:05 PM Michael Stapelberg
> >> <[email protected]> wrote:
> >>
> >>> Hey Arend
> >>>
> >>> I recently bisected a user-reported WiFi breakage on the Raspberry Pi
> >>> 3B+ to your commit
> >>> https://github.com/torvalds/linux/commit/d6a5c562214f26e442c8ec3ff1e28e16675d1bcf
> >>> https://lore.kernel.org/r/[email protected]
> >>>
> >>> On our end, the issue was reported here:
> >>> https://github.com/gokrazy/wifi/issues/3
> >>>
> >>> As of Linux 6.2.6, this seems to still be unfixed, so I wanted to
> >>> check in and see if you could take a look please?
> >>
> >> I have a suspicion, but can you provide the kernel .config file so I can
> >> see if I am on the right track.
> >>
> >>> + brcmfmac: brcmf_fwvid_request_module: mod=wcc: failed -2
> >>> + ieee80211 phy0: brcmf_attach: brcmf_fwvid_attach failed
> >>> + brcmfmac: brcmf_sdio_firmware_callback: brcmf_attach failed
> >>
> >> Regards,
> >> Arend
>
>
>

2023-03-15 08:33:46

[permalink] [raw]

Subject: Re: wifi breakage due to commit "wifi: brcmfmac: add support for vendor-specific firmware api"

On March 15, 2023 8:58:43 AM Michael Stapelberg
<[email protected]> wrote:

> Hey Arend
>
> Thanks for taking a look.
>
> The brcmfmac-wcc module is present on my installation, but I’m not
> currently loading it.
> (I only load brcmutil, then brcmfmac.)
>
> But, even if I load it, the wlan0 interface still doesn’t show up.
>
> In the modules.dep, I see that brcmfmac-wcc should be loaded after
> brcmutil and brcmfmac.
> Is that accurate?

Yes. Actually brcmfmac loads brcmfmac-wcc by itself.

> My installation is somewhat unusual in that I use a custom userland
> (see https://gokrazy.org/).

Bit tricky choice of name :-p I will check it out.

Regards,
Arend

>
> Thanks
> Best regards
> Michael
>
>
> On Wed, 15 Mar 2023 at 00:21, Arend Van Spriel
> <[email protected]> wrote:
>>
>> On March 14, 2023 11:49:05 PM Michael Stapelberg
>> <[email protected]> wrote:
>>
>>> Of course, find it attached. Thanks.
>>
>> Strange. Could it be there is some build config needed to install the
>> additional kernel modules (brcmfmac-wcc.ko, etc.) on RPi image?
>>
>> Regards,
>> Arend
>>
>>>
>>> On Tue, 14 Mar 2023 at 23:26, Arend Van Spriel
>>> <[email protected]> wrote:
>>>>
>>>> On March 14, 2023 8:39:05 PM Michael Stapelberg
>>>> <[email protected]> wrote:
>>>>
>>>>> Hey Arend
>>>>>
>>>>> I recently bisected a user-reported WiFi breakage on the Raspberry Pi
>>>>> 3B+ to your commit
>>>>> https://github.com/torvalds/linux/commit/d6a5c562214f26e442c8ec3ff1e28e16675d1bcf
>>>>> https://lore.kernel.org/r/[email protected]
>>>>>
>>>>> On our end, the issue was reported here:
>>>>> https://github.com/gokrazy/wifi/issues/3
>>>>>
>>>>> As of Linux 6.2.6, this seems to still be unfixed, so I wanted to
>>>>> check in and see if you could take a look please?
>>>>
>>>> I have a suspicion, but can you provide the kernel .config file so I can
>>>> see if I am on the right track.
>>>>
>>>>> + brcmfmac: brcmf_fwvid_request_module: mod=wcc: failed -2
>>>>> + ieee80211 phy0: brcmf_attach: brcmf_fwvid_attach failed
>>>>> + brcmfmac: brcmf_sdio_firmware_callback: brcmf_attach failed
>>>>
>>>> Regards,
>>>> Arend

Attachments:

2023-03-15 08:54:08

[permalink] [raw]

Subject: Re: wifi breakage due to commit "wifi: brcmfmac: add support for vendor-specific firmware api"

Hey Arend

On Wed, 15 Mar 2023 at 09:33, Arend Van Spriel
<[email protected]> wrote:
>
> On March 15, 2023 8:58:43 AM Michael Stapelberg
> <[email protected]> wrote:
>
> > Hey Arend
> >
> > Thanks for taking a look.
> >
> > The brcmfmac-wcc module is present on my installation, but I’m not
> > currently loading it.
> > (I only load brcmutil, then brcmfmac.)
> >
> > But, even if I load it, the wlan0 interface still doesn’t show up.
> >
> > In the modules.dep, I see that brcmfmac-wcc should be loaded after
> > brcmutil and brcmfmac.
> > Is that accurate?
>
> Yes. Actually brcmfmac loads brcmfmac-wcc by itself.

Hmm, when I run lsmod after booting, I don’t see it:

/tmp/breakglass1452371198 # lsmod
hci_uart 61440 0 - Live 0xffffba314a2cd000
btbcm 24576 1 hci_uart, Live 0xffffba314a2c6000
bluetooth 786432 3 hci_uart,btbcm, Live 0xffffba314a205000
brcmfmac 258048 0 - Live 0xffffba314a1c5000
brcmutil 20480 1 brcmfmac, Live 0xffffba314a1bf000
ecdh_generic 16384 1 bluetooth, Live 0xffffba314a1ba000
ecc 36864 1 ecdh_generic, Live 0xffffba314a1b0000
/tmp/breakglass1452371198 #

How does the “load by itself” mechanism work?
Maybe I’m missing a userspace component for it.

Thanks

>
>
> > My installation is somewhat unusual in that I use a custom userland
> > (see https://gokrazy.org/).
>
> Bit tricky choice of name :-p I will check it out.
>
> Regards,
> Arend
>
> >
> > Thanks
> > Best regards
> > Michael
> >
> >
> > On Wed, 15 Mar 2023 at 00:21, Arend Van Spriel
> > <[email protected]> wrote:
> >>
> >> On March 14, 2023 11:49:05 PM Michael Stapelberg
> >> <[email protected]> wrote:
> >>
> >>> Of course, find it attached. Thanks.
> >>
> >> Strange. Could it be there is some build config needed to install the
> >> additional kernel modules (brcmfmac-wcc.ko, etc.) on RPi image?
> >>
> >> Regards,
> >> Arend
> >>
> >>>
> >>> On Tue, 14 Mar 2023 at 23:26, Arend Van Spriel
> >>> <[email protected]> wrote:
> >>>>
> >>>> On March 14, 2023 8:39:05 PM Michael Stapelberg
> >>>> <[email protected]> wrote:
> >>>>
> >>>>> Hey Arend
> >>>>>
> >>>>> I recently bisected a user-reported WiFi breakage on the Raspberry Pi
> >>>>> 3B+ to your commit
> >>>>> https://github.com/torvalds/linux/commit/d6a5c562214f26e442c8ec3ff1e28e16675d1bcf
> >>>>> https://lore.kernel.org/r/[email protected]
> >>>>>
> >>>>> On our end, the issue was reported here:
> >>>>> https://github.com/gokrazy/wifi/issues/3
> >>>>>
> >>>>> As of Linux 6.2.6, this seems to still be unfixed, so I wanted to
> >>>>> check in and see if you could take a look please?
> >>>>
> >>>> I have a suspicion, but can you provide the kernel .config file so I can
> >>>> see if I am on the right track.
> >>>>
> >>>>> + brcmfmac: brcmf_fwvid_request_module: mod=wcc: failed -2
> >>>>> + ieee80211 phy0: brcmf_attach: brcmf_fwvid_attach failed
> >>>>> + brcmfmac: brcmf_sdio_firmware_callback: brcmf_attach failed
> >>>>
> >>>> Regards,
> >>>> Arend
>
>
>

2023-03-15 09:12:50

[permalink] [raw]

Subject: Re: wifi breakage due to commit "wifi: brcmfmac: add support for vendor-specific firmware api"

On 3/15/2023 9:53 AM, Michael Stapelberg wrote:
> Hey Arend
>
> On Wed, 15 Mar 2023 at 09:33, Arend Van Spriel
> <[email protected]> wrote:
>>
>> On March 15, 2023 8:58:43 AM Michael Stapelberg
>> <[email protected]> wrote:
>>
>>> Hey Arend
>>>
>>> Thanks for taking a look.
>>>
>>> The brcmfmac-wcc module is present on my installation, but I’m not
>>> currently loading it.
>>> (I only load brcmutil, then brcmfmac.)
>>>
>>> But, even if I load it, the wlan0 interface still doesn’t show up.
>>>
>>> In the modules.dep, I see that brcmfmac-wcc should be loaded after
>>> brcmutil and brcmfmac.
>>> Is that accurate?
>>
>> Yes. Actually brcmfmac loads brcmfmac-wcc by itself.
>
> Hmm, when I run lsmod after booting, I don’t see it:
>
> /tmp/breakglass1452371198 # lsmod
> hci_uart 61440 0 - Live 0xffffba314a2cd000
> btbcm 24576 1 hci_uart, Live 0xffffba314a2c6000
> bluetooth 786432 3 hci_uart,btbcm, Live 0xffffba314a205000
> brcmfmac 258048 0 - Live 0xffffba314a1c5000
> brcmutil 20480 1 brcmfmac, Live 0xffffba314a1bf000
> ecdh_generic 16384 1 bluetooth, Live 0xffffba314a1ba000
> ecc 36864 1 ecdh_generic, Live 0xffffba314a1b0000
> /tmp/breakglass1452371198 #

That is expected as the log shows:

>>>>>>> + brcmfmac: brcmf_fwvid_request_module: mod=wcc: failed -2
>>>>>>> + ieee80211 phy0: brcmf_attach: brcmf_fwvid_attach failed

So in brcmf_fwvid_request_module it tries to load brcmfmac-wcc, but for
some reason it fails.

> How does the “load by itself” mechanism work?
> Maybe I’m missing a userspace component for it.

I think it works pretty similar to firmware loading. These days firmware
loading does not rely on a user-space helper and I assumed the same is
true for module loading. Will look into it.

Regards,
Arend

> Thanks
>
>>
>>
>>> My installation is somewhat unusual in that I use a custom userland
>>> (see https://gokrazy.org/).
>>
>> Bit tricky choice of name :-p I will check it out.
>>
>> Regards,
>> Arend
>>
>>>
>>> Thanks
>>> Best regards
>>> Michael
>>>
>>>
>>> On Wed, 15 Mar 2023 at 00:21, Arend Van Spriel
>>> <[email protected]> wrote:
>>>>
>>>> On March 14, 2023 11:49:05 PM Michael Stapelberg
>>>> <[email protected]> wrote:
>>>>
>>>>> Of course, find it attached. Thanks.
>>>>
>>>> Strange. Could it be there is some build config needed to install the
>>>> additional kernel modules (brcmfmac-wcc.ko, etc.) on RPi image?
>>>>
>>>> Regards,
>>>> Arend
>>>>
>>>>>
>>>>> On Tue, 14 Mar 2023 at 23:26, Arend Van Spriel
>>>>> <[email protected]> wrote:
>>>>>>
>>>>>> On March 14, 2023 8:39:05 PM Michael Stapelberg
>>>>>> <[email protected]> wrote:
>>>>>>
>>>>>>> Hey Arend
>>>>>>>
>>>>>>> I recently bisected a user-reported WiFi breakage on the Raspberry Pi
>>>>>>> 3B+ to your commit
>>>>>>> https://github.com/torvalds/linux/commit/d6a5c562214f26e442c8ec3ff1e28e16675d1bcf
>>>>>>> https://lore.kernel.org/r/[email protected]
>>>>>>>
>>>>>>> On our end, the issue was reported here:
>>>>>>> https://github.com/gokrazy/wifi/issues/3
>>>>>>>
>>>>>>> As of Linux 6.2.6, this seems to still be unfixed, so I wanted to
>>>>>>> check in and see if you could take a look please?
>>>>>>
>>>>>> I have a suspicion, but can you provide the kernel .config file so I can
>>>>>> see if I am on the right track.
>>>>>>
>>>>>>> + brcmfmac: brcmf_fwvid_request_module: mod=wcc: failed -2
>>>>>>> + ieee80211 phy0: brcmf_attach: brcmf_fwvid_attach failed
>>>>>>> + brcmfmac: brcmf_sdio_firmware_callback: brcmf_attach failed
>>>>>>
>>>>>> Regards,
>>>>>> Arend
>>
>>
>>

Attachments:

2023-03-15 09:59:19

by Johannes Berg

[permalink] [raw]

Subject: Re: wifi breakage due to commit "wifi: brcmfmac: add support for vendor-specific firmware api"

On Wed, 2023-03-15 at 10:12 +0100, Arend van Spriel wrote:
>
> I think it works pretty similar to firmware loading. These days firmware
> loading does not rely on a user-space helper and I assumed the same is
> true for module loading. Will look into it.
>

As far as I know you still need the modprobe helper in

/proc/sys/kernel/modprobe

johannes

2023-03-15 10:02:12

[permalink] [raw]

Subject: Re: wifi breakage due to commit "wifi: brcmfmac: add support for vendor-specific firmware api"

Attachments:

2023-03-15 10:12:29

[permalink] [raw]

Subject: Re: wifi breakage due to commit "wifi: brcmfmac: add support for vendor-specific firmware api"

On 3/15/2023 10:57 AM, Johannes Berg wrote:
> On Wed, 2023-03-15 at 10:12 +0100, Arend van Spriel wrote:
>>
>> I think it works pretty similar to firmware loading. These days firmware
>> loading does not rely on a user-space helper and I assumed the same is
>> true for module loading. Will look into it.
>>
>
> As far as I know you still need the modprobe helper in
>
> /proc/sys/kernel/modprobe

Hi Michael,

With hint from Johannes I found this Kconfig option:

config MODPROBE_PATH
string "Path to modprobe binary"
default "/sbin/modprobe"
help
When kernel code requests a module, it does so by calling
the "modprobe" userspace utility. This option allows you to
set the path where that binary is found. This can be changed
at runtime via the sysctl file
/proc/sys/kernel/modprobe. Setting this to the empty string
removes the kernel's ability to request modules (but
userspace can still load modules explicitly).

Now checking the .config that you sent it says:

CONFIG_MODPROBE_PATH="/sbin/modprobe"

So my suspicion is that modprobe is not at that given location?

Regards,
Arend

Attachments:

2023-03-15 19:53:55

[permalink] [raw]

Subject: Re: wifi breakage due to commit "wifi: brcmfmac: add support for vendor-specific firmware api"

Hey Arend

Thanks for looking into it.

Your suspicion was spot on: we don’t have /sbin/modprobe (we don’t
even have /sbin!)

I addressed this issue by loading the brcmfmac-wcc module where we
also load the others:
https://github.com/gokrazy/wifi/commit/f611656338b68faa5f42bc2c8bbf1e0231a74a50

I’m not entirely sure why I concluded in my earlier testing that
loading the module doesn’t help.
Perhaps the issue was that I was manually loading the module after the
wifi program
tried to use the interface already and had it fail its firmware load —
in either case,
loading brcmfmac-wcc from the wifi program seems to work fine.

I also realize that the error message gives a number of pointers once
you know how to interpret it:

+ brcmfmac: brcmf_fwvid_request_module: mod=wcc: failed -2

I now understand this was trying to tell me that
request_module("brcmfmac-wcc") failed
with error code -2 (ENOENT), which means that the kernel’s
request_module() function
could not find CONFIG_MODPROBE_PATH.

For comparison, when the user space tool exists but fails, including
when it can’t find
the requested module (ENOENT), it always returns exit code 1:
https://salsa.debian.org/md/kmod/-/blob/7d761268ebb68245daf47167e418823c68c3eb98/tools/modprobe.c#L1025

I think it would be nice if the Linux kernel could print a more
descriptive error message in this case.
I’m not sure what the kernel conventions are: should the caller of
request_module() produce a good error message?
Ideally, the message would translate -2 into ENOENT, and mention the
name “brcmfmac-wcc”.

Or maybe we could have a kconfig option to indicate there is no
CONFIG_MODPROBE_PATH,
and instead make any request_module() call fail loudly with a
descriptive error in dmesg?

Thanks for considering
Best regards
Michael

On Wed, 15 Mar 2023 at 11:11, Arend van Spriel
<[email protected]> wrote:
>
> On 3/15/2023 10:57 AM, Johannes Berg wrote:
> > On Wed, 2023-03-15 at 10:12 +0100, Arend van Spriel wrote:
> >>
> >> I think it works pretty similar to firmware loading. These days firmware
> >> loading does not rely on a user-space helper and I assumed the same is
> >> true for module loading. Will look into it.
> >>
> >
> > As far as I know you still need the modprobe helper in
> >
> > /proc/sys/kernel/modprobe
>
> Hi Michael,
>
> With hint from Johannes I found this Kconfig option:
>
> config MODPROBE_PATH
> string "Path to modprobe binary"
> default "/sbin/modprobe"
> help
> When kernel code requests a module, it does so by calling
> the "modprobe" userspace utility. This option allows you to
> set the path where that binary is found. This can be changed
> at runtime via the sysctl file
> /proc/sys/kernel/modprobe. Setting this to the empty string
> removes the kernel's ability to request modules (but
> userspace can still load modules explicitly).
>
> Now checking the .config that you sent it says:
>
> CONFIG_MODPROBE_PATH="/sbin/modprobe"
>
> So my suspicion is that modprobe is not at that given location?
>
> Regards,
> Arend

2023-03-16 06:17:12

[permalink] [raw]

Subject: Re: wifi breakage due to commit "wifi: brcmfmac: add support for vendor-specific firmware api"

On March 15, 2023 8:53:50 PM Michael Stapelberg
<[email protected]> wrote:

> Hey Arend
>
> Thanks for looking into it.
>
> Your suspicion was spot on: we don’t have /sbin/modprobe (we don’t
> even have /sbin!)
>
> I addressed this issue by loading the brcmfmac-wcc module where we
> also load the others:
> https://github.com/gokrazy/wifi/commit/f611656338b68faa5f42bc2c8bbf1e0231a74a50
>
> I’m not entirely sure why I concluded in my earlier testing that
> loading the module doesn’t help.
> Perhaps the issue was that I was manually loading the module after the
> wifi program
> tried to use the interface already and had it fail its firmware load —
> in either case,
> loading brcmfmac-wcc from the wifi program seems to work fine.

I was going to say it seems racy, but I am not sure why that works at all.
In brcmf_fwvid_request_module it says:

ret = request_module("brcmfmac-%s", fwvid_list[fwvid].name);
if (ret)
goto fail;

and the missing modprobe would always cause request_module() to fail (see
[1]) hence the device probe would fail. What am I missing here?

Regards,
Arend

[1] https://elixir.bootlin.com/linux/latest/source/kernel/kmod.c#L138

>
> I also realize that the error message gives a number of pointers once
> you know how to interpret it:
>
> + brcmfmac: brcmf_fwvid_request_module: mod=wcc: failed -2
>
> I now understand this was trying to tell me that
> request_module("brcmfmac-wcc") failed
> with error code -2 (ENOENT), which means that the kernel’s
> request_module() function
> could not find CONFIG_MODPROBE_PATH.
>
> For comparison, when the user space tool exists but fails, including
> when it can’t find
> the requested module (ENOENT), it always returns exit code 1:
> https://salsa.debian.org/md/kmod/-/blob/7d761268ebb68245daf47167e418823c68c3eb98/tools/modprobe.c#L1025
>
> I think it would be nice if the Linux kernel could print a more
> descriptive error message in this case.
> I’m not sure what the kernel conventions are: should the caller of
> request_module() produce a good error message?
> Ideally, the message would translate -2 into ENOENT, and mention the
> name “brcmfmac-wcc”.
>
> Or maybe we could have a kconfig option to indicate there is no
> CONFIG_MODPROBE_PATH,
> and instead make any request_module() call fail loudly with a
> descriptive error in dmesg?
>
> Thanks for considering
> Best regards
> Michael
>
> On Wed, 15 Mar 2023 at 11:11, Arend van Spriel
> <[email protected]> wrote:
>>
>> On 3/15/2023 10:57 AM, Johannes Berg wrote:
>>> On Wed, 2023-03-15 at 10:12 +0100, Arend van Spriel wrote:
>>>>
>>>> I think it works pretty similar to firmware loading. These days firmware
>>>> loading does not rely on a user-space helper and I assumed the same is
>>>> true for module loading. Will look into it.
>>>
>>> As far as I know you still need the modprobe helper in
>>>
>>> /proc/sys/kernel/modprobe
>>
>> Hi Michael,
>>
>> With hint from Johannes I found this Kconfig option:
>>
>> config MODPROBE_PATH
>> string "Path to modprobe binary"
>> default "/sbin/modprobe"
>> help
>> When kernel code requests a module, it does so by calling
>> the "modprobe" userspace utility. This option allows you to
>> set the path where that binary is found. This can be changed
>> at runtime via the sysctl file
>> /proc/sys/kernel/modprobe. Setting this to the empty string
>> removes the kernel's ability to request modules (but
>> userspace can still load modules explicitly).
>>
>> Now checking the .config that you sent it says:
>>
>> CONFIG_MODPROBE_PATH="/sbin/modprobe"
>>
>> So my suspicion is that modprobe is not at that given location?
>>
>> Regards,
>> Arend

Attachments:

2023-03-16 07:54:31

[permalink] [raw]

Subject: Re: wifi breakage due to commit "wifi: brcmfmac: add support for vendor-specific firmware api"

Hmm, I’m not sure either.

My guess would be that the module loading happens only when actually
initializing the device,
not when merely loading the kernel module? At least I don’t see any
module-related errors in dmesg:

[ 5.603264] brcmfmac: brcmf_fw_alloc_request: using
brcm/brcmfmac43455-sdio for chip BCM4345/6
[ 5.612966] brcmfmac mmc1:0001:1: Direct firmware load for
brcm/brcmfmac43455-sdio.raspberrypi,3-model-b-plus.bin failed with
error -2
[ 5.704359] Bluetooth: Core ver 2.22
[ 5.708263] NET: Registered PF_BLUETOOTH protocol family
[ 5.715884] Bluetooth: HCI device and connection manager initialized
[ 5.722720] Bluetooth: HCI socket layer initialized
[ 5.728316] Bluetooth: L2CAP socket layer initialized
[ 5.733655] Bluetooth: SCO socket layer initialized
[ 5.851351] Bluetooth: HCI UART driver ver 2.3
[ 5.856653] Bluetooth: HCI UART protocol H4 registered
[ 5.862449] Bluetooth: HCI UART protocol Broadcom registered
[ 5.863571] hci_uart_bcm serial0-0: supply vbat not found, using
dummy regulator
[ 5.878107] hci_uart_bcm serial0-0: supply vddio not found, using
dummy regulator
[ 5.888944] brcmfmac_wcc: brcmf_wcc_attach: executing
[ 5.912344] brcmfmac: brcmf_c_preinit_dcmds: Firmware: BCM4345/6
wl0: Mar 2 2020 23:30:41 version 7.45.202 (r724630 CY) FWID
01-72f6ece2
[ 6.006077] uart-pl011 3f201000.serial: no DMA platform data
[ 6.246580] Bluetooth: hci0: BCM: chip id 107
[ 6.251455] Bluetooth: hci0: BCM: features 0x2f
[ 6.257631] Bluetooth: hci0: BCM4345C0
[ 6.261481] Bluetooth: hci0: BCM4345C0 (003.001.025) build 0000
[ 6.267670] Bluetooth: hci0: BCM: firmware Patch file not found, tried:
[ 6.274395] Bluetooth: hci0: BCM:
'brcm/BCM4345C0.raspberrypi,3-model-b-plus.hcd'
[ 6.282062] Bluetooth: hci0: BCM: 'brcm/BCM4345C0.hcd'
[ 6.287369] Bluetooth: hci0: BCM: 'brcm/BCM.raspberrypi,3-model-b-plus.hcd'
[ 6.294496] Bluetooth: hci0: BCM: 'brcm/BCM.hcd'
[ 7.903878] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 9.926759] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready

On Thu, 16 Mar 2023 at 07:17, Arend Van Spriel
<[email protected]> wrote:
>
> On March 15, 2023 8:53:50 PM Michael Stapelberg
> <[email protected]> wrote:
>
> > Hey Arend
> >
> > Thanks for looking into it.
> >
> > Your suspicion was spot on: we don’t have /sbin/modprobe (we don’t
> > even have /sbin!)
> >
> > I addressed this issue by loading the brcmfmac-wcc module where we
> > also load the others:
> > https://github.com/gokrazy/wifi/commit/f611656338b68faa5f42bc2c8bbf1e0231a74a50
> >
> > I’m not entirely sure why I concluded in my earlier testing that
> > loading the module doesn’t help.
> > Perhaps the issue was that I was manually loading the module after the
> > wifi program
> > tried to use the interface already and had it fail its firmware load —
> > in either case,
> > loading brcmfmac-wcc from the wifi program seems to work fine.
>
> I was going to say it seems racy, but I am not sure why that works at all.
> In brcmf_fwvid_request_module it says:
>
> ret = request_module("brcmfmac-%s", fwvid_list[fwvid].name);
> if (ret)
> goto fail;
>
> and the missing modprobe would always cause request_module() to fail (see
> [1]) hence the device probe would fail. What am I missing here?
>
> Regards,
> Arend
>
> [1] https://elixir.bootlin.com/linux/latest/source/kernel/kmod.c#L138
>
> >
> > I also realize that the error message gives a number of pointers once
> > you know how to interpret it:
> >
> > + brcmfmac: brcmf_fwvid_request_module: mod=wcc: failed -2
> >
> > I now understand this was trying to tell me that
> > request_module("brcmfmac-wcc") failed
> > with error code -2 (ENOENT), which means that the kernel’s
> > request_module() function
> > could not find CONFIG_MODPROBE_PATH.
> >
> > For comparison, when the user space tool exists but fails, including
> > when it can’t find
> > the requested module (ENOENT), it always returns exit code 1:
> > https://salsa.debian.org/md/kmod/-/blob/7d761268ebb68245daf47167e418823c68c3eb98/tools/modprobe.c#L1025
> >
> > I think it would be nice if the Linux kernel could print a more
> > descriptive error message in this case.
> > I’m not sure what the kernel conventions are: should the caller of
> > request_module() produce a good error message?
> > Ideally, the message would translate -2 into ENOENT, and mention the
> > name “brcmfmac-wcc”.
> >
> > Or maybe we could have a kconfig option to indicate there is no
> > CONFIG_MODPROBE_PATH,
> > and instead make any request_module() call fail loudly with a
> > descriptive error in dmesg?
> >
> > Thanks for considering
> > Best regards
> > Michael
> >
> > On Wed, 15 Mar 2023 at 11:11, Arend van Spriel
> > <[email protected]> wrote:
> >>
> >> On 3/15/2023 10:57 AM, Johannes Berg wrote:
> >>> On Wed, 2023-03-15 at 10:12 +0100, Arend van Spriel wrote:
> >>>>
> >>>> I think it works pretty similar to firmware loading. These days firmware
> >>>> loading does not rely on a user-space helper and I assumed the same is
> >>>> true for module loading. Will look into it.
> >>>
> >>> As far as I know you still need the modprobe helper in
> >>>
> >>> /proc/sys/kernel/modprobe
> >>
> >> Hi Michael,
> >>
> >> With hint from Johannes I found this Kconfig option:
> >>
> >> config MODPROBE_PATH
> >> string "Path to modprobe binary"
> >> default "/sbin/modprobe"
> >> help
> >> When kernel code requests a module, it does so by calling
> >> the "modprobe" userspace utility. This option allows you to
> >> set the path where that binary is found. This can be changed
> >> at runtime via the sysctl file
> >> /proc/sys/kernel/modprobe. Setting this to the empty string
> >> removes the kernel's ability to request modules (but
> >> userspace can still load modules explicitly).
> >>
> >> Now checking the .config that you sent it says:
> >>
> >> CONFIG_MODPROBE_PATH="/sbin/modprobe"
> >>
> >> So my suspicion is that modprobe is not at that given location?
> >>
> >> Regards,
> >> Arend
>
>
>

2023-03-16 09:40:30

[permalink] [raw]

Subject: Re: wifi breakage due to commit "wifi: brcmfmac: add support for vendor-specific firmware api"

On 3/16/2023 8:54 AM, Michael Stapelberg wrote:
> Hmm, I’m not sure either.

Actually, the check I referred to only bails out when
CONFIG_MODPROBE_PATH is unset. In your case it is set but it is not
existing. I dived into request_module() and it ends up scheduling a work
to call the modprobe which will fail. However, brcmfmac will wait for
brcmfmac-wcc to register itself so doing the insmod of brcmfmac-wcc will
do the trick. However, now I am back at saying this can be racy. It
would be better to come up with a modprobe.go for your user-space
environment and put that in kernel config.

Regards,
Arend

Attachments:

2023-03-17 08:07:13

[permalink] [raw]

Subject: Re: wifi breakage due to commit "wifi: brcmfmac: add support for vendor-specific firmware api"

Can you outline under what conditions the race can happen?

I’ll take care of implementing a userspace modprobe helper.

Thanks

On Thu, 16 Mar 2023 at 10:40, Arend van Spriel
<[email protected]> wrote:
>
> On 3/16/2023 8:54 AM, Michael Stapelberg wrote:
> > Hmm, I’m not sure either.
>
> Actually, the check I referred to only bails out when
> CONFIG_MODPROBE_PATH is unset. In your case it is set but it is not
> existing. I dived into request_module() and it ends up scheduling a work
> to call the modprobe which will fail. However, brcmfmac will wait for
> brcmfmac-wcc to register itself so doing the insmod of brcmfmac-wcc will
> do the trick. However, now I am back at saying this can be racy. It
> would be better to come up with a modprobe.go for your user-space
> environment and put that in kernel config.
>
> Regards,
> Arend
>

2023-03-17 09:14:03

[permalink] [raw]

Subject: Re: wifi breakage due to commit "wifi: brcmfmac: add support for vendor-specific firmware api"

On 3/17/2023 9:04 AM, Michael Stapelberg wrote:
> Can you outline under what conditions the race can happen?

I was concerned about brcmfmac-wcc doing register before brcmfmac is
waiting for that. Looking more into it I think this is not a concern. We
are using a completion construct which will synchronize properly.

Regards,
Arend

Attachments: