LinuxLists.cc - [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Mon, 2008-07-14 at 16:41 -0700, Andrew Morton wrote:
> On Mon, 14 Jul 2008 16:23:26 -0700
> David Woodhouse <[email protected]> wrote:
>
> > Linus, please pull from the for-2.6.27 branch of:
> > git://git.infradead.org/users/dwmw2/firmware-2.6.git for-2.6.27
>
> The firmware flamewars seem to have subsided lately. Is everyone happy
> with this now?

I've deliberately left drivers/net out of this pull request... all
_other_ maintainers seemed quite happy to see their drivers updated to
use request_firmware().

Let's see how it pans out. I don't expect any problems -- after all,
modern drivers have been using request_firmware() for a _long_ time now,
and the distributions cope quite happily with it already. We're ready to
ship the firmware in a separate package in Fedora as soon as Linus pulls
from my tree. Once the dust has settled and it's all working fine, we
can look at converting drivers/net and drivers/scsi too.

--
dwmw2

2008-07-14 23:54:10

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Mon, Jul 14, 2008 at 4:47 PM, David Woodhouse <[email protected]> wrote:
> On Mon, 2008-07-14 at 16:41 -0700, Andrew Morton wrote:
>> On Mon, 14 Jul 2008 16:23:26 -0700
>> David Woodhouse <[email protected]> wrote:
>>
>> > Linus, please pull from the for-2.6.27 branch of:
>> > git://git.infradead.org/users/dwmw2/firmware-2.6.git for-2.6.27
>>
>> The firmware flamewars seem to have subsided lately. Is everyone happy
>> with this now?
>
> I've deliberately left drivers/net out of this pull request... all
> _other_ maintainers seemed quite happy to see their drivers updated to
> use request_firmware().
>
> Let's see how it pans out. I don't expect any problems -- after all,
> modern drivers have been using request_firmware() for a _long_ time now,
> and the distributions cope quite happily with it already. We're ready to
> ship the firmware in a separate package in Fedora as soon as Linus pulls
> from my tree. Once the dust has settled and it's all working fine, we
> can look at converting drivers/net and drivers/scsi too.

so after this patchset, if i put qllogic fw in firmware/, that fw will
work or not...

YH

2008-07-15 00:00:27

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Mon, 14 Jul 2008 16:41:19 -0700
Andrew Morton <[email protected]> wrote:

> On Mon, 14 Jul 2008 16:23:26 -0700
> David Woodhouse <[email protected]> wrote:
>
> > Linus, please pull from the for-2.6.27 branch of:
> > git://git.infradead.org/users/dwmw2/firmware-2.6.git
> > for-2.6.27
>
> The firmware flamewars seem to have subsided lately. Is everyone
> happy with this now?
>

this seems to have left the contentious pieces out....

--
If you want to reach me at my work email, use [email protected]
For development, discussion and tips for power savings,
visit http://www.lesswatts.org

2008-07-15 00:06:04

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Mon, 14 Jul 2008, Arjan van de Ven wrote:

> On Mon, 14 Jul 2008 16:41:19 -0700
> Andrew Morton <[email protected]> wrote:
>
>> On Mon, 14 Jul 2008 16:23:26 -0700
>> David Woodhouse <[email protected]> wrote:
>>
>>> Linus, please pull from the for-2.6.27 branch of:
>>> git://git.infradead.org/users/dwmw2/firmware-2.6.git
>>> for-2.6.27
>>
>> The firmware flamewars seem to have subsided lately. Is everyone
>> happy with this now?
>>
>
> this seems to have left the contentious pieces out....

almost, the one thing that this could have done would be to offer an
option to bundle the firmware with the module. currently it offers the
option to load the firmware at the same time as the module, not _quite_
the same thing.

there was also the issue that was raised about how to handle firmware
during suspend/resume. I don't remember seeing a happy solution to that
one.

David Lang

2008-07-15 00:23:40

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Mon, 2008-07-14 at 16:53 -0700, Yinghai Lu wrote:
> so after this patchset, if i put qllogic fw in firmware/, that fw will
> work or not...

The qlogic driver already uses request_firmware(), like most modern
drivers do. If you want to include that firmware in your kernel image
(assuming it's in /lib/firmware already), do something like this:

CONFIG_EXTRA_FIRMWARE="ql2500_fw.bin"
CONFIG_EXTRA_FIRMWARE_DIR=/lib/firmware

Then it'll find that firmware file and pull it in to your kernel, and
you can set CONFIG_SCSI_QLA_FC=y and expect it to work. Select the right
firmware for your device, of course.

--
dwmw2

2008-07-15 00:31:39

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Mon, Jul 14, 2008 at 5:23 PM, David Woodhouse <[email protected]> wrote:
> On Mon, 2008-07-14 at 16:53 -0700, Yinghai Lu wrote:
>> so after this patchset, if i put qllogic fw in firmware/, that fw will
>> work or not...
>
> The qlogic driver already uses request_firmware(), like most modern
> drivers do. If you want to include that firmware in your kernel image
> (assuming it's in /lib/firmware already), do something like this:
>
> CONFIG_EXTRA_FIRMWARE="ql2500_fw.bin"
> CONFIG_EXTRA_FIRMWARE_DIR=/lib/firmware
>
> Then it'll find that firmware file and pull it in to your kernel, and
> you can set CONFIG_SCSI_QLA_FC=y and expect it to work. Select the right
> firmware for your device, of course.
>

CONFIG_EXTRA_FIRMWARE="ql2100_fw.bin ql2300_fw.bin ql2400_fw.bin
ql2200_fw.bin ql2322_fw.bin ql2500_fw.bin"
CONFIG_EXTRA_FIRMWARE_DIR="firmware"

make[1]: *** No rule to make target
`include/config/builtin/firmware/dir.h', needed by
`firmware/ql2100_fw.bin.gen.S'. Stop.
make: *** [firmware] Error 2

YH

2008-07-15 00:35:00

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Mon, 14 Jul 2008, [email protected] wrote:
>
> there was also the issue that was raised about how to handle firmware during
> suspend/resume. I don't remember seeing a happy solution to that one.

I don't understand why this even keeps coming up.

The only reliable way to handle it is to load the firmware before the
suspend happens - preferably by simply never unloading it in the first
place.

Anything else is fraught with problems.

Stuff that expects to eject itself anyway(*) over a suspend event can
obviously decide to throw it away and require a reload at resume time (by
virtue of simply re-discovering the whole device!).

Everybody else should just keep the firmware in memory in the driver.

Does it waste some ram? Sure. Tough.

Linus

(*) eg some random USB device that really _really_ doesn't care and isn't
expected to be involved in any filesystem access - say a webcam.

2008-07-15 00:37:19

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Mon, 2008-07-14 at 17:06 -0700, [email protected] wrote:
> On Mon, 14 Jul 2008, Arjan van de Ven wrote:
>
> > On Mon, 14 Jul 2008 16:41:19 -0700
> > Andrew Morton <[email protected]> wrote:
> >
> >> On Mon, 14 Jul 2008 16:23:26 -0700
> >> David Woodhouse <[email protected]> wrote:
> >>
> >>> Linus, please pull from the for-2.6.27 branch of:
> >>> git://git.infradead.org/users/dwmw2/firmware-2.6.git
> >>> for-2.6.27
> >>
> >> The firmware flamewars seem to have subsided lately. Is everyone
> >> happy with this now?
> >>
> >
> > this seems to have left the contentious pieces out....
>
> almost, the one thing that this could have done would be to offer an
> option to bundle the firmware with the module. currently it offers the
> option to load the firmware at the same time as the module, not _quite_
> the same thing.

I see no real point in that. If you have userspace to load modules, then
you have userspace to load firmware.

We made 'make modules_install' install the firmware in /lib/firmware for
you at the same time as it installs the modules, to avoid problems if
people forgot to run 'make firmware_install'. That really ought to be
enough.

I know Jeff complained that it wasn't, and insisted that he _needed_ to
be able to scp a single .ko file around which contained both, and the
world was broken if he could not -- but I disagree with him.

But since I wanted this tree to be uncontentious and obviously the
correct thing to do, I've dropped the drivers/net changes for now
anyway.

It's odd that this request has suddenly come out of the blue when we've
been using request_firmware() from modules for years already.

> there was also the issue that was raised about how to handle firmware
> during suspend/resume. I don't remember seeing a happy solution to that
> one.

My b43 seems to work fine on suspend/resume, as do most other modern
drivers which use request_firmware(). There is certainly no fundamental,
generic problem with suspend/resume vs. the firmware loader.

If you see a problem with a specific driver which we've converted, do
please point it out.

--
dwmw2

2008-07-15 00:41:19

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Mon, 14 Jul 2008, David Woodhouse wrote:
>
> I see no real point in that. If you have userspace to load modules, then
> you have userspace to load firmware.

David, the case is different: the module is loaded _across_ the
suspend/resume, but the device needs to have its firmware reloaded at
resume time because it lost power.

And since the device may be resumed before the root filesystem disk
device, it needs to do so _without_ calling request_firmware() at resume
time.

The above is _normal_.

And the fact is, the only way to handle it sanely is to do the
request_firmware() before suspending.

I _really_ don't understand why this keeps coming up all the time. It's a
no-brainer, with a single possible solution, yet people continue to whine
about it.

Linus

2008-07-15 00:44:21

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Mon, 2008-07-14 at 17:40 -0700, Linus Torvalds wrote:
>
> On Mon, 14 Jul 2008, David Woodhouse wrote:
> >
> > I see no real point in that. If you have userspace to load modules, then
> > you have userspace to load firmware.
>
> David, the case is different: the module is loaded _across_ the
> suspend/resume, but the device needs to have its firmware reloaded at
> resume time because it lost power.

Either I put my reply in response to the wrong quotation, or you thought
I did.

I intended the above to be a response to the "please put firmware _into_
the .ko file" request. Not to the suspend/resume thing.

Yes, I agree that suspend/resume is a non-issue.

--
dwmw2

2008-07-15 00:50:39

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Mon, Jul 14, 2008 at 5:23 PM, David Woodhouse <[email protected]> wrote:
> On Mon, 2008-07-14 at 16:53 -0700, Yinghai Lu wrote:
>> so after this patchset, if i put qllogic fw in firmware/, that fw will
>> work or not...
>
> The qlogic driver already uses request_firmware(), like most modern
> drivers do. If you want to include that firmware in your kernel image
> (assuming it's in /lib/firmware already), do something like this:
>
> CONFIG_EXTRA_FIRMWARE="ql2500_fw.bin"
> CONFIG_EXTRA_FIRMWARE_DIR=/lib/firmware
>
> Then it'll find that firmware file and pull it in to your kernel, and
> you can set CONFIG_SCSI_QLA_FC=y and expect it to work. Select the right
> firmware for your device, of course.

MK_FW firmware/tigon/tg3.bin.gen.S
make[1]: *** No rule to make target `firmware/tigon/tg3.bin', needed
by `firmware/tigon/tg3.bin.gen.o'. Stop.
make: *** [firmware] Error 2

YH

2008-07-15 00:50:55

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Mon, 2008-07-14 at 17:31 -0700, Yinghai Lu wrote:
> make[1]: *** No rule to make target
> `include/config/builtin/firmware/dir.h', needed by
> `firmware/ql2100_fw.bin.gen.S'. Stop.
> make: *** [firmware] Error 2

Oh, pants. Sorry, I renamed the CONFIG_BUILTIN_FIRMWARE option to
CONFIG_EXTRA_FIRMWARE some time back, but all my own test builds had the
include/config/builtin/firmware/dir.h file still lying around so I
didn't notice that I'd forgotten to change that manual dependency. And
evidently nobody using linux-next actually tried to include arbitrary
_other_ firmware into their tree. This patch should fix it...

(Why _do_ we have stray files left in include/config/ when config
options go away, btw? Shouldn't they get deleted?)

----
Subject: firmware: Correct dependency on CONFIG_EXTRA_FIRMWARE_DIR

When CONFIG_EXTRA_FIRMWARE_DIR gets changed, the filename in the .S file
(which uses .incbin to include the binary) needs to change. When we
renamed the BUILTIN_FIRMWARE_DIR option to EXTRA_FIRMWARE_DIR, we forgot
to update the manual dependency in firmware/Makefile, so it was
depending on a non-existent file in include/config/

Signed-off-by: David Woodhouse <[email protected]>

diff --git a/firmware/Makefile b/firmware/Makefile
index 5a1e61c..ec4f0cc 100644
--- a/firmware/Makefile
+++ b/firmware/Makefile
@@ -146,7 +146,7 @@ $(patsubst %,$(obj)/%.gen.S, $(fw-shipped-y)): %: $(wordsize_deps) \
| $(objtree)/$$(dir %)
$(call cmd,fwbin,$(patsubst %.gen.S,%,$@))
$(patsubst %,$(obj)/%.gen.S, $(fw-external-y)): %: $(wordsize_deps) \
- include/config/builtin/firmware/dir.h | $(objtree)/$$(dir %)
+ include/config/extra/firmware/dir.h | $(objtree)/$$(dir %)
$(call cmd,fwbin,$(fwabs)/$(patsubst $(obj)/%.gen.S,%,$@))

# The .o files depend on the binaries directly; the .S files don't.

--
dwmw2

2008-07-15 00:51:26

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Mon, 14 Jul 2008, Linus Torvalds wrote:

> On Mon, 14 Jul 2008, [email protected] wrote:
>>
>> there was also the issue that was raised about how to handle firmware during
>> suspend/resume. I don't remember seeing a happy solution to that one.
>
> I don't understand why this even keeps coming up.
>
> The only reliable way to handle it is to load the firmware before the
> suspend happens - preferably by simply never unloading it in the first
> place.
>
> Anything else is fraught with problems.
>
> Stuff that expects to eject itself anyway(*) over a suspend event can
> obviously decide to throw it away and require a reload at resume time (by
> virtue of simply re-discovering the whole device!).
>
> Everybody else should just keep the firmware in memory in the driver.
>
> Does it waste some ram? Sure. Tough.

I agree with this, but the proponents of the seperate firmware are listing
the fact that the firmware doesn't tie up ram as one of the big reasons
for making the change.

David Lang

2008-07-15 00:52:31

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Mon, 2008-07-14 at 17:50 -0700, Yinghai Lu wrote:
> MK_FW firmware/tigon/tg3.bin.gen.S
> make[1]: *** No rule to make target `firmware/tigon/tg3.bin', needed
> by `firmware/tigon/tg3.bin.gen.o'. Stop.
> make: *** [firmware] Error 2

I didn't include the tg3 patch in what I just asked Linus to pull. Are
you working from linux-next?

--
dwmw2

2008-07-15 00:53:20

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Mon, Jul 14, 2008 at 5:52 PM, David Woodhouse <[email protected]> wrote:
> On Mon, 2008-07-14 at 17:50 -0700, Yinghai Lu wrote:
>> MK_FW firmware/tigon/tg3.bin.gen.S
>> make[1]: *** No rule to make target `firmware/tigon/tg3.bin', needed
>> by `firmware/tigon/tg3.bin.gen.o'. Stop.
>> make: *** [firmware] Error 2
>
> I didn't include the tg3 patch in what I just asked Linus to pull. Are
> you working from linux-next?

linus tree.

YH

2008-07-15 01:03:18

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Mon, 14 Jul 2008, [email protected] wrote:
> >
> > Does it waste some ram? Sure. Tough.
>
> I agree with this, but the proponents of the seperate firmware are listing the
> fact that the firmware doesn't tie up ram as one of the big reasons for making
> the change.

That's a totally bogus argument.

The fact is, if you build it into your module, you'll waste at _least_ as
much ram as if you just load it once at module load time.

So there is no actual valid reason to object to "request_firmware()".

I don't know why people get confused about this. I suspect that people
kind of expect that since they need to reload the firmware when resuming
the device, they should also do the "request_firmware()" at resume time.

Maybe it's worth explicitly documenting that request_firmware()/release()
should be done as a module init/exit (or a device detect-eject) time
option. Quite frankly, I think the current firmware docs are actually
actively misleading, because they link the request_firmware() with the
copying to device: quoting from Documentation/firmware_class/README:

High level behavior (driver code):
==================================

if(request_firmware(&fw_entry, $FIRMWARE, device) == 0)
copy_fw_to_device(fw_entry->data, fw_entry->size);
release(fw_entry);

and that is a fundamentally broken world-view.

The logic _should_ be that the firmware is requested at module init or
device discovery, and the release is done at module exit or device eject.

The "request_firmware()" should absolutely *not* be mentally tied to
"copy_fw_to_device" at all. They are very distinct issues, and in fact
must be totally separate for any driver that supports hotplug.

Linus

2008-07-15 01:08:41

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Mon, Jul 14, 2008 at 5:50 PM, David Woodhouse <[email protected]> wrote:
> On Mon, 2008-07-14 at 17:31 -0700, Yinghai Lu wrote:
>> make[1]: *** No rule to make target
>> `include/config/builtin/firmware/dir.h', needed by
>> `firmware/ql2100_fw.bin.gen.S'. Stop.
>> make: *** [firmware] Error 2
>
> Oh, pants. Sorry, I renamed the CONFIG_BUILTIN_FIRMWARE option to
> CONFIG_EXTRA_FIRMWARE some time back, but all my own test builds had the
> include/config/builtin/firmware/dir.h file still lying around so I
> didn't notice that I'd forgotten to change that manual dependency. And
> evidently nobody using linux-next actually tried to include arbitrary
> _other_ firmware into their tree. This patch should fix it...
>
> (Why _do_ we have stray files left in include/config/ when config
> options go away, btw? Shouldn't they get deleted?)
>
> ----
> Subject: firmware: Correct dependency on CONFIG_EXTRA_FIRMWARE_DIR
>
> When CONFIG_EXTRA_FIRMWARE_DIR gets changed, the filename in the .S file
> (which uses .incbin to include the binary) needs to change. When we
> renamed the BUILTIN_FIRMWARE_DIR option to EXTRA_FIRMWARE_DIR, we forgot
> to update the manual dependency in firmware/Makefile, so it was
> depending on a non-existent file in include/config/
>
> Signed-off-by: David Woodhouse <[email protected]>
>
> diff --git a/firmware/Makefile b/firmware/Makefile
> index 5a1e61c..ec4f0cc 100644
> --- a/firmware/Makefile
> +++ b/firmware/Makefile
> @@ -146,7 +146,7 @@ $(patsubst %,$(obj)/%.gen.S, $(fw-shipped-y)): %: $(wordsize_deps) \
> | $(objtree)/$$(dir %)
> $(call cmd,fwbin,$(patsubst %.gen.S,%,$@))
> $(patsubst %,$(obj)/%.gen.S, $(fw-external-y)): %: $(wordsize_deps) \
> - include/config/builtin/firmware/dir.h | $(objtree)/$$(dir %)
> + include/config/extra/firmware/dir.h | $(objtree)/$$(dir %)
> $(call cmd,fwbin,$(fwabs)/$(patsubst $(obj)/%.gen.S,%,$@))
>
> # The .o files depend on the binaries directly; the .S files don't.
>
>
> --
> dwmw2
>

thanks. it works

MK_FW firmware/ql2100_fw.bin.gen.S
MK_FW firmware/ql2300_fw.bin.gen.S
MK_FW firmware/ql2400_fw.bin.gen.S
MK_FW firmware/ql2200_fw.bin.gen.S
MK_FW firmware/ql2322_fw.bin.gen.S
MK_FW firmware/ql2500_fw.bin.gen.S
AS firmware/ql2100_fw.bin.gen.o
AS firmware/ql2300_fw.bin.gen.o
AS firmware/ql2400_fw.bin.gen.o
AS firmware/ql2200_fw.bin.gen.o
AS firmware/ql2322_fw.bin.gen.o
AS firmware/ql2500_fw.bin.gen.o
LD firmware/built-in.o

calling qla2x00_module_init+0x0/0x131
QLogic Fibre Channel HBA Driver: 8.02.01-k4
ACPI: PCI Interrupt 0000:0c:02.0[A] -> GSI 57 (level, low) -> IRQ 57
qla2xxx 0000:0c:02.0: Found an ISP2312, irq 57, iobase 0xffffc2002d00c000
qla2xxx 0000:0c:02.0: Configuring PCI space...
qla2xxx 0000:0c:02.0: Configure NVRAM parameters...
qla2xxx 0000:0c:02.0: Verifying loaded RISC code...
firmware: using built-in firmware ql2300_fw.bin
qla2xxx 0000:0c:02.0: Allocated (412 KB) for firmware dump...

YH

2008-07-15 01:13:33

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Mon, 2008-07-14 at 17:50 -0700, Yinghai Lu wrote:
> MK_FW firmware/tigon/tg3.bin.gen.S
> make[1]: *** No rule to make target `firmware/tigon/tg3.bin', needed
> by `firmware/tigon/tg3.bin.gen.o'. Stop.
> make: *** [firmware] Error 2

I apologise; a line intended for the tg3 conversion accidentally slipped
into firmware/Makefile along with the dabusb patch, when I cherry-picked
from the old tree to omit the drivers/net patches. Again, it didn't show
up in my own testing because I already _had_ the file in question.

(Sorry, Linus. This and the previous fix are now in the for-2.6.27
branch of the same repo as before:
git://git.infradead.org/users/dwmw2/firmware-2.6.git for-2.6.27
)

----
Subject: Fix accidental reference to tg3 firmware

We're not updating the tg3 driver to use request_firmware() yet, but a
reference to its firmware accidentally slipped in as part of commit
c4667746 ("dabusb: use request_firmware()"). Remove it again.

Signed-off-by: David Woodhouse <[email protected]>

diff --git a/firmware/Makefile b/firmware/Makefile
index 10028ac..8dbdd3f 100644
--- a/firmware/Makefile
+++ b/firmware/Makefile
@@ -34,8 +34,6 @@ fw-shipped-$(CONFIG_SND_SB16_CSP) += sb16/mulaw_main.csp sb16/alaw_main.csp \
sb16/ima_adpcm_capture.csp
fw-shipped-$(CONFIG_SND_YMFPCI) += yamaha/ds1_ctrl.fw yamaha/ds1_dsp.fw \
yamaha/ds1e_ctrl.fw
-fw-shipped-$(CONFIG_TIGON3) += tigon/tg3.bin tigon/tg3_tso.bin \
- tigon/tg3_tso5.bin
fw-shipped-$(CONFIG_USB_DABUSB) += dabusb/firmware.fw dabusb/bitstream.bin
fw-shipped-$(CONFIG_USB_EMI26) += emi26/loader.fw emi26/firmware.fw \
emi26/bitstream.fw

--
dwmw2

2008-07-15 01:17:37

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Mon, 14 Jul 2008, Linus Torvalds wrote:

> On Mon, 14 Jul 2008, [email protected] wrote:
>>>
>>> Does it waste some ram? Sure. Tough.
>>
>> I agree with this, but the proponents of the seperate firmware are listing the
>> fact that the firmware doesn't tie up ram as one of the big reasons for making
>> the change.
>
> That's a totally bogus argument.
>
> The fact is, if you build it into your module, you'll waste at _least_ as
> much ram as if you just load it once at module load time.
>
> So there is no actual valid reason to object to "request_firmware()".

you misunderstood me. the people pushing request_firmware() are doing so
on the basis that they won't have to use kernel ram to hold the firmware.
the people pushing for having the option of building the firmware into the
module are acknowleding that this may use a little more ram, but they see
it as being more reliable.

> I don't know why people get confused about this. I suspect that people
> kind of expect that since they need to reload the firmware when resuming
> the device, they should also do the "request_firmware()" at resume time.

according to David W they would, becouse the driver would not keep the
firmware in kernel memory after it's initialized, so if you need to
reinitialize the hardware after a resume you would need to do
request_firmware() at resume time (the option of doing request_firmware()
just before going to sleep has been floated as an option to avoid this
problem)

> Maybe it's worth explicitly documenting that request_firmware()/release()
> should be done as a module init/exit (or a device detect-eject) time
> option. Quite frankly, I think the current firmware docs are actually
> actively misleading, because they link the request_firmware() with the
> copying to device: quoting from Documentation/firmware_class/README:
>
> High level behavior (driver code):
> ==================================
>
> if(request_firmware(&fw_entry, $FIRMWARE, device) == 0)
> copy_fw_to_device(fw_entry->data, fw_entry->size);
> release(fw_entry);
>
> and that is a fundamentally broken world-view.

but that is what David W has been telling everyone is one of the big
benifits of using request_firmware(), that the memory gets released once
the device is initialized. (as always, it's possible I have misunderstood
the argument, but I can't see any other way that it would save memory)

> The logic _should_ be that the firmware is requested at module init or
> device discovery, and the release is done at module exit or device eject.

this would go a long way to satisfying the people who have been objecting
to the request_firmware() push. It would solve the various
reinitialization problems (at resume, and at other times for some
drivers).

the only thing it doesn't satisfy is the people who don't want to be
forced to have seperate firmware files out on the filesystem.

As I understand the arguments, the reason to not force the firmeware files
to be seperate is that it adds complexity and introduces the possibility
that the firmware may be incompatible with the driver, forcing the
creation of some sort of userspace/filesystem versioning.

the reson to force the firmware files to be seperate is that it avoids
using kernel memory to hold them and it allows for the firmware to be
removed from the kernel tree entirely and shipped seperately (apparently
Fedora is eager to do the latter for perceived licensing reasons)

> The "request_firmware()" should absolutely *not* be mentally tied to
> "copy_fw_to_device" at all. They are very distinct issues, and in fact
> must be totally separate for any driver that supports hotplug.

in the last flamewar it seemed pretty clear that the expectation was that
a hotplug event would trigger a new request_firmware() call out to
userspace.

David Lang

2008-07-15 01:44:30

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

From: Andrew Morton <[email protected]>
Date: Mon, 14 Jul 2008 16:41:19 -0700

> On Mon, 14 Jul 2008 16:23:26 -0700
> David Woodhouse <[email protected]> wrote:
>
> > Linus, please pull from the for-2.6.27 branch of:
> > git://git.infradead.org/users/dwmw2/firmware-2.6.git for-2.6.27
>
> The firmware flamewars seem to have subsided lately. Is everyone happy
> with this now?

I'm not happy for network drivers, although I'm happy to see that
David respected out NACKs on tg3/bnx2/etc. and didn't include those
bits.

I still haven't seen an answer to the chip reset issues. The argument
has been that this stuff is going to save kernel memory when the
driver isn't loading firmware, but that's not really possible.

When a lot of these network drivers reset, due to a transmit timeout
or other exception, we need to load the firmware again.

So this means, that if request_firmware() dies or fails for some
reason in these scenerios, we can't reset the network card. Something
as simple as a memory or other allocation failure deep inside of
request_firmware() means we lose the networking interface.

To me this seems like a very non-resilient setup. It makes sense
to just keep the firmware in-core so we can load it without having
to even think about possible failures.

And that to me basically makes this transformation pointless. It's
in fact a regression.

Furthermore, the issue of suspend and resume was brought up. What if
request_firmware() fails then? And can request_firmware() even be
allowed in that context? What if the disk on which the firmware
resides hasn't been resumed yet?

In response to the suspend/resume queries, a lot of special voodoo was
mentioned that perhaps would allow request_firmware() to get invoked
in the correct context and with the firmware partition's block device
resumed already.

But who the heck is going to write all of that code and fix up all
of these drivers so that they can resume reliably again?

And for what?

None of the people who wrote and maintain these effected networking
drivers want these changes installed. And they are the ones who will
have to diagnose strange request_firmware() failures that lead to
non-functioning devices during resume or after a transmit timeout.

2008-07-15 01:46:18

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

David Woodhouse wrote:
> Linus, please pull from the for-2.6.27 branch of:
> git://git.infradead.org/users/dwmw2/firmware-2.6.git for-2.6.27
>
> This does the following (not quite in this order):
>
> - Provides a method for firmware files to be built into the kernel, so
> that drivers can be updated to use request_firmware() without forcing
> people to use an initrd.
>
> - In doing that, mark fw->data as 'const', and fix a few drivers to cope.
>
> - Update a bunch of drivers to use request_firmware(), and move their
> existing firmware, previously hardcoded in all kinds of interesting
> ways, into the firmware/ directory of the kernel tree.
>
> - Add a FIRMWARE_IN_KERNEL config option which allows the firmware for
> the above (and below) drivers to continue to be built into the kernel.
>
> - Add a simple method for representing Intel HEX records in binary form,
> since a bunch of drivers need that kind of thing. And a tool to
> convert HEX files into that binary form, for the kernel to load.
>
> - Drop special cases from a few drivers which could either use
> request_firmware() or build their firmware in; they don't need to
> do that for themselves any more.
>
> - Implement 'make firmware_install' target, which installs all firmware
> shipped with the kernel.
>
> - Modify 'make modules_install' to install any firmware required by the
> modules it installs (where that firmware is shipped in the kernel tree).
>
> There are more drivers to be converted to request_firmware() -- notably
> drivers/net and drivers/scsi, but it's probably best to ensure that the
> infrastructure is all in place and working correctly before we update
> those. (Although since modern drivers have been using request_firmware()
> for some time, I don't think there's likely to be a problem there; we'll
> probably clean up the stragglers in time for 2.6.28.)
>
> David Howells (5):
> Fix a const pointer usage warning in the Digigram VX soundcard driver
> Fix a const assignment in moxa_load_fw()
> Fix a const pointer error in the Conexant cx23418 MPEG encoder driver
> Fix a const pointer usage warning in the Digigram pcxhr soundcard driver
> Fix a const pointer usage warning in the Digigram miXart soundcard driver
>
> Full shortlog and diffstat below, although the really interesting part
> is probably between commits fd4f80de..59890f74 -- seen at
> http://git.infradead.org/users/dwmw2/firmware-2.6.git?a=shortlog;h=59890f74

After the long threads, you would think you would have at least
mentioned controversy attached to these changes?

Or perhaps mention that you are specifically excluding the ability to
build firmware into modules -- the most reliable method available for
those who use modules, MORE reliable than a system where firmware and
kernel module are separated.

Or some of the objections raised, that were not met?

Or even _acknowledge_ that these changes have a high probability of
creating a non-working driver, and therefore advising distros with Big
Fat Warnings and Be Carefuls sprinkled liberally?

Or perhaps describe the outside-the-kernel migration path for distros?

Isn't it silly to create firmware/ top-level dir, when the firmware more
naturally lives in the same dir as the driver to which it is intimately
tied? We'll just have to start creating firmware/net firmware/char
firmware/media etc. after a while.

What about keeping binary objects as binary objects? Surely git can
handle binary file. And you are already converting each firmware's data
from format A to format B. Might as well convert it to unpacked,
un-ASCII'd format that will be loaded directly, as that can be easier in
some cases.

I think it's just downright shady to push this crap without mentioning
any of the swirl of issues surrounding this, that are quite relevant to
its inclusion.

It should be obvious that building firmware into a kernel module (or the
kernel itself, if you don't use modules) is _the_ most reliable method
of ensuring your driver will work [if it requires firmware].

Jeff

2008-07-15 01:50:27

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

From: Linus Torvalds <[email protected]>
Date: Mon, 14 Jul 2008 17:33:58 -0700 (PDT)

> The only reliable way to handle it is to load the firmware before the
> suspend happens - preferably by simply never unloading it in the first
> place.

That's what we were told was the huge benefit of moving some
of these drivers over to this request_firmware() stuff, it would
save memory.

To me, especially for the networking drivers for which I am familiar,
this transformation seems completely pointless from a technical
perspective. In fact it's a regression as-is.

2008-07-15 01:52:18

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

From: [email protected]
Date: Mon, 14 Jul 2008 17:51:37 -0700 (PDT)

> I agree with this, but the proponents of the seperate firmware are listing
> the fact that the firmware doesn't tie up ram as one of the big reasons
> for making the change.

Exactly.

Otherwise these firmware changes are utterly pointless.

2008-07-15 01:55:04

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

Arjan van de Ven wrote:
> On Mon, 14 Jul 2008 16:41:19 -0700
> Andrew Morton <[email protected]> wrote:
>
>> On Mon, 14 Jul 2008 16:23:26 -0700
>> David Woodhouse <[email protected]> wrote:
>>
>>> Linus, please pull from the for-2.6.27 branch of:
>>> git://git.infradead.org/users/dwmw2/firmware-2.6.git
>>> for-2.6.27
>> The firmware flamewars seem to have subsided lately. Is everyone
>> happy with this now?
>>
>
> this seems to have left the contentious pieces out....

Correction: It seems to have completely ignored contentious issues,
which is not the same thing.

It -removes- the ability to build firmware into the kernel module -- a
currently deployed, highly reliable configuration.

I'm not surprised that fact was snipped in David's submission --
wouldn't want to mention that you are unconditionally replacing a
reliable method of firmware loading with a less reliable one.

Building the firmware into the kernel is far less useful than it
appears: often users lack the ability to maintain their own vmlinuz
kernel image build, but do build updated drivers. e.g. driver disks for
Red Hat Enterprise Linux.

IMO the newly added /inability/ to build firmware into kernel modules is
a clear regression.

Jeff

2008-07-15 02:09:41

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Mon, 14 Jul 2008, [email protected] wrote:

> On Mon, 14 Jul 2008, Linus Torvalds wrote:
> >
> > That's a totally bogus argument.
>
> you misunderstood me. the people pushing request_firmware() are doing so on
> the basis that they won't have to use kernel ram to hold the firmware. the
> people pushing for having the option of building the firmware into the module
> are acknowleding that this may use a little more ram, but they see it as being
> more reliable.

I'm just saying that it's a totally bogus argument to claim that it takes
less memory - Either way.

As to reliability, I don't buy that, especially with a generic interface,
and with a way to link the thing in-kernel anyway. Using common
infrastructure is going to be more reliable.

> > I don't know why people get confused about this. I suspect that people
> > kind of expect that since they need to reload the firmware when resuming
> > the device, they should also do the "request_firmware()" at resume time.
>
> according to David W they would, becouse the driver would not keep the
> firmware in kernel memory after it's initialized

And if so, David W is a total moran, and shouldn't have been doing this.

The fact is, there _are_ good arguments for request_firmware(), but they
have nothing what-so-ever to do with memory use or anything like that.

The argument for request_firmware() is that it's a good _single_ interface
to the whole firmware issue, allowing us to split up the driver from the
firmware without every driver having to do some hack of its own.

Not memory use.

So next time you see somebody arguing about memory use (either way), just
slap them, and tell them Linus told you to.

Linus

2008-07-15 02:12:42

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

David Woodhouse wrote:
> On Mon, 2008-07-14 at 17:06 -0700, [email protected] wrote:
>> On Mon, 14 Jul 2008, Arjan van de Ven wrote:
>>
>>> On Mon, 14 Jul 2008 16:41:19 -0700
>>> Andrew Morton <[email protected]> wrote:
>>>
>>>> On Mon, 14 Jul 2008 16:23:26 -0700
>>>> David Woodhouse <[email protected]> wrote:
>>>>
>>>>> Linus, please pull from the for-2.6.27 branch of:
>>>>> git://git.infradead.org/users/dwmw2/firmware-2.6.git
>>>>> for-2.6.27
>>>> The firmware flamewars seem to have subsided lately. Is everyone
>>>> happy with this now?
>>>>
>>> this seems to have left the contentious pieces out....
>> almost, the one thing that this could have done would be to offer an
>> option to bundle the firmware with the module. currently it offers the
>> option to load the firmware at the same time as the module, not _quite_
>> the same thing.
>
> I see no real point in that. If you have userspace to load modules, then
> you have userspace to load firmware.

You have been provided with several counter-examples here.

Driver disks, initrd, and other such constructs are not necessarily the
land of happy, fat and healthy userspace that you believe it universally
to be. Image build scripts need to be aware that they need to copy
firmware. Some already know -- but many, particularly with networking
-- were such simple one-driver affairs that nobody bothered to script in
the extra smarts.

> We made 'make modules_install' install the firmware in /lib/firmware for
> you at the same time as it installs the modules, to avoid problems if
> people forgot to run 'make firmware_install'. That really ought to be
> enough.

And it was like pulling teeth, just get that change in, ensuring the
normal build process will not suddenly produce non-working drivers...
which it did for several kernel hackers. As predicted.

> I know Jeff complained that it wasn't, and insisted that he _needed_ to
> be able to scp a single .ko file around which contained both, and the
> world was broken if he could not -- but I disagree with him.

It's simple logic: existing systems DO copy around modules. You seem
to have forgotten an example not so easily derided: driver disks and
their build scripts, which are used all over the place.

Particularly in cases, like enterprise distros, where you are updating a
driver but never touch the main kernel image.

That will continue to be true even when enterprise distros are based off
of 2.6.27 or later.

> But since I wanted this tree to be uncontentious and obviously the
> correct thing to do, I've dropped the drivers/net changes for now
> anyway.
>
> It's odd that this request has suddenly come out of the blue when we've
> been using request_firmware() from modules for years already.

Why is it out of the blue to worry about a working driver suddenly
ceasing to work?

This has nothing to do with request_firmware() itself -- its about
having the infrastructure in place so that users do not notice the switch.

Jeff

2008-07-15 02:13:41

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Mon, 14 Jul 2008, Jeff Garzik wrote:
>
> IMO the newly added /inability/ to build firmware into kernel modules is a
> clear regression.

IMO you're being stupid.

How about explainign why it makes any difference what-so-ever?

If you can load the module, you can load the firmware. Claiming anything
else is just _stupid_.

And if you cannot see the advantage of having _one_ interface to firmware
loading, instead of each driver picking a random approach, I don't know
what to say.

The fact that some people have been confused and apparently blathered
about memory savings is totally immaterial. Ignore the stupid people.

Linus

2008-07-15 02:17:47

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

From: Jeff Garzik <[email protected]>
Date: Mon, 14 Jul 2008 22:12:24 -0400

> David Woodhouse wrote:
> > But since I wanted this tree to be uncontentious and obviously the
> > correct thing to do, I've dropped the drivers/net changes for now
> > anyway.
> >
> > It's odd that this request has suddenly come out of the blue when we've
> > been using request_firmware() from modules for years already.
>
> Why is it out of the blue to worry about a working driver suddenly
> ceasing to work?
>
> This has nothing to do with request_firmware() itself -- its about
> having the infrastructure in place so that users do not notice the switch.

And I want to reiterate my adversion to the folks who keep saying
that being opposed to request_firmware() is just like being opposed
to modules.

That's is very far from the truth.

When module support was added, guess what? I could still build a
completely static kernel image like I always could.

And in fact, to this day, that's what I personally do because that's
how I like my kernels.

But this request_firmware() change does not allow one to get what he
could get before, which is a completely self-contained driver module
object file.

This is the difference between providing an option and making
something mandatory. This firmware split up is now mandatory.

2008-07-15 02:19:59

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

From: Linus Torvalds <[email protected]>
Date: Mon, 14 Jul 2008 19:13:16 -0700 (PDT)

> On Mon, 14 Jul 2008, Jeff Garzik wrote:
> >
> > IMO the newly added /inability/ to build firmware into kernel modules is a
> > clear regression.
>
> IMO you're being stupid.
>
> How about explainign why it makes any difference what-so-ever?

Just like Jeff, I think it does.

> If you can load the module, you can load the firmware. Claiming anything
> else is just _stupid_.

It's about having one foo.ko file that is self contained.

We've always been able to do that.

Just like how when module support was added to the kernel, and we
could still build everything statically into one kernel image if we
wanted to, we likewise should still be able to get everything
necessary for a driver module into a single foo.ko file.

2008-07-15 02:21:54

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On 15-07-08 03:52, David Miller wrote:

> From: [email protected] Date: Mon, 14 Jul 2008 17:51:37 -0700 (PDT)
>
>> I agree with this, but the proponents of the seperate firmware are
>> listing the fact that the firmware doesn't tie up ram as one of the
>> big reasons for making the change.
>
> Exactly.
>
> Otherwise these firmware changes are utterly pointless.

The point of them is legal. Which everyone knows so could perhaps that
particular dance be done away with?

The current patches allow to

1) load the firmware from userspace
2) load the firmware from the kernel image (vmlinux)

They lack the ability to load the firmware from the module image.

From a technical standpoint that seems an odd setup. As a driver author
you'd either be satisfied with loading the firmware from userspace, or
if for example you feel you're very much tied to your specific firmware,
you want it to be as close to you as possible -- in the kernel image
while you yourself are a module makes little sense. You couldn't for
example just copy the single module around.

That seems a clear technical point to fix but after that, the point is
legal. At least for existing drivers, this consists of distributions
worrying about GPL consequences and wanting to seperate out the existing
firmware blobs.

They do worry -- whatever those worries might be worth -- so there's the
point. You may feel it not a good point but that's something other than
no point existing.

I followed along on these flamewars and I'd say to reject these current
patches on technical grounds until the ability to build the firmware
into the module exists and merge it after -- even though people might
feel there's little or no upside to them, I'd say there's little of no
downside either once that ability is in.

Rene.

2008-07-15 02:22:56

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

Linus Torvalds wrote:
> On Mon, 14 Jul 2008, Jeff Garzik wrote:
>> IMO the newly added /inability/ to build firmware into kernel modules is a
>> clear regression.
>
> IMO you're being stupid.
>
> How about explainign why it makes any difference what-so-ever?
>
> If you can load the module, you can load the firmware. Claiming anything
> else is just _stupid_.

False -- for every router build script, driver disk build script, etc.
that has not yet been updated to copy both a module and the
/lib/firmware that the kernel tree is newly spitting out.

And for every case, the breakage is completely silent, until one boots
into a firmware-less image and loads drivers.

> And if you cannot see the advantage of having _one_ interface to firmware
> loading, instead of each driver picking a random approach, I don't know
> what to say.

It's a great approach! As I've said over and over and over (and now
over) again.

My complaints are about --not breaking stuff--, not request_firmware().

You have to look at the build process, second stage image builders,
embedded system image creators, and other "hangers on" that do not look
and behave like a mainstream Linux distro.

It is a mistake to assume that all systems are _already_ prepared to
install and digest new /lib/firmware files that the kernel build now
spits out.

And the consequences of that assumption is a non-working driver, often a
non-booting system, not just a few quirks here and there.

Jeff

2008-07-15 02:24:20

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Mon, 2008-07-14 at 19:17 -0700, David Miller wrote:
> When module support was added, guess what? I could still build a
> completely static kernel image like I always could.
>
> And in fact, to this day, that's what I personally do because that's
> how I like my kernels.

Good. You can still do precisely that, and build the firmware into your
kernel. You can have exactly what you like. Hey, you can build even
_more_ firmware into your kernel now. You can have NFS-root on devices
you previously had to use an initrd for. hth.

> But this request_firmware() change does not allow one to get what he
> could get before, which is a completely self-contained driver module
> object file.
>
> This is the difference between providing an option and making
> something mandatory. This firmware split up is now mandatory.

In all the years we've been using request_firmware(), nobody ever asked
for a way to build the firmware _into_ the .ko file, until now. Why is
it suddenly so important for a small handful of older network drivers,
when nobody else has ever seen the need for it -- even in modern network
drivers?

--
dwmw2

2008-07-15 02:24:38

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

From: Rene Herman <[email protected]>
Date: Tue, 15 Jul 2008 04:23:14 +0200

> On 15-07-08 03:52, David Miller wrote:
>
> > From: [email protected] Date: Mon, 14 Jul 2008 17:51:37 -0700 (PDT)
> >
> >> I agree with this, but the proponents of the seperate firmware are
> >> listing the fact that the firmware doesn't tie up ram as one of the
> >> big reasons for making the change.
> >
> > Exactly.
> >
> > Otherwise these firmware changes are utterly pointless.
>
> The point of them is legal.

Thanks for proving something I tried to establish for weeks
but which Alan Cox, David W., and others vehemently denied.

They states that it was being done on a technical basis rather
than being predominantly a legal one.

2008-07-15 02:31:18

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On 15-07-08 04:24, David Miller wrote:

> From: Rene Herman <[email protected]>
> Date: Tue, 15 Jul 2008 04:23:14 +0200
>
>> On 15-07-08 03:52, David Miller wrote:
>>
>>> From: [email protected] Date: Mon, 14 Jul 2008 17:51:37 -0700 (PDT)
>>>
>>>> I agree with this, but the proponents of the seperate firmware are
>>>> listing the fact that the firmware doesn't tie up ram as one of the
>>>> big reasons for making the change.
>>> Exactly.
>>>
>>> Otherwise these firmware changes are utterly pointless.
>> The point of them is legal.
>
> Thanks for proving something I tried to establish for weeks
> but which Alan Cox, David W., and others vehemently denied.
>
> They states that it was being done on a technical basis rather
> than being predominantly a legal one.

Yes, they were obstinate or dishonest (I won't say "respectively"). As
to "proving" though, I cannot prove anything, being a mere observer.

At this point I really believe this discussion should be about the other
part of my reply -- the point mostly put forward by Jef Garzik about the
firmware inside the module image. Without that ability, I don't believe
these are good patches.

_With_ that ability, I myself do. Let's allow allow everyone their own
level of fear, uncertainty and doubt.

Rene

2008-07-15 02:34:51

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Mon, 14 Jul 2008, Jeff Garzik wrote:
>
> My complaints are about --not breaking stuff--, not request_firmware().

Guys, request_firmware() has been around for about five years now.

If this discussion had happened in 2003, I'd agree.

As it is, I think it's time to just face it - many people do want to have
a unified interface (and yes, I obviously count myself in that group), and
you can't just continue do nothing and ignore that.

Your argument seems to be that you don't want to break anything, but
you've also clearly not wanted to _fix_ anything for the last five years.

At some point, somebody has to just do it. And five years is _way_ long
enough.

Linus

2008-07-15 02:41:26

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

From: Linus Torvalds <[email protected]>
Date: Mon, 14 Jul 2008 19:33:25 -0700 (PDT)

>
>
> On Mon, 14 Jul 2008, Jeff Garzik wrote:
> >
> > My complaints are about --not breaking stuff--, not request_firmware().
>
> Guys, request_firmware() has been around for about five years now.

Jeff's not against request_firmware() in an of itself.

He's against the fact that it's not possible to build a
self-contained foo.ko module file any longer, and that
would definitely be possible with using request_firmware().

2008-07-15 02:44:55

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Mon, 14 Jul 2008, David Miller wrote:
>
> They states that it was being done on a technical basis rather
> than being predominantly a legal one.

No.

Yes, the original reason for request_firmware() was obviously very much
partly legal.

HOWEVER.

Once you have a model that is required (for whatever reasons) for some
drivers, we're much better off using the _same_ model for all drivers,
whether it is necessary for legal reasons for those other drivers.

Put this way: if you do a distro, you _need_ to support firmware loading
anyway. And once you do that, it's just annoying how some drivers then do
something odd and special for the same thing, for no real good reason.

It's like having each wireless driver do their own 802.11 stack. It works,
and all those driver writers seem to be always convinced that it's the
right thing to do. But as a network maintainer, do you like it?

Now, replace "802.11 stack" with "firmware loader", and then replace
"network maintainer" with "kernel maintainer".

And maybe you can see why I like just making everybody do
request_firmware().

Do you call that a legal reason? Or a technical one?

Linus

2008-07-15 02:46:16

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

From: Linus Torvalds <[email protected]>
Date: Mon, 14 Jul 2008 19:39:03 -0700 (PDT)

> Put this way: if you do a distro, you _need_ to support firmware loading
> anyway. And once you do that, it's just annoying how some drivers then do
> something odd and special for the same thing, for no real good reason.

In what way is it annoying?

Most distribution people aren't even aware that drivers like tg3 and
bnx2 even have firmware. In fact it's self contained and less for
them to worry about.

The correct firmware is right there in the driver, it's what the
driver maintainer tested as a unit, and everything.

It neither gets in the distro guys way nor causes any functional
problems.

It may cause potential legal issues, but that is another matter.

> It's like having each wireless driver do their own 802.11 stack. It works,
> and all those driver writers seem to be always convinced that it's the
> right thing to do. But as a network maintainer, do you like it?
>
> Now, replace "802.11 stack" with "firmware loader", and then replace
> "network maintainer" with "kernel maintainer".

Distro folks do ship external drivers that have their own 802.11
stack, so this is a very good example in fact :-)

I think the modular vs. static kernel image is a better analogy.
At least with that I have a real choice, with this stuff I suddently
won't.

Regardless of what you or I think about these changes, the fact is
that there were definitely some areas of contention and even some
level of misrepresentation of these changes. And given that, some
of us feel that merging this in right now was on the premature side.

2008-07-15 02:47:20

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Mon, 14 Jul 2008, Linus Torvalds wrote:

> On Mon, 14 Jul 2008, [email protected] wrote:
>
>> On Mon, 14 Jul 2008, Linus Torvalds wrote:
>>>
>>> That's a totally bogus argument.
>>
>> you misunderstood me. the people pushing request_firmware() are doing so on
>> the basis that they won't have to use kernel ram to hold the firmware. the
>> people pushing for having the option of building the firmware into the module
>> are acknowleding that this may use a little more ram, but they see it as being
>> more reliable.
>
> I'm just saying that it's a totally bogus argument to claim that it takes
> less memory - Either way.
>
> As to reliability, I don't buy that, especially with a generic interface,
> and with a way to link the thing in-kernel anyway. Using common
> infrastructure is going to be more reliable.
>
>>> I don't know why people get confused about this. I suspect that people
>>> kind of expect that since they need to reload the firmware when resuming
>>> the device, they should also do the "request_firmware()" at resume time.
>>
>> according to David W they would, becouse the driver would not keep the
>> firmware in kernel memory after it's initialized
>
> And if so, David W is a total moran, and shouldn't have been doing this.
>
> The fact is, there _are_ good arguments for request_firmware(), but they
> have nothing what-so-ever to do with memory use or anything like that.
>
> The argument for request_firmware() is that it's a good _single_ interface
> to the whole firmware issue, allowing us to split up the driver from the
> firmware without every driver having to do some hack of its own.

I don't think anyone objects to request_firmware() as a common interface.
they object to forcing the firmware to be seperate.

in the monolithic kernel case the people pushing this bowed to preasure
and improved request_firmware() so that the firmware could be embedded in
the kernel image as opposed to the initial approach of requireing an
initramfs.

Jeff and David M are arguing that it should be possible to also embed the
firmware into the module .ko file. If that option was available I don't
think there would be opposition to having the one interface. the lack of
that option is causing them to object and point out cases where it's not
as good as previously.

David Lang

> Not memory use.
>
> So next time you see somebody arguing about memory use (either way), just
> slap them, and tell them Linus told you to.
>
> Linus
>

2008-07-15 02:49:21

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Mon, 2008-07-14 at 22:22 -0400, Jeff Garzik wrote:
> You have to look at the build process, second stage image builders,
> embedded system image creators, and other "hangers on" that do not look
> and behave like a mainstream Linux distro.
>
> It is a mistake to assume that all systems are _already_ prepared to
> install and digest new /lib/firmware files that the kernel build now
> spits out.

I didn't just assume it. I checked the major distributions -- and they
_do_ check the MODULE_FIRMWARE() tags in the modules they include, and
include the corresponding firmware. The firmware loader has been around
for a _long_ time now, and they've had to support this for almost as
long.

Yes, there are occasional bugs -- and I'm sure there _are_ some minor
distributions or 'image creators' out there which still don't cope.

But that isn't a new problem -- and it's certainly not _caused_ by us
updating a small handful of the older drivers to match what modern
drivers do.

--
dwmw2

2008-07-15 02:52:43

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Mon, 14 Jul 2008, David Woodhouse wrote:

> On Mon, 2008-07-14 at 19:17 -0700, David Miller wrote:
>> When module support was added, guess what? I could still build a
>> completely static kernel image like I always could.
>>
>> And in fact, to this day, that's what I personally do because that's
>> how I like my kernels.
>
> Good. You can still do precisely that, and build the firmware into your
> kernel. You can have exactly what you like. Hey, you can build even
> _more_ firmware into your kernel now. You can have NFS-root on devices
> you previously had to use an initrd for. hth.

we recognise that a monolithic kernel can have the firmware embedded in it
(and I thank you for doing so, this will help me with my Intel wireless
cards). the objection is that you can't embed firmware into modules.

>> But this request_firmware() change does not allow one to get what he
>> could get before, which is a completely self-contained driver module
>> object file.
>>
>> This is the difference between providing an option and making
>> something mandatory. This firmware split up is now mandatory.
>
> In all the years we've been using request_firmware(), nobody ever asked
> for a way to build the firmware _into_ the .ko file, until now. Why is
> it suddenly so important for a small handful of older network drivers,
> when nobody else has ever seen the need for it -- even in modern network
> drivers?

the reason the objections are showing up now is that the drivers that
previously had the firmware embedded in them (so that nobody but the
driver author needed to even know that there _was_ firmware involved) are
now being converted to use request_firmware(). that change is breaking the
invisability of the firmware that they have cultivated. if they had the
option to keep things as they are (even while you provide the option for
others to move the firmware out) they could not object like they are.

David Lang

2008-07-15 02:53:18

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Mon, 14 Jul 2008, David Miller wrote:
>
> Jeff's not against request_firmware() in an of itself.
>
> He's against the fact that it's not possible to build a
> self-contained foo.ko module file any longer, and that
> would definitely be possible with using request_firmware().

.. but is it really that big of a deal?

If it were about not being able to build a self-contained non-modular
"vmlinux", I'd be upset too.

But once you can load a module, you can load the firmware. You just have
to _remember_ to move it along with the module.

I mean really, you're the one who was complaining that there was no
technical reason, but now you're apparently using the "alzheimer's
defense".

Besides, it's not even true that foo.ko modules are self-contained. We've
for years and years (pretty much since day one) had nesting module models,
where in order to load foo.ko you need to load baz.ko first. Yes, the
firmware file is named differently, but it _is_ different.

Linus

2008-07-15 02:54:59

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Mon, 2008-07-14 at 18:50 -0700, David Miller wrote:
> From: Linus Torvalds <[email protected]>
> Date: Mon, 14 Jul 2008 17:33:58 -0700 (PDT)
>
> > The only reliable way to handle it is to load the firmware before the
> > suspend happens - preferably by simply never unloading it in the first
> > place.
>
> That's what we were told was the huge benefit of moving some
> of these drivers over to this request_firmware() stuff, it would
> save memory.

There _are_ RAM benefits for a number of drivers, even when you load the
firmware just once at initialisation time and keep it resident all the
time the device is active (and no, I won't slap myself for saying it).

The tg3 driver is a prime example -- it contains three separate firmware
images for different revisions of the card, but you'll almost never
actually need all three. Mostly you'll only need _one_ of them.

--
dwmw2

2008-07-15 03:04:58

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On 15-07-08 04:39, Linus Torvalds wrote:

> On Mon, 14 Jul 2008, David Miller wrote:
>> They states that it was being done on a technical basis rather
>> than being predominantly a legal one.
>
> No.
>
> Yes, the original reason for request_firmware() was obviously very much
> partly legal.
>
> HOWEVER.
>
> Once you have a model that is required (for whatever reasons) for some
> drivers, we're much better off using the _same_ model for all drivers,
> whether it is necessary for legal reasons for those other drivers.
>
> Put this way: if you do a distro, you _need_ to support firmware loading
> anyway. And once you do that, it's just annoying how some drivers then do
> something odd and special for the same thing, for no real good reason.

There's little alternative infrastructure in declaring a static array of
unsigned char though.

Yes, it still makes sense to make everyone use request_firmware() if
only because these other drivers now all of a sudden have a more
accessible way of updating their firmwares, but David's objective is
legal here.

Which is fine.

But the very clear _technical_ objection about this not providing for
keeping modules and their firmware together for those that really do
want it to be that way was not at all addressed. Note, this while still
using request_firmware(), just at the option of the kernel builder with
the firmware compiled into the module. Right now, this allows the
firmware to be compiled into the vmlinux only which makes fairly little
sense if the driver itself is a module.

Rene.

2008-07-15 03:05:50

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

Linus Torvalds wrote:
>
> On Mon, 14 Jul 2008, David Miller wrote:
>> Jeff's not against request_firmware() in an of itself.
>>
>> He's against the fact that it's not possible to build a
>> self-contained foo.ko module file any longer, and that
>> would definitely be possible with using request_firmware().
>
> .. but is it really that big of a deal?
>
> If it were about not being able to build a self-contained non-modular
> "vmlinux", I'd be upset too.
>
> But once you can load a module, you can load the firmware. You just have
> to _remember_ to move it along with the module.

And failure to "remember" to update associated build image scripts
silently and without complaint producing a non-working driver. For
drivers that, in many cases, have worked for 5 or 10 or more years
without needing additional files.

So it's not about remembering. You can't just wish new userland and
build processes upon the universe.

Heck, you yourself have in the past supported building current kernels
on older userlands -- this flies in the face of that.

> Besides, it's not even true that foo.ko modules are self-contained. We've
> for years and years (pretty much since day one) had nesting module models,
> where in order to load foo.ko you need to load baz.ko first. Yes, the
> firmware file is named differently, but it _is_ different.

It's not only named differently, it is in a markedly different directory
hierarchy that isn't automatically picked up alongside *.ko.

But fundamentally, why remove a capability that works, thereby depriving
system builders of the ability to _choose_ when, if ever, to prefer
/lib/firmware to built-in firmware?

Let's standardize on a request_firmware() sure, one that does not remove
that choice.

Jeff

2008-07-15 03:15:56

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

Linus Torvalds wrote:
>
> On Mon, 14 Jul 2008, Jeff Garzik wrote:
>> My complaints are about --not breaking stuff--, not request_firmware().
>
> Guys, request_firmware() has been around for about five years now.
>
> If this discussion had happened in 2003, I'd agree.
>
> As it is, I think it's time to just face it - many people do want to have
> a unified interface (and yes, I obviously count myself in that group), and
> you can't just continue do nothing and ignore that.
>
> Your argument seems to be that you don't want to break anything, but
> you've also clearly not wanted to _fix_ anything for the last five years.
>
> At some point, somebody has to just do it. And five years is _way_ long
> enough.

In threads of years past, you've agreed with the plainly obvious --
built-in firmware is obviously more reliable, and is more likely to
produce a working driver.

It doesn't break anything to keep this feature.

Keeping this feature doesn't prevent any other fixes.

Keeping this feature closes the window for potential regressions.

And keeping this feature means Linux users can make a _choice_, rather
than forcing /lib/firmware on them.

Do you really think embedded system and micro distro authors are in 100%
agreement that /lib/firmware is best for their situation? Really?

Jeff

2008-07-15 03:31:31

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Mon, 14 Jul 2008, Jeff Garzik wrote:
>
> Do you really think embedded system and micro distro authors are in 100%
> agreement that /lib/firmware is best for their situation? Really?

Umm. Why are you bringing up these total red herrings?

Those embedded systems can just build everything into one image, and are
much better off using no modules at all.

But how about instead of complaining, help add the infrastructure then, so
that you can insert those firmware things into modules and still use just
one single interface?

Move _forward_, not backwards.

Linus

2008-07-15 04:46:34

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

Hi,

just noticed this thread which frightens me...

On Mon, Jul 14, 2008 at 07:23:55PM -0700, David Woodhouse wrote:
> In all the years we've been using request_firmware(), nobody ever asked
> for a way to build the firmware _into_ the .ko file, until now. Why is
> it suddenly so important for a small handful of older network drivers,
> when nobody else has ever seen the need for it -- even in modern network
> drivers?

Because most often, you need to bring the network driver from any location
to your machine by any method *before* you can transfer anything easily.
It's already painful to have to load modules from userland, please do not
complicate the installation process. I can assure you that it's painful
to boot a system and see that neither network nor USB work. When you
find a way to pass a network driver there through any method (included
writing it onto a CD), you're finally safe. If now you have to iterate
because once you modprobe, you notice that *some* firmware files are
missing, it'll get really really annoying.

You say that it is suddenly important, but I disagree. The only network
driver I have which needs a firmware is myri10ge. And guess what ? Getting
it to work on test platforms was a real hassle. Upon modprobe, the system
would wait 2 minutes (I never understood why we have to wait when a file
is missing, but that's another story of crappy design). From the first
second, I could think "oh sh*t, the firmware again...". The I had to get
the firmware files, hotplug script and also the script to configure /proc
and /sys entries just to be able to load this driver. Fortunately, I could
transfer these files via the other NIC which did not require a firmware to
be loaded.

It was so much hard to get it to work on test platforms that I finally
found how to rebuild the myri10ge driver with the firmware included in
it, and now it has changed my life. Simply modprobe and that's all.
Firmwares are so much annoying that I think that those should just be
optional, and be built into modules *by default*. There should be a
really valid reason for wanting to extract them. Upgradability is not
one, because there are more frequent changes in modules than in firmwares.

Just my 2 cents,
Willy

2008-07-15 04:57:17

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Mon, 14 Jul 2008 19:45:57 -0700 (PDT)
David Miller <[email protected]> wrote:

> From: Linus Torvalds <[email protected]>
> Date: Mon, 14 Jul 2008 19:39:03 -0700 (PDT)
>
> > Put this way: if you do a distro, you _need_ to support firmware
> > loading anyway. And once you do that, it's just annoying how some
> > drivers then do something odd and special for the same thing, for
> > no real good reason.
>
> In what way is it annoying?
>
> Most distribution people aren't even aware that drivers like tg3 and
> bnx2 even have firmware. In fact it's self contained and less for
> them to worry about.

and.. after this patch that still seems to be the case, unless I'm
looking at it really cross eyed.
Nothing in this patch makes it impossible to do so.. or changes the
drivers you mention. in fact it's the opposite almost: it improves
request_firmware() usability significantly. (by allowing vmlinux build
in etc).

--
If you want to reach me at my work email, use [email protected]
For development, discussion and tips for power savings,
visit http://www.lesswatts.org

2008-07-15 04:59:42

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Mon, Jul 14, 2008 at 07:51:55PM -0700, Linus Torvalds wrote:
>
>
> On Mon, 14 Jul 2008, David Miller wrote:
> >
> > Jeff's not against request_firmware() in an of itself.
> >
> > He's against the fact that it's not possible to build a
> > self-contained foo.ko module file any longer, and that
> > would definitely be possible with using request_firmware().
>
> .. but is it really that big of a deal?
>
> If it were about not being able to build a self-contained non-modular
> "vmlinux", I'd be upset too.
>
> But once you can load a module, you can load the firmware. You just have
> to _remember_ to move it along with the module.

In order to transfer it, right now you feed it through a working device.
When that device itself requires firmware to work, you will suddenly
discover that it becomes harder and harder to get any communication device
to work on your target system.

Then, in order to load it, you need to have a properly set up system
(hotplug scripts, firmware at the proper location, etc...). This is
really not convenient on embedded systems or live CD systems. In fact,
among all my machines, only my notebook is currently able to load a
firmware.

Also, there's a versionning problem. It's easy to rename a driver
"foo.ko" => "foo-1.ko" and just load the exact file or hold several
versions there by simply moving a symlink. Having to fiddle around
with firmware files depending on the driver version will get much
harder.

Also, when you're cleaning up a system after installation, how do
you know which firmware files you need to keep and which ones you
can safely remove ?

Just like Jeff and Davem, I really want to be able to continue to
use my drivers like I have done since modules were introduced.
Switching a single-file to a multi-file/multi-dir installation would
be a huge maintenance regression IMHO.

Willy

2008-07-15 05:09:33

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Mon, 2008-07-14 at 21:56 -0700, Arjan van de Ven wrote:
> On Mon, 14 Jul 2008 19:45:57 -0700 (PDT)
> David Miller <[email protected]> wrote:
>
> > From: Linus Torvalds <[email protected]>
> > Date: Mon, 14 Jul 2008 19:39:03 -0700 (PDT)
> >
> > > Put this way: if you do a distro, you _need_ to support firmware
> > > loading anyway. And once you do that, it's just annoying how some
> > > drivers then do something odd and special for the same thing, for
> > > no real good reason.
> >
> > In what way is it annoying?
> >
> > Most distribution people aren't even aware that drivers like tg3 and
> > bnx2 even have firmware. In fact it's self contained and less for
> > them to worry about.
>
> and.. after this patch that still seems to be the case, unless I'm
> looking at it really cross eyed.
> Nothing in this patch makes it impossible to do so.. or changes the
> drivers you mention.

Well, the HEAD of my tree (and linux-next) does change tg3. I just
haven't asked Linus to pull that part, because I wanted to keep the
contentious part separate from the simple and obvious parts.

(Even though the tg3 fix _is_ simple and obvious, to me. But that
evidently doesn't stop it from being contentious too. So it can wait.)

--
dwmw2

2008-07-15 05:09:51

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Mon, Jul 14, 2008 at 08:29:13PM -0700, Linus Torvalds wrote:
>
>
> On Mon, 14 Jul 2008, Jeff Garzik wrote:
> >
> > Do you really think embedded system and micro distro authors are in 100%
> > agreement that /lib/firmware is best for their situation? Really?
>
> Umm. Why are you bringing up these total red herrings?

It's not red herring, Linus. I (as a user, as an embedded system developer
and as a micro distro maintainer) do really care.

> Those embedded systems can just build everything into one image, and are
> much better off using no modules at all.

While it is sometimes acceptable to have everything built into one image,
sometimes your embedded system is modular in that the same image works
on several hardware variants. In this case, it's largely preferable to
work with modules.

> But how about instead of complaining, help add the infrastructure then, so
> that you can insert those firmware things into modules and still use just
> one single interface?

I think that would make everyone happy. It would be worth taking a
look at myri10ge version 1.4.2 to see how they managed to support
both mechanisms.

> Move _forward_, not backwards.

This is just a matter of point of view. From my POV, having to feed
several unrelated files into a system to be able to get it to support
any device is moving backwards, compared to the currently clean all-in-1
driver model.

BTW, I still did not understand what existing problem we are trying
to solve by this move "forward".

Willy

2008-07-15 05:13:50

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Mon, Jul 14, 2008 at 07:48:32PM -0700, David Woodhouse wrote:
> On Mon, 2008-07-14 at 22:22 -0400, Jeff Garzik wrote:
> > You have to look at the build process, second stage image builders,
> > embedded system image creators, and other "hangers on" that do not look
> > and behave like a mainstream Linux distro.
> >
> > It is a mistake to assume that all systems are _already_ prepared to
> > install and digest new /lib/firmware files that the kernel build now
> > spits out.
>
> I didn't just assume it. I checked the major distributions -- and they
> _do_ check the MODULE_FIRMWARE() tags in the modules they include, and
> include the corresponding firmware. The firmware loader has been around
> for a _long_ time now, and they've had to support this for almost as
> long.

David, "being around" and "being heavily used" are not the same. My
experience with firmware is limited to WiFi drivers. I've never seen
that annoying thing on a server yet. Not that it does not exist there,
just that fortunately I've not being facing it.

> Yes, there are occasional bugs -- and I'm sure there _are_ some minor
> distributions or 'image creators' out there which still don't cope.
>
> But that isn't a new problem -- and it's certainly not _caused_ by us
> updating a small handful of the older drivers to match what modern
> drivers do.

Pardon? If by "updating" you mean you keep backwards compatibility, OK.
But if by "updating" you mean "breaking existing behaviour", yes, it
is your change which will cause the problems.

Willy

2008-07-15 05:20:08

by Benjamin Herrenschmidt

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Mon, 2008-07-14 at 17:44 -0700, David Woodhouse wrote:
> On Mon, 2008-07-14 at 17:40 -0700, Linus Torvalds wrote:
> >
> > On Mon, 14 Jul 2008, David Woodhouse wrote:
> > >
> > > I see no real point in that. If you have userspace to load modules, then
> > > you have userspace to load firmware.
> >
> > David, the case is different: the module is loaded _across_ the
> > suspend/resume, but the device needs to have its firmware reloaded at
> > resume time because it lost power.
>
> Either I put my reply in response to the wrong quotation, or you thought
> I did.
>
> I intended the above to be a response to the "please put firmware _into_
> the .ko file" request. Not to the suspend/resume thing.
>
> Yes, I agree that suspend/resume is a non-issue.

Besides, request_firmware() from a resume() method is broken. Has always
been and will probably always be.

Ben.

2008-07-15 05:20:33

by Benjamin Herrenschmidt

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

>
> Thanks for proving something I tried to establish for weeks
> but which Alan Cox, David W., and others vehemently denied.
>
> They states that it was being done on a technical basis rather
> than being predominantly a legal one.

Despite the risk of going against the current here, I do like David
patches for one major reason: I love my kernels with everything
built-in, that's how I can netboot them etc... without having to bother
with installing modules.

David's patches allow me to do that with drivers that would have
-required- disk based firmwares beforehand. It allows things like
rooting off a block device that needs a firmware without an initrd,
etc....

As long as the firmwares are shipped with the kernel tree, I'm happpy,
and I don't care much about building the firmware inside the .ko's. Now,
if somebody care, I suppose it's a feature that can be added fairly
transparently.

Cheers,
Ben.

2008-07-15 05:30:45

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 2008-07-15 at 15:15 +1000, Benjamin Herrenschmidt wrote:
> >
> > Thanks for proving something I tried to establish for weeks
> > but which Alan Cox, David W., and others vehemently denied.
> >
> > They states that it was being done on a technical basis rather
> > than being predominantly a legal one.
>
> Despite the risk of going against the current here, I do like David
> patches for one major reason: I love my kernels with everything
> built-in, that's how I can netboot them etc... without having to bother
> with installing modules.

I think you're definitely _with_ the current on that one.

> David's patches allow me to do that with drivers that would have
> -required- disk based firmwares beforehand. It allows things like
> rooting off a block device that needs a firmware without an initrd,
> etc....
>
> As long as the firmwares are shipped with the kernel tree, I'm happpy,

They don't even need to be shipped with the kernel tree. You can set
CONFIG_EXTRA_FIRMWARE_DIR to point to wherever you keep them
(like /lib/firmware) and it'll pull them in from there. That's _why_
it's useful for drivers which would have required an initrd before, and
which don't have their firmware shipped in the kernel tree.

--
dwmw2

2008-07-15 05:31:30

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On 15-07-08 06:56, Arjan van de Ven wrote:

> On Mon, 14 Jul 2008 19:45:57 -0700 (PDT)
> David Miller <[email protected]> wrote:
>
>> From: Linus Torvalds <[email protected]>
>> Date: Mon, 14 Jul 2008 19:39:03 -0700 (PDT)
>>
>>> Put this way: if you do a distro, you _need_ to support firmware
>>> loading anyway. And once you do that, it's just annoying how some
>>> drivers then do something odd and special for the same thing, for
>>> no real good reason.
>> In what way is it annoying?
>>
>> Most distribution people aren't even aware that drivers like tg3 and
>> bnx2 even have firmware. In fact it's self contained and less for
>> them to worry about.
>
> and.. after this patch that still seems to be the case, unless I'm
> looking at it really cross eyed.
> Nothing in this patch makes it impossible to do so.. or changes the
> drivers you mention. in fact it's the opposite almost: it improves
> request_firmware() usability significantly. (by allowing vmlinux build
> in etc).

Only by virtue of those specific drivers having been skipped for now.
For those that weren't skipped, the only way to have them still be self
contained is to have the kernel itself compile in all firmware for all
modules after which the module will find it there. Distributions will of
course not do this, not does it make any sense for own kernels.

Rene.

2008-07-15 05:32:35

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 2008-07-15 at 07:06 +0200, Willy Tarreau wrote:
> This is just a matter of point of view. From my POV, having to feed
> several unrelated files into a system to be able to get it to support
> any device is moving backwards, compared to the currently clean
> all-in-1 driver model.

So you disapprove of having to load mac80211.ko before your wireless NIC
driver? Or sungem_phy.ko before sungem.ko? Or libphy.ko before various
other drivers?

Or having to install /sbin/modprobe before you can load modules?

--
dwmw2

2008-07-15 05:34:17

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Mon, 14 Jul 2008, David Woodhouse wrote:

> On Mon, 2008-07-14 at 18:50 -0700, David Miller wrote:
>> From: Linus Torvalds <[email protected]>
>> Date: Mon, 14 Jul 2008 17:33:58 -0700 (PDT)
>>
>>> The only reliable way to handle it is to load the firmware before the
>>> suspend happens - preferably by simply never unloading it in the first
>>> place.
>>
>> That's what we were told was the huge benefit of moving some
>> of these drivers over to this request_firmware() stuff, it would
>> save memory.
>
> There _are_ RAM benefits for a number of drivers, even when you load the
> firmware just once at initialisation time and keep it resident all the
> time the device is active (and no, I won't slap myself for saying it).
>
> The tg3 driver is a prime example -- it contains three separate firmware
> images for different revisions of the card, but you'll almost never
> actually need all three. Mostly you'll only need _one_ of them.

and when a card gets hotplugged into the system for a driver that's
already in memory handling other cards you intend that it will have to go
out to userspace to find the firmware?

how much memory are you expecting to save this way?

David Lang

2008-07-15 05:38:14

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Mon, 14 Jul 2008, David Woodhouse wrote:

> On Tue, 2008-07-15 at 07:06 +0200, Willy Tarreau wrote:
>> This is just a matter of point of view. From my POV, having to feed
>> several unrelated files into a system to be able to get it to support
>> any device is moving backwards, compared to the currently clean
>> all-in-1 driver model.
>
> So you disapprove of having to load mac80211.ko before your wireless NIC
> driver? Or sungem_phy.ko before sungem.ko? Or libphy.ko before various
> other drivers?
>
> Or having to install /sbin/modprobe before you can load modules?

I actually have systems that don't have modprobe (Tivo series 1 boxes,
they have the insmod/rmmod binary , but not modproble)

2008-07-15 05:38:29

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Mon, Jul 14, 2008 at 10:32:10PM -0700, David Woodhouse wrote:
> On Tue, 2008-07-15 at 07:06 +0200, Willy Tarreau wrote:
> > This is just a matter of point of view. From my POV, having to feed
> > several unrelated files into a system to be able to get it to support
> > any device is moving backwards, compared to the currently clean
> > all-in-1 driver model.
>
> So you disapprove of having to load mac80211.ko before your wireless NIC
> driver?

for this one, I don't know, as it is typically something which might
be installed without me noticing.

> Or sungem_phy.ko before sungem.ko?

that one might be more annoying, however, having the same name and being
in the same directory as its brother makes it quite obvious. The ones
which had already caught me were mii, crc32, etc...

> Or libphy.ko before various other drivers?

cf above.

> Or having to install /sbin/modprobe before you can load modules?

David, please do not exagerate. I hoped you were able to see the
difference between being lazy and encountering difficulties to
set up a system.

Willy

2008-07-15 05:45:14

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Mon, 14 Jul 2008, David Woodhouse wrote:

> On Mon, 2008-07-14 at 21:56 -0700, Arjan van de Ven wrote:
>> On Mon, 14 Jul 2008 19:45:57 -0700 (PDT)
>> David Miller <[email protected]> wrote:
>>
>>> From: Linus Torvalds <[email protected]>
>>> Date: Mon, 14 Jul 2008 19:39:03 -0700 (PDT)
>>>
>>>> Put this way: if you do a distro, you _need_ to support firmware
>>>> loading anyway. And once you do that, it's just annoying how some
>>>> drivers then do something odd and special for the same thing, for
>>>> no real good reason.
>>>
>>> In what way is it annoying?
>>>
>>> Most distribution people aren't even aware that drivers like tg3 and
>>> bnx2 even have firmware. In fact it's self contained and less for
>>> them to worry about.
>>
>> and.. after this patch that still seems to be the case, unless I'm
>> looking at it really cross eyed.
>> Nothing in this patch makes it impossible to do so.. or changes the
>> drivers you mention.
>
> Well, the HEAD of my tree (and linux-next) does change tg3. I just
> haven't asked Linus to pull that part, because I wanted to keep the
> contentious part separate from the simple and obvious parts.
>
> (Even though the tg3 fix _is_ simple and obvious, to me. But that
> evidently doesn't stop it from being contentious too. So it can wait.)

a techinical question here (I know, how dare I ask a technical question in
the middle of the flamefest ;-)

linus pointed out that the documentation reccomended the
request_firmware();load_firmware();release() approach and stated that that
approach was the wrong way to do things, instead doing a request_firmware
early and release when the module is unloaded.

does this patch series follow the documented reccomendation? or does it
follow the more concervative approach Linus pointed out? (it's far faster
to ask then to search Internet archives for the patches)

David Lang

2008-07-15 06:03:36

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Mon, 2008-07-14 at 22:45 -0700, [email protected] wrote:
> linus pointed out that the documentation reccomended the
> request_firmware();load_firmware();release() approach and stated that that
> approach was the wrong way to do things, instead doing a request_firmware
> early and release when the module is unloaded.
>
> does this patch series follow the documented reccomendation? or does it
> follow the more concervative approach Linus pointed out? (it's far faster
> to ask then to search Internet archives for the patches)

Mostly it follows the documented recommendation, since most of the
touched drivers are USB drivers and you end up re-enumerating and
starting from scratch on resume anyway. And the remainder are so old
that they don't have suspend/resume support anyway. Remember, we're only
really updating the older drivers; newer drivers tend to use
request_firmware() already, and have done for years.

The only one I remember offhand that loads the firmware and keeps it
around is tg3 -- and I didn't ask Linus to pull that one. In fact, we
only did it that way for tg3 since that driver seems to doing _all_ its
chip reset and firmware reload (including a boatload of udelays) within
a spinlock. That in itself is probably not optimal, but it wasn't really
within the scope of what we were doing to fix it.

--
dwmw2

2008-07-15 06:23:09

by Oliver Neukum

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

Am Dienstag 15 Juli 2008 08:03:15 schrieb David Woodhouse:
> Mostly it follows the documented recommendation, since most of the
> touched drivers are USB drivers and you end up re-enumerating and
> starting from scratch on resume anyway. And the remainder are so old
> that they don't have suspend/resume support anyway. Remember, we're only
> really updating the older drivers; newer drivers tend to use
> request_firmware() already, and have done for years.

As for USB that was before persistance. It now allows you to even
support a loss of power without reprobing, provided

a) the devices does not change IDs
b) you have firmware around when reset_resume() is called

Regards
Oliver

2008-07-15 07:22:17

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

Linus Torvalds wrote:
>
> On Mon, 14 Jul 2008, Jeff Garzik wrote:
>> Do you really think embedded system and micro distro authors are in 100%
>> agreement that /lib/firmware is best for their situation? Really?
>
> Umm. Why are you bringing up these total red herrings?
>
> Those embedded systems can just build everything into one image, and are
> much better off using no modules at all.

Oh come on, embedded systems use modules all the time, and surely you
know that. My router running OpenWRT says

root@gw:~# ipkg list | grep -c kmod
91

Therefore, simple kernel replacement can create a regression by doing

1) build 2.6.26 kernel, including all associated kernel module packages
("kmod-*")
2) install new kernel and associated kmod packages
3) watch system work as expected
4) build 2.6.27 kernel, including all associated kernel module packages
5) install new kernel and associated kmod packages
6) watch system fail

Why fail? The package file lists for kmod-* ipkg's are unaware of any
firmware needs newly added in 2.6.27.

So the driver gets packaged, but ipkg build process is completely
unaware of the additional firmware requirement until a manifest is
updated. The normal build process appears to succeed -- yet at next
boot you see that it failed.

Similar breakage for driver disks.

Similar breakage for older versions of mainstream distros (won't get
copied into initrd, with obvious results).

> But how about instead of complaining, help add the infrastructure then, so
> that you can insert those firmware things into modules and still use just
> one single interface?
>
> Move _forward_, not backwards.

Does that mean you agree it's a regression? :) Sure, I'd be happy to
help fix the damage.

I am still surprised this was merged at all in its state, and am a bit
disappointed. This one

* removed choice, by removing ability to build firmware into modules

* forced a flag day build process change upon all distros/builders who
switch to >= 2.6.27. no build script updates == non-working drivers.

* the legal subtext of these changes was not mentioned at all

Generally, we don't do that. Generally, we give distros the ability to
choose between old-way and new-way during a time of transition.
Generally we make it easy for newer kernels to work on existing systems,
maximizing the [test] audience.

Jeff

2008-07-15 07:37:26

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 2008-07-15 at 03:21 -0400, Jeff Garzik wrote:
> * forced a flag day build process change upon all distros/builders who
> switch to >= 2.6.27. no build script updates == non-working drivers.

Jeff, we don't usually call it a 'flag day' when it's just the normal
and uninteresting case that you update the upstream version of the
package, and you need minor tweaks to how you package it. That happens
all the time. A general rule of thumb is to stop drinking at least 4
hours before undertaking a full version upgrade of _any_ package you
maintain.

You could argue that converting a wireless driver to use mac80211 would
break ipkg distributions too. And call that a 'flag day'.

But that would be silly. Wouldn't it?

--
dwmw2

2008-07-15 07:51:25

by Ingo Molnar

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

* David Woodhouse <[email protected]> wrote:

> On Mon, 2008-07-14 at 17:31 -0700, Yinghai Lu wrote:
> > make[1]: *** No rule to make target
> > `include/config/builtin/firmware/dir.h', needed by
> > `firmware/ql2100_fw.bin.gen.S'. Stop.
> > make: *** [firmware] Error 2
>
> Oh, pants. Sorry, I renamed the CONFIG_BUILTIN_FIRMWARE option to
> CONFIG_EXTRA_FIRMWARE some time back, but all my own test builds had
> the include/config/builtin/firmware/dir.h file still lying around so I
> didn't notice that I'd forgotten to change that manual dependency. And
> evidently nobody using linux-next actually tried to include arbitrary
> _other_ firmware into their tree. This patch should fix it...

here's another bugreport, i get this build failure with latest -git
(which includes your fix above):

CALL scripts/checksyscalls.sh
CHK include/linux/compile.h
make[1]: *** No rule to make target `firmware/tigon/tg3.bin', needed by `firmware/tigon/tg3.bin.gen.o'. Stop.
make: *** [firmware] Error 2

with:

http://redhat.com/~mingo/misc/config-Tue_Jul_15_09_38_58_CEST_2008.bad

such configs worked before.

(i'd exclude TIGON3 from my qa builds but i've got tg3 hardware.)

Ingo

2008-07-15 07:57:35

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

Linus Torvalds wrote:
> On Mon, 14 Jul 2008, [email protected] wrote:
>> On Mon, 14 Jul 2008, Linus Torvalds wrote:
>>> That's a totally bogus argument.
>> you misunderstood me. the people pushing request_firmware() are doing so on
>> the basis that they won't have to use kernel ram to hold the firmware. the
>> people pushing for having the option of building the firmware into the module
>> are acknowleding that this may use a little more ram, but they see it as being
>> more reliable.
>
> I'm just saying that it's a totally bogus argument to claim that it takes
> less memory - Either way.

Agreed.

> As to reliability, I don't buy that, especially with a generic interface,
> and with a way to link the thing in-kernel anyway. Using common
> infrastructure is going to be more reliable.

A more complex, multi-file inter-dependent system is more reliable than
a single-file driver with built-in firmware, doing the same thing?

Come _on_.

There are good arguments for request_firmware(), but that ain't it.

> The argument for request_firmware() is that it's a good _single_ interface
> to the whole firmware issue, allowing us to split up the driver from the
> firmware without every driver having to do some hack of its own.

Agreed.

Jeff

2008-07-15 08:07:52

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

David Woodhouse wrote:
> On Tue, 2008-07-15 at 03:21 -0400, Jeff Garzik wrote:
>> * forced a flag day build process change upon all distros/builders who
>> switch to >= 2.6.27. no build script updates == non-working drivers.
>
> Jeff, we don't usually call it a 'flag day' when it's just the normal
> and uninteresting case that you update the upstream version of the
> package, and you need minor tweaks to how you package it. That happens
> all the time. A general rule of thumb is to stop drinking at least 4
> hours before undertaking a full version upgrade of _any_ package you
> maintain.
>
> You could argue that converting a wireless driver to use mac80211 would
> break ipkg distributions too. And call that a 'flag day'.
>
> But that would be silly. Wouldn't it?

Normally, one can upgrade the kernel, not upgrade userland, and happily
continue working.

With this change, there are several exceptions where this is not the
case, across all driver types -- media, wan, char, net, scsi, ...

Package manifests, build scripts and other details MUST be updated, or
these drivers will not work as they did in kernels <= 2.6.26.

That is not an opinion, that is a simple fact of life with the kernel as
of 50515af207d410c9f228380e529c56f43c3de0bd.

And yes, for the people who have to clean up the mess, I call that a
flag day.

Jeff

2008-07-15 08:15:42

by Ingo Molnar

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

* David Woodhouse <[email protected]> wrote:

> On Tue, 2008-07-15 at 03:21 -0400, Jeff Garzik wrote:
> > * forced a flag day build process change upon all distros/builders
> > who switch to >= 2.6.27. no build script updates == non-working
> > drivers.
>
> Jeff, we don't usually call it a 'flag day' when it's just the normal
> and uninteresting case that you update the upstream version of the
> package, and you need minor tweaks to how you package it. [...]

well, i have no opinion one way or another about the firmware issue, but
breaking the build and breaking previously working setups is not a
"minor tweak".

> [...] That happens all the time. [...]

No, you are quite wrong, it does not happen all the time.

It happens occasionally out of 10,000 changes per kernel cycle. It
happens unintentionally most of the time and it's called a regression.

The difference here is that you seem to have broken my setup fully
intentionally.

I have no problem with going forward, but i have a problem with breaking
working setups and i have a problem with forced migration to something
that doesnt work by default.

If there's a fix i can apply and test i'm a happy camper, but all this
arguing in these threads with zero patches posted makes me quite
nervous.

Ingo

2008-07-15 08:25:53

by Ingo Molnar

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

* Ingo Molnar <[email protected]> wrote:

> If there's a fix i can apply and test i'm a happy camper, but all this
> arguing in these threads with zero patches posted makes me quite
> nervous.

ah, Yinghai forwarded me the fix below - i am a happy camper :-)

btw.:

| from the old tree to omit the drivers/net patches. Again, it didn't
| show up in my own testing because I already _had_ the file in
| question.

6 out of 7 test-distros i have didnt have the tg3.bin file.

Ingo

---------- Forwarded message ----------
From: David Woodhouse <[email protected]>
Date: Mon, Jul 14, 2008 at 6:13 PM
Subject: Re: [GIT *] Allow request_firmware() to be satisfied from
in-kernel, use it in more drivers.
To: Yinghai Lu <[email protected]>
Cc: Andrew Morton <[email protected]>,
[email protected], [email protected],
[email protected]

On Mon, 2008-07-14 at 17:50 -0700, Yinghai Lu wrote:
> MK_FW firmware/tigon/tg3.bin.gen.S
> make[1]: *** No rule to make target `firmware/tigon/tg3.bin', needed
> by `firmware/tigon/tg3.bin.gen.o'. Stop.
> make: *** [firmware] Error 2

I apologise; a line intended for the tg3 conversion accidentally slipped
into firmware/Makefile along with the dabusb patch, when I cherry-picked
from the old tree to omit the drivers/net patches. Again, it didn't show
up in my own testing because I already _had_ the file in question.

(Sorry, Linus. This and the previous fix are now in the for-2.6.27
branch of the same repo as before:
git://git.infradead.org/users/dwmw2/firmware-2.6.git for-2.6.27
)

----
Subject: Fix accidental reference to tg3 firmware

We're not updating the tg3 driver to use request_firmware() yet, but a
reference to its firmware accidentally slipped in as part of commit
c4667746 ("dabusb: use request_firmware()"). Remove it again.

Signed-off-by: David Woodhouse <[email protected]>

diff --git a/firmware/Makefile b/firmware/Makefile
index 10028ac..8dbdd3f 100644
--- a/firmware/Makefile
+++ b/firmware/Makefile
@@ -34,8 +34,6 @@ fw-shipped-$(CONFIG_SND_SB16_CSP) +=
sb16/mulaw_main.csp sb16/alaw_main.csp \
sb16/ima_adpcm_capture.csp
fw-shipped-$(CONFIG_SND_YMFPCI) += yamaha/ds1_ctrl.fw yamaha/ds1_dsp.fw \
yamaha/ds1e_ctrl.fw
-fw-shipped-$(CONFIG_TIGON3) += tigon/tg3.bin tigon/tg3_tso.bin \
- tigon/tg3_tso5.bin
fw-shipped-$(CONFIG_USB_DABUSB) += dabusb/firmware.fw dabusb/bitstream.bin
fw-shipped-$(CONFIG_USB_EMI26) += emi26/loader.fw emi26/firmware.fw \
emi26/bitstream.fw

--
dwmw2

2008-07-15 09:57:21

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

Hi David,

> I'm not happy for network drivers, although I'm happy to see that
> David respected out NACKs on tg3/bnx2/etc. and didn't include those
> bits.
>
> I still haven't seen an answer to the chip reset issues. The argument
> has been that this stuff is going to save kernel memory when the
> driver isn't loading firmware, but that's not really possible.
>
> When a lot of these network drivers reset, due to a transmit timeout
> or other exception, we need to load the firmware again.
>
> So this means, that if request_firmware() dies or fails for some
> reason in these scenerios, we can't reset the network card. Something
> as simple as a memory or other allocation failure deep inside of
> request_firmware() means we lose the networking interface.
>
> To me this seems like a very non-resilient setup. It makes sense
> to just keep the firmware in-core so we can load it without having
> to even think about possible failures.

actually when we wrote request_firmware() a long time ago, the idea was
to support various kind of setups (although until now only the on-demand
got used).

So one idea was to allow minimal inclusion of firmware as binary blobs
in the driver and allow overwriting them by loading newer version from
the filesystem. This was intended to ensure minimal working drivers. The
main background was that some devices work with a small firmware and
provide basic features and only need extra firmware for advanced
features. Everything suppose to be hidden behind request_firmware() and
a method to register binary blobs as firmware from within a driver. So
the driver developer really doesn't have to invent anything by himself.

We are not there yet and at the moment, I am not sure if this is still a
valid requirement, but David's work goes into the right direction and is
a good start.

Regards

Marcel

2008-07-15 14:41:17

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 2008-07-15 at 08:23 +0200, Oliver Neukum wrote:
> Am Dienstag 15 Juli 2008 08:03:15 schrieb David Woodhouse:
> > Mostly it follows the documented recommendation, since most of the
> > touched drivers are USB drivers and you end up re-enumerating and
> > starting from scratch on resume anyway. And the remainder are so old
> > that they don't have suspend/resume support anyway. Remember, we're only
> > really updating the older drivers; newer drivers tend to use
> > request_firmware() already, and have done for years.
>
> As for USB that was before persistance. It now allows you to even
> support a loss of power without reprobing, provided
>
> a) the devices does not change IDs
> b) you have firmware around when reset_resume() is called

c) the driver in question actually _has_ a reset_resume() method

The old, unloved drivers which I've touched as part of this patch set
did not. In general the newer, loved drivers use request_firmware()
already.

--
dwmw2

2008-07-15 14:56:54

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 15 Jul 2008, Willy Tarreau wrote:
> >
> > But once you can load a module, you can load the firmware. You just have
> > to _remember_ to move it along with the module.
>
> In order to transfer it, right now you feed it through a working device.
> When that device itself requires firmware to work, you will suddenly
> discover that it becomes harder and harder to get any communication device
> to work on your target system.

Willy, stop this blathering.

I suspect I will have to just delete this thread unread because it's so
full of total crap.

This whole "working device" argument is total bullshit.

If the driver was a module before, it needed a module load to become
working. If you could load the module, you could load the firmware. Thus
the transfer is non-issue.

And if the driver was built-in, you can still build in the firmware.

And by proof of induction - your "now you feed it through a working
device" - this also holds for that "a working device" driver. The device
you load the module off has exactly the same rules: you can build in the
firmware.

So it didn't get any harder at all. Except for the fact that you need to
remember to install the firmware.

Which is just about the same thing as asking people to remember to do
"make modules_install" to get a working system. Yes, if you have a driver
as a module, you need to install that module for the device to work. Yes,
it's definitely "harder", but seriously..

Linus

2008-07-15 15:05:39

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 15 Jul 2008, Jeff Garzik wrote:
>
> A more complex, multi-file inter-dependent system is more reliable than a
> single-file driver with built-in firmware, doing the same thing?

I'm not interested in your made-up arguments.

Please just build everything statically. Don't use modules. They are
clearly too complex for you, adn everybody is happy.

Linus

2008-07-15 15:09:36

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 15 Jul 2008, Linus Torvalds wrote:

> On Tue, 15 Jul 2008, Willy Tarreau wrote:
>>>
>>> But once you can load a module, you can load the firmware. You just have
>>> to _remember_ to move it along with the module.
>>
>> In order to transfer it, right now you feed it through a working device.
>> When that device itself requires firmware to work, you will suddenly
>> discover that it becomes harder and harder to get any communication device
>> to work on your target system.
>
> Willy, stop this blathering.
>
> I suspect I will have to just delete this thread unread because it's so
> full of total crap.
>
> This whole "working device" argument is total bullshit.
>
> If the driver was a module before, it needed a module load to become
> working. If you could load the module, you could load the firmware. Thus
> the transfer is non-issue.
>
> And if the driver was built-in, you can still build in the firmware.
>
> And by proof of induction - your "now you feed it through a working
> device" - this also holds for that "a working device" driver. The device
> you load the module off has exactly the same rules: you can build in the
> firmware.
>
> So it didn't get any harder at all. Except for the fact that you need to
> remember to install the firmware.
>
> Which is just about the same thing as asking people to remember to do
> "make modules_install" to get a working system. Yes, if you have a driver
> as a module, you need to install that module for the device to work. Yes,
> it's definitely "harder", but seriously..

has a standard been defined for how to maintain different versions of
firmware for different versions of drivers yet? driver maintainers do not
test drivers with all different firmware versions, just the one(s) that
ship with that kernel version. there are also reports from maintainers
that some driver/firmware combos do not have full backwards compatability
so you can't just use the new firmware with the old drivers (causing
unexpected glitches when switching between kernel versions)

that was one of the things that was identified in the last flamewar that
was handwaved away but nothing was defined.

yes it's something that other drivers that use request_firmware() have not
run into problems with, but the maintainers of drivers that have embedded
the firmware into the driver up until now have stated that they see this
as a problem.

it's also not that hard to do (it's just someone making a statement of
what the standard will be that others can agree with), but it needs to be
done.

the trivial way out is to define a firmware tree per kernel version
(similar to how modules are handled), but that would offend some people (I
think it was Alan Cox who was arguing in the last flamewar about how much
bandwidth it would save to not distribute firmware with each kernel
version), and it would definantly be a waste of disk space since the
firmware changes far less frequently then the modules do.

David Lang

2008-07-15 15:12:03

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 15 Jul 2008, Jeff Garzik wrote:
>
> With this change, there are several exceptions where this is not the case,
> across all driver types -- media, wan, char, net, scsi, ...
>
> Package manifests, build scripts and other details MUST be updated, or these
> drivers will not work as they did in kernels <= 2.6.26.

Umm. I complained six months ago when the e1000e driver took over, and the
e1000 driver stopped working for me, and you suddenly had to know which
version of the e1000 you had - even though there was not any sane way to
tell them apart.

Ingo _still_ complains about that occasionally.

What did you do? You didn't care.

Now you suddenly do care that some driver needs additional love and
learning.

I call bullshit. You don't like the change - ok. But stop making bogus
arguments about it.

Linus

2008-07-15 15:57:23

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

Linus Torvalds wrote:
> On Mon, 14 Jul 2008, Jeff Garzik wrote:
>> IMO the newly added /inability/ to build firmware into kernel modules
>> is a clear regression.
>
> IMO you're being stupid.
>
> How about explainign why it makes any difference what-so-ever?
>
> If you can load the module, you can load the firmware. Claiming anything
> else is just _stupid_.

Sure, in theory it's that simple. Here's a concrete example that shows how
things are harder in practice.

I use the 'make deb-pkg' target (from scripts/package) to build my Debian
kernel packages from git. So that needs to be adapted to include
/lib/firmware. No real problem so far.

So then I build 2.6.27-rc1 and install it. Great.

You release 2.6.27-rc2 and I build it. Ouch! It fails to install, at least
if I want to install it _alongside_ 2.6.27-rc1 or other kernels (which I
do!). Why does it fail? Because dpkg's package management does not allow
one package to overwrite files already "owned" by another package.

So, how is this solved by Debian for already existing firmware packages?
Basically by making a separate package for each firmware file (or
driver). This works because there are not too many of them, but having a
huge number of tiny packages is a nightmare by itself.

But anyway, the dep-pkg target will have to be made smarter than it is now
if it's to deal with this [1]. And at least currently it is broken.

If I were able to compile firmware into the modules, the problem would be
solved in one go.

I don't know how the Debian kernel team plans to deal with this for distro
kernel packages. They probably _do_ want to keep them separate [2]. Maybe
by grouping firmware for really common drivers into
firmware-basic-drivers or something along those lines.

Cheers,
FJP

[1] Only quick solution I see is to have it install the firmware in a
versioned directory and have the postinst copy things from there to
/lib/firmware.
[2] As one of the developers for Debian Installer I'm not looking forward
to the complications that is going to cause for us and users.

2008-07-15 16:01:52

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 15 Jul 2008, [email protected] wrote:
>
> has a standard been defined for how to maintain different versions of firmware
> for different versions of drivers yet?

No. We haven't had that possibility, probably largely because we haven't
even really had the ability (Davem and Jeff talk about "choice" and how
request_firmware() supposedly removes it, but they ignore the fact that it
very much cuts the other way too - request_firmware() _adds_ choice and
makes it easier for people to try to upgrade firmware without upgrading
the kernel).

> that was one of the things that was identified in the last flamewar that was
> handwaved away but nothing was defined.

I don't think we can - or want to - define things until we've seen the
problems and advantages in practice over a longer time.

Do we want a versioning scheme? Probably not - exactly because people may
well want to update (or downgrade) firmware from vendors independently of
the kernel.

But maybe people will eventually want some kind of blacklisting scheme
where the driver can say things like "I know I won't work with versions
that match this checksum" and cause a nice big warning at load-time.

And I think we can all _imagine_ scenarios in all kinds of possible ways
for things to work better or fail spectaclarly, and even come up with
anecdotal evidence where scheme X (out of any number of random variations)
is the "only possible one to work". I'd argue that we simply don't have
any realistic way to decide which ones make sense.

For example, we can be in the situation where we have some case where
"only this particular firmware is known to work, and others are known to
break", and people can still take two totally different mental approaches
to the problem.

One approach is the "strict control one", where people argue that the
above problem means that we absolutely *must* have very strict versioning,
and we just always pair things up exactly.

But the other argument is that firmware is part of hardware, and the
driver should strive to work with different versions of vendor firmware,
the same way it should strive to work with different versions of chip
hardware.

Which one is right? Probably neither. Sometimes one, sometimes the other,
and in both cases it should probably not be black-and-white.

IOW, I suspect that in the long run we want to say

"Ok, there's no way we can support _all_ firmware versions, but on the
other hand, we also don't want to force particular versions and not let
people experiment.

So drivers should strive to be flexible, and driver writers should not
always just say 'there is only one "blessed" firmware', but on the
other hand we cannot require drivers to work around horrible known bugs
in firmware when a fixed firmware is available"

but the exact details will have to depend on particular drivers.

And we should obviously have _a_ version of the firmware available with
the kernel when that is possible. But I'd hate for it to be 1:1 with a
particular driver version - because at that point it smells of being a
single work, and if it is more than mere aggregation it's no longer viable
with most of our firmware (I don't think we have source for more than one
or two cases).

Linus

2008-07-15 16:05:27

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 15 Jul 2008 17:57:09 +0200
Frans Pop <[email protected]> wrote:

> Linus Torvalds wrote:
> > On Mon, 14 Jul 2008, Jeff Garzik wrote:
> >> IMO the newly added /inability/ to build firmware into kernel
> >> modules is a clear regression.
> >
> > IMO you're being stupid.
> >
> > How about explainign why it makes any difference what-so-ever?
> >
> > If you can load the module, you can load the firmware. Claiming
> > anything else is just _stupid_.
>
> Sure, in theory it's that simple. Here's a concrete example that
> shows how things are harder in practice.
>
> I use the 'make deb-pkg' target (from scripts/package) to build my
> Debian kernel packages from git. So that needs to be adapted to
> include /lib/firmware. No real problem so far.
>
> So then I build 2.6.27-rc1 and install it. Great.
>
> You release 2.6.27-rc2 and I build it. Ouch! It fails to install, at
> least if I want to install it _alongside_ 2.6.27-rc1 or other kernels
> (which I do!). Why does it fail? Because dpkg's package management
> does not allow one package to overwrite files already "owned" by
> another package.
>

question: how do you deal with this with the dozens of drivers that use
request_firmware() even before yesterday ?
the answer is critical in how to deal with this "new" situation as
well ;-)

2008-07-15 16:17:37

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 15 Jul 2008, Frans Pop wrote:
>
> So, how is this solved by Debian for already existing firmware packages?
> Basically by making a separate package for each firmware file (or
> driver). This works because there are not too many of them, but having a
> huge number of tiny packages is a nightmare by itself.

Why don't you just take the kernel-supplied firmware and make it part of
the kernel package? The same way the kernel-supplied modules are part of
it?

You're sounding like

"Mommy, mommy, it hurts when I repeatedly hit my head against the wall
and stick forks in my eyes!"

and quite frankly, it doesn't make me go "Oh, poor baby" at all.

Linus

2008-07-15 16:22:11

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 2008-07-15 at 09:01 -0700, Linus Torvalds wrote:
> (I don't think we have source for more than one or two cases).

We don't. As part of the cleanup I've been trying to document what we
know about the origins of the firmware, in the firmware/WHENCE file.
Unfortunately, mostly that's just "unknown".

We have managed to dig out the original source for one or two hex arrays
though.

--
dwmw2

2008-07-15 16:25:11

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tuesday 15 July 2008, Arjan van de Ven wrote:
> Frans Pop <[email protected]> wrote:
> > You release 2.6.27-rc2 and I build it. Ouch! It fails to install, at
> > least if I want to install it _alongside_ 2.6.27-rc1 or other kernels
> > (which I do!). Why does it fail? Because dpkg's package management
> > does not allow one package to overwrite files already "owned" by
> > another package.
>
> question: how do you deal with this with the dozens of drivers that use
> request_firmware() even before yesterday ?
> the answer is critical in how to deal with this "new" situation as
> well ;-)

For Debian packages that was in the part of the mail you snipped.

The simple answer for my current use of dep-pkg for custom kernels is:
I don't _have_ any hardware that requires firmware that's split out with
current kernels.
Maybe I'll be lucky and will find that I don't have any with 2.6.27, but
I'm not betting on that.

2008-07-15 16:34:40

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 15 Jul 2008, Linus Torvalds wrote:

> On Tue, 15 Jul 2008, Frans Pop wrote:
>>
>> So, how is this solved by Debian for already existing firmware packages?
>> Basically by making a separate package for each firmware file (or
>> driver). This works because there are not too many of them, but having a
>> huge number of tiny packages is a nightmare by itself.
>
> Why don't you just take the kernel-supplied firmware and make it part of
> the kernel package? The same way the kernel-supplied modules are part of
> it?

David W has made it very clear that the eventual intent of this patch
series is to remove all firmware from the kernel tree and have it shipped
as a seperate package.

David Lang

> You're sounding like
>
> "Mommy, mommy, it hurts when I repeatedly hit my head against the wall
> and stick forks in my eyes!"
>
> and quite frankly, it doesn't make me go "Oh, poor baby" at all.
>
> Linus
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>

2008-07-15 16:37:19

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tuesday 15 July 2008, you wrote:
> > You release 2.6.27-rc2 and I build it. Ouch! It fails to install, at
> > least if I want to install it _alongside_ 2.6.27-rc1 or other kernels
> > (which I do!). Why does it fail? Because dpkg's package management
> > does not allow one package to overwrite files already "owned" by
> > another package.
>
> question: how do you deal with this with the dozens of drivers that use
> request_firmware() even before yesterday ?
> the answer is critical in how to deal with this "new" situation as
> well ;-)

For Debian packages that's in the part of the mail you snipped.

The simple answer for my current use of dep-pkg is: I don't _have_ any
hardware that requires firmware that's split out with current kernels.

2008-07-15 16:37:38

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 15 Jul 2008, Linus Torvalds wrote:
>
> Why don't you just take the kernel-supplied firmware and make it part of
> the kernel package? The same way the kernel-supplied modules are part of
> it?

And, btw, if there is something we can do to help, we obviously should.

So ask yourself (and DavidW) things like

- Is "make firmware_install" not enough for the package build system?

- Does it need some other way to figure out what firmware is needed for
just the configured set of drivers, eg "make firmware_list"?

- Do we need to make the firmware loader have multiple directories it
looks at? Do we want first a kernel-version-specific one, then a
generic fallback one, so that we can easily have per-kernel firmware,
the same way we have per-kernel modules?

- Add your interesting question here.

There are obviously all these kinds of _productive_ questions you can ask.
But people seem to be mostly in an unproductive "Mommy, it hurts when I do
something stupid" mode.

Linus

2008-07-15 16:37:55

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tuesday 15 July 2008, you wrote:
> On Tue, 15 Jul 2008, Frans Pop wrote:
> > So, how is this solved by Debian for already existing firmware
> > packages? Basically by making a separate package for each firmware
> > file (or driver). This works because there are not too many of them,
> > but having a huge number of tiny packages is a nightmare by itself.
>
> Why don't you just take the kernel-supplied firmware and make it part
> of the kernel package? The same way the kernel-supplied modules are
> part of it?

Important note: this is not me, this is the Debian kernel team based on
Debian Free Software Guidelines (DFSG). If it were purely up to me I'd be
a lot more pragmatic.

1) Because Debian _wants_ them split out for DFSG compliance reasons.
Most of the current firmware packages are kept in the "non-free" section
of the Debian archive while the kernel itself lives in "main".
As long as firmware could not be split out, the compliance problem was the
source of many discussions, but for the most part ignored in practice,
with the exception of a few drivers with really problematic firmware
licences.

2) Because of the overwrite and version management problems.
/lib/firmware as a single dumping ground for firmware for all kernel
versions really sucks from that PoV. One of the huge strengths of Debian
is its ability to clean up after itself when packages are removed.
This is partially solved by giving each firmware file it's own package
because then you can use the versioning of the firmware itself in the
package versions, which allows proper file management by the packaging
system. But as I said, I'm not sure that still works if their number
suddenly explodes.

2008-07-15 16:42:40

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 15 Jul 2008, [email protected] wrote:
>
> David W has made it very clear that the eventual intent of this patch series
> is to remove all firmware from the kernel tree and have it shipped as a
> seperate package.

I don't think that kind of black-and-white model is necessarily
attainable. Nor do I think it's even _good_.

It's probably the right thign for _some_ firmware, assuming we have models
that just make it easy enough - for drivers where it's generally better
for distros to get the firmware from vendors rather than the kernel.

But there is certainly nothing inherently good in forcing a split.

I think both sides here need to calm down and stop being so extreme.

I do want request_firmware() to be the way to load firmware (and possibly
other configuration data too, for that matter - I don't think it's
necessarily wrong for people to "misuse" it for other long-term data
despite the name), and I do want people to be able to update firmware
easily without having to force a kernel bump, but I also don't think it
means that firmware has to go away from the kernel sources.

So I think the _infrastructure_ is important. But the extreme "you have to
split it off entirely" mindset is equally as stupid as the "you must
compile it into the module" mindset.

Linus

2008-07-15 16:52:27

by Oliver Neukum

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

Am Dienstag 15 Juli 2008 16:40:54 schrieb David Woodhouse:
> On Tue, 2008-07-15 at 08:23 +0200, Oliver Neukum wrote:
> > Am Dienstag 15 Juli 2008 08:03:15 schrieb David Woodhouse:
> > > Mostly it follows the documented recommendation, since most of the
> > > touched drivers are USB drivers and you end up re-enumerating and
> > > starting from scratch on resume anyway. And the remainder are so old
> > > that they don't have suspend/resume support anyway. Remember, we're only
> > > really updating the older drivers; newer drivers tend to use
> > > request_firmware() already, and have done for years.
> >
> > As for USB that was before persistance. It now allows you to even
> > support a loss of power without reprobing, provided
> >
> > a) the devices does not change IDs
> > b) you have firmware around when reset_resume() is called
>
> c) the driver in question actually _has_ a reset_resume() method
>
> The old, unloved drivers which I've touched as part of this patch set
> did not. In general the newer, loved drivers use request_firmware()
> already.
>

Generally we love all drivers and welcome new additions.
Which did you touch? I lost track.

Regards
Oliver

2008-07-15 16:53:50

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

Linus Torvalds wrote:
>
> On Tue, 15 Jul 2008, Jeff Garzik wrote:
>> With this change, there are several exceptions where this is not the case,
>> across all driver types -- media, wan, char, net, scsi, ...
>>
>> Package manifests, build scripts and other details MUST be updated, or these
>> drivers will not work as they did in kernels <= 2.6.26.
>
> Umm. I complained six months ago when the e1000e driver took over, and the
> e1000 driver stopped working for me, and you suddenly had to know which
> version of the e1000 you had - even though there was not any sane way to
> tell them apart.
>
> Ingo _still_ complains about that occasionally.
>
> What did you do? You didn't care.

Completely false -- what was the outcome? The breakage complained about
was fixed, we made sure Ingo had proper defconfig entries, the changes
to e1000e were pushed back.

That's a lot of taking-feedback-from-others that you define as "didn't
care."

Jeff

2008-07-15 16:48:37

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tuesday 15 July 2008, Linus Torvalds wrote:
> There are obviously all these kinds of _productive_ questions you can
> ask. But people seem to be mostly in an unproductive "Mommy, it hurts
> when I do something stupid" mode.

Of course, but the main reason I replied in the first place was because
your "there are no real problems" attitude in this discussion was
annoying me.

And even in your first reply to me you basically ignored that fact that at
least one kernel packaging tool and one that is *part of the kernel tree
itself* is now broken.

There *are* real problems and some of them *could* be solved by allowing
firmware to be compiled into modules. Sure, other solutions are possible
too and if I actually run into the problems I foresee with dpk-deb you
can be certain I will try to help solve them.

But if you want to invite people to hold a productive discussion, it helps
if you don't completely ignore their concerns out of hand.

2008-07-15 17:02:13

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

Linus Torvalds wrote:
>
> On Tue, 15 Jul 2008, Jeff Garzik wrote:
>> A more complex, multi-file inter-dependent system is more reliable than a
>> single-file driver with built-in firmware, doing the same thing?
>
> I'm not interested in your made-up arguments.

It's a simple question... you claimed A was more reliable than B, even
though A is a more complex system than B.

I'm curious how that works, especially given that you have claimed the
_exact opposite_ in years past, by pointing out how firmware separation
could mean no-boot.

> Please just build everything statically. Don't use modules. They are
> clearly too complex for you, adn everybody is happy.

Can't you make a simple, objective, technical comparison between two
systems?

firmware-outside-driver has more utility and flexibility, but more
things can go wrong, like driver present/firmware absent case that
happens every day with today's wireless drivers.

firmware-inside-driver guarantees without _any_ doubt that the firmware
is present, if the driver is guarantee. There is no guarantee more
solid than that.

Jeff

2008-07-15 17:08:48

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 15 Jul 2008, Frans Pop wrote:
>
> Of course, but the main reason I replied in the first place was because
> your "there are no real problems" attitude in this discussion was
> annoying me.

I really don't think there are any real problems.

The problems are all due to inertia - people (and scripts) are used to one
way of doing things. Those aren't really technical issues, and more
importantly, they are not issues that _can_ be solved any other way than
just doing it, and fixing the fallout.

Trying to be "proactive" doesn't work. Waiting for people (and scripts) to
get used to a new model before merging it also doesn't work. And
complaining about a change that pretty obviously is needed also doesn't
work.

So let's just do it. And then, when specific scripts are encountered, fix
them. And yes, fixing the scripts may _well_ imply adding more support for
it in the kernel. It could be either in build and installation scripts, or
in udev infrastructure, or in request_firmware() capability itself.

For example, some people hate udev, and they may have some really good
reason why they really don't want to rely on that in their setup. So maybe
we should teach request_firmware() to try to look up (binary) firmware
files directly from the filesystem before it does a udev callback (in
fact, maybe it even already does - I really didn't check very closely).

None of these are "problems".

But people who don't even want to look for solutions - _that_ is a
problem.

Linus

2008-07-15 17:10:44

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 15 Jul 2008, Jeff Garzik wrote:
>
> I'm curious how that works, especially given that you have claimed the _exact
> opposite_ in years past, by pointing out how firmware separation could mean
> no-boot.

Umm. Firmware separation _could_ mean non-boot.

DavidW &co fixed it. You can now build it in.

You seem to ignore that things change, and _have_ changed.

Linus

2008-07-15 17:17:16

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

Linus Torvalds wrote:
> I call bullshit. You don't like the change - ok.

SIGH. Wrong, wrong, wrong. I have always thought request_firmware()
was a good idea, even if it decreased reliability. Because we need that
ability and that API.

The changes are good,

They just don't need to break stuff.

Which they do.

> But stop making bogus
> arguments about it.

How many more real world examples of breakage do you need?

* This breaks newer kernel / older userland
* This breaks anywhere with a file list for modules' contents (ipkg
kernel module packages)
* This breaks driver disk build packages
* This breaks for the oddball, not just me, who
* This breaks for an unknown amount of people who have been assuming
that their driver will always work when modprobe'd, even though they are
completely unaware of its new firmware needs

All these are cases in use today. Real users, Linus. Real distros.

Jeff

P.S. And as I stated in another email, I'm happy to help fix this
regression.

2008-07-15 17:26:00

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tuesday 15 July 2008, you wrote:
> So I think the _infrastructure_ is important. But the extreme "you have
> to split it off entirely" mindset is equally as stupid as the "you must
> compile it into the module" mindset.

Agreed. All I've wanted to say is that having the _option_ to continue to
compile it into the module can simplify certain use cases, including my
current use of the kernel's own deb-pkg target for private builds of the
kernel for testing purposes.

One thing I would dearly like to see is the ability to choose a different
location for firmware for my own custom built kernels than the location
used for firmware from official Debian kernel packages so that at least
those two don't clash.
Doesn't look like that is possible with current git head.

2008-07-15 17:27:18

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 15 Jul 2008, Jeff Garzik wrote:
>
> The changes are good,
>
> They just don't need to break stuff.
>
> Which they do.

If you were serious about that, then you'd be suggesting _fixes_.

I'm not seeing you do that. I'm just seeing non-productive "we should
never have changed" emails.

See my emails about what can be _productive_ avenues to explore. As it is,
I have yet to see a _single_ productive email from you.

Really, Jeff.

Linus

2008-07-15 17:31:05

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

Linus Torvalds wrote:
>
> On Tue, 15 Jul 2008, Jeff Garzik wrote:
>> I'm curious how that works, especially given that you have claimed the _exact
>> opposite_ in years past, by pointing out how firmware separation could mean
>> no-boot.
>
> Umm. Firmware separation _could_ mean non-boot.
>
> DavidW &co fixed it. You can now build it in.
>
> You seem to ignore that things change, and _have_ changed.

Already addressed -- you and David seem to be ignoring the common case
where you may update the driver but not the kernel, making that feature
far less useful than it appears.

Inability to update the kernel image (vmlinuz) is the case with just
about every enterprise distro out there, including ones to be based on
>= 2.6.27.

Driver disks and our own in-tree build-module-out-of-tree support are
built precisely for this purpose: to build a new driver against an
existing kernel, because so often you cannot update the core vmlinuz image.

(rather, you _can_ update vmlinuz, but that would fail on the next
automated update, so its not a realistic option)

Jeff

2008-07-15 17:34:25

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 15 Jul 2008 18:36:56 +0200
Frans Pop <[email protected]> wrote:

> On Tuesday 15 July 2008, you wrote:
> > > You release 2.6.27-rc2 and I build it. Ouch! It fails to install,
> > > at least if I want to install it _alongside_ 2.6.27-rc1 or other
> > > kernels (which I do!). Why does it fail? Because dpkg's package
> > > management does not allow one package to overwrite files already
> > > "owned" by another package.
> >
> > question: how do you deal with this with the dozens of drivers that
> > use request_firmware() even before yesterday ?
> > the answer is critical in how to deal with this "new" situation as
> > well ;-)
>
> For Debian packages that's in the part of the mail you snipped.
>
> The simple answer for my current use of dep-pkg is: I don't _have_
> any hardware that requires firmware that's split out with current
> kernels.

... and do you have any with current -git ?

--
If you want to reach me at my work email, use [email protected]
For development, discussion and tips for power savings,
visit http://www.lesswatts.org

2008-07-15 17:36:38

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, Jul 15, 2008 at 07:55:37AM -0700, Linus Torvalds wrote:
> Willy, stop this blathering.
>
> I suspect I will have to just delete this thread unread because it's so
> full of total crap.
>
> This whole "working device" argument is total bullshit.
>
> If the driver was a module before, it needed a module load to become
> working. If you could load the module, you could load the firmware. Thus
> the transfer is non-issue.

Sometimes your responses leave me speechless! It's not only a problem of
transfer, but also a problem of selection of what to copy overthere. It
has happened to me more than once when installing a machine to proceed this
way :
- leave the data center
- go back to the office to find the driver somewhere
- burn a CD with it (I once even had to feed it through a 9600 bauds Annex
port concentrator).
- go back to the DC and copy it from the CD. I must say that at this stage
when you see warnings because it's not the exact same kernel version, you
don't mind anymore and force the load in order to be able to stick an IP
address on your machine.

Now when you load it and you see it requires a firmware, you're back to #1,
without even the certainty that the firmware you get will :
a) be the right one
b) work
c) be enough

So please don't say that the transfer is a non-issue. It is what magnifies
the added complexity.

Instead, I would have loved to see multiboot-compatible module loading.
When your kernel does not support your fresh new RAID card and panics at
boot, you simply put the driver on a floppy (or USB stick), pre-load the
driver from GRUB and boot the kernel. I don't see how this would be possible
anymore with separate firmwares.

Don't get me wrong: I'm not against the ability to separate firmware from
modules, I'm against making that mandatory when it obviously complexifies
a lot of things without any obvious advantage in return.

> Which is just about the same thing as asking people to remember to do
> "make modules_install" to get a working system. Yes, if you have a driver
> as a module, you need to install that module for the device to work. Yes,
> it's definitely "harder", but seriously..

Please don't compare an upgrade of a working desktop or developer machine with
an installation from scratch of a server in an uncomfortable location. Those
are two completely different things.

Willy

2008-07-15 17:36:52

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 2008-07-15 at 09:41 -0700, Linus Torvalds wrote:
> So I think the _infrastructure_ is important. But the extreme "you
> have to split it off entirely" mindset is equally as stupid as the
> "you must compile it into the module" mindset.

I wouldn't necessarily argue that we _have_ to split it off, but I'd
certainly like our infrastructure to be mature enough that it could cope
with doing so.

I plan to make a 'linux-firmware-from-linus' git tree which is generated
automatically from yours by a script, much like the kernel-headers git
tree is, commit by commit with an automatic transform on the tree
objects.

Once we have that tree being created, we could pull from it into a
canonical 'linux-firmware' tree, and _that_ tree could also start to
include firmware belonging to companies who have said that we may
distribute their firmware, but _not_ as part of the Linux kernel.

We can see how that tree works out in practice, and how/whether people
start using it. And then maybe we'll be in a position to have a
_discussion_ about how best to ship the firmware. But for now it's
premature.

--
dwmw2

2008-07-15 17:44:10

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 15 Jul 2008, Jeff Garzik wrote:
>
> Already addressed -- you and David seem to be ignoring the common case where
> you may update the driver but not the kernel, making that feature far less
> useful than it appears.

This is the "I'm too stupid to live" argument. I've already answered that.
What's so hard with "make firmware_install"?

Linus

2008-07-15 17:44:43

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 2008-07-15 at 19:35 +0200, Willy Tarreau wrote:
> It's not only a problem of transfer, but also a problem of selection
> of what to copy overthere.

This is why the MODULE_FIRMWARE() tag exists.

Most scripts tend to get this right already -- they _have_ to, since
modern drivers have been using request_firmware() for some time already.

> Please don't compare an upgrade of a working desktop or developer
> machine with an installation from scratch of a server in an
> uncomfortable location. Those are two completely different things.

If your setup doesn't cope with MODULE_FIRMWARE(), it's likely to break
when you next deploy a new server anyway -- because your new server will
have new hardware, and your new hardware will have a modern driver. And
modern drivers use request_firmware() _anyway_.

--
dwmw2

2008-07-15 17:46:22

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 15 Jul 2008, Linus Torvalds wrote:
>
> This is the "I'm too stupid to live" argument. I've already answered that.
> What's so hard with "make firmware_install"?

Btw, that is by no means the only way to handle it either. As I said in
other emails - about trying to find _productive_ questions to answer -
maybe we could also just install it in the modules directory or some
similar per-kernel location, and just teach request_firmware() to load
binary blobs directly.

Then - horror of horrors - we could even make "make modules_install" do
things like that automatically for you with some Makefile rule trickery.

See? There are _productive_ things you can do or ask others to do.

But singing the same "I'm so stupid that I cannot install the firmware"
song over and over again doesn't make me think you are worth listening to.

Linus

2008-07-15 17:55:21

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

Linus Torvalds wrote:
>
> On Tue, 15 Jul 2008, Jeff Garzik wrote:
>> The changes are good,
>>
>> They just don't need to break stuff.
>>
>> Which they do.
>
> If you were serious about that, then you'd be suggesting _fixes_.

The fix was suggested from day one: permit firmware-in-module, which
I've already said I will work on... provided its not dead on arrival.

Other fixes I've stated in these threads I want to tackle:

* newly separated firmware should live alongside the driver

* driver author should be able to deliver binary firmware direct from
vendor, without any ihex or C-source-code mangling.

> I'm not seeing you do that. I'm just seeing non-productive "we should
> never have changed" emails.

The emails are "this change should not contain obvious breakages"

To review the parts you've apparently missed, the problems and
regressions I have found during review were

- Kconfig produced non-working drivers. 10+ skilled kernel hackers hit
this. I complained, was called silly, pointless, and having a tantrum.
Then, the issue was fixed, and kernel hackers could boot again.

- the normal build process produced non-working drivers. A couple more
skilled kernel hackers hit this one. Build process regression was
OBVIOUS. Again, fixing this was called pointless and silly. Again, it
got fixed. Again, kernel hackers started working again.

Now we have one last major regression hole to fix -- firmware-in-module.

> See my emails about what can be _productive_ avenues to explore. As it is,
> I have yet to see a _single_ productive email from you.

Strange, considering that it has been me who pointed out problems that
would -- and did -- hit other kernel hackers, and those problems after
much complaining and insulting got fixed.

Jeff

2008-07-15 17:56:36

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 15 Jul 2008, Linus Torvalds wrote:

> On Tue, 15 Jul 2008, [email protected] wrote:
>>
>> has a standard been defined for how to maintain different versions of firmware
>> for different versions of drivers yet?
>
> No. We haven't had that possibility, probably largely because we haven't
> even really had the ability (Davem and Jeff talk about "choice" and how
> request_firmware() supposedly removes it, but they ignore the fact that it
> very much cuts the other way too - request_firmware() _adds_ choice and
> makes it easier for people to try to upgrade firmware without upgrading
> the kernel).

and they aren't objecting to this choice being added. they just want to
keep the choice of working the other way (although they implicitly are
saying that they won't be doing a lot of testing of different firmware
versions becouse they do want to work the other way)

David Lang

2008-07-15 17:58:08

by Henrique de Moraes Holschuh

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tuesday 15 July 2008, Arjan van de Ven wrote:
> > The simple answer for my current use of dep-pkg is: I don't _have_
> > any hardware that requires firmware that's split out with current
> > kernels.
>
> ... and do you have any with current -git ?

Haven't tried that yet. Only just finished building .26 for my boxes and
have other priorities ATM anyway.
You'll hear about it on the list when I do :-)

2008-07-15 18:05:56

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 15 Jul 2008, Frans Pop wrote:
> So then I build 2.6.27-rc1 and install it. Great.
>
> You release 2.6.27-rc2 and I build it. Ouch! It fails to install, at least
> if I want to install it _alongside_ 2.6.27-rc1 or other kernels (which I
> do!). Why does it fail? Because dpkg's package management does not allow
> one package to overwrite files already "owned" by another package.

I believe I read in the previous thread that some distros are already using
/lib/firmware/<kernel version>/.

There was also the suggestion of moving the entire set of kernel-packaged
firmware to /lib/modules/<kernel version>/firmware, probably while keeping
/lib/firmware as a second place to look for firmware so that we don't hose
any system.

> If I were able to compile firmware into the modules, the problem would be
> solved in one go.

And this thread would have been shorter, even. I hope someone decides to
write that support instead of complaining ;-)

But I do feel we still need a smarter userspace firmware loader to make
firmware packaging less insane. The "current aproach" you described, with
one firmware per package, is not good. It doesn't allow for multiple
versions of the firmware to be installed should you need it.

> I don't know how the Debian kernel team plans to deal with this for distro
> kernel packages. They probably _do_ want to keep them separate [2]. Maybe
> by grouping firmware for really common drivers into
> firmware-basic-drivers or something along those lines.

I sure hope we go with the more proper fix (a version-enabled firmware
loader that can do the unversioned /lib/firmware as well). It is far more
resilient in the long run.

> [1] Only quick solution I see is to have it install the firmware in a
> versioned directory and have the postinst copy things from there to
> /lib/firmware.

Yuck. That would work only if you never needed two active copies of the
kernel [with different firmware files] active at the same time.

> [2] As one of the developers for Debian Installer I'm not looking forward
> to the complications that is going to cause for us and users.

That was my point. These firmware loading changes are good, but there is a
lot of crap missing (most of it NOT in the kernel) before it can be exposed
to ordinary users.

And without firmware-in-the-module support (which is the ONLY scenario where
the entire userspace will not notice anything different), this WILL cause
problem to distros, we will need to scramble up and fix it all before we can
package 2.6.26. I don't know if this is a big problem, though. The work
will need to be done sooner or later anyway, and we should have enough time
to do so as long as we don't care for packaging the early -rc.

--
"One disk to rule them all, One disk to find them. One disk to bring
them all and in the darkness grind them. In the Land of Redmond
where the shadows lie." -- The Silicon Valley Tarot
Henrique Holschuh

2008-07-15 18:09:54

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 2008-07-15 at 18:52 +0200, Oliver Neukum wrote:
> Am Dienstag 15 Juli 2008 16:40:54 schrieb David Woodhouse:
> > On Tue, 2008-07-15 at 08:23 +0200, Oliver Neukum wrote:
> > > Am Dienstag 15 Juli 2008 08:03:15 schrieb David Woodhouse:
> > > > Mostly it follows the documented recommendation, since most of the
> > > > touched drivers are USB drivers and you end up re-enumerating and
> > > > starting from scratch on resume anyway. And the remainder are so old
> > > > that they don't have suspend/resume support anyway. Remember, we're only
> > > > really updating the older drivers; newer drivers tend to use
> > > > request_firmware() already, and have done for years.
> > >
> > > As for USB that was before persistance. It now allows you to even
> > > support a loss of power without reprobing, provided
> > >
> > > a) the devices does not change IDs
> > > b) you have firmware around when reset_resume() is called
> >
> > c) the driver in question actually _has_ a reset_resume() method
> >
> > The old, unloved drivers which I've touched as part of this patch set
> > did not. In general the newer, loved drivers use request_firmware()
> > already.
> >
>
> Generally we love all drivers and welcome new additions.
> Which did you touch? I lost track.

Ignoring the simple bits to make fw->data const? See
http://git.infradead.org/users/dwmw2/firmware-2.6.git?a=shortlog&h=d71792ac
or git-diff --stat 59890f74..d71792ac | grep -v firmware/

21 drivers, mostly older USB drivers and including four sound drivers
which were already using request_firmware() optionally. The HEAD of my
tree contains a few more updates in drivers/scsi and drivers/net, but
out of respect for Dave's and Jeff's objections I left those out of the
pull request for now. Hopefully the noise will die down when the sky
doesn't _actually_ fall on our heads, and we can merge the remaining
changes into 2.6.28.

drivers/atm/Makefile | 6 +-
drivers/atm/ambassador.c | 140 +-
drivers/atm/ambassador.h | 11 -
drivers/atm/atmsar11.data | 2063 -----
drivers/atm/atmsar11.regions | 6 -
drivers/atm/atmsar11.start | 4 -
drivers/char/dsp56k.c | 84 +-
drivers/char/ip2/fip_firm.h | 2149 -----
drivers/char/ip2/ip2base.c | 5 +-
drivers/char/ip2/ip2main.c | 47 +-
drivers/media/dvb/ttusb-budget/dvb-ttusb-budget.c | 16 +-
.../media/dvb/ttusb-budget/dvb-ttusb-dspbootcode.h | 1644 ----
drivers/media/video/cpia2/cpia2_core.c | 46 +-
drivers/media/video/cpia2/cpia2patch.h | 233 -
drivers/media/video/dabfirmware.h | 1415 ----
drivers/media/video/dabusb.c | 44 +-
drivers/media/video/usbvideo/vicam.c | 317 +-
drivers/net/tokenring/smctr.c | 56 +-
drivers/net/tokenring/smctr.h | 2 -
drivers/net/tokenring/smctr_firmware.h | 978 ---
drivers/net/usb/kaweth.c | 43 +-
drivers/net/usb/kawethfw.h | 557 --
drivers/usb/misc/emi26.c | 96 +-
drivers/usb/misc/emi26_fw.h | 5779 -------------
drivers/usb/misc/emi62.c | 130 +-
drivers/usb/misc/emi62_fw_m.h | 8853 --------------------
drivers/usb/misc/emi62_fw_s.h | 8837 -------------------
drivers/usb/serial/Kconfig | 24 +-
drivers/usb/serial/io_edgeport.c | 183 +-
drivers/usb/serial/io_fw_boot.h | 556 --
drivers/usb/serial/io_fw_boot2.h | 546 --
drivers/usb/serial/io_fw_down.h | 1229 ---
drivers/usb/serial/io_fw_down2.h | 1133 ---
drivers/usb/serial/io_fw_down3.h | 847 --
drivers/usb/serial/io_ti.c | 73 +-
drivers/usb/serial/keyspan.c | 78 +-
drivers/usb/serial/keyspan.h | 84 -
drivers/usb/serial/keyspan_mpr_fw.h | 286 -
drivers/usb/serial/keyspan_pda.S | 1124 ---
drivers/usb/serial/keyspan_pda.c | 51 +-
drivers/usb/serial/keyspan_pda_fw.h | 99 -
drivers/usb/serial/keyspan_usa18x_fw.h | 447 -
drivers/usb/serial/keyspan_usa19_fw.h | 285 -
drivers/usb/serial/keyspan_usa19qi_fw.h | 284 -
drivers/usb/serial/keyspan_usa19qw_fw.h | 448 -
drivers/usb/serial/keyspan_usa19w_fw.h | 446 -
drivers/usb/serial/keyspan_usa28_fw.h | 466 -
drivers/usb/serial/keyspan_usa28x_fw.h | 447 -
drivers/usb/serial/keyspan_usa28xa_fw.h | 449 -
drivers/usb/serial/keyspan_usa28xb_fw.h | 448 -
drivers/usb/serial/keyspan_usa49w_fw.h | 464 -
drivers/usb/serial/keyspan_usa49wlc_fw.h | 476 --
drivers/usb/serial/ti_fw_3410.h | 885 --
drivers/usb/serial/ti_fw_5052.h | 885 --
drivers/usb/serial/ti_usb_3410_5052.c | 40 +-
drivers/usb/serial/whiteheat.c | 77 +-
drivers/usb/serial/whiteheat_fw.h | 1669 ----
drivers/usb/serial/xircom_pgs.S | 1192 ---
drivers/usb/serial/xircom_pgs_fw.h | 103 -
sound/isa/Kconfig | 9 -
sound/isa/sb/sb16_csp.c | 22 -
sound/isa/sb/sb16_csp_codecs.h | 949 ---
sound/pci/Kconfig | 30 -
sound/pci/korg1212/korg1212-firmware.h | 987 ---
sound/pci/korg1212/korg1212.c | 18 -
sound/pci/maestro3.c | 171 -
sound/pci/ymfpci/ymfpci_image.h | 1565 ----
sound/pci/ymfpci/ymfpci_main.c | 63 -

--
dwmw2

2008-07-15 18:12:06

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

Linus Torvalds wrote:
>
> On Tue, 15 Jul 2008, Linus Torvalds wrote:
>> This is the "I'm too stupid to live" argument. I've already answered that.
>> What's so hard with "make firmware_install"?
>
> Btw, that is by no means the only way to handle it either. As I said in
> other emails - about trying to find _productive_ questions to answer -
> maybe we could also just install it in the modules directory or some
> similar per-kernel location, and just teach request_firmware() to load
> binary blobs directly.
>
> Then - horror of horrors - we could even make "make modules_install" do
> things like that automatically for you with some Makefile rule trickery.
>
> See? There are _productive_ things you can do or ask others to do.
>
> But singing the same "I'm so stupid that I cannot install the firmware"
> song over and over again doesn't make me think you are worth listening to.

Can't you see that changing build processes takes time? Fixing driver
disk creation tools take time? Validating that each driver is packaged
with its correct firmware takes time?

With each driver converted, it is entirely possible that all users are
completely unaware that it even had a firmware, much less that it now
needs 'make firmware_install' to continue working.

If one is completely unaware of a driver's firmware need, (a) you don't
know you need to install it and (b) you don't find out you need it until
you boot and the driver fails to work.

I am trying to work to minimize (a) and (b) here, because as deployed,
there are large chances for failure here, across a random selection of
drivers.

Jeff

P.S. AFAIK, 'make modules_install' installs firmware, unless that
change was not pushed upstream.

2008-07-15 18:15:17

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

Henrique de Moraes Holschuh wrote:
> On Tue, 15 Jul 2008, Frans Pop wrote:
>> If I were able to compile firmware into the modules, the problem would be
>> solved in one go.
>
> And this thread would have been shorter, even. I hope someone decides to
> write that support instead of complaining ;-)

I've volunteered -- but if Linus and David will reject my code out of
hand, there is not much point in writing it.

>> [2] As one of the developers for Debian Installer I'm not looking forward
>> to the complications that is going to cause for us and users.
>
> That was my point. These firmware loading changes are good, but there is a
> lot of crap missing (most of it NOT in the kernel) before it can be exposed
> to ordinary users.

100% agreed.

That's the point I've been trying to make all along.

The end goal is good, but it's not at all ready for normal users.

> And without firmware-in-the-module support (which is the ONLY scenario where
> the entire userspace will not notice anything different), this WILL cause
> problem to distros, we will need to scramble up and fix it all before we can
> package 2.6.26. I don't know if this is a big problem, though. The work
> will need to be done sooner or later anyway, and we should have enough time
> to do so as long as we don't care for packaging the early -rc.

Agreed.

Jeff

2008-07-15 18:20:50

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

Linus Torvalds wrote:
>
> On Tue, 15 Jul 2008, Jeff Garzik wrote:
>> Already addressed -- you and David seem to be ignoring the common case where
>> you may update the driver but not the kernel, making that feature far less
>> useful than it appears.
>
> This is the "I'm too stupid to live" argument. I've already answered that.
> What's so hard with "make firmware_install"?

So hard that several kernel hackers wound up with non-working drivers
(read the pre-push bug reports).

Which is why 'make modules_install' installs the firmware, or at least
it did before David W pushed upstream.

That was a key build process regression fix (thanks David!).

Sorry Linus, the end change is a good idea, but you cannot hand-wave
away breakage just by calling people stupid.

Jeff

2008-07-15 18:21:20

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 2008-07-15 at 15:05 -0300, Henrique de Moraes Holschuh wrote:
>
> I believe I read in the previous thread that some distros are already using
> /lib/firmware/<kernel version>/.
>
> There was also the suggestion of moving the entire set of kernel-packaged
> firmware to /lib/modules/<kernel version>/firmware, probably while keeping
> /lib/firmware as a second place to look for firmware so that we don't hose
> any system.

You can do that right now; I expect Ubuntu will, since their userspace
already looks like that. You just need to set $(INSTALL_FW_PATH) when
you install the firmware (or modules).

The Ubuntu folks have chosen not to share their changes with upstream,
though -- and if we were to _impose_ that policy, then we would be
requiring changes to userspace which are not yet widespread. That's a
bad idea. What we're doing right now is _entirely_ compatible with what
userspace already has to deal with for modern drivers, and that's very
important.

There may be some merit in Ubuntu's policy, but we need userspace to
change first, long before we can change the default in the kernel.

But certainly the kernel _allows_ that, if you want it.

--
dwmw2

2008-07-15 18:22:35

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 15 Jul 2008, Jeff Garzik wrote:
>
> Can't you see that changing build processes takes time? Fixing driver disk
> creation tools take time? Validating that each driver is packaged with its
> correct firmware takes time?

Umm. Can't you see that if we don't start doing it, it will never get done
AT ALL?

Yes, this will take time.

But that's not an argument for not merging it. Quite the reverse.

Linus

2008-07-15 18:25:24

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 15 Jul 2008, Jeff Garzik wrote:

> Henrique de Moraes Holschuh wrote:
> > On Tue, 15 Jul 2008, Frans Pop wrote:
> > > If I were able to compile firmware into the modules, the problem would be
> > > solved in one go.
> >
> > And this thread would have been shorter, even. I hope someone decides to
> > write that support instead of complaining ;-)
>
> I've volunteered -- but if Linus and David will reject my code out of hand,
> there is not much point in writing it.

Quite frankly, I don't think it's something to reject out of hand, but it
does sound pretty damn stupid.

Why compile them into the module, when that's just the _inflexible_ model?

Why not just install them with the module (perhaps in the module
directory, perhaps not), and just load them?

Really. You _do_ have do to "make modules_install" or similar. That has
_already_ got inter-file dependencies (ie some modules simply depend on
other modules). I'm not seeing why you cannot accept change.

Linus

2008-07-15 18:26:47

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 15 Jul 2008, Jeff Garzik wrote:
>
> Which is why 'make modules_install' installs the firmware, or at least it did
> before David W pushed upstream.

So you're literally just about making this be "make modules_install"
rather than "make firmware_install"

Ok. Are you going to be happy if "make modules_install" just copies the
firmware files of the affected modules too?

Or are you just whining because you want to whine, and will make up some
new argument then?

Linus

2008-07-15 18:31:17

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 2008-07-15 at 11:25 -0700, Linus Torvalds wrote:
>
> On Tue, 15 Jul 2008, Jeff Garzik wrote:
> >
> > Which is why 'make modules_install' installs the firmware, or at least it did
> > before David W pushed upstream.
>
> So you're literally just about making this be "make modules_install"
> rather than "make firmware_install"
>
> Ok. Are you going to be happy if "make modules_install" just copies the
> firmware files of the affected modules too?

I did that already. I'm not _entirely_ sure it was the right thing to
do, but it does help to reduce surprises when people don't realise that
a driver they're using now has separate firmware.

I don't believe it was the result of a constructive request from Jeff; I
think someone else suggested it in response to his whining. It may even
have been my idea, but I don't recall.

This is the current situation:

'make modules_install' will install the firmware required by the modules
it installs. It'll install that to $(INSTALL_MOD_PATH)/lib/firmware/
unless you override that by setting $(INSTALL_FW_PATH). So you can put
it into a kernel-specific directory if you want to.

'make firmware_install' will install _all_ available firmware files
to /lib/firmware (or $(INSTALL_FW_PATH)), regardless of architecture or
configuration. That's what Fedora is using for its 'kernel-firmware'
package, which is arch-independent and required by the various kernel
binary packages.

--
dwmw2

2008-07-15 18:34:09

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

Linus Torvalds wrote:
>
> On Tue, 15 Jul 2008, Jeff Garzik wrote:
>> Can't you see that changing build processes takes time? Fixing driver disk
>> creation tools take time? Validating that each driver is packaged with its
>> correct firmware takes time?
>
> Umm. Can't you see that if we don't start doing it, it will never get done
> AT ALL?
>
> Yes, this will take time.

Which, by implication, means that userland is not -already- prepared.

Which means that if you don't have a recent userland of a mainstream
distro, the result is non-working drivers.

Which is the type of regression I thought we did not want.

> But that's not an argument for not merging it. Quite the reverse.

This is not an either/or proposition, and that is what you are missing.

The change should go in _while at the same time_ not creating regressions.

The change is not the problem.

That the change exists without the ability to recreate similar outputs
(i.e. firmware in module) is the problem.

Jeff

2008-07-15 18:38:21

by Chris Friesen

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

Linus Torvalds wrote:

> Why compile them into the module, when that's just the _inflexible_ model?
>
> Why not just install them with the module (perhaps in the module
> directory, perhaps not), and just load them?
>
> Really. You _do_ have do to "make modules_install" or similar. That has
> _already_ got inter-file dependencies (ie some modules simply depend on
> other modules). I'm not seeing why you cannot accept change.

Even if the end goal is to support more flexible firmware loading, it
would seem to make sense to allow building the firmware into the
module--if only as an interim step to avoid breaking existing setups.

Chris

2008-07-15 18:45:11

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

Linus Torvalds wrote:
>
> On Tue, 15 Jul 2008, Jeff Garzik wrote:
>> Which is why 'make modules_install' installs the firmware, or at least it did
>> before David W pushed upstream.
>
> So you're literally just about making this be "make modules_install"
> rather than "make firmware_install"
>
> Ok. Are you going to be happy if "make modules_install" just copies the
> firmware files of the affected modules too?

Re-read the above, and what you snipped. My statement was

* upstream _already_ does this (installs firmware via modules_install)

* it is a good thing, because it means existing build scripts and
processes will Do The Right Thing and produce a working driver

* thus eliminating the most common problem likely to be encountered

That does not eliminate all problems, because build processes still have
hardcoded assumptions about outputs.

Restoring firmware-in-module ability closes the last logical hole, last
fundamental flaw in the scheme, by permitting distros to reproduce
precisely what they built in 2.6.26.

All of the regressions examples I am citing are cured by
firmware-in-module, because they are all examples of problems that occur
when you remove the ability to reproduce the same functional pieces as
2.6.26.

Jeff

2008-07-15 18:54:54

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 15 Jul 2008, Jeff Garzik wrote:
>
> That does not eliminate all problems, because build processes still have
> hardcoded assumptions about outputs.

Tough. They need to get fixed.

What's the downside of fixing them, again?

Linus

2008-07-15 18:56:00

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

Linus Torvalds wrote:
>
> On Tue, 15 Jul 2008, Jeff Garzik wrote:
>
>> Henrique de Moraes Holschuh wrote:
>>> On Tue, 15 Jul 2008, Frans Pop wrote:
>>>> If I were able to compile firmware into the modules, the problem would be
>>>> solved in one go.
>>> And this thread would have been shorter, even. I hope someone decides to
>>> write that support instead of complaining ;-)
>> I've volunteered -- but if Linus and David will reject my code out of hand,
>> there is not much point in writing it.
>
> Quite frankly, I don't think it's something to reject out of hand, but it
> does sound pretty damn stupid.
>
> Why compile them into the module, when that's just the _inflexible_ model?

Because it works today.

Because that is what people and their scripts expect, today.

Because people should not be forced to fix all their firmware-related
breakage immediately, just to boot 2.6.27.

> I'm not seeing why you cannot accept change.

It is highly feasible to do the change without breaking stuff and
removing choice.

Beer can be great tasting AND less filling.

Jeff

2008-07-15 18:58:50

by Theodore Ts'o

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, Jul 15, 2008 at 02:44:46PM -0400, Jeff Garzik wrote:
> All of the regressions examples I am citing are cured by
> firmware-in-module, because they are all examples of problems that occur
> when you remove the ability to reproduce the same functional pieces as
> 2.6.26.

Jeff,

I think you've said this before, but let me try to help you
cut to the chase. You are willing to *implement*
firmware-in-the-module for request_firmware(), but you want a
commitment that David Woodhouse and Linus will accept such a patch
before you go off and write it. Is that right?

If so, may I suggest you start implementing, instead of
sending e-mails? For all the time people have spent arguing about it,
maybe it could have been implemented already.

Once it has been implemented, do you have any further
objections aka ideas about how request_firmware() could be improved?

- Ted
(who is hoping we can guide
this thread onto slightly more
productive avenues)

2008-07-15 19:04:19

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 15 Jul 2008, Jeff Garzik wrote:
>
> Because people should not be forced to fix all their firmware-related breakage
> immediately, just to boot 2.6.27.

You're continuing to make an argument that doesn't seem to backed up with
any actual real problems.

Can you point to real breakage? For real people?

It sounds like your whole argument literally boils down to "one or two
people doing something really stupid or odd cannot just fix their setup".

What is the real-life situation where copying the firmware with the
modules (but still as separate files) actually breaks?

IOW, what _is_ this theoretical breakage? And why is it so deadly
suddenly? Give us real examples that somehow cannot be fixed?

Linus

2008-07-15 19:08:30

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 2008-07-15 at 14:55 -0400, Jeff Garzik wrote:
> Linus Torvalds wrote:
> >
> > On Tue, 15 Jul 2008, Jeff Garzik wrote:
> >
> >> Henrique de Moraes Holschuh wrote:
> >>> On Tue, 15 Jul 2008, Frans Pop wrote:
> >>>> If I were able to compile firmware into the modules, the problem would be
> >>>> solved in one go.
> >>> And this thread would have been shorter, even. I hope someone decides to
> >>> write that support instead of complaining ;-)
> >> I've volunteered -- but if Linus and David will reject my code out of hand,
> >> there is not much point in writing it.
> >
> > Quite frankly, I don't think it's something to reject out of hand, but it
> > does sound pretty damn stupid.
> >
> > Why compile them into the module, when that's just the _inflexible_ model?
>
> Because it works today.
>
> Because that is what people and their scripts expect, today.
>
> Because people should not be forced to fix all their firmware-related
> breakage immediately, just to boot 2.6.27.

On one hand, they don't -- the drivers I've updated in 2.6.27 aren't the
ones they need to boot. I deliberately left drivers/net and drivers/scsi
till later. I don't _expect_ any problems, but there's no rush -- we can
iron out any kinks with the drivers I've done already, then merge the
remaining updates into 2.6.28.

One the other hand, they _do_ need to fix such breakage immediately, if
they have it -- because what I'm doing here is nothing new. Most modern
drivers use request_firmware() already.

So if your userspace has this hypothetical "firmware-related breakage"
of which you speak, then it already won't boot on machines with qla2xxx,
which was converted to use request_firmware() in 2005. To pick but one
example out of many.

--
dwmw2

2008-07-15 19:09:44

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

Linus Torvalds wrote:
>
> On Tue, 15 Jul 2008, Jeff Garzik wrote:
>> That does not eliminate all problems, because build processes still have
>> hardcoded assumptions about outputs.
>
> Tough. They need to get fixed.
>
> What's the downside of fixing them, again?

Yes, they need fixing.

But without firmware-in-module this would be akin to adding kernel
module support in version 1.0.1, and forcing all distros to fix their
stuff immediately in order to upgrade from version 1.0.0.

Permitting firmware-in-module means you do not force each distro, build
script, and user immediately into the new regime, with the associated
long list of breakage examples.

It's just sad because it feels "not the Linux way".

The way this was rolled out just feels "harder", far less friendly on
the distros -- our customers -- than most other transitions. Not as
hard as vmlinuz->kmodule and IDE->libata transitions, I grant, but still
not the way we usually do things.

Nobody was ever objecting to request_firmware(). It was always about
the removal of firmware-in-module and associated forced breakage on
distros and build processes. The fallout of _that_ is the rich source
of breakage examples I've been pulling from.

Jeff

2008-07-15 19:13:54

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tuesday 15 July 2008, David Woodhouse wrote:
> On Tue, 2008-07-15 at 15:05 -0300, Henrique de Moraes Holschuh wrote:
> > I believe I read in the previous thread that some distros are already
> > using /lib/firmware/<kernel version>/.
> >
> > There was also the suggestion of moving the entire set of
> > kernel-packaged firmware to /lib/modules/<kernel version>/firmware,
> > probably while keeping /lib/firmware as a second place to look for
> > firmware so that we don't hose any system.
>
> You can do that right now; I expect Ubuntu will, since their userspace
> already looks like that. You just need to set $(INSTALL_FW_PATH) when
> you install the firmware (or modules).

With the disclaimer that I really haven't researched this all that much I
think we'll end up finding that a multi-level model is needed.

Something like:
- /lib/firmware: intended for externally provided firmware and local
overrides; try to load from there first (version checks may fail)
- /lib/modules/firmware: not sure if this level is necessary, but
could be useful for e.g. firmware for out-of-tree drivers
- /lib/modules/$kver/firmware or
/lib/modules/$kver/$modulepath: firmware built (and possibly packaged
together with the kernel

Some scheme like this would introduce at least some sanity when it comes
to keeping control of what is installed and provided (and may therefore
be removed) by what and to avoid the question "how the hell did that file
get there and do I still actually need it with my current kernels?".

It sounds to me like $(INSTALL_FW_PATH) is a good start, but maybe even
too flexible as it does not define a standard: it allows user spaces of
different distros to diverge which at some point will create its own
nightmares.
It's also not flexible enough as it only supports a single location for
firmware while clearly there can be different sources of firmware.

Cheers,
FJP

2008-07-15 19:14:48

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

Theodore Tso wrote:
> On Tue, Jul 15, 2008 at 02:44:46PM -0400, Jeff Garzik wrote:
>> All of the regressions examples I am citing are cured by
>> firmware-in-module, because they are all examples of problems that occur
>> when you remove the ability to reproduce the same functional pieces as
>> 2.6.26.
>
> Jeff,
>
> I think you've said this before, but let me try to help you
> cut to the chase. You are willing to *implement*
> firmware-in-the-module for request_firmware(), but you want a
> commitment that David Woodhouse and Linus will accept such a patch
> before you go off and write it. Is that right?
>
> If so, may I suggest you start implementing, instead of
> sending e-mails? For all the time people have spent arguing about it,
> maybe it could have been implemented already.

Already started, in fact, since Linus said he would not reject it out of
hands.

Obviously that is not acceptance; I know as well as any that acceptance
does not occur until the moment of upstream merge.

> Once it has been implemented, do you have any further
> objections aka ideas about how request_firmware() could be improved?

Further objections? None major.

The two other minor problems I feel need solving, but are not related to
breakage or regressions, are:

1) firmware should be able to live alongside the driver. We don't need
to start growing firmware/net alongside drivers/net, firmware/scsi
alongside drivers/scsi, firmware/char alongside drivers/char, etc.

2) firmware should be able to be shipped in final format (binary),
rather than ihex.

I feel strongly that 2.6.27 should not be released without
firmware-in-module, for all the reasons mentioned in these threads.
It's a major regression, IMO, without firmware-in-module ability.

The other stuff (#1, #2 in list above) is small potatoes.

Jeff

2008-07-15 19:19:24

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 15 Jul 2008, Jeff Garzik wrote:
>
> But without firmware-in-module this would be akin to adding kernel module
> support in version 1.0.1, and forcing all distros to fix their stuff
> immediately in order to upgrade from version 1.0.0.

You're just making that up. The distro's can fix it long before they use
the kernel.

And the users we can just fix by making request_firmware() look up the
files itself, if it doesn't already.

You're just making up your bogus arguments.

Linus

2008-07-15 19:20:39

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

Hi Frans,

> > On Mon, 14 Jul 2008, Jeff Garzik wrote:
> >> IMO the newly added /inability/ to build firmware into kernel modules
> >> is a clear regression.
> >
> > IMO you're being stupid.
> >
> > How about explainign why it makes any difference what-so-ever?
> >
> > If you can load the module, you can load the firmware. Claiming anything
> > else is just _stupid_.
>
> Sure, in theory it's that simple. Here's a concrete example that shows how
> things are harder in practice.
>
> I use the 'make deb-pkg' target (from scripts/package) to build my Debian
> kernel packages from git. So that needs to be adapted to include
> /lib/firmware. No real problem so far.
>
> So then I build 2.6.27-rc1 and install it. Great.
>
> You release 2.6.27-rc2 and I build it. Ouch! It fails to install, at least
> if I want to install it _alongside_ 2.6.27-rc1 or other kernels (which I
> do!). Why does it fail? Because dpkg's package management does not allow
> one package to overwrite files already "owned" by another package.
>
> So, how is this solved by Debian for already existing firmware packages?
> Basically by making a separate package for each firmware file (or
> driver). This works because there are not too many of them, but having a
> huge number of tiny packages is a nightmare by itself.
>
> But anyway, the dep-pkg target will have to be made smarter than it is now
> if it's to deal with this [1]. And at least currently it is broken.
>
> If I were able to compile firmware into the modules, the problem would be
> solved in one go.
>
> I don't know how the Debian kernel team plans to deal with this for distro
> kernel packages. They probably _do_ want to keep them separate [2]. Maybe
> by grouping firmware for really common drivers into
> firmware-basic-drivers or something along those lines.
>
> Cheers,
> FJP
>
> [1] Only quick solution I see is to have it install the firmware in a
> versioned directory and have the postinst copy things from there to
> /lib/firmware.

using /lib/firmware/`uname -r`/ is actually not a bad idea. You only
have to fix udev to actually include this in the list of directories to
look for firmware files. Also Ubuntu is already doing this.

Regards

Marcel

2008-07-15 19:21:23

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 15 Jul 2008, Jeff Garzik wrote:
>
> Already started, in fact, since Linus said he would not reject it out of
> hands.

Btw, I need to see it actually solve a real problem. As it is, I really
think you've just painted yourself into a corner where you can't just
admit that we can just load the firmware directly.

Could you _please_ entertain at least thinking about that option?

Linus

2008-07-15 19:22:33

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

Hi Linus,

> > David W has made it very clear that the eventual intent of this patch series
> > is to remove all firmware from the kernel tree and have it shipped as a
> > seperate package.
>
> I don't think that kind of black-and-white model is necessarily
> attainable. Nor do I think it's even _good_.
>
> It's probably the right thign for _some_ firmware, assuming we have models
> that just make it easy enough - for drivers where it's generally better
> for distros to get the firmware from vendors rather than the kernel.
>
> But there is certainly nothing inherently good in forcing a split.
>
> I think both sides here need to calm down and stop being so extreme.
>
> I do want request_firmware() to be the way to load firmware (and possibly
> other configuration data too, for that matter - I don't think it's
> necessarily wrong for people to "misuse" it for other long-term data
> despite the name), and I do want people to be able to update firmware
> easily without having to force a kernel bump, but I also don't think it
> means that firmware has to go away from the kernel sources.

actually the PCMCIA subsystem is already using request_firmware() to
load external CIS data to overwrite broken ones.

Regards

Marcel

2008-07-15 19:24:36

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 15 Jul 2008, Linus Torvalds wrote:
>
> Could you _please_ entertain at least thinking about that option?

.. and the reason I ask you to do that is quite frankly, you seem to want
to just hide the thing under a rug - and _encouraging_ people to not even
try to work with a new and more flexible model.

For example, there's nothing wrong with having people replace their
firmware when they want to. Equally, there's nothing wrong in having
distros put stable firmware into another package, so that you don't need
to have <n> different versions of the firmware in <n> different kernel
module directories just because you have <n> different versions of the
module itself.

So why do you want to work so hard at not just encouraging people to get
the firmware loaded dynamically from the filesystem?

Linus

2008-07-15 19:26:44

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

Hi Linus,

> > Why don't you just take the kernel-supplied firmware and make it part of
> > the kernel package? The same way the kernel-supplied modules are part of
> > it?
>
> And, btw, if there is something we can do to help, we obviously should.
>
> So ask yourself (and DavidW) things like
>
> - Is "make firmware_install" not enough for the package build system?
>
> - Does it need some other way to figure out what firmware is needed for
> just the configured set of drivers, eg "make firmware_list"?
>
> - Do we need to make the firmware loader have multiple directories it
> looks at? Do we want first a kernel-version-specific one, then a
> generic fallback one, so that we can easily have per-kernel firmware,
> the same way we have per-kernel modules?

as just mentioned in another post. If you look at Ubuntu, their udev
helper that actually looks for the firmware and loads it into the kernel
does exactly this. Historically we have also old paths since the
agreement on /lib/firmware came later, but that is userspace policy
anyway and not part of the kernel.

Regards

Marcel

2008-07-15 19:27:57

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 15 Jul 2008, Marcel Holtmann wrote:
>
> using /lib/firmware/`uname -r`/ is actually not a bad idea. You only
> have to fix udev to actually include this in the list of directories to
> look for firmware files. Also Ubuntu is already doing this.

I really don't think we need to even "fix udev".

Why don't we just load it ourselves? Esepcially as there are probably
places that try to avoid udev entirely, or at least use a very
cut-down-version.

We should be fairly trivially able to be _entirely_ backwards compatible
with any sane setup (not the _sane_ part! It implies that people don't
copy individual modules around by hand!), with no actual breakage or need
for distros to even update anything at all - just make the kernel able to
look up binary blobs in the same place it installed them.

That sounds like the RightThing(tm) to do _regardless_ of any other
issues, doesn't it? If the kernel installs it in some known place, why
should it not just read them from that known place?

Linus

2008-07-15 19:28:24

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 2008-07-15 at 14:58 -0400, Theodore Tso wrote:
> On Tue, Jul 15, 2008 at 02:44:46PM -0400, Jeff Garzik wrote:
> > All of the regressions examples I am citing are cured by
> > firmware-in-module, because they are all examples of problems that occur
> > when you remove the ability to reproduce the same functional pieces as
> > 2.6.26.
>
> Jeff,
>
> I think you've said this before, but let me try to help you
> cut to the chase. You are willing to *implement*
> firmware-in-the-module for request_firmware(), but you want a
> commitment that David Woodhouse and Linus will accept such a patch
> before you go off and write it. Is that right?

The more important question is whether _Linus_ will take it, and of
course I can't answer for him.

Personally I think it's pointless. But since it's only _mildly_
counterproductive, if you really want to do it then I wouldn't actively
oppose it unless it turns out to be a mess to implement, or unless it's
limited to working only with the in-tree firmware.

If you implement it cleanly and optionally (and defaulting off), and if
it works properly with out-of-tree firmware too (so I can build a
libertas usb8xxx.ko module with the firmware included, for example),
then I certainly wouldn't throw my toys out of the pram over it.
Ideally, it would work for out-of-tree drivers (make M=... modules) too.

I was thinking that we could maybe post-process .ko files (which we do
already for other reasons), and add a special section in them containing
the firmware which is mentioned in MODULE_FIRMWARE() tags. The special
section is how we do it for the kernel too. That would take a little bit
of code in module.c to keep track of those sections and add them to a
list somewhere that firmware_class.c can see them, and then you have to
think about lifetime and locking issues.

But those were just preliminary thoughts -- I wasn't really planning to
implement it so I didn't go through it in detail.

An alternative approach might be to turn firmware blobs into actual
modules, which just register themselves with the firmware loader in
module_init(). I'm not sure I like that approach though, and depmod
wouldn't 'notice' them automatically so it wouldn't work with current
userspace. I've tried to be very careful to ensure that anything we do
here is entirely compatible with existing userspace.

--
dwmw2

2008-07-15 19:29:19

by Peter Breuer

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

In article <[email protected]> Linus wrote:
> Why not just install them with the module (perhaps in the module
> directory, perhaps not), and just load them?

I also don't get why one can't put them in a module wrapper in some OTHER
dir as

__init__ fwFOO = something_huge_made_from_a_string ;
EXPORT_SYMBOL(fwFOO);

and have the real driver module just reference fwFOO in its init sequence.
Modprobe can deal with where to look for it. No kernel source need be
tainted.

Peter

2008-07-15 19:31:28

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

Hi Linus,

> > using /lib/firmware/`uname -r`/ is actually not a bad idea. You only
> > have to fix udev to actually include this in the list of directories to
> > look for firmware files. Also Ubuntu is already doing this.
>
> I really don't think we need to even "fix udev".

fixing udev means adding an extra path to it :)

> Why don't we just load it ourselves? Esepcially as there are probably
> places that try to avoid udev entirely, or at least use a very
> cut-down-version.
>
> We should be fairly trivially able to be _entirely_ backwards compatible
> with any sane setup (not the _sane_ part! It implies that people don't
> copy individual modules around by hand!), with no actual breakage or need
> for distros to even update anything at all - just make the kernel able to
> look up binary blobs in the same place it installed them.
>
> That sounds like the RightThing(tm) to do _regardless_ of any other
> issues, doesn't it? If the kernel installs it in some known place, why
> should it not just read them from that known place?

Personally I am against hacking file loading from within the kernel.
This is why we involved udev (or hotplug in the beginning). It allows
you to have userspace policy for the search paths etc.

Regards

Marcel

2008-07-15 19:33:17

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 15 Jul 2008, Linus Torvalds wrote:
>
> [..] just make the kernel able to look up binary blobs in the same place
> it installed them.

Obviously, we'd want to make it dynamically configurable, ie the same way
we specify where to find 'modprobe', let people specify (in
/proc/kernel/firmware-dir or something) where to find the firmware. Just
default it to something sane (that obviously matches where we installed
them).

I'm not a huge udevd fan, obviously. I think we can short-circuit this all
in the name of expediency, with udevd as a fallback, not as the only way.

Linus

2008-07-15 19:37:45

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 2008-07-15 at 12:27 -0700, Linus Torvalds wrote:
>
> On Tue, 15 Jul 2008, Marcel Holtmann wrote:
> >
> > using /lib/firmware/`uname -r`/ is actually not a bad idea. You only
> > have to fix udev to actually include this in the list of directories to
> > look for firmware files. Also Ubuntu is already doing this.
>
> I really don't think we need to even "fix udev".
>
> Why don't we just load it ourselves? Esepcially as there are probably
> places that try to avoid udev entirely, or at least use a very
> cut-down-version.
>
> We should be fairly trivially able to be _entirely_ backwards compatible
> with any sane setup (not the _sane_ part! It implies that people don't
> copy individual modules around by hand!), with no actual breakage or need
> for distros to even update anything at all - just make the kernel able to
> look up binary blobs in the same place it installed them.
>
> That sounds like the RightThing(tm) to do _regardless_ of any other
> issues, doesn't it? If the kernel installs it in some known place, why
> should it not just read them from that known place?

I'm unconvinced that the kernel should be setting this kind of policy.

But I suppose if you make it tunable in sysfs and just switch to calling
do_filp_open() directly from firmware_class.c instead of punting to
userspace, that might work.

It leaves you with less flexibility -- it would prevent stuff like the
udev trick I posted a week or so back to fix the Intel microcode loader
by automatically generating the needed binary blob on the fly from
microcode.dat, for example.

We also have a long tradition of pointing and laughing at people who
want to call open() from within the kernel. But it _could_ work,
certainly.

I'm not really convinced it helps though. The script which creates an
initrd still needs to look at the MODULE_FIRMWARE() tag and include the
right firmware file. If that's broken, you're screwed anyway. And that
was true in 2.6.25 too.

--
dwmw2

2008-07-15 19:39:29

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

Hi Henrique,

> > You release 2.6.27-rc2 and I build it. Ouch! It fails to install, at least
> > if I want to install it _alongside_ 2.6.27-rc1 or other kernels (which I
> > do!). Why does it fail? Because dpkg's package management does not allow
> > one package to overwrite files already "owned" by another package.
>
> I believe I read in the previous thread that some distros are already using
> /lib/firmware/<kernel version>/.
>
> There was also the suggestion of moving the entire set of kernel-packaged
> firmware to /lib/modules/<kernel version>/firmware, probably while keeping
> /lib/firmware as a second place to look for firmware so that we don't hose
> any system.

can we please leave the /lib/modules/*/firmware idea out of it. While
this thread is about the kernel drivers that need firmware loading, but
it is not the whole story. We do have hardware that needs a firmware
loading process that takes place entirely in userspace. In some USB and
UART cases we can do the firmware loading step before any kernel driver
takes control of that device and currently these firmware loading
helpers look into /lib/firmware/ where every firmware file should be
placed.

Also the /lib/firmware directory was chosen a long time ago and you find
the discussion on the linux-hotplug mailing list archive if you are
interested.

That said, using /lib/firmware/`uname -r`/ like Ubuntu supports right
now and then fall back to /lib/firmware looks like a sane approach to
me. However as David mentioned, this needs patches to udev first before
we go any step further.

Regards

Marcel

2008-07-15 19:40:06

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 15 Jul 2008, Marcel Holtmann wrote:
>
> Personally I am against hacking file loading from within the kernel.
> This is why we involved udev (or hotplug in the beginning). It allows
> you to have userspace policy for the search paths etc.

Well, I'm personally against _forcing_ people to use udev.

Yes, desktop distros generally will do so, since there are lots of complex
issues etc. But I don't think we've generally forced it on people if they
don't want it (ie I haven't tried it personally since the distro I used
started using udev, but I think you can still just ignore udev events and
set everything up statically).

And I don't think that's wrong. Which implies that setup things should
still generally at least allow us to avoid udev.

(But maybe I'm wrong, and everybody already uses udev just because they
couldn't be bothered not to).

Linus

2008-07-15 19:41:59

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 15 Jul 2008, Linus Torvalds wrote:

> On Tue, 15 Jul 2008, Jeff Garzik wrote:
>>
>> Because people should not be forced to fix all their firmware-related breakage
>> immediately, just to boot 2.6.27.
>
> You're continuing to make an argument that doesn't seem to backed up with
> any actual real problems.
>
> Can you point to real breakage? For real people?

several kernel devs ran into this with -next (Jeff can name names if
needed)

> It sounds like your whole argument literally boils down to "one or two
> people doing something really stupid or odd cannot just fix their setup".
>
> What is the real-life situation where copying the firmware with the
> modules (but still as separate files) actually breaks?
>
> IOW, what _is_ this theoretical breakage? And why is it so deadly
> suddenly? Give us real examples that somehow cannot be fixed?

there are no cases that cannot somehow be fixed. the desire is to make it
possible to avoid having to find and fix everything before booting 2.6.27,
while still making it possible to compile a kernel with all the new
capabilities for those who want to fix everything now.

David Lang

2008-07-15 19:44:45

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

Hi Linus,

> > Personally I am against hacking file loading from within the kernel.
> > This is why we involved udev (or hotplug in the beginning). It allows
> > you to have userspace policy for the search paths etc.
>
> Well, I'm personally against _forcing_ people to use udev.
>
> Yes, desktop distros generally will do so, since there are lots of complex
> issues etc. But I don't think we've generally forced it on people if they
> don't want it (ie I haven't tried it personally since the distro I used
> started using udev, but I think you can still just ignore udev events and
> set everything up statically).
>
> And I don't think that's wrong. Which implies that setup things should
> still generally at least allow us to avoid udev.
>
> (But maybe I'm wrong, and everybody already uses udev just because they
> couldn't be bothered not to).

I actually think that is the case here. udev is so lightweight and
really slim nowadays that not using it is a big drawback. There might be
some system out there that are still using a static /dev directory, but
for sure there are not many and even the embedded world doesn't really
mind.

Also I think the /sbin/hotplug thing is still available (not verified
this) and thus calling a firmware loading helper script with a udev
replacement would be possible. Someone correct me here if I am wrong.

Regards

Marcel

2008-07-15 19:50:33

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 15 Jul 2008, David Woodhouse wrote:
>
> I'm unconvinced that the kernel should be setting this kind of policy.

I do think it should probably be conditional, but I think that's true of
udevd itself too, so hey, it cuts both ways.

> But I suppose if you make it tunable in sysfs and just switch to calling
> do_filp_open() directly from firmware_class.c instead of punting to
> userspace, that might work.

Yup. And then you can disable it either statically (config option) or by
writing an invalid path into /proc/sys/kernel/firmware-dir or whatever.

Or you can just decide that if you find something in the kernel-specific
firmware directory, then it should always take precedence over whatever
udev rules. Which sounds good to me anyway.

Maybe you really do have some very kernel-specific issue (ie you're trying
a new driver that can handle a new experimental firmware, but you don't
want your old fall-back kernels to use it because you just fixed the bug
that makes it work again).

Requiring you to write udevd scripts for that sounds insane. I wouldn't
even know where to start.

So kernel-specific directories do make sense. As does the whole "I don't
want to handle the pain that is udev scripts".

Linus

2008-07-15 19:54:09

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

Hi Linus,

> Requiring you to write udevd scripts for that sounds insane. I wouldn't
> even know where to start.
>
> So kernel-specific directories do make sense. As does the whole "I don't
> want to handle the pain that is udev scripts".

you don't have to. We extend udev once and then it will always work. As
mentioned before, Ubuntu is already doing this. They never bothered to
submit this upstream and I didn't really care since I have all my
firmware files in /lib/firmware and had no need to add versions to it.

Regards

Marcel

2008-07-15 19:58:50

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 2008-07-15 at 12:49 -0700, Linus Torvalds wrote:
> So kernel-specific directories do make sense. As does the whole "I
> don't want to handle the pain that is udev scripts".

I'm not violently opposed to kernel-specific directories. Ubuntu does it
that way already, and it may well make sense. I'm not sure it's really
necessary either, but it's an option.

Unless we really are going to make the kernel open files for itself,
though, I think it's too early for us to think about making that change
in the kernel. We'd need userspace to cope with it that way _first_.

(When I say 'in the kernel', I think the only place this 'policy' is yet
implemented is the default for $(INSTALL_FW_PATH) in modules_install and
firmware_install make targets, right?)

--
dwmw2

2008-07-15 20:00:21

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

Linus Torvalds wrote:
>
> On Tue, 15 Jul 2008, Jeff Garzik wrote:
>> Because people should not be forced to fix all their firmware-related breakage
>> immediately, just to boot 2.6.27.
>
> You're continuing to make an argument that doesn't seem to backed up with
> any actual real problems.
>
> Can you point to real breakage? For real people?

Already posted a list.

> It sounds like your whole argument literally boils down to "one or two
> people doing something really stupid or odd cannot just fix their setup".
>
> What is the real-life situation where copying the firmware with the
> modules (but still as separate files) actually breaks?

The breakage comes from all the existing-at-this-point-today processes
that will not copy the firmware with the module.

We're not talking about one or two people, we are talking about projects
being actively used.

Which in turn means not only those projects need to immediately fix
stuff to make 2.6.27 work, but users who build and test kernels are left
to pick up the pieces when their drivers break too.

> IOW, what _is_ this theoretical breakage? And why is it so deadly
> suddenly? Give us real examples that somehow cannot be fixed?

It's not theoretical, we have already run into non-working drivers due
to build process changes. And those were the smart guys, the kernel
hackers.

All of the examples I have listed can certainly be fixed -- well, except
the "new kernel, old system" case -- sometimes easily.

* the consequences of the breakage -- 100% non-working driver, possibly
unbootable system

* the likelihood of breakage -- anything that is not a recent version of
a mainstream distro WILL have problems, because it simply does not know
about the firmware that must follow the module whereever it goes.

* the need to immediately add/fix userland and build processes, just to
keep the same driver set working.

* the need for time, to figure out if you are even affected by this change

* the need for time, to plan the best method to implement firmware
deployment in your distro

Without firmware-in-module, it is a basic fact that you are forcing
everyone to fix their stuff, simply to be able to use 2.6.27 like they
did 2.6.26.

That's unfriendly to distros, users, and kernel testers.

Jeff

2008-07-15 20:00:41

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

Hi David,

> > > Which is why 'make modules_install' installs the firmware, or at least it did
> > > before David W pushed upstream.
> >
> > So you're literally just about making this be "make modules_install"
> > rather than "make firmware_install"
> >
> > Ok. Are you going to be happy if "make modules_install" just copies the
> > firmware files of the affected modules too?
>
> I did that already. I'm not _entirely_ sure it was the right thing to
> do, but it does help to reduce surprises when people don't realise that
> a driver they're using now has separate firmware.
>
> I don't believe it was the result of a constructive request from Jeff; I
> think someone else suggested it in response to his whining. It may even
> have been my idea, but I don't recall.
>
> This is the current situation:
>
> 'make modules_install' will install the firmware required by the modules
> it installs. It'll install that to $(INSTALL_MOD_PATH)/lib/firmware/
> unless you override that by setting $(INSTALL_FW_PATH). So you can put
> it into a kernel-specific directory if you want to.
>
> 'make firmware_install' will install _all_ available firmware files
> to /lib/firmware (or $(INSTALL_FW_PATH)), regardless of architecture or
> configuration. That's what Fedora is using for its 'kernel-firmware'
> package, which is arch-independent and required by the various kernel
> binary packages.

I really think we should use /lib/firmware/`uname -r`/. I do see the
point here that I don't wanna overwrite existing firmware from other
installed kernels. Especially if modules_install will install the
firmware files.

So in case of non-Ubuntu distros we have to push a fix to udev, but that
is not a big deal. It should be a one-line change if I am not mistaken.

Regards

Marcel

2008-07-15 20:06:38

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 2008-07-15 at 22:00 +0200, Marcel Holtmann wrote:
> I really think we should use /lib/firmware/`uname -r`/. I do see the
> point here that I don't wanna overwrite existing firmware from other
> installed kernels. Especially if modules_install will install the
> firmware files.
>
> So in case of non-Ubuntu distros we have to push a fix to udev, but that
> is not a big deal. It should be a one-line change if I am not mistaken.

My concern with that is that even though it's a one-line change, it's a
one-line change which people don't already _have_. We really do need to
remain compatible with existing setups.

For _now_, I think it's much better to leave it in /lib/firmware where
existing upstream udev will find it. Let's change udev ASAP and then we
can talk about when it makes sense to change the default setting of
$(INSTALL_FW_PATH) to match.

I'm not too worried about overwriting existing firmware
in /lib/firmware. We very rarely change firmware anyway, when we _do_
change it it's almost always compatible rather than having ABI changes.
On the rare occasions that remain, the incompatibly changed firmware
really ought to have a new filename so that older drivers don't pick it
up when they want the old one.

It's not a _complete_ non-problem, but it's fairly close. And the
incompatibility with existing upstream udev would be much more of an
issue, I think.

--
dwmw2

2008-07-15 20:08:33

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 15 Jul 2008, Marcel Holtmann wrote:
>
> you don't have to. We extend udev once and then it will always work.

Umm. The thing is, people running new kernels with old user land is not
just supposed to work, it's _really_ supposed to work.

It's what I do. Something that breaks that has to have damn good reasons
to break it.

So I do not disagree with Jeff on that point _at_all_. I'm in violent
agreement with Jeff on the fact that we should not require system updates
for the kernel to do the right thing.

The thing I disagree with Jeff on is that he then seems to turn that into
something very negative ("let's not separate the firmware at all").

And I'd much rather just fix it. And that means that if people can point
to udevd's that get confused - or lack of udevd's entirely - both of which
sound very likely to me, then we should have a graceful fallback position.

And just supporting the notion of loading the firmware directly sounds
like an obvious such case. It may not be the _only_ solution, for example,
which is why I'd actually like to see people point to the _actual_
reported problems.

IOW, the problems shouldn't a "don't do this" thing. They should be a "ok,
that problem happened, we can solve it by doing X".

Linus

2008-07-15 20:10:15

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

Hi David,

> > So kernel-specific directories do make sense. As does the whole "I
> > don't want to handle the pain that is udev scripts".
>
> I'm not violently opposed to kernel-specific directories. Ubuntu does it
> that way already, and it may well make sense. I'm not sure it's really
> necessary either, but it's an option.

then lets get the udev extension merged upstream (since it doesn't do
any harm anyway) and then install everything in /lib/firmware/`uname -r`
and see how that works out.

So Ubuntu will obviously not break since it does this already. For
Fedora deploying new packages for 8 and 9 should not be a problem at all
once the maintainer is convinced that it is a good idea. They update
packages all the time anyway. For the other distros, I don't know.

Regards

Marcel

2008-07-15 20:16:47

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 15 Jul 2008, Jeff Garzik wrote:
>
> Already posted a list.

No you didn't.

You posted a list of problems you have from doing odd and stupid things.
Not the reports, not the explanations of what those odd and stupid
things actually were in practice, and why they had to be that stupid.

Quite frankly, I suspect that it's all a matter of "people compiled their
own kernels, and didn't copy the new files, because they just didn't
realize they needed to".

Which we really don't even need kernel fixes fot, it just needs some
education. BUT. If we knew what the details were, we could make the need
for education even less, so that it would just work with whatever they do.

But yes, if it fundamentally boils down do "you had a script that copied
named files by hand", then that script needs fixing. But that's somethign
that somebody who compiles his own kernel (especially in an odd
environment, apparently) just needs to do.

Sometimes build updates need people to fix things. We find gcc bugs, we
require people to update. The whole "new kernels should work with old
distributions" does not mean that people building the kernels might not
have to do _something_ special.

Linus

2008-07-15 20:18:17

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 15 Jul 2008, Marcel Holtmann wrote:
>
> So Ubuntu will obviously not break since it does this already. For
> Fedora deploying new packages for 8 and 9 should not be a problem at all
> once the maintainer is convinced that it is a good idea. They update
> packages all the time anyway. For the other distros, I don't know.

Umm. Again, Marcel. We don't require new user land to work.

But just loading the thing on our own does solve this problem. But udev we
simply cannot require people to magically have a new version of.

Linus

2008-07-15 20:21:18

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

Hi Linus,

> > you don't have to. We extend udev once and then it will always work.
>
> Umm. The thing is, people running new kernels with old user land is not
> just supposed to work, it's _really_ supposed to work.
>
> It's what I do. Something that breaks that has to have damn good reasons
> to break it.

so request_firmware() is well established and works. It is used by a
bunch of drivers at the moment.

If we do wanna make more heavy use of it (which seems everybody is in
agreement) then we need to make some userspace changes like adding the
kernel version to keep it sane. This can be done step by step to make
sure we keep the current status quo.

And as David mentioned multiple times, he left drivers/{net,scsi} out of
it for now.

> So I do not disagree with Jeff on that point _at_all_. I'm in violent
> agreement with Jeff on the fact that we should not require system updates
> for the kernel to do the right thing.
>
> The thing I disagree with Jeff on is that he then seems to turn that into
> something very negative ("let's not separate the firmware at all").
>
> And I'd much rather just fix it. And that means that if people can point
> to udevd's that get confused - or lack of udevd's entirely - both of which
> sound very likely to me, then we should have a graceful fallback position.
>
> And just supporting the notion of loading the firmware directly sounds
> like an obvious such case. It may not be the _only_ solution, for example,
> which is why I'd actually like to see people point to the _actual_
> reported problems.

I really don't see that problem here since request_firmware() has been
around for so long now and so many drivers are using it nowadays that I
don't see your point here on how it will break.

However if you wanna make the kernel do the loading from the filesystem
directly, I am actually fine with it. We were opposed to that idea,
because it was always kinda hackish to have the kernel load files and it
felt wrong to do it this way. Policy decisions on directory layouts
belong into the userspace and not the kernel. At least that is my take
on this.

Regards

Marcel

2008-07-15 20:25:30

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

Hi Linus,

> > So Ubuntu will obviously not break since it does this already. For
> > Fedora deploying new packages for 8 and 9 should not be a problem at all
> > once the maintainer is convinced that it is a good idea. They update
> > packages all the time anyway. For the other distros, I don't know.
>
> Umm. Again, Marcel. We don't require new user land to work.
>
> But just loading the thing on our own does solve this problem. But udev we
> simply cannot require people to magically have a new version of.

we might have to do a step by step upgrade for this. Something like
install everything in /lib/firmware/`uname -r`/ and create symlinks for
a grace period of the next sixt month or something like this.

These are details that have be sorted out, but should not stop us to get
this right.

And before you say anything. You can expect that firmware loading
from /lib/firmware works almost everywhere. At least this has been for
the last 3 years or so. Remember that we do have request_firmware() even
in the latest 2.4 kernels.

Regards

Marcel

2008-07-15 20:27:19

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

Linus Torvalds wrote:
>
> On Tue, 15 Jul 2008, Marcel Holtmann wrote:
>> you don't have to. We extend udev once and then it will always work.
>
> Umm. The thing is, people running new kernels with old user land is not
> just supposed to work, it's _really_ supposed to work.
>
> It's what I do. Something that breaks that has to have damn good reasons
> to break it.
>
> So I do not disagree with Jeff on that point _at_all_. I'm in violent
> agreement with Jeff on the fact that we should not require system updates
> for the kernel to do the right thing.
>
> The thing I disagree with Jeff on is that he then seems to turn that into
> something very negative ("let's not separate the firmware at all").

How many times do I have to say you misunderstand?

I'm happy with separating the firmware at the source code level. I
never objected to that.

All I want is the _option_ to compile firmware into a module. We are
talking about compiled output, here.

That _option_ has the desireable properties of

* working on older userland, even with DavidW's tg3 and scsi patches

* retaining an option several kernel hackers and users find useful

* permitting use of 2.6.27 without _immediately_ having to fix stuff

> And I'd much rather just fix it. And that means that if people can point
> to udevd's that get confused - or lack of udevd's entirely - both of which
> sound very likely to me, then we should have a graceful fallback position.
>
> And just supporting the notion of loading the firmware directly sounds
> like an obvious such case. It may not be the _only_ solution, for example,
> which is why I'd actually like to see people point to the _actual_
> reported problems.
>
> IOW, the problems shouldn't a "don't do this" thing. They should be a "ok,
> that problem happened, we can solve it by doing X".

Stop using strawmen, please.

I never said "don't do that". Ever.

I said "when you do that, don't remove ability to compile firmware into
the driver."

And as has been stated repeatedly, the compiling firmware into the
driver clearly DOES NOT exclude the more flexible, load-from-userland
request_firmware() model.

To be extremely concrete, firmware-in-module is

* add Kconfig option (kernel-wide or per-driver, dunno) asking
"build firmware into drivers, as before?"

* tweak build process to build firmware into foo.ko output,
probably in a specially marked ELF section

* get request_firmware() to automatically notice that the
MODULE_FIRMWARE() was built into this driver, and to
look at the special ELF section for its data

See? That is not a "don't do that." That has never been "don't do that."

That is the one major request here, one that enables us to avoid all
those potential breakages I've been listing, and the many more breakages
of which I'm unaware. It enables use of older userlands.

And yes, it enables "weirdos" like me to continue building the firmware
into the driver, because it has a proven track record of high
reliability and simplicity.

All without stomping on the flexibility of request_firmware().

Jeff

2008-07-15 20:30:38

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

Hi David,

> > I really think we should use /lib/firmware/`uname -r`/. I do see the
> > point here that I don't wanna overwrite existing firmware from other
> > installed kernels. Especially if modules_install will install the
> > firmware files.
> >
> > So in case of non-Ubuntu distros we have to push a fix to udev, but that
> > is not a big deal. It should be a one-line change if I am not mistaken.
>
> My concern with that is that even though it's a one-line change, it's a
> one-line change which people don't already _have_. We really do need to
> remain compatible with existing setups.
>
> For _now_, I think it's much better to leave it in /lib/firmware where
> existing upstream udev will find it. Let's change udev ASAP and then we
> can talk about when it makes sense to change the default setting of
> $(INSTALL_FW_PATH) to match.

fair enough. What about keeping symlink in /lib/firmware, but actually
doing the install in a versioned directory?

> I'm not too worried about overwriting existing firmware
> in /lib/firmware. We very rarely change firmware anyway, when we _do_
> change it it's almost always compatible rather than having ABI changes.
> On the rare occasions that remain, the incompatibly changed firmware
> really ought to have a new filename so that older drivers don't pick it
> up when they want the old one.

Actually in some case we do. I think about the case where I have an two
years old kernel that came with the distro and would need that firmware
to make the driver work. However now we have the new firmware and it
falls over.

Breaking a distro install is worse then breaking a new kernel. At least
this is how I see it. I do like to have the distro setup as fallback or
rescue when I mess up.

Regards

Marcel

2008-07-15 20:32:27

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 15 Jul 2008, Marcel Holtmann wrote:
>
> so request_firmware() is well established and works. It is used by a
> bunch of drivers at the moment.

That's not the question.

The question is whether it works with the *new* users.

Which it would apparently NOT do for everybody if we put the new drivers
in any other place than /lib/firmware, very much including a kernel-
specific subdirectory there.

Linus

2008-07-15 20:33:40

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

Marcel Holtmann wrote:
> I really don't see that problem here since request_firmware() has been
> around for so long now and so many drivers are using it nowadays that I
> don't see your point here on how it will break.

There is no fundamental breakage in the new request_firmware() system.

However, it is misleading to state

what I'm doing here is nothing new. Most modern
drivers use request_firmware() already.

because we are not talking about new drivers, we are talking about
existing drivers changing en masse from "not requiring firmware" state
to "requiring firmware" state.

We are also talking about a totally new source of critically important
data (the firmware, coming from the kernel tree), and scripts must be
updated to take that into account.

Sure, in many cases the fix may even be a one-liner... but you still
need that fix, in order to continue to use drivers that were working in
2.6.26.

Jeff

2008-07-15 20:44:24

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tuesday, 15 of July 2008, David Woodhouse wrote:
> On Mon, 2008-07-14 at 16:41 -0700, Andrew Morton wrote:
> > On Mon, 14 Jul 2008 16:23:26 -0700
> > David Woodhouse <[email protected]> wrote:
> >
> > > Linus, please pull from the for-2.6.27 branch of:
> > > git://git.infradead.org/users/dwmw2/firmware-2.6.git for-2.6.27
> >
> > The firmware flamewars seem to have subsided lately. Is everyone happy
> > with this now?
>
> I've deliberately left drivers/net out of this pull request... all
> _other_ maintainers seemed quite happy to see their drivers updated to
> use request_firmware().

Have you fixed the issue with FORCE in line 117 of scripts/Makefile.host?

Rafael

2008-07-15 20:49:18

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, Jul 15, 2008 at 04:26:41PM -0400, Jeff Garzik wrote:
> To be extremely concrete, firmware-in-module is
>
> * add Kconfig option (kernel-wide or per-driver, dunno) asking
> "build firmware into drivers, as before?"
>
> * tweak build process to build firmware into foo.ko output,
> probably in a specially marked ELF section
>
> * get request_firmware() to automatically notice that the
> MODULE_FIRMWARE() was built into this driver, and to
> look at the special ELF section for its data

Jeff, just thinking, wouldn't it be slightly easier to move the firmware
in a separate module on its own and just add a dependency, so that foo.ko
automatically loads foo-fw.ko ? I know it will be slightly differente, but
would not change in-site deployment workflows nor installed scripts.

Regards,
Willy

2008-07-15 20:49:51

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 2008-07-15 at 22:45 +0200, Rafael J. Wysocki wrote:
> Have you fixed the issue with FORCE in line 117 of
> scripts/Makefile.host?

I need to hassle Sam about that.

Sam, why do we have FORCE on the rules for $(hostprogs-y)? It forces the
ihex2fw tool to be rebuilt, which in turn forces all .HEX->.fw
conversions to happen again, which means that 'make modules_install'
wants to write to the source tree. That's a problem in some strange
situations like Rafael's, where he builds on another host and exports
the tree read-only by NFS.

Can we do this? ...

diff --git a/scripts/Makefile.host b/scripts/Makefile.host
index 1ac414f..c0b6a86 100644
--- a/scripts/Makefile.host
+++ b/scripts/Makefile.host
@@ -114,7 +114,7 @@ hostcxx_flags = -Wp,-MD,$(depfile) $(__hostcxx_flags)
quiet_cmd_host-csingle = HOSTCC $@
cmd_host-csingle = $(HOSTCC) $(hostc_flags) -o $@ $< \
$(HOST_LOADLIBES) $(HOSTLOADLIBES_$(@F))
-$(host-csingle): $(obj)/%: $(src)/%.c FORCE
+$(host-csingle): $(obj)/%: $(src)/%.c
$(call if_changed_dep,host-csingle)

# Link an executable based on list of .o files, all plain c

--
dwmw2

2008-07-15 20:56:05

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

Linus Torvalds wrote:
>
> On Tue, 15 Jul 2008, Jeff Garzik wrote:
>> Already posted a list.
>
> No you didn't.
>
> You posted a list of problems you have from doing odd and stupid things.
> Not the reports, not the explanations of what those odd and stupid
> things actually were in practice, and why they had to be that stupid.
>
> Quite frankly, I suspect that it's all a matter of "people compiled their
> own kernels, and didn't copy the new files, because they just didn't
> realize they needed to".

No, I spelled out a list of projects, what would break, and how it would
break.

Again,

* Red Hat driver disks. The build process is unaware of the need to
bundle firmware. Any use with Fedora (or, later, RHEL) will result in a
non-working driver, for the simple reason that firmware is not copied
onto the image.

* Embedded projects like OpenWRT, that divide up kernel modules into
multiple packages. Since it's not just one big package, you have to
update each package's list of files to include the newly split-out
firmwares. Failure to do so results in a successful build set of
packages... of drivers that lack their firmware.

* Router and embedded system image builders like tomsrtbt, with similar
problem to RH driver disks: they simply do not know that a firmware
needs to be copied into the image, alongside the tg3 driver.

* Older userlands where, axiomatically, the mkinitrd and other firmware
fixes have not been made, and may never be made.

Yes, it is a matter of education.

Yes, it is a matter of fixes.

That does not change the fact that 2.6.27 will mean non-working drivers
for these situations, and there is a simple remedy for this entire class
of problems: retain ability to compile firmware into the module.

That does not change the fact that useful testers may be turned away
from 2.6.2[78] while waiting for Red Hat or OpenWRT or whomever to fix
their stuff.

Jeff

2008-07-15 20:57:53

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 2008-07-15 at 13:49 -0700, David Woodhouse wrote:
> Sam, why do we have FORCE on the rules for $(hostprogs-y)? It forces the
> ihex2fw tool to be rebuilt, which in turn forces all .HEX->.fw
> conversions to happen again, which means that 'make modules_install'
> wants to write to the source tree. That's a problem in some strange
> situations like Rafael's, where he builds on another host and exports
> the tree read-only by NFS.
>
> Can we do this? ...

Alternatively, Rafael, does this do it for you?

--- a/firmware/Makefile
+++ b/firmware/Makefile
@@ -150,11 +150,11 @@ $(obj)/%: $(obj)/%.ihex | $(objtree)/$(obj)/$$(dir %)
# is actually meaningful, because the firmware has to be loaded in a certain
# order rather than as a single binary blob. Thus, we convert them into our
# more compact binary representation of ihex records (<linux/ihex.h>)
-$(obj)/%.fw: $(obj)/%.HEX $(obj)/ihex2fw | $(objtree)/$(obj)/$$(dir %)
+$(obj)/%.fw: $(obj)/%.HEX | $(obj)/ihex2fw $(objtree)/$(obj)/$$(dir %)
$(call cmd,ihex2fw)

# .H16 is our own modified form of Intel HEX, with 16-bit length for records.
-$(obj)/%.fw: $(obj)/%.H16 $(obj)/ihex2fw | $(objtree)/$(obj)/$$(dir %)
+$(obj)/%.fw: $(obj)/%.H16 | $(obj)/ihex2fw $(objtree)/$(obj)/$$(dir %)
$(call cmd,h16tofw)

$(firmware-dirs):

--
dwmw2

2008-07-15 21:17:51

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

Linus Torvalds wrote:
>
> On Tue, 15 Jul 2008, Jeff Garzik wrote:
>> Already started, in fact, since Linus said he would not reject it out of
>> hands.
>
> Btw, I need to see it actually solve a real problem. As it is, I really
> think you've just painted yourself into a corner where you can't just
> admit that we can just load the firmware directly.

Kernel 2.6.26. Older userland w/ initrd requirement, plus module w/
compiled-in firmware.

Simply dropping 2.6.27 into that situation will produce an unbootable
system, since the firmware would not get copied into the initrd.

That's not just a kernel hacker situation. Plenty of sites, even a few
large Red Hat deployments, run newer kernels on a known-stable, older
userland. RH does not actively support swap-in-your-own-kernel,
preferring our own tested kernel, but new-kernel/older-userland is not
uncommon.

With upstream as it is today, there are just too many little ways the
firmware can fail to follow the driver it once lived inside. And the
consequences of failure are delayed (until next boot), silent (until
next boot), and notable (may produce non-booting system, certainly a
non-working driver).

Jeff

2008-07-15 21:19:23

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

Marcel Holtmann wrote:
> And as David mentioned multiple times, he left drivers/{net,scsi} out of
> it for now.

True, but we obviously must consider them -- and booting-related
implications -- since that is the obvious direction everybody is headed
towards.

Jeff

2008-07-15 21:24:46

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

Willy Tarreau wrote:
> On Tue, Jul 15, 2008 at 04:26:41PM -0400, Jeff Garzik wrote:
>> To be extremely concrete, firmware-in-module is
>>
>> * add Kconfig option (kernel-wide or per-driver, dunno) asking
>> "build firmware into drivers, as before?"
>>
>> * tweak build process to build firmware into foo.ko output,
>> probably in a specially marked ELF section
>>
>> * get request_firmware() to automatically notice that the
>> MODULE_FIRMWARE() was built into this driver, and to
>> look at the special ELF section for its data
>
> Jeff, just thinking, wouldn't it be slightly easier to move the firmware
> in a separate module on its own and just add a dependency, so that foo.ko
> automatically loads foo-fw.ko ? I know it will be slightly differente, but
> would not change in-site deployment workflows nor installed scripts.

Quite true. That's definitely an option, but I feel that building the
firmware into the driver module itself would be about the same level of
difficulty, but carry with it additional benefits:

Users and distros can be _certain_ their driver setup will not break due
to these changes, if you have Kconfig options available to reproduce
exactly what 2.6.26 produced [firmware compiled into the driver itself].

If Kconfig options are set such that the outputs are the same in 2.6.26
and 2.6.27 -- vmlinuz and kernel modules -- then that should close
windows of regression both real and theoretical, by producing _exactly_
the same outputs.

Jeff

2008-07-15 21:25:01

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 15 Jul 2008, David Woodhouse wrote:

> On Tue, 2008-07-15 at 22:00 +0200, Marcel Holtmann wrote:
>> I really think we should use /lib/firmware/`uname -r`/. I do see the
>> point here that I don't wanna overwrite existing firmware from other
>> installed kernels. Especially if modules_install will install the
>> firmware files.
>>
>> So in case of non-Ubuntu distros we have to push a fix to udev, but that
>> is not a big deal. It should be a one-line change if I am not mistaken.
>
> My concern with that is that even though it's a one-line change, it's a
> one-line change which people don't already _have_. We really do need to
> remain compatible with existing setups.

when I read this I had to double check who the mail was from. this sounds
like exactly the statement that Jeff has been making in his arguments to
allow for the firmware-in-the-modules approach.

David Lang

2008-07-15 21:25:31

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 2008-07-15 at 17:14 -0400, Jeff Garzik wrote:
> Linus Torvalds wrote:
> >
> > On Tue, 15 Jul 2008, Jeff Garzik wrote:
> >> Already started, in fact, since Linus said he would not reject it out of
> >> hands.
> >
> > Btw, I need to see it actually solve a real problem. As it is, I really
> > think you've just painted yourself into a corner where you can't just
> > admit that we can just load the firmware directly.
>
> Kernel 2.6.26. Older userland w/ initrd requirement, plus module w/
> compiled-in firmware.
>
> Simply dropping 2.6.27 into that situation will produce an unbootable
> system, since the firmware would not get copied into the initrd.

Kernel 2.6.15 with ancient userland. You update to 2.6.16, which
includes commit 5433383e ('convert qla2xxx to request_firmware()').

Same problem? Only three years ago. And not really a problem back then
either, because _even_ then you had to have a fairly ancient userland
for it not to work.

You're far too late to start whining about it now, Jeff. We're just
cleaning up the last drivers now. The rest have been where we want them
to be for _years_.

--
dwmw2

2008-07-15 21:26:25

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 15 Jul 2008, Jeff Garzik wrote:
>
> Kernel 2.6.26. Older userland w/ initrd requirement, plus module w/
> compiled-in firmware.
>
> Simply dropping 2.6.27 into that situation will produce an unbootable system,
> since the firmware would not get copied into the initrd.

Umm. Neither would the module.

So I assume you are doing the module load from the disk, not the initrd?

Or are you rebuilding the module into the initrd? If so, why didn't you do
the firmware too and my "load directly from file" idea which is pretty
simple? IOW, what's the fundamnetal problem here?

But hey, if you _do_ the "link it together", go ahead. I don't see why
it's so magically superior, but I wanted it for the compiled-in driver
case, so it's not fundamentally different from that either. And I can't
really find the energy to care any more.

Linus

2008-07-15 21:27:47

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 15 Jul 2008, Linus Torvalds wrote:

> On Tue, 15 Jul 2008, Marcel Holtmann wrote:
>>
>> you don't have to. We extend udev once and then it will always work.
>
> Umm. The thing is, people running new kernels with old user land is not
> just supposed to work, it's _really_ supposed to work.
>
> It's what I do. Something that breaks that has to have damn good reasons
> to break it.
>
> So I do not disagree with Jeff on that point _at_all_. I'm in violent
> agreement with Jeff on the fact that we should not require system updates
> for the kernel to do the right thing.
>
> The thing I disagree with Jeff on is that he then seems to turn that into
> something very negative ("let's not separate the firmware at all").

he's not saying not to seperate the firmware at all, he's asking for the
_option_ of compiling a kernel that doesn't have the firmware seperate
from the modules.

a kernel compiled with this option would just drop-in to an older distro
with zero impact. newer distros that have updated their userspace tools
could compile with different options and have the firmware seperate.

David Lang

2008-07-15 21:40:30

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

Linus Torvalds wrote:
>
> On Tue, 15 Jul 2008, Jeff Garzik wrote:
>> Kernel 2.6.26. Older userland w/ initrd requirement, plus module w/
>> compiled-in firmware.
>>
>> Simply dropping 2.6.27 into that situation will produce an unbootable system,
>> since the firmware would not get copied into the initrd.
>
> Umm. Neither would the module.

Sure it would.

Normal kernel build with modules, older userland (with older initrd).

'make modules_install && make install' will
* install modules in /lib
* install firmware in /lib [2.6.27 only]
* install kernel
* update grub
* rebuild initrd

Do this, under both 2.6.26 and 2.6.27.

2.6.26 will put a working driver into initrd.

2.6.27 will put a dead driver into initrd (because no firmware got
copied into initrd image).

> Or are you rebuilding the module into the initrd? If so, why didn't you do
> the firmware too and my "load directly from file" idea which is pretty
> simple? IOW, what's the fundamnetal problem here?

The firmware doesn't get copied into the initrd, because it is an
automated process (older mkinitrd) that is unaware of the need to copy
the firmware into the initrd.

Do the same build process on 2.6.26 and 2.6.27, and you WILL get
different results on older userlands, for the same simple reason: the
scripts do not know to copy the firmware when they copy the kernel module.

> But hey, if you _do_ the "link it together", go ahead. I don't see why

Great! I will :)

> it's so magically superior, but I wanted it for the compiled-in driver
> case, so it's not fundamentally different from that either.

It's not magically superior, just a necessary tool that helps avoid a
whole class of regressions, keeps us working on older userland, etc., etc.

Jeff

2008-07-15 21:41:30

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[email protected] wrote:
> he's not saying not to seperate the firmware at all, he's asking for the
> _option_ of compiling a kernel that doesn't have the firmware seperate
> from the modules.
>
> a kernel compiled with this option would just drop-in to an older distro
> with zero impact. newer distros that have updated their userspace tools
> could compile with different options and have the firmware seperate.

Correct.

Thanks :)

Jeff

2008-07-15 21:45:32

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 15 Jul 2008, [email protected] wrote:
>
> a kernel compiled with this option would just drop-in to an older distro with
> zero impact. newer distros that have updated their userspace tools could
> compile with different options and have the firmware seperate.

The 'zero impact' is what doesn't make sense here.

You are supposed to be able to run ol distributions, yes.

But that doesn't mean that you can necessarily just plop things in the
same way as you always did before.

For example, you have to rewrite your distro's initrd if you are using
modules. You cannot just re-use the modules in the distro initrd. So doing
a new kernel has _never_ been 'zero impact' in the sense that you could
just switch vmlinux files around.

(Btw, I personally actually want my kernel to be _truly_ zero impact, but
that also means that I don't use modules - because that way I really can
avoid changing even the initrd image too. But that also already works)

Why is it suddenly so important that a kernel be 'zero impact' for that
module case, when it's never been zero impact for that case before? You
had to rewrite the initrd to begin with, but now you're not willing to do
it again, just because you have to rewrite it slightly _differently_?

THAT is what I find so odd. The inability to accept just a slight change
in kernel build.

But whatever. This really isn't worth it. The request_firmware() thing
will clearly happen regardless, and as long as the backwards compat code
is small and Jeff writes it, what do I care? Even if I think it looks
largely pointless..

Linus

2008-07-15 21:47:28

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

From: David Woodhouse <[email protected]>
Date: Tue, 15 Jul 2008 14:24:21 -0700

> Kernel 2.6.15 with ancient userland. You update to 2.6.16, which
> includes commit 5433383e ('convert qla2xxx to request_firmware()').
>
> Same problem?

For the record I did get burnt by that one and I did make a huge stink
about it.

I used to be able to avoid initrd entirely on every single one of my
boxes, even my machines with qla2xxx based root filesystems, and that
changeset basically meant the end of that.

I simply don't do much hacking on that particular machine any more,
it's not worth the pain just to get a kernel that boots on it.

Yes, I know I can do stupid things like use the dist's initrd image
and just ignore the module version warnings, which is what Linus
happens to do, but that's just too gross for my tastes.

2008-07-15 22:13:21

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 2008-07-15 at 17:39 -0400, Jeff Garzik wrote:
>
> 'make modules_install && make install' will
> * install modules in /lib
> * install firmware in /lib [2.6.27 only]
> * install kernel
> * update grub
> * rebuild initrd
>
> Do this, under both 2.6.26 and 2.6.27.
>
> 2.6.26 will put a working driver into initrd.
>
> 2.6.27 will put a dead driver into initrd (because no firmware got
> copied into initrd image).

I believe your claim is untrue. It uses the distribution's tool to
create the initrd, doesn't it? And those tools _do_ handle the
MODULE_FIRMWARE() tags and pull in appropriate firmware. They've had to
get that right for _years_ already.

--
dwmw2

2008-07-15 22:19:55

by Adrian Bunk

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, Jul 15, 2008 at 09:44:45PM +0200, Marcel Holtmann wrote:
> Hi Linus,
>
> > > Personally I am against hacking file loading from within the kernel.
> > > This is why we involved udev (or hotplug in the beginning). It allows
> > > you to have userspace policy for the search paths etc.
> >
> > Well, I'm personally against _forcing_ people to use udev.
> >
> > Yes, desktop distros generally will do so, since there are lots of complex
> > issues etc. But I don't think we've generally forced it on people if they
> > don't want it (ie I haven't tried it personally since the distro I used
> > started using udev, but I think you can still just ignore udev events and
> > set everything up statically).
> >
> > And I don't think that's wrong. Which implies that setup things should
> > still generally at least allow us to avoid udev.
> >
> > (But maybe I'm wrong, and everybody already uses udev just because they
> > couldn't be bothered not to).
>
> I actually think that is the case here. udev is so lightweight and
> really slim nowadays that not using it is a big drawback. There might be
> some system out there that are still using a static /dev directory, but
> for sure there are not many and even the embedded world doesn't really
> mind.
>...

Depends on which embedded systems you look at.

For MID devices that's definitely true, but there are also today still
cases where customers want fast startup times on < 200 MHz embedded
devices, and when every second counts you don't want to add a
"Waiting for /dev to be fully populated" from udev to the bootup.

(And on my computer (whose Linux installation predates the inclusion of
devfsd (sic) into the kernel) the static /dev also still works fine.)

> Regards
>
> Marcel

cu
Adrian

--

"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed

2008-07-15 22:21:00

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tuesday 15 July 2008, you wrote:
> Why is it suddenly so important that a kernel be 'zero impact' for that
> module case, when it's never been zero impact for that case before? You
> had to rewrite the initrd to begin with, but now you're not willing to
> do it again, just because you have to rewrite it slightly
> _differently_?

Because in most cases users do not write and maintain the scripts used to
build custom kernels themselves. They use (huge and way too) tools like
Debian's 'make-kpkg' for that.

Because users may want to build a custom 2.6.27 in for example a Debian
Etch environment using the make-kpkg package that is available in that
release of Debian. Or even a Debian Lenny environment (Lenny is about to
be frozen so is very unlikely to include whatever may be needed for
this).

Why is it good to extend the lifetime of the i386 and amd64 symlinks for
x86 images by an extra few years so that older versions of kernel
packaging tools - like make-kpkg - don't break, but is breaking the exact
same tools with this firmware change suddenly OK?

Users _want_ to be able to build new kernels in older environments. Why
take that option away from them? Why force them through the pain of
partial upgrades of their userspace if it can so easily be avoided?

>From my PoV adding this option is a no-brainer: if Jeff is willing to do
the work, then what harm can it do? The benefits to me are obvious.
I really don't get what you have against it.

Cheers,
FJP

P.S. Sorry to keep coming up with Debian examples, but that's the
environment I know.

2008-07-15 22:25:23

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 15 Jul 2008, Linus Torvalds wrote:

> On Tue, 15 Jul 2008, [email protected] wrote:
>>
>> a kernel compiled with this option would just drop-in to an older distro with
>> zero impact. newer distros that have updated their userspace tools could
>> compile with different options and have the firmware seperate.
>
> The 'zero impact' is what doesn't make sense here.
>
> You are supposed to be able to run ol distributions, yes.
>
> But that doesn't mean that you can necessarily just plop things in the
> same way as you always did before.
>
> For example, you have to rewrite your distro's initrd if you are using
> modules. You cannot just re-use the modules in the distro initrd. So doing
> a new kernel has _never_ been 'zero impact' in the sense that you could
> just switch vmlinux files around.

this time I will call 'strawman'. we've been required to put in the
appropriate modules ever since modules were introduced. as Jeff gave
examples of the userspace tools have been taking care of that for many
years (and on distros where they don't take care of that you go with
monolithic kernels), but they don't take care of firmware in many cases.

> (Btw, I personally actually want my kernel to be _truly_ zero impact, but
> that also means that I don't use modules - because that way I really can
> avoid changing even the initrd image too. But that also already works)

I do the same thing 99% of the time.

> Why is it suddenly so important that a kernel be 'zero impact' for that
> module case, when it's never been zero impact for that case before? You
> had to rewrite the initrd to begin with, but now you're not willing to do
> it again, just because you have to rewrite it slightly _differently_?

becouse the tools that wrote the initrd already put the modules in. I
don't maintain those tools, they came with the distro. we're just asking
to not require those tools to be updated immediatly.

David Lang

> THAT is what I find so odd. The inability to accept just a slight change
> in kernel build.
>
> But whatever. This really isn't worth it. The request_firmware() thing
> will clearly happen regardless, and as long as the backwards compat code
> is small and Jeff writes it, what do I care? Even if I think it looks
> largely pointless..
>
> Linus
>
>
>

2008-07-15 22:27:18

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 2008-07-15 at 15:24 -0700, [email protected] wrote:
>
> > Why is it suddenly so important that a kernel be 'zero impact' for that
> > module case, when it's never been zero impact for that case before? You
> > had to rewrite the initrd to begin with, but now you're not willing to do
> > it again, just because you have to rewrite it slightly _differently_?
>
> becouse the tools that wrote the initrd already put the modules in. I
> don't maintain those tools, they came with the distro. we're just asking
> to not require those tools to be updated immediatly.

The tools that write the initrd already put the firmware in too.

--
dwmw2

2008-07-15 22:43:44

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 2008-07-15 at 14:47 -0700, David Miller wrote:
> From: David Woodhouse <[email protected]>
> Date: Tue, 15 Jul 2008 14:24:21 -0700
>
> > Kernel 2.6.15 with ancient userland. You update to 2.6.16, which
> > includes commit 5433383e ('convert qla2xxx to request_firmware()').
> >
> > Same problem?
>
> For the record I did get burnt by that one and I did make a huge stink
> about it.
>
> I used to be able to avoid initrd entirely on every single one of my
> boxes, even my machines with qla2xxx based root filesystems, and that
> changeset basically meant the end of that.

Until now. The only significant _generic_ change that just went into
Linus' kernel is the fact that you can now build that firmware _into_
your kernel, and you can boot qla2xxx without an initrd again.

HTH.

--
dwmw2

2008-07-15 22:45:24

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

From: David Woodhouse <[email protected]>
Date: Tue, 15 Jul 2008 15:42:32 -0700

> On Tue, 2008-07-15 at 14:47 -0700, David Miller wrote:
> > From: David Woodhouse <[email protected]>
> > Date: Tue, 15 Jul 2008 14:24:21 -0700
> >
> > > Kernel 2.6.15 with ancient userland. You update to 2.6.16, which
> > > includes commit 5433383e ('convert qla2xxx to request_firmware()').
> > >
> > > Same problem?
> >
> > For the record I did get burnt by that one and I did make a huge stink
> > about it.
> >
> > I used to be able to avoid initrd entirely on every single one of my
> > boxes, even my machines with qla2xxx based root filesystems, and that
> > changeset basically meant the end of that.
>
> Until now. The only significant _generic_ change that just went into
> Linus' kernel is the fact that you can now build that firmware _into_
> your kernel, and you can boot qla2xxx without an initrd again.

That's great.

Even better would be if the qla2xxx firmware was the tree like it used
to be too. Then it truly would be fully back to how things used to be.

2008-07-15 22:47:30

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

David Woodhouse wrote:
> On Tue, 2008-07-15 at 17:39 -0400, Jeff Garzik wrote:
>> 'make modules_install && make install' will
>> * install modules in /lib
>> * install firmware in /lib [2.6.27 only]
>> * install kernel
>> * update grub
>> * rebuild initrd
>>
>> Do this, under both 2.6.26 and 2.6.27.
>>
>> 2.6.26 will put a working driver into initrd.
>>
>> 2.6.27 will put a dead driver into initrd (because no firmware got
>> copied into initrd image).
>
> I believe your claim is untrue. It uses the distribution's tool to
> create the initrd, doesn't it? And those tools _do_ handle the
> MODULE_FIRMWARE() tags and pull in appropriate firmware.

slackware's CURRENT mkinitrd and kernel-modules scripts appear to
disprove your claims. And yes, they use a modern kernel, 2.6.24.

> They've had to
> get that right for _years_ already.

You should not just /assume/ that everything is ok, simply because time
passed since request_firmware() was added. Or let impatience blind you
to the breakage that will result from your change.

Further, there are several factors that clearly differentiate a driver
that has been using request_firmware() for a while, and newly converting
a large block of drivers en masse.

Jeff

2008-07-15 22:51:19

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 15 Jul 2008, David Woodhouse wrote:

> On Tue, 2008-07-15 at 15:24 -0700, [email protected] wrote:
>>
>>> Why is it suddenly so important that a kernel be 'zero impact' for that
>>> module case, when it's never been zero impact for that case before? You
>>> had to rewrite the initrd to begin with, but now you're not willing to do
>>> it again, just because you have to rewrite it slightly _differently_?
>>
>> becouse the tools that wrote the initrd already put the modules in. I
>> don't maintain those tools, they came with the distro. we're just asking
>> to not require those tools to be updated immediatly.
>
> The tools that write the initrd already put the firmware in too.

not always, as Jeff showed.

David Lang

2008-07-15 23:05:42

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

David Woodhouse wrote:
> Until now. The only significant _generic_ change that just went into
> Linus' kernel is the fact that you can now build that firmware _into_
> your kernel, and you can boot qla2xxx without an initrd again.

Yes, that is quite nice...

It will be nice to be able to do the same with modules too :)

Jeff

2008-07-15 23:09:20

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, Jul 15, 2008 at 04:55:45PM -0400, Jeff Garzik wrote:

> No, I spelled out a list of projects, what would break, and how it would
> break.
>
> Again,
>
> * Red Hat driver disks. The build process is unaware of the need to
> bundle firmware. Any use with Fedora (or, later, RHEL) will result in a
> non-working driver, for the simple reason that firmware is not copied
> onto the image.

For old RHEL - they won't support 2.6.27, so moot.
For next-gen RHEL - they need fixing anyway.
For Fedora, they don't (and hopefully won't) exist.

Dave

--
http://www.codemonkey.org.uk

2008-07-15 23:12:14

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, Jul 15, 2008 at 12:57:46PM -0700, David Woodhouse wrote:
> On Tue, 2008-07-15 at 12:49 -0700, Linus Torvalds wrote:
> > So kernel-specific directories do make sense. As does the whole "I
> > don't want to handle the pain that is udev scripts".
>
> I'm not violently opposed to kernel-specific directories. Ubuntu does it
> that way already, and it may well make sense. I'm not sure it's really
> necessary either, but it's an option.

Some of the wireless drivers have an issue where firmware x works with
driver y, so it's useful for that case even if nothing else uses it.

Dave

--
http://www.codemonkey.org.uk

2008-07-15 23:34:43

by Alexandre Oliva

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Jul 14, 2008, Linus Torvalds <[email protected]> wrote:

> [...] I like just making everybody do request_firmware().

> Do you call that a legal reason? Or a technical one?

\o.

Hmm, let's se...

How about legal?

If they don't budge in, you can terminate their license to Linux,
because they are distributing your GPLed work combined with software
that is not under a GPL-compatible license.

:-P :-D

--
Alexandre Oliva http://www.lsd.ic.unicamp.br/~oliva/
Free Software Evangelist oliva@{lsd.ic.unicamp.br, gnu.org}
FSFLA Board Member ¡Sé Libre! => http://www.fsfla.org/
Red Hat Compiler Engineer aoliva@{redhat.com, gcc.gnu.org}

2008-07-15 23:43:57

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tuesday, 15 of July 2008, David Woodhouse wrote:
> On Tue, 2008-07-15 at 13:49 -0700, David Woodhouse wrote:
> > Sam, why do we have FORCE on the rules for $(hostprogs-y)? It forces the
> > ihex2fw tool to be rebuilt, which in turn forces all .HEX->.fw
> > conversions to happen again, which means that 'make modules_install'
> > wants to write to the source tree. That's a problem in some strange
> > situations like Rafael's, where he builds on another host and exports
> > the tree read-only by NFS.
> >
> > Can we do this? ...
>
> Alternatively, Rafael, does this do it for you?

No, it doesn't.

> --- a/firmware/Makefile
> +++ b/firmware/Makefile
> @@ -150,11 +150,11 @@ $(obj)/%: $(obj)/%.ihex | $(objtree)/$(obj)/$$(dir %)
> # is actually meaningful, because the firmware has to be loaded in a certain
> # order rather than as a single binary blob. Thus, we convert them into our
> # more compact binary representation of ihex records (<linux/ihex.h>)
> -$(obj)/%.fw: $(obj)/%.HEX $(obj)/ihex2fw | $(objtree)/$(obj)/$$(dir %)
> +$(obj)/%.fw: $(obj)/%.HEX | $(obj)/ihex2fw $(objtree)/$(obj)/$$(dir %)
> $(call cmd,ihex2fw)
>
> # .H16 is our own modified form of Intel HEX, with 16-bit length for records.
> -$(obj)/%.fw: $(obj)/%.H16 $(obj)/ihex2fw | $(objtree)/$(obj)/$$(dir %)
> +$(obj)/%.fw: $(obj)/%.H16 | $(obj)/ihex2fw $(objtree)/$(obj)/$$(dir %)
> $(call cmd,h16tofw)
>
> $(firmware-dirs):
>

Rafael

2008-07-15 23:44:31

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 15 Jul 2008, [email protected] wrote:
>
> becouse the tools that wrote the initrd already put the modules in. I don't
> maintain those tools, they came with the distro. we're just asking to not
> require those tools to be updated immediatly.

But mkinitrd (which is the _only_ thing that people tend to use to write
initrd's - is there even anything else) has already been doing this for
years, as has been pointed out several times.

This is why I harped on the fact that you already rebuilt your initrd
image.

So this really isn't a "updated immediately" issue, afaik. Googling for

mkinitrd MODULE_FIRMWARE

shows a patch from two years ago as the #1 hit in order to make the
aic94xx driver work.

Loking down a bit, there's a hotplug discussion from early 2005 (gmane
says "3 years, 20 weeks, 3 days, 6 hours and 39 minutes ago" in the
header).

Quite frankly, if it's still a problem, there's simply something _wrong_
with the distribution. And yeah, maybe people need to update their kernel
building tools.

I already pointed out how the kernel development team quite often says "we
will no longer build with gcc-.xyz because it's too old and buggy". The
build tools requirements are simply *different* from the runtime tools.

If it's literally just an issue of an mkinitrd that is too damn old, why
don't we just make that test at kernel build time. *EXACTLY* the same way
we test for compilers that are too old, and refuse to build with them?

(And no, I have no idea which version we should test for. In fact,
mkinitrd seems to be singularly hard to test versions for, in that it
seems to want the user to be root even just to give the version output.
Ooh).

Linus

2008-07-15 23:47:17

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

Dave Jones wrote:
> On Tue, Jul 15, 2008 at 04:55:45PM -0400, Jeff Garzik wrote:
>
> > No, I spelled out a list of projects, what would break, and how it would
> > break.
> >
> > Again,
> >
> > * Red Hat driver disks. The build process is unaware of the need to
> > bundle firmware. Any use with Fedora (or, later, RHEL) will result in a
> > non-working driver, for the simple reason that firmware is not copied
> > onto the image.
>
> For old RHEL - they won't support 2.6.27, so moot.
> For next-gen RHEL - they need fixing anyway.
> For Fedora, they don't (and hopefully won't) exist.

Grin,
http://docs.fedoraproject.org/fedora-install-guide-en/fc5/sn-bootoptions-hardware.html

Or DKMS, which should package firmware:
http://linux.dell.com/projects.shtml

2008-07-16 00:09:00

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, Jul 15, 2008 at 07:46:33PM -0400, Jeff Garzik wrote:
> Dave Jones wrote:
> > On Tue, Jul 15, 2008 at 04:55:45PM -0400, Jeff Garzik wrote:
> >
> > > No, I spelled out a list of projects, what would break, and how it would
> > > break.
> > >
> > > Again,
> > >
> > > * Red Hat driver disks. The build process is unaware of the need to
> > > bundle firmware. Any use with Fedora (or, later, RHEL) will result in a
> > > non-working driver, for the simple reason that firmware is not copied
> > > onto the image.
> >
> > For old RHEL - they won't support 2.6.27, so moot.
> > For next-gen RHEL - they need fixing anyway.
> > For Fedora, they don't (and hopefully won't) exist.
>
> Grin,
> http://docs.fedoraproject.org/fedora-install-guide-en/fc5/sn-bootoptions-hardware.html

I don't think any of this stuff has ever been actually used in a Fedora context.
We certainly haven't released any official ones ttbomk.

> Or DKMS, which should package firmware:
> http://linux.dell.com/projects.shtml

Not quite the same thing as a driver disk, but I'll humour you..
That handles MODULE_FIRMWARE fine afaik. Though again, this doesn't see much
actual use in a Fedora context.

Dave

--
http://www.codemonkey.org.uk

2008-07-16 00:13:17

by Alexandre Oliva

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Jul 15, 2008, David Woodhouse <[email protected]> wrote:

> On Tue, 2008-07-15 at 07:06 +0200, Willy Tarreau wrote:
>> This is just a matter of point of view. From my POV, having to feed
>> several unrelated files into a system to be able to get it to support
>> any device is moving backwards, compared to the currently clean
>> all-in-1 driver model.

> So you disapprove of having to load mac80211.ko before your wireless NIC
> driver? Or sungem_phy.ko before sungem.ko? Or libphy.ko before various
> other drivers?

Hey, how about a config option that turns firwmare files into
stand-alone kernel modules that register the blob into the
request_firmware internal list, and MODULE_FIRMWARE into a module
dependency?

--
Alexandre Oliva http://www.lsd.ic.unicamp.br/~oliva/
Free Software Evangelist oliva@{lsd.ic.unicamp.br, gnu.org}
FSFLA Board Member ¡Sé Libre! => http://www.fsfla.org/
Red Hat Compiler Engineer aoliva@{redhat.com, gcc.gnu.org}

2008-07-16 00:21:17

by Chris Adams

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

Once upon a time, Jeff Garzik <[email protected]> said:
>* Red Hat driver disks. The build process is unaware of the need to
>bundle firmware. Any use with Fedora (or, later, RHEL) will result in a
>non-working driver, for the simple reason that firmware is not copied
>onto the image.

Then Red Hat driver disks are already broken, as there are current
modules (e.g. qla2xxx) that require external firmware.

>* Embedded projects like OpenWRT, that divide up kernel modules into
>multiple packages. Since it's not just one big package, you have to
>update each package's list of files to include the newly split-out
>firmwares. Failure to do so results in a successful build set of
>packages... of drivers that lack their firmware.

AFAIK OpenWRT and the like don't just throw in random new kernel
releases without at least reading release notes and testing.
--
Chris Adams <[email protected]>
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.

2008-07-16 00:23:10

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Wednesday 16 July 2008, Linus Torvalds wrote:
> On Tue, 15 Jul 2008, [email protected] wrote:
> > becouse the tools that wrote the initrd already put the modules in. I
> > don't maintain those tools, they came with the distro. we're just
> > asking to not require those tools to be updated immediatly.
>
> But mkinitrd (which is the _only_ thing that people tend to use to
> write initrd's - is there even anything else) has already been doing
> this for years, as has been pointed out several times.

Debian doe not have mkinitrd but initramfs-tools [1] and yaird [2]. The
status of the last is uncertain (not currently included for Lenny).

Where did you get this strange notion that mkinitrd is the _only_ (your
emphasis) thing people use?
Point is that I find it really strange that - in the extremely rich and
varied world that is based on Linux - you seem so certain that every tool
out there already does the right thing for all cases, or can be trivially
updated to do so. I just don't believe that.

I even _know_ that yaird is broken wrt. firmware [3]. It's one of the
reasons it's unlikely to be included in Lenny.

But hell, I'll see if I can give this a try tomorrow. Get out of this
crazy make-belief discussion and provide some facts. Who knows, maybe
I'll be surprised and the existing Debian toolsets will do the right
thing. I seriously doubt it though.

[1] http://packages.debian.org/sid/initramfs-tools
[2] http://packages.debian.org/sid/yaird
[3] Only works if config files are edited manually which makes it useless
during e.g. new installs.

2008-07-16 00:28:37

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 15 Jul 2008, Alexandre Oliva wrote:
>
> Hey, how about a config option that turns firwmare files into
> stand-alone kernel modules that register the blob into the
> request_firmware internal list, and MODULE_FIRMWARE into a module
> dependency?

I think that should work fine from a technical angle, and I don't disagree
per se, but it really is very ugly.

Wouldn't it be much simpler to just require people to have a reasonably
up-to-date mkinitrd. Apparently it's not available _everywhere_, but if we
check for this up-front (even if we can apparently only do so at "make
install" time due to the root requirements), wouldn't that really solve
the issue with much less work?

Linus

2008-07-16 00:29:26

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Wed, 16 Jul 2008, Frans Pop wrote:
>
> Where did you get this strange notion that mkinitrd is the _only_ (your
> emphasis) thing people use?

Heh. Maybe from the fact that I don't personally use modules or initrd at
all, and have no interest in doing so ;)

Ok, so people use other things. Grr. I guess we can't just require updated
tools, because we don't know what the tools are.

Linus

2008-07-16 00:43:27

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

Linus Torvalds wrote:
>
> On Wed, 16 Jul 2008, Frans Pop wrote:
>> Where did you get this strange notion that mkinitrd is the _only_ (your
>> emphasis) thing people use?
>
> Heh. Maybe from the fact that I don't personally use modules or initrd at
> all, and have no interest in doing so ;)
>
> Ok, so people use other things. Grr. I guess we can't just require updated
> tools, because we don't know what the tools are.

Yes, that's part of the problem :/

Everybody seems to love rolling their own stuff in this area, partly
necessitated I suppose by the wide variety of ways to boot on various
platforms.

Jeff

2008-07-16 00:52:17

by Theodore Ts'o

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, Jul 15, 2008 at 04:42:52PM -0700, Linus Torvalds wrote:
> On Tue, 15 Jul 2008, [email protected] wrote:
> >
> > becouse the tools that wrote the initrd already put the modules in. I don't
> > maintain those tools, they came with the distro. we're just asking to not
> > require those tools to be updated immediatly.
>
> But mkinitrd (which is the _only_ thing that people tend to use to write
> initrd's - is there even anything else) has already been doing this for
> years, as has been pointed out several times.

Actually, there is also mkinitramfs (which is what Debian and Ubuntu
use), and Yaird (Yet Another mkinitrd). Also, the distro's tend to
heavily customize mkinitrd in general, so there is no guarantee that
just because two distro's use some tool that happens to be named
mkinitrd that the two tools have much in common with each other....

This is highly unfortunate, and I've often argued that we probably
would have been better off if we had included an mkinitrd tool as part
of the kernel sources to make life easier instead of having every
single distro do their own thing; it would also make it easier to run
a newer kernel on an older enterprise distro --- I remember the pain
trying to make a 2.6.16 kernel boot on a RHEL4 base system two years
ago; I don't want to even think about what would happen with something
more modern at this point.

> If it's literally just an issue of an mkinitrd that is too damn old, why
> don't we just make that test at kernel build time. *EXACTLY* the same way
> we test for compilers that are too old, and refuse to build with them?

For one reason, because there's more than one mkinitrd. FC9 ships
with mkinitrd 6.0.52; OpenSuSE ships with mkinitrd 2.1, and the
sources don't look even vaguely similar to one another. For example,
Red Hat's mkinitrd uses nash, which is a maddingly useless shell that
made it near-impossible to debug why a 2.6.16 kernel wasn't booting on
a RHEL-4 system on a LS-21 blade. OpenSuSE's mkinitrd ships with
bash, and Debian/Ubuntu will give you a busybox shell, which makes
life *much* easier to debug when you're trying to figure out why your
latest bleeding edge kernel isn't able to mount the root partition out
of initrd.

- Ted

P.S. For bonus points, it would be nice if initrd's included fsck,
since it then it would be possible to safely fsck the root partition
and not require a reboot if the root partition was modified. But
given that every single distro seems to ship their own initrd, not to
mention their own init scripts, it's hard to try to push for this sort
of change. The amount of distro-specific engineering work probably
makes it not worth it.

2008-07-16 01:04:19

by Sven-Haegar Koch

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 15 Jul 2008, David Woodhouse wrote:

> On Tue, 2008-07-15 at 17:39 -0400, Jeff Garzik wrote:
> >
> > 'make modules_install && make install' will
> > * install modules in /lib
> > * install firmware in /lib [2.6.27 only]
> > * install kernel
> > * update grub
> > * rebuild initrd
> >
> > Do this, under both 2.6.26 and 2.6.27.
> >
> > 2.6.26 will put a working driver into initrd.
> >
> > 2.6.27 will put a dead driver into initrd (because no firmware got
> > copied into initrd image).
>
> I believe your claim is untrue. It uses the distribution's tool to
> create the initrd, doesn't it? And those tools _do_ handle the
> MODULE_FIRMWARE() tags and pull in appropriate firmware. They've had to
> get that right for _years_ already.

Something from the non-development world:

At least the initramfs-tools (which contains mkinitrd) of Debian Etch do
not do this (the current testing/development version Lenny does) - no
firmware support in the initrd at all - and not needed for booting until
now.
(It has /lib/firmware support after the initrd for the not-so-important
stuff like usb/sound/video/wireless)

Yes, it may be nearly ancient by now, but for servers it contains all we
need, and most certainly would not want to use something that changes and
needs checking/hand-fixing every month - I expect this version to be still
in use a year from now, then just gradually being updated to the next
debian version.

Currently we use 2.6.24, and I expect to use some newer kernel to support
newer replacement hardware before all installs are upgraded. So having the
firmware-inside-module support would be a help for us when modules are
starting to be needed for disk or network access, keeping from the need to
try backporting huge initrd/make-kpkg changes.

c'ya
sven

--

The Internet treats censorship as a routing problem, and routes around
it. (John Gilmore on http://www.cygnus.com/~gnu/)

2008-07-16 01:14:30

by Alexey Dobriyan

[permalink] [raw]

Subject: Prosaic firmware issues

Could somebody from firmware people fix boring stuff, please:

1) after allyesconfig build, git-status show plenty of crap:

# firmware/atmsar11.fw
# firmware/atmsar11.fw.gen.S
# firmware/cpia2/stv0672_vp4.bin.gen.S
# firmware/dabusb/bitstream.bin.gen.S
# firmware/dabusb/firmware.fw
# firmware/dabusb/firmware.fw.gen.S
# firmware/edgeport/boot.fw
# firmware/edgeport/boot.fw.gen.S
# firmware/edgeport/boot2.fw
# firmware/edgeport/boot2.fw.gen.S
# firmware/edgeport/down.fw
# firmware/edgeport/down.fw.gen.S
# firmware/edgeport/down2.fw
# firmware/edgeport/down2.fw.gen.S
# firmware/edgeport/down3.bin.gen.S
# firmware/emi26/bitstream.fw
# firmware/emi26/bitstream.fw.gen.S
# firmware/emi26/firmware.fw
# firmware/emi26/firmware.fw.gen.S
# firmware/emi26/loader.fw
# firmware/emi26/loader.fw.gen.S
# firmware/emi62/bitstream.fw
# firmware/emi62/bitstream.fw.gen.S
# firmware/emi62/loader.fw
# firmware/emi62/loader.fw.gen.S
# firmware/emi62/midi.fw
# firmware/emi62/midi.fw.gen.S
# firmware/emi62/spdif.fw
# firmware/emi62/spdif.fw.gen.S
# firmware/ess/maestro3_assp_kernel.fw
# firmware/ess/maestro3_assp_kernel.fw.gen.S
# firmware/ess/maestro3_assp_minisrc.fw
# firmware/ess/maestro3_assp_minisrc.fw.gen.S
# firmware/ihex2fw
# firmware/intelliport2.bin.gen.S
# firmware/kaweth/new_code.bin.gen.S
# firmware/kaweth/new_code_fix.bin.gen.S
# firmware/kaweth/trigger_code.bin.gen.S
# firmware/kaweth/trigger_code_fix.bin.gen.S
# firmware/keyspan/mpr.fw
# firmware/keyspan/mpr.fw.gen.S
# firmware/keyspan/usa18x.fw
# firmware/keyspan/usa18x.fw.gen.S
# firmware/keyspan/usa19.fw
# firmware/keyspan/usa19.fw.gen.S
# firmware/keyspan/usa19qi.fw
# firmware/keyspan/usa19qi.fw.gen.S
# firmware/keyspan/usa19qw.fw
# firmware/keyspan/usa19qw.fw.gen.S
# firmware/keyspan/usa19w.fw
# firmware/keyspan/usa19w.fw.gen.S
# firmware/keyspan/usa28.fw
# firmware/keyspan/usa28.fw.gen.S
# firmware/keyspan/usa28x.fw
# firmware/keyspan/usa28x.fw.gen.S
# firmware/keyspan/usa28xa.fw
# firmware/keyspan/usa28xa.fw.gen.S
# firmware/keyspan/usa28xb.fw
# firmware/keyspan/usa28xb.fw.gen.S
# firmware/keyspan/usa49w.fw
# firmware/keyspan/usa49w.fw.gen.S
# firmware/keyspan/usa49wlc.fw
# firmware/keyspan/usa49wlc.fw.gen.S
# firmware/keyspan_pda/keyspan_pda.fw
# firmware/keyspan_pda/keyspan_pda.fw.gen.S
# firmware/keyspan_pda/xircom_pgs.fw
# firmware/keyspan_pda/xircom_pgs.fw.gen.S
# firmware/korg/k1212.dsp
# firmware/korg/k1212.dsp.gen.S
# firmware/ti_3410.fw
# firmware/ti_3410.fw.gen.S
# firmware/ti_5052.fw
# firmware/ti_5052.fw.gen.S
# firmware/ttusb-budget/dspbootcode.bin.gen.S
# firmware/vicam/firmware.fw
# firmware/vicam/firmware.fw.gen.S
# firmware/whiteheat.fw
# firmware/whiteheat.fw.gen.S
# firmware/whiteheat_loader.fw
# firmware/whiteheat_loader.fw.gen.S
# firmware/yamaha/ds1_ctrl.fw
# firmware/yamaha/ds1_ctrl.fw.gen.S
# firmware/yamaha/ds1_dsp.fw
# firmware/yamaha/ds1_dsp.fw.gen.S
# firmware/yamaha/ds1e_ctrl.fw
# firmware/yamaha/ds1e_ctrl.fw.gen.S

2) after "make mrproper" after allyesconfig, some files aren't removed,
use "git-ls-files -o" to see them.

# firmware/atmsar11.fw
# firmware/dabusb/firmware.fw
# firmware/emi26/bitstream.fw
# firmware/emi26/firmware.fw
# firmware/emi26/loader.fw
# firmware/emi62/bitstream.fw
# firmware/emi62/loader.fw
# firmware/emi62/midi.fw
# firmware/emi62/spdif.fw
# firmware/ess/maestro3_assp_kernel.fw
# firmware/ess/maestro3_assp_minisrc.fw
# firmware/korg/k1212.dsp
# firmware/yamaha/ds1_ctrl.fw
# firmware/yamaha/ds1_dsp.fw
# firmware/yamaha/ds1e_ctrl.fw

2008-07-16 01:43:33

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, Jul 15, 2008 at 08:51:33PM -0400, Theodore Tso wrote:
> For one reason, because there's more than one mkinitrd. FC9 ships
> with mkinitrd 6.0.52; OpenSuSE ships with mkinitrd 2.1, and the
> sources don't look even vaguely similar to one another.

Right. Other than name, they've historically shared nothing.
Both have grown through the evolution of multiple distros, requiring
different workarounds in each due to differences in CONFIG_ options
in the kernel between vendors for eg.

Whilst it would be great for unified development on the tools that
create the early boot process, I think it's a non-starter due to the
fact that you can't really do it without throwing out everything you
already have today. The same reason imo, that hpa's klibc work hasn't
gained mass-appeal from vendors.

Even if we had a 'reference' mkinitrd in the kernel, it would be pretty much
useless until it reached feature parity with every distros existing tools,
and if everyone uses those instead of furthering the reference implementation,
it fails on the starting blocks.

Dave

--
http://www.codemonkey.org.uk

2008-07-16 01:46:55

[permalink] [raw]

Subject: Re: Prosaic firmware issues

On Wed, 16 Jul 2008, Alexey Dobriyan wrote:
>
> Could somebody from firmware people fix boring stuff, please:
>
> 1) after allyesconfig build, git-status show plenty of crap:

Something like the appended, perhaps?

(I'm not going to commit it, I want some firmware person to double-check
it).

> 2) after "make mrproper" after allyesconfig, some files aren't removed,
> use "git-ls-files -o" to see them.

Ok, that needs some Makefile magic. git magic I can do.

Linus

---
firmware/.gitignore | 2 ++
1 files changed, 2 insertions(+), 0 deletions(-)
create mode 100644 firmware/.gitignore

diff --git a/firmware/.gitignore b/firmware/.gitignore
new file mode 100644
index 0000000..9ade43a
--- /dev/null
+++ b/firmware/.gitignore
@@ -0,0 +1,2 @@
+*.gen.S
+*.fw

2008-07-16 02:39:55

by Theodore Ts'o

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, Jul 15, 2008 at 09:36:19PM -0400, Dave Jones wrote:
> Whilst it would be great for unified development on the tools that
> create the early boot process, I think it's a non-starter due to the
> fact that you can't really do it without throwing out everything you
> already have today. The same reason imo, that hpa's klibc work hasn't
> gained mass-appeal from vendors.

Yeah, it would have been much better if we had included a reference
mkinitrd from the beginning, instead of saying, "Kernel doesn't set
policy", and hoping the distro's would converge on something similar.
The reality though is to the extent that we want to make changes like
request_firmware() we are actually better off setting policy for stuff
which is this low-level.

> Even if we had a 'reference' mkinitrd in the kernel, it would be
> pretty much useless until it reached feature parity with every
> distros existing tools, and if everyone uses those instead of
> furthering the reference implementation, it fails on the starting
> blocks.

In order to work now it would need to have a relatively simple way for
distro's to hook in their own enhancements, and someone would have to
do most of the work of migrating at least the major distro's to use
the new reference "official" mkinitrd. It's something we should have
done from the very beginning, and it's a lot harder to fix up that
mistake years later.

- Ted

2008-07-16 03:08:44

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

Dave Jones wrote:
> Whilst it would be great for unified development on the tools that
> create the early boot process, I think it's a non-starter due to the
> fact that you can't really do it without throwing out everything you
> already have today. The same reason imo, that hpa's klibc work hasn't
> gained mass-appeal from vendors.

Quite true and IMO disappointing, because klibc would make great
infrastructure for rolling out things just like this.

Jeff

2008-07-16 04:43:32

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, Jul 15, 2008 at 08:43:07PM -0400, Jeff Garzik wrote:
> Linus Torvalds wrote:
> >
> >On Wed, 16 Jul 2008, Frans Pop wrote:
> >>Where did you get this strange notion that mkinitrd is the _only_ (your
> >>emphasis) thing people use?
> >
> >Heh. Maybe from the fact that I don't personally use modules or initrd at
> >all, and have no interest in doing so ;)
> >
> >Ok, so people use other things. Grr. I guess we can't just require updated
> >tools, because we don't know what the tools are.
>
> Yes, that's part of the problem :/
>
> Everybody seems to love rolling their own stuff in this area, partly
> necessitated I suppose by the wide variety of ways to boot on various
> platforms.

Exactly. To give you an idea, I have a lot of servers running on a
"firmware" which consists in two parts :
- a bzImage containing an initramfs with modules and a few scripts
- an initrd which is in fact the rootfs

When the kernel boots, it mounts the initrd, does a pivot_root and mounts
its modules into /boot/$(uname -r). All my kernels run modules from /boot
and not from /lib/modules, because it makes them more convenient to add
and remove. So /lib/modules is just a symlink to /boot.

The above process is very convenient, as it is compatible with a lot of
boot methods : hard disk, CDROM, USB stick, PXE, etc...

And moreover, I can have multiple kernels with only one rootfs (SMP, etc).
Introducing firmware there would mean a major thinking and rework of the
build (and possibly boot) process. The least invasive would probably be
to stuff them next to the modules in $(uname -r)/firmware, but even then,
it will take a lot of time to switch to a new system.

And judging from what I see around, I'm far from being the only one not
using mkinitrd.

Willy

2008-07-16 04:54:53

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On 15-07-08 21:14, Jeff Garzik wrote:

> Theodore Tso wrote:

>> If so, may I suggest you start implementing, instead of sending
>> e-mails? For all the time people have spent arguing about it,
>> maybe it could have been implemented already.
>
> Already started, in fact, since Linus said he would not reject it out
> of hands.

Great, as I really hope this gets in. Was looking but am very scared of
the linker goo...

I'm going to like the feature in itself once it has grown the ability
for firmware in the module and really do not right now. One of the
drivers I care about (old ISA SB16 CSP code) was also affected and my
old 486 system with the hardware in it is _still_ lauging its little
balls of at the people stating that "if you have userspace to load the
module you have userspace to support at the hotplug/udev goo to load the
firmware".

Given that one needs to be rather irrelevant to care about the SB16 CSP
in the first place it wasn't enough to complain really, but yes, I'm
really quite sure that I'm going to use the feature of building it into
the module more than once while running around with floppies trying to
support that old stuff.

Rene.

2008-07-16 05:54:41

[permalink] [raw]

Subject: Re: Prosaic firmware issues

On Tue, 2008-07-15 at 18:45 -0700, Linus Torvalds wrote:
>
> On Wed, 16 Jul 2008, Alexey Dobriyan wrote:
> >
> > Could somebody from firmware people fix boring stuff, please:
> >
> > 1) after allyesconfig build, git-status show plenty of crap:
>
> Something like the appended, perhaps?
>
> (I'm not going to commit it, I want some firmware person to double-check
> it).

Generally sane, but wants a few more things added.

> > 2) after "make mrproper" after allyesconfig, some files aren't removed,
> > use "git-ls-files -o" to see them.
>
> Ok, that needs some Makefile magic. git magic I can do.

Ah, that's caused by a stray := where there should be a +=, so the
firmware files listed before that get dropped from the list of files to
be cleaned.

Alexey, thanks for the report. Please could you verify that this fixes
it for you?

----
Subject: Fix 'make clean' and .gitignore for firmware/ directory.

Provide a .gitignore file, and fix a typo which prevented some of the
generated binary files from being removed on 'make clean'.

Signed-off-by: David Woodhouse <[email protected]>

diff --git a/firmware/.gitignore b/firmware/.gitignore
new file mode 100644
index 0000000..d9c6901
--- /dev/null
+++ b/firmware/.gitignore
@@ -0,0 +1,6 @@
+*.gen.S
+*.fw
+*.bin
+*.csp
+*.dsp
+ihex2fw
diff --git a/firmware/Makefile b/firmware/Makefile
index 981bd52..55ca321 100644
--- a/firmware/Makefile
+++ b/firmware/Makefile
@@ -73,7 +73,7 @@ fw-shipped-$(CONFIG_USB_SERIAL_KEYSPAN_USA28X) += keyspan/usa28x.fw
fw-shipped-$(CONFIG_USB_SERIAL_KEYSPAN_USA49W) += keyspan/usa49w.fw
fw-shipped-$(CONFIG_USB_SERIAL_KEYSPAN_USA49WLC) += keyspan/usa49wlc.fw
else
-fw-shipped- := keyspan/mpr.fw keyspan/usa18x.fw keyspan/usa19.fw \
+fw-shipped- += keyspan/mpr.fw keyspan/usa18x.fw keyspan/usa19.fw \
keyspan/usa19qi.fw keyspan/usa19qw.fw keyspan/usa19w.fw \
keyspan/usa28.fw keyspan/usa28xa.fw keyspan/usa28xb.fw \
keyspan/usa28x.fw keyspan/usa49w.fw keyspan/usa49wlc.fw

--
dwmw2

2008-07-16 06:04:20

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

Chris Adams wrote:
> Once upon a time, Jeff Garzik <[email protected]> said:
>> * Red Hat driver disks. The build process is unaware of the need to
>> bundle firmware. Any use with Fedora (or, later, RHEL) will result in a
>> non-working driver, for the simple reason that firmware is not copied
>> onto the image.
>
> Then Red Hat driver disks are already broken, as there are current
> modules (e.g. qla2xxx) that require external firmware.

That doesn't change the problem any. There are still users of this
thing, regardless of what appellations are attached.

>> * Embedded projects like OpenWRT, that divide up kernel modules into
>> multiple packages. Since it's not just one big package, you have to
>> update each package's list of files to include the newly split-out
>> firmwares. Failure to do so results in a successful build set of
>> packages... of drivers that lack their firmware.
>
> AFAIK OpenWRT and the like don't just throw in random new kernel
> releases without at least reading release notes and testing.

Sure. The difference here is that we usually take care to make sure
there is some way for the distributor to continue configuring their
system in the manner known to work, while developing and testing new stuff.

But in this case, that option was not available to distributors (but
hopefully it will be, in another day or two).

Jeff

2008-07-16 08:02:53

by Gabor Gombas

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Wed, Jul 16, 2008 at 02:55:32AM +0200, Sven-Haegar Koch wrote:

> Something from the non-development world:
>
> At least the initramfs-tools (which contains mkinitrd) of Debian Etch do
> not do this (the current testing/development version Lenny does) - no
> firmware support in the initrd at all - and not needed for booting until
> now.
> (It has /lib/firmware support after the initrd for the not-so-important
> stuff like usb/sound/video/wireless)
>
> Yes, it may be nearly ancient by now, but for servers it contains all we
> need, and most certainly would not want to use something that changes and
> needs checking/hand-fixing every month - I expect this version to be still
> in use a year from now, then just gradually being updated to the next
> debian version.
>
> Currently we use 2.6.24, and I expect to use some newer kernel to support
> newer replacement hardware before all installs are upgraded. So having the
> firmware-inside-module support would be a help for us when modules are
> starting to be needed for disk or network access, keeping from the need to
> try backporting huge initrd/make-kpkg changes.

You can easyly install initramfs-tools from lenny on an etch box. It
only depends on a newer klibc, so it won't disturb any other services.

Btw., the Debian-provided 2.6.25 kernel already lacks the firmware for
some popular modules (like bnx2). So if you're using Debian, you as a
sysadmin already have to handle the "firmware is in a separate package"
issue.

Gabor

--
---------------------------------------------------------
MTA SZTAKI Computer and Automation Research Institute
Hungarian Academy of Sciences
---------------------------------------------------------

2008-07-16 09:09:53

by Peter Zijlstra

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 2008-07-15 at 23:07 -0400, Jeff Garzik wrote:
> Dave Jones wrote:
> > Whilst it would be great for unified development on the tools that
> > create the early boot process, I think it's a non-starter due to the
> > fact that you can't really do it without throwing out everything you
> > already have today. The same reason imo, that hpa's klibc work hasn't
> > gained mass-appeal from vendors.
>
> Quite true and IMO disappointing, because klibc would make great
> infrastructure for rolling out things just like this.

Indeed, for one i've been told it would enable cross compiling of
initrds, without which I deem initrd useless junk - whoever still
compiles their kernel on the machine it will run on? :-)

2008-07-16 14:21:58

by Alexey Dobriyan

[permalink] [raw]

Subject: Re: Prosaic firmware issues

On Tue, Jul 15, 2008 at 10:54:04PM -0700, David Woodhouse wrote:
> On Tue, 2008-07-15 at 18:45 -0700, Linus Torvalds wrote:
> >
> > On Wed, 16 Jul 2008, Alexey Dobriyan wrote:
> > >
> > > Could somebody from firmware people fix boring stuff, please:
> > >
> > > 1) after allyesconfig build, git-status show plenty of crap:
> >
> > Something like the appended, perhaps?
> >
> > (I'm not going to commit it, I want some firmware person to double-check
> > it).
>
> Generally sane, but wants a few more things added.
>
> > > 2) after "make mrproper" after allyesconfig, some files aren't removed,
> > > use "git-ls-files -o" to see them.
> >
> > Ok, that needs some Makefile magic. git magic I can do.
>
> Ah, that's caused by a stray := where there should be a +=, so the
> firmware files listed before that get dropped from the list of files to
> be cleaned.
>
> Alexey, thanks for the report. Please could you verify that this fixes
> it for you?

It does, thank you.

2008-07-16 15:06:47

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Tue, 2008-07-15 at 13:49 -0700, David Woodhouse wrote:
> I need to hassle Sam about that.

I think I have it. Rafael, does _this_ fix it for you?

diff --git a/scripts/Makefile.fwinst b/scripts/Makefile.fwinst
index 3d2f460..994060d 100644
--- a/scripts/Makefile.fwinst
+++ b/scripts/Makefile.fwinst
@@ -43,3 +43,15 @@ __fw_install: $(installed-fw)
__fw_modinst: $(mod-fw)

FORCE:
+
+# Read all saved command lines and dependencies for the $(targets) we
+# may be building using $(if_changed{,_dep}). As an optimization, we
+# don't need to read them if the target does not exist; we will rebuild
+# anyway in that case.
+
+targets := $(wildcard $(sort $(targets)))
+cmd_files := $(wildcard $(foreach f,$(targets),$(dir $(f)).$(notdir $(f)).cmd))
+
+ifneq ($(cmd_files),)
+ include $(cmd_files)
+endif

--
dwmw2

2008-07-16 21:26:37

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Wednesday, 16 of July 2008, David Woodhouse wrote:
> On Tue, 2008-07-15 at 13:49 -0700, David Woodhouse wrote:
> > I need to hassle Sam about that.
>
> I think I have it. Rafael, does _this_ fix it for you?

No, but the error message seems to be different:

HOSTCC firmware/ihex2fw
/home/rafael/src/linux-next/firmware/ihex2fw.c:268: fatal error: opening dependency file firmware/.ihex2fw.d: Read-only file system
compilation terminated.

Removing the FORCE from line 117 of scripts/Makefile.host makes it work.

Of course, I still have to run "make ... firmware_install" locally on the
server.

> diff --git a/scripts/Makefile.fwinst b/scripts/Makefile.fwinst
> index 3d2f460..994060d 100644
> --- a/scripts/Makefile.fwinst
> +++ b/scripts/Makefile.fwinst
> @@ -43,3 +43,15 @@ __fw_install: $(installed-fw)
> __fw_modinst: $(mod-fw)
>
> FORCE:
> +
> +# Read all saved command lines and dependencies for the $(targets) we
> +# may be building using $(if_changed{,_dep}). As an optimization, we
> +# don't need to read them if the target does not exist; we will rebuild
> +# anyway in that case.
> +
> +targets := $(wildcard $(sort $(targets)))
> +cmd_files := $(wildcard $(foreach f,$(targets),$(dir $(f)).$(notdir $(f)).cmd))
> +
> +ifneq ($(cmd_files),)
> + include $(cmd_files)
> +endif
>

2008-07-16 22:05:21

by dcg

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

El Tue, 15 Jul 2008 16:26:41 -0400, Jeff Garzik <[email protected]> escribió:

> And yes, it enables "weirdos" like me to continue building the firmware
> into the driver, because it has a proven track record of high
> reliability and simplicity.

I'm one of those "weirdos", and I'd like to explain from my user-POV (a user
who usually compiles and tests -rcs) why I prefer firmware-in-module I don't
find it "weird".

I use modules as much as I can. Apparently many people here uses non-modular
.configs. Me, I learnt lot of time ago that we live in a hotplugged world
were you can't predict what hardware you are going to need to plug, and you
don't want to tell your girlfriend that you need to compile a "kernel module"
because she brought a USB memory stick and you didn't compiled the support
when you compiled your kernel. And I don't like doing "lspci" every time I
want to know what network card I have in a box. So I compile everything as
a module (except the support for all sata/ide controllers and ext3, to
avoid the nightmares of initrds) and I let udev do all the job. I only use
external firmware when it's strictly needed for legal reasons.

Yes, just like I can do "make modules_install", I can do "make
firmware_install". It's not hard - just an extra command. But it's an
extra one. And I'm lazy to the point that I'm _not_ willing to type
that extra command. You could install the firmware automatically with
"make modules_install". Still, I'd need to cleanup frequently
/lib/firmware/<version> because of all the crap (and MB) that old
-rcs and -next's versions will leave there.

Those are _extra_ steps. Steps that force me to learn things that
I didn't need to know before. I didn't even know if my network
card had firmware or not. I never cared, and I don't want to start
caring about it now.

With firmware-in-module, I can switch a simple config option that will
get me back to my ignorance of firmware related things. And Jeff will
be my hero of the month for bringing back the simplicity of the old
times, while the rest of the people that want to force me to learn about
firmware (iow, the people who thinks that firmware-in-module is worthless)
remember me of the microkernel people, who could write books and give
speeches in universities about how insanely great is to have fulfilled
one of the most important problems in the story of CS - separating
firmware from the driver code. Despite of the fact that it does not
bring ANYTHING new for me, excepting added complexity.

I'll continue using external firmware only when for legal reasons it
can't be redistributed, of course, but fortunately for me that's an
exception on my hardware devices, not the rule.

2008-07-16 22:42:48

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Wed, 2008-07-16 at 23:28 +0200, Rafael J. Wysocki wrote:
> On Wednesday, 16 of July 2008, David Woodhouse wrote:
> > On Tue, 2008-07-15 at 13:49 -0700, David Woodhouse wrote:
> > > I need to hassle Sam about that.
> >
> > I think I have it. Rafael, does _this_ fix it for you?
>
> No, but the error message seems to be different:
>
> HOSTCC firmware/ihex2fw
> /home/rafael/src/linux-next/firmware/ihex2fw.c:268: fatal error: opening dependency file firmware/.ihex2fw.d: Read-only file system
> compilation terminated.

Hm, can you do that again with V=2?

> Removing the FORCE from line 117 of scripts/Makefile.host makes it work.
>
> Of course, I still have to run "make ... firmware_install" locally on the
> server.

Yeah, that part is easy enough to fix, once the unwanted rebuilding is
fixed.

--
dwmw2

2008-07-17 18:30:10

by Alan

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

> * the legal subtext of these changes was not mentioned at all

The only legal subtext has been some mad fantasy proposed by Dave Miller
that David Woodhouse current employer and previous one were both in some
conspiracy to push firmware elsewhere. It would be very helpful if you
stuck to the real world.

The only legal agenda people I know actually don't like request_firmware
because it makes it easier to have non-free firmware files. In their
strange world they'd rather delete the driver

2008-07-17 20:40:36

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Thursday, 17 of July 2008, David Woodhouse wrote:
> On Wed, 2008-07-16 at 23:28 +0200, Rafael J. Wysocki wrote:
> > On Wednesday, 16 of July 2008, David Woodhouse wrote:
> > > On Tue, 2008-07-15 at 13:49 -0700, David Woodhouse wrote:
> > > > I need to hassle Sam about that.
> > >
> > > I think I have it. Rafael, does _this_ fix it for you?
> >
> > No, but the error message seems to be different:
> >
> > HOSTCC firmware/ihex2fw
> > /home/rafael/src/linux-next/firmware/ihex2fw.c:268: fatal error: opening dependency file firmware/.ihex2fw.d: Read-only file system
> > compilation terminated.
>
> Hm, can you do that again with V=2?

HOSTCC firmware/ihex2fw - due to: FORCE
/home/rafael/src/linux-next/firmware/ihex2fw.c:268: fatal error: opening dependency file firmware/.ihex2fw.d: Read-only file system
compilation terminated.

Thanks,
Rafael

2008-07-17 21:25:56

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Thu, 2008-07-17 at 22:42 +0200, Rafael J. Wysocki wrote:
>
> HOSTCC firmware/ihex2fw - due to: FORCE
> /home/rafael/src/linux-next/firmware/ihex2fw.c:268: fatal error:
> opening dependency file firmware/.ihex2fw.d: Read-only file system
> compilation terminated.

I'm very confused. I can do this...

make INSTALL_FW_PATH=/tmp/foo firmware_install
su otheruser
touch firmware/.ihex2fw.cmd (gives EPERM as expected)
make INSTALL_FW_PATH=/tmp/foo2 firmware_install

And it _works_ for me, after I've made that change in
scripts/Makefile.fwinst. But I don't have .ihex2fw.d -- I
have .ihex2fw.cmd. What is in your .ihex2fw.d? And does it work if you
include _that_ from firmware/Makefile?

/me wonders where Sam is hiding...

--
dwmw2

2008-07-17 21:31:47

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

From: Alan Cox <[email protected]>
Date: Tue, 15 Jul 2008 13:42:46 +0100

> > * the legal subtext of these changes was not mentioned at all
>
> The only legal subtext has been some mad fantasy proposed by Dave Miller
> that David Woodhouse current employer and previous one were both in some
> conspiracy to push firmware elsewhere. It would be very helpful if you
> stuck to the real world.

That was wrong, but there have been other legal subtexts.

request_firmware() itself, although partially done for technical reasons,
had some very real legal impetus.

Linus has even said that in this thread.

2008-07-17 22:23:42

[permalink] [raw]

Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

On Thursday, 17 of July 2008, David Woodhouse wrote:
> On Thu, 2008-07-17 at 22:42 +0200, Rafael J. Wysocki wrote:
> >
> > HOSTCC firmware/ihex2fw - due to: FORCE
> > /home/rafael/src/linux-next/firmware/ihex2fw.c:268: fatal error:
> > opening dependency file firmware/.ihex2fw.d: Read-only file system
> > compilation terminated.
>
> I'm very confused. I can do this...
>
> make INSTALL_FW_PATH=/tmp/foo firmware_install
> su otheruser
> touch firmware/.ihex2fw.cmd (gives EPERM as expected)
> make INSTALL_FW_PATH=/tmp/foo2 firmware_install
>
> And it _works_ for me, after I've made that change in
> scripts/Makefile.fwinst. But I don't have .ihex2fw.d -- I
> have .ihex2fw.cmd. What is in your .ihex2fw.d?

There's no '.ihex2fw.d' under 'firmware' in my O= path.

I just do 'make modules_install' on the NFS client, after I have run
'make O=... ... firmware_install' on the server and it prints the above error
message.

Should I add anything to the 'make modules_install' command line?

Rafael

2008-07-18 04:15:32