Date: Wed, 28 Jun 2017 00:24:19 +0200
From: "Luis R. Rodriguez" <mcgrof@kernel.org>
To: Jakub Kicinski <jakub.kicinski@netronome.com>
Cc: "Luis R. Rodriguez" <mcgrof@kernel.org>,
        Bjorn Andersson <bjorn.andersson@linaro.org>,
        Arend Van Spriel <arend.vanspriel@broadcom.com>,
        Tom Gundersen <teg@jklm.no>, Daniel Wagner <wagi@monom.org>,
        Ming Lei <ming.lei@redhat.com>, yi1.li@linux.intel.com,
        takahiro.akashi@linaro.org, nbroeking@me.com,
        Greg Kroah-Hartman <gregkh@linuxfoundation.org>, mfuzzey@parkeon.com,
        ebiederm@xmission.com, dmitry.torokhov@gmail.com, dwmw2@infradead.org,
        jewalt@lgsinnovations.com, rafal@milecki.pl, rjw@rjwysocki.net,
        atull@kernel.org, moritz.fischer@ettus.com, pmladek@suse.com,
        johannes.berg@intel.com, emmanuel.grumbach@intel.com,
        luciano.coelho@intel.com, luto@kernel.org,
        torvalds@linux-foundation.org, keescook@chromium.org,
        dhowells@redhat.com, pjones@redhat.com, hdegoede@redhat.com,
        alan@linux.intel.com, tytso@mit.edu, paul.gortmaker@windriver.com,
        mtosatti@redhat.com, mawilcox@microsoft.com, stephen.boyd@linaro.org,
        markivx@codeaurora.org, linux-kernel@vger.kernel.org,
        oss-drivers@netronome.com, systemd-devel@lists.freedesktop.org
Subject: Re: [PATCH] firmware: wake all waiters
Message-ID: <20170627222419.GY21846@wotan.suse.de>
References: <20170623233702.20564-1-jakub.kicinski@netronome.com>
 <20170626212036.GE21846@wotan.suse.de>
 <20170626191009.0c11eed0@cakuba.netronome.com>
 <20170627163942.GQ21846@wotan.suse.de>
 <20170627142553.0fe417b3@cakuba.netronome.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20170627142553.0fe417b3@cakuba.netronome.com>
User-Agent: Mutt/1.6.0 (2016-04-01)
Sender: linux-kernel-owner@vger.kernel.org
Content-Length: 3486
Lines: 76

On Tue, Jun 27, 2017 at 02:25:53PM -0700, Jakub Kicinski wrote:
> On Tue, 27 Jun 2017 18:39:42 +0200, Luis R. Rodriguez wrote:
> > 
> > > The problem is that advanced NICs are quite programmable [1] and
> > > depending on use case one may want to load different firmware files.  
> > 
> > Right, so in the 802.11 world some devices might use different firmware for
> > different modes of operation, STA, AP, Mesh, but this is all very protocol
> > specific, so userspace could tickle the kernel about a mode.
> > 
> > Do your use cases have protocol definitions which can be exposed in userspace?
> > Or are these just fw variants with different bells and whistles? How man
> > different use cases are we talking about?
> 
> Right now we have three modes that come from Netronome itself, a "basic
> NIC" one, and two advanced for TC flower/Open vSwitch acceleration and
> for eBPF offload.  I was hoping some enumeration scheme could work here,
> but I really can't come up with one.

How about just supporting 3 firmware names, with the first two being optional,
but if found one of those two is found it would use that one. Then only if
both of these are not present would a default be looked for and used?

In terms of interface, a simple symlink / renaming scheme would suffice to
support this. No custom hooks at all.

> To be honest waiting for rootfs to be available is lower on my list of
> priorities, but it's definitely nice to have.  I also don't care about
> supporting more complex rootfs setups, simply trying whatever comes
> after initramfs covers 99.9% use cases.  0.1% can load the FW manually/
> rebind the driver IMHO.

Right, the only issue with firmwared is you'd expect folks would have
it deployed, and that's not the case today, but when you do control the
ecosystem you can certainly use it as an option. Let me know if you
do try it out.

> > Be careful how you do this as you'll have to support it in the driver forever
> > if you use something like sysfs I think, otherwise you will break some
> > userspace. However if you use debugfs I think its understood that's loose API.
> 
> Unfortunately the netdev community does not like debugfs.  I would
> prefer to extend the firmware subsystem if possible and use the
> existing sysfs interface, just in a new "mode".

I don't think this is required, you can simply use different filenames as
noted above.

> > > Current firmware subsystem doesn't seem to cater to this use case to
> > > well.  
> > 
> > Its a matter of asking and talking. I've provided references of things to
> > try to address the hacky -EPROBE_DEFER. It does however require a userspace
> > daemon used, so it does require use of the uevent fallback mechanism.
> 
> Do you know how systemd developers feel about the issue (CCed)?  Given
> that it seems to dominate in data center OSes now I'm slightly worried
> having to push Big Linux Vendors to package some seemingly
> embedded-centric software just to make advanced NICs run :(

firmwared was written by a systemd developer :)

I think it was first packaged into systemd, and then it was split out to
help those who want it external.

> > >  - how to make sure different cards, which request the same file name
> > >    can be served different default firmwares...  
> > 
> > I believe your patch + the error path fix will handle this now, no?
> 
> I'm not sure.  I think it would work if I set FW_OPT_NOCACHE, though.
> I need to test that.

Why do you need FW_OPT_NOCACHE?

 Luis