Hi all,
Sorry for the long delay in getting back to this. I'm meaning to write
some code soon also for this, to illustrate better, but I figured I'd
still get some thoughts out before I do that.
After more discussion (@Intel) and the previous thread(s), I've pretty
much come to the conclusion that we should have a small subsystem for
WWAN, rather than fudging everything like we previously did.
We can debate whether or not that should use 'real' netlink or generic
netlink - personally I know the latter better and I think it has some
real advantages like easier message parsing (it's automatic more or
less) including strict checking and automatic policy introspection (I
recently wrote the code for this and it's plugged into generic netlink
family, for other netlink families it needs more hand-written code). But
I could possibly be convinced of doing something else, and/or perhaps
building more infrastructure for 'real' netlink to realize those
benefits there as well.
In terms of what I APIs are needed, the kernel-driver side and userspace
side go pretty much hand in hand (the wwan subsystem just providing the
glue), so what I say below is pretty much both a method/function call
(kernel internal API) or a netlink message (userspace API).
1) I think a generic abstraction of WWAN device that is not a netdev
is needed. Yes, on the one hand it's quite nice to be able to work on
top of a given netdev, but it's also limiting because it requires the
data flow to go through there, and packets that are tagged in some
way be exchanged there.
For VLANs this can be out-of-band (in a sense) with hw-accel, but for
rmnet-style it's in-band, and that limits what we can do.
Now, of course this doesn't mean there shouldn't be a netdev created
by default in most cases, but it gives us a way to do additional
things that we cannot do with *just* a netdev.
From a driver POV though, it would register a new "wwan_device", and
then get some generic callback to create a netdev on top, maybe by
default from the subsystem or from the user.
2) Clearly, one needs to be able to create PDN netdevs, with the PDN
given to the command. Here's another advantage: If these are created
on top of another abstraction, not another netdev, they can have
their own queues, multiqueue RX etc. much more easily.
Also, things like the "if I have just a single channel, drop the mux
headers" can then be entirely done in the driver, and the default
netdev no longer has the possibility of muxed and IP frames on the
same datapath.
This also enables more things like handling checksum offload directly
in the driver, which doesn't behave so well with VLANs I think.
All of that will just be easier for 5G too, I believe, with
acceleration being handled per PDN, multi-queue working without
ndo_select_queue, etc.
Quite possibly there might be some additional (vendor-dependent?)
configuration for when such netdevs are created, but we need to
figure out if that really needs to be at creation time, or just
ethtool later or something like that. I guess it depends on how
generic it needs to be.
3) Separately, we need to have an ability to create "generalized control
channels". I'm thinking there would be a general command "create
control channel" with a given type (e.g. ATCMD, RPC, MBIM, GNSS) plus
a list of vendor-specific channels (e.g. for tracing).
I'm unsure where this channel should really go - somehow it seems to
me that for many (most?) of these registering them as a serial line
would be most appropriate, but some, especially vendor-defined
channels like tracing, would probably better use a transport that's
higher bandwidth than, e.g. netdevs.
One way I thought of doing this was to create an abstraction in the
wwan framework that lets the driver use SKBs anyway (i.e. TX and RX
on these channels using SKBs) and then translate them to some channel
in the framework - that way, we can even select at runtime if we want
a netdev (not really plugged into the network stack, ARPHDR_VOID?) or
some other kind of transport. Building that would allow us to add
transport types in the future too.
I guess such a channel should also be created by default, if it's
not already created by the driver in some out-of-band way anyway (and
most likely it shouldn't be, but I guess drivers might have some
entirely different communication channels for AT CMDs?)
4) There was a question about something like pure IP channels that
belong to another PDN and apparently now separate netdevs might be
used, but it seems to me that could just be a queue reserved on the
regular netdevs and then when you say ("enable video streaming on
wwan1 interface") that can do some magic to classify the video
packets (DSCP?) to another hardware queue for better QoS.
Anyway, if all of this doesn't seem completely outlandish I'll try to
write some code to illustrate it (sooner, rather than later).
Thanks,
johannes
Hi Johannes,
> Sorry for the long delay in getting back to this. I'm meaning to write
> some code soon also for this, to illustrate better, but I figured I'd
> still get some thoughts out before I do that.
>
> After more discussion (@Intel) and the previous thread(s), I've pretty
> much come to the conclusion that we should have a small subsystem for
> WWAN, rather than fudging everything like we previously did.
>
> We can debate whether or not that should use 'real' netlink or generic
> netlink - personally I know the latter better and I think it has some
> real advantages like easier message parsing (it's automatic more or
> less) including strict checking and automatic policy introspection (I
> recently wrote the code for this and it's plugged into generic netlink
> family, for other netlink families it needs more hand-written code). But
> I could possibly be convinced of doing something else, and/or perhaps
> building more infrastructure for 'real' netlink to realize those
> benefits there as well.
>
>
> In terms of what I APIs are needed, the kernel-driver side and userspace
> side go pretty much hand in hand (the wwan subsystem just providing the
> glue), so what I say below is pretty much both a method/function call
> (kernel internal API) or a netlink message (userspace API).
>
> 1) I think a generic abstraction of WWAN device that is not a netdev
> is needed. Yes, on the one hand it's quite nice to be able to work on
> top of a given netdev, but it's also limiting because it requires the
> data flow to go through there, and packets that are tagged in some
> way be exchanged there.
> For VLANs this can be out-of-band (in a sense) with hw-accel, but for
> rmnet-style it's in-band, and that limits what we can do.
>
> Now, of course this doesn't mean there shouldn't be a netdev created
> by default in most cases, but it gives us a way to do additional
> things that we cannot do with *just* a netdev.
>
> From a driver POV though, it would register a new "wwan_device", and
> then get some generic callback to create a netdev on top, maybe by
> default from the subsystem or from the user.
Have you actually looked at Phonet or CAIF.
And netdev by default seems like repeating the same mistake we have done with WiFi. Your default context in LTE cases is only available when actually registered to the LTE bearer. It is pretty much pointless to have a netdev if you are not registered to the network.
You have to do a lot of initial modem setup before you ever get to the having your default context connected. Have a look at oFono and what it does to bring up the modem.
> 2) Clearly, one needs to be able to create PDN netdevs, with the PDN
> given to the command. Here's another advantage: If these are created
> on top of another abstraction, not another netdev, they can have
> their own queues, multiqueue RX etc. much more easily.
>
> Also, things like the "if I have just a single channel, drop the mux
> headers" can then be entirely done in the driver, and the default
> netdev no longer has the possibility of muxed and IP frames on the
> same datapath.
>
> This also enables more things like handling checksum offload directly
> in the driver, which doesn't behave so well with VLANs I think.
>
> All of that will just be easier for 5G too, I believe, with
> acceleration being handled per PDN, multi-queue working without
> ndo_select_queue, etc.
>
> Quite possibly there might be some additional (vendor-dependent?)
> configuration for when such netdevs are created, but we need to
> figure out if that really needs to be at creation time, or just
> ethtool later or something like that. I guess it depends on how
> generic it needs to be.
I think you need to provide actually a lot more details on how queue control inside Linux would be helpful and actually work in the first place. I don’t see how Linux will be ever in charge and not the modem do this all for you.
> 3) Separately, we need to have an ability to create "generalized control
> channels". I'm thinking there would be a general command "create
> control channel" with a given type (e.g. ATCMD, RPC, MBIM, GNSS) plus
> a list of vendor-specific channels (e.g. for tracing).
>
> I'm unsure where this channel should really go - somehow it seems to
> me that for many (most?) of these registering them as a serial line
> would be most appropriate, but some, especially vendor-defined
> channels like tracing, would probably better use a transport that's
> higher bandwidth than, e.g. netdevs.
>
> One way I thought of doing this was to create an abstraction in the
> wwan framework that lets the driver use SKBs anyway (i.e. TX and RX
> on these channels using SKBs) and then translate them to some channel
> in the framework - that way, we can even select at runtime if we want
> a netdev (not really plugged into the network stack, ARPHDR_VOID?) or
> some other kind of transport. Building that would allow us to add
> transport types in the future too.
>
> I guess such a channel should also be created by default, if it's
> not already created by the driver in some out-of-band way anyway (and
> most likely it shouldn't be, but I guess drivers might have some
> entirely different communication channels for AT CMDs?)
I would just use sockets like Phonet and CAIF.
> 4) There was a question about something like pure IP channels that
> belong to another PDN and apparently now separate netdevs might be
> used, but it seems to me that could just be a queue reserved on the
> regular netdevs and then when you say ("enable video streaming on
> wwan1 interface") that can do some magic to classify the video
> packets (DSCP?) to another hardware queue for better QoS.
>
>
>
> Anyway, if all of this doesn't seem completely outlandish I'll try to
> write some code to illustrate it (sooner, rather than later).
Frankly I have a problem if this is designed from the hardware point of view. Unless you are familiar with how 3GPP works and what a telephony stack like oFono has to do, there is really no point in trying to create a WWAN subsystem.
Anything defined needs to be defined in terms of 3GPP and not what current drivers have hacked together.
Regards
Marcel
On Mon, 2019-05-27 at 15:20 +0200, Johannes Berg wrote:
> Hi all,
>
> Sorry for the long delay in getting back to this. I'm meaning to write
> some code soon also for this, to illustrate better, but I figured I'd
> still get some thoughts out before I do that.
>
> After more discussion (@Intel) and the previous thread(s), I've pretty
> much come to the conclusion that we should have a small subsystem for
> WWAN, rather than fudging everything like we previously did.
>
> We can debate whether or not that should use 'real' netlink or generic
> netlink - personally I know the latter better and I think it has some
> real advantages like easier message parsing (it's automatic more or
> less) including strict checking and automatic policy introspection (I
> recently wrote the code for this and it's plugged into generic netlink
> family, for other netlink families it needs more hand-written code). But
> I could possibly be convinced of doing something else, and/or perhaps
> building more infrastructure for 'real' netlink to realize those
> benefits there as well.
>
>
> In terms of what I APIs are needed, the kernel-driver side and userspace
> side go pretty much hand in hand (the wwan subsystem just providing the
> glue), so what I say below is pretty much both a method/function call
> (kernel internal API) or a netlink message (userspace API).
>
> 1) I think a generic abstraction of WWAN device that is not a netdev
> is needed. Yes, on the one hand it's quite nice to be able to work on
> top of a given netdev, but it's also limiting because it requires the
> data flow to go through there, and packets that are tagged in some
> way be exchanged there.
> For VLANs this can be out-of-band (in a sense) with hw-accel, but for
> rmnet-style it's in-band, and that limits what we can do.
>
> Now, of course this doesn't mean there shouldn't be a netdev created
> by default in most cases, but it gives us a way to do additional
> things that we cannot do with *just* a netdev.
>
> From a driver POV though, it would register a new "wwan_device", and
> then get some generic callback to create a netdev on top, maybe by
> default from the subsystem or from the user.
>
> 2) Clearly, one needs to be able to create PDN netdevs, with the PDN
> given to the command. Here's another advantage: If these are created
> on top of another abstraction, not another netdev, they can have
> their own queues, multiqueue RX etc. much more easily.
>
> Also, things like the "if I have just a single channel, drop the mux
> headers" can then be entirely done in the driver, and the default
> netdev no longer has the possibility of muxed and IP frames on the
> same datapath.
>
> This also enables more things like handling checksum offload directly
> in the driver, which doesn't behave so well with VLANs I think.
>
> All of that will just be easier for 5G too, I believe, with
> acceleration being handled per PDN, multi-queue working without
> ndo_select_queue, etc.
>
> Quite possibly there might be some additional (vendor-dependent?)
> configuration for when such netdevs are created, but we need to
> figure out if that really needs to be at creation time, or just
> ethtool later or something like that. I guess it depends on how
> generic it needs to be.
I'm pretty sure it would have to be async via netlink or ethtool or
whatever later, because the control plane (ModemManager,
libmbim/libqmi, oFono, etc) is what sets up the PDP/EPS context and
thus the data channel. A netdev (or a queue on that netdev) would be a
representation of that data channel, but that's not something the
kernel knows beforehand.
> 3) Separately, we need to have an ability to create "generalized control
> channels". I'm thinking there would be a general command "create
> control channel" with a given type (e.g. ATCMD, RPC, MBIM, GNSS) plus
> a list of vendor-specific channels (e.g. for tracing).
>
> I'm unsure where this channel should really go - somehow it seems to
> me that for many (most?) of these registering them as a serial line
> would be most appropriate, but some, especially vendor-defined
> channels like tracing, would probably better use a transport that's
> higher bandwidth than, e.g. netdevs.
There's only a couple protocols that are run over serial transport,
namely AT, DM/DIAG, and Sierra's CnS.
Most of the rest like QMI and MBIM are packet-based protocols that can
use different transports. QMI for example can use CDC-WDM for USB
devices but on SoCs will use Qualcomm's SMD I believe.
Should you really need to account for these specially, or would some
kind of sysfs linkage like SET_NETDEV_DEV() be more appropriate?
Really all you want to do is (a) identify which WWAN device a given
control/data channel is for and (b) perhaps tag different control/data
channels with attributes like primary/secondary/gps/sim/etc often
through USB attributes or hardcoded data on SoCs.
> One way I thought of doing this was to create an abstraction in the
> wwan framework that lets the driver use SKBs anyway (i.e. TX and RX
> on these channels using SKBs) and then translate them to some channel
> in the framework - that way, we can even select at runtime if we want
> a netdev (not really plugged into the network stack, ARPHDR_VOID?) or
> some other kind of transport. Building that would allow us to add
> transport types in the future too.
I'm not quite sure what you mean here, can you explain?
> I guess such a channel should also be created by default, if it's
> not already created by the driver in some out-of-band way anyway (and
> most likely it shouldn't be, but I guess drivers might have some
> entirely different communication channels for AT CMDs?)
For existing devices we're not talking about monolithic drivers here
(excepting 'hso') which I guess is part of the issue. A given device
might be driven by 2 or 3 different kernel drivers (usb-serial derived,
netdev, cdc-wdm) and they all expose different channels and none of
them coordinate. You have to do sysfs matching up the family tree to
find out they are associated with each other. If the kernel can take
over some of that responsibility great.
But the diversity is large. Any given TTY representing an AT channel
could be from USB drivers (usb-serial, cdc-wdm, vendor-specific driver
like sierra/hso, option) or PCI (nozomi) or platform stuff (Qualcomm
SoC ones). You can also do AT-over-QMI if you want.
I think it's worth discussing how this could be better unified but
maybe small steps to get there are more feasible.
> 4) There was a question about something like pure IP channels that
> belong to another PDN and apparently now separate netdevs might be
> used, but it seems to me that could just be a queue reserved on the
> regular netdevs and then when you say ("enable video streaming on
> wwan1 interface") that can do some magic to classify the video
> packets (DSCP?) to another hardware queue for better QoS.
Most stuff is pure IP these days (both for QMI/rmnet and MBIM at least)
and having ethernet encapsulation is kinda pointless. But anyway you'd
need some mechanism to map that particular queue to a given channel/PDN
created by the control channel.
But classification is mostly done in the hardware/network because
setting different QoS for a given PDP/EPS context is basically saying
how much airtime the queue gets. No amount of kernel involvement is
going to change what the network lets you transmit. And I honestly
don't know how the firmware responds when its internal queues for a
given context are full that would tell a kernel queue to stop sending
more.
Dan
>
>
> Anyway, if all of this doesn't seem completely outlandish I'll try to
> write some code to illustrate it (sooner, rather than later).
>
> Thanks,
> johannes
>
Hi Marcel,
> Have you actually looked at Phonet or CAIF.
Phonet is just a specific protocol spoken by a specific modem (family)
for their control plane. Not all modems are going to be speaking this.
Same for CAIF, really. I don't really see all that much that's generic
(enough) here. It's unfortunate that in all this time nobody ever
bothered to even try though, and just invented all their own mechanisms
to precisely mirror the hardware and firmware they were working with.
Theoretically now, you could build a driver that still speaks CAIF or
phonet and then translates to a specific modem, but what would the point
be?
Now, I'm looking more at the data plan than the control plane, so I
guess you could argue I should've not just mentioned MBIM and AT
commands, but also something like Phonet/CAIF.
> And netdev by default seems like repeating the same mistake we have
> done with WiFi. Your default context in LTE cases is only available
> when actually registered to the LTE bearer. It is pretty much
> pointless to have a netdev if you are not registered to the network.
I partially agree with this.
Of course, for WiFi, that's just wrong since the control path is on the
netdev. Without a netdev, nothing works, and so not having one by
default just adds something pointless to the setup necessary to bring up
anything at all. It can be argued whether not allowing to remove it is
right, but that just detracts from the discussion at hand and there's
also a lot of history here.
I do understand (and mostly agree) that having a netdev by default
that's not connected to anything and has no functionality until you've
done some out-of-band (as far as the netdev is concerned) setup is
fairly much pointless, but OTOH having a netdev is something that people
seem to want for various reasons (discovery, ethtool, ...).
> You have to do a lot of initial modem setup before you ever get to the
> having your default context connected. Have a look at oFono and what
> it does to bring up the modem.
Sure.
> > 2) Clearly, one needs to be able to create PDN netdevs, with the PDN
> > given to the command. Here's another advantage: If these are created
> > on top of another abstraction, not another netdev, they can have
> > their own queues, multiqueue RX etc. much more easily.
[...]
> I think you need to provide actually a lot more details on how queue
> control inside Linux would be helpful and actually work in the first
> place. I don’t see how Linux will be ever in charge and not the modem
> do this all for you.
I think you misunderstand. I wasn't really talking about *queue control*
(packet dequeue etc.) although this is *also* something that could be
interesting, since the modem can only control packets that ever made it
to the hardware.
I was more thinking of what I actually wrote - "have their own queues,
multiqueue RX etc." - i.e. control the software layer of the queues in
the driver, rather than having all of that managed in some stacked
netdev like VLAN.
For example, with stacked netdevs like VLANs it gets difficult (and
awkward from a network stack perspective) to put frames for different
connections (stacked netdevs) into different hardware queues and manage
flow control correctly.
Similarly, if one connection has maybe multiple hardware queues (say for
a specific video stream) disentangling that when you have stacked
netdevs is much harder than when they're all separate.
(And, of course, there's little point in having the underlying netdev to
start with since it speaks a device-specific protocol the network stack
will never understand.)
> > 3) Separately, we need to have an ability to create "generalized control
> > channels". I'm thinking there would be a general command "create
> > control channel" with a given type (e.g. ATCMD, RPC, MBIM, GNSS) plus
> > a list of vendor-specific channels (e.g. for tracing).
[...]
> > I guess such a channel should also be created by default, if it's
> > not already created by the driver in some out-of-band way anyway (and
> > most likely it shouldn't be, but I guess drivers might have some
> > entirely different communication channels for AT CMDs?)
>
> I would just use sockets like Phonet and CAIF.
It was in fact one of the options I considered, but it seems to have
very little traction with any other userspace, and having a separate
socket family yet again also seems a bit pointless. I guess for devices
that do already speak Phonet or CAIF that would make sense. Possible to
some extent, but not really useful, would be to terminate the Phonet or
CAIF protocol inside the driver or stack, but then you end up having to
emulate some specific firmware behaviour ...
So ultimately it boils down to what protocols you want to support and
what kind of API they typically use. I guess I don't have enough hubris
to propose unifying the whole command set and how you talk to your
device ;-)
I suppose you could have a socket type for AT commands, but is that
really all that useful? Or a socket type that muxes the different
control channels you might have, which gets close to phonet/caif.
> Frankly I have a problem if this is designed from the hardware point
> of view. Unless you are familiar with how 3GPP works and what a
> telephony stack like oFono has to do, there is really no point in
> trying to create a WWAN subsystem.
>
> Anything defined needs to be defined in terms of 3GPP and not what
> current drivers have hacked together.
That objection makes no sense to me. 3GPP doesn't define how the data
plane is implemented in your device/OS. There's a data plane, it carries
IP packets, but how you get those to your applications?
After all, I'm not really proposing that we put oFono or something like
it into the kernel - far from it! I'm only proposing that we kill the
many various ways of creating and managing the necessary netdevs (VLANs,
sysfs, rmnet, ...) from a piece of software like oFono (or libmbim or
whatever else).
Apart from CAIF and phonet, oFono doesn't even try to do this though,
afaict, so I guess it relies on the default netdev created, or some out-
of-band configuration is still needed?
johannes
Hi Dan,
> > Quite possibly there might be some additional (vendor-dependent?)
> > configuration for when such netdevs are created, but we need to
> > figure out if that really needs to be at creation time, or just
> > ethtool later or something like that. I guess it depends on how
> > generic it needs to be.
>
> I'm pretty sure it would have to be async via netlink or ethtool or
> whatever later, because the control plane (ModemManager,
> libmbim/libqmi, oFono, etc) is what sets up the PDP/EPS context and
> thus the data channel. A netdev (or a queue on that netdev) would be a
> representation of that data channel, but that's not something the
> kernel knows beforehand.
Right.
It just seemed that people do want to have a netdev (if only to see that
their driver loaded and use ethtool to dump the firmware version), and
then you can reassign it to some actual context later?
It doesn't really matter much. If you have a control channel and higher-
level abstraction (wwan device) then having the netdev is probably more
of a nuisance and mostly irrelevant, just might be useful for legacy
reasons.
> > 3) Separately, we need to have an ability to create "generalized control
> > channels". I'm thinking there would be a general command "create
> > control channel" with a given type (e.g. ATCMD, RPC, MBIM, GNSS) plus
> > a list of vendor-specific channels (e.g. for tracing).
> >
> > I'm unsure where this channel should really go - somehow it seems to
> > me that for many (most?) of these registering them as a serial line
> > would be most appropriate, but some, especially vendor-defined
> > channels like tracing, would probably better use a transport that's
> > higher bandwidth than, e.g. netdevs.
>
> There's only a couple protocols that are run over serial transport,
> namely AT, DM/DIAG, and Sierra's CnS.
Right.
> Most of the rest like QMI and MBIM are packet-based protocols that can
> use different transports. QMI for example can use CDC-WDM for USB
> devices but on SoCs will use Qualcomm's SMD I believe.
Right, though transport and protocol are sort of different issues.
> Should you really need to account for these specially, or would some
> kind of sysfs linkage like SET_NETDEV_DEV() be more appropriate?
>
> Really all you want to do is (a) identify which WWAN device a given
> control/data channel is for and (b) perhaps tag different control/data
> channels with attributes like primary/secondary/gps/sim/etc often
> through USB attributes or hardcoded data on SoCs.
Ah, that brings up something I completely forgot.
I was thinking only of the case that the control channel(s) to the
device is/are all managed by the *kernel* driver, i.e. you'd have some
device-specific driver that has an interface into userspace to talk to
the modem's control channel (and that we could abstract).
However, yes, that's not true - many will be like USB where the control
channel is driven by a generic kernel driver (e.g. maybe usb-serial) or
no kernel driver at all, and then this linkage is probably the right
approach. Need to think about it.
OTOH, there will be device-specific ways to add more control channels
(say for debug/trace purposes etc.) and those would not have a "natural"
interface to userspace like control channels with generic/no drivers.
> > One way I thought of doing this was to create an abstraction in the
> > wwan framework that lets the driver use SKBs anyway (i.e. TX and RX
> > on these channels using SKBs) and then translate them to some channel
> > in the framework - that way, we can even select at runtime if we want
> > a netdev (not really plugged into the network stack, ARPHDR_VOID?) or
> > some other kind of transport. Building that would allow us to add
> > transport types in the future too.
>
> I'm not quite sure what you mean here, can you explain?
I was just thinking of the mechanics of doing this in the driver (while,
like I said above, completely forgetting about the non-monolithic driver
case). It's not really that important.
> > I guess such a channel should also be created by default, if it's
> > not already created by the driver in some out-of-band way anyway (and
> > most likely it shouldn't be, but I guess drivers might have some
> > entirely different communication channels for AT CMDs?)
>
> For existing devices we're not talking about monolithic drivers here
> (excepting 'hso') which I guess is part of the issue.
Right, and doesn't help I forgot about this ;-)
> A given device
> might be driven by 2 or 3 different kernel drivers (usb-serial derived,
> netdev, cdc-wdm) and they all expose different channels and none of
> them coordinate. You have to do sysfs matching up the family tree to
> find out they are associated with each other. If the kernel can take
> over some of that responsibility great.
Right. I guess it's hard for the kernel to take responsibility unless we
teach all the components (usb-serial, ...) that certain devices are
modems and should get some (additional?) registration?
> But the diversity is large. Any given TTY representing an AT channel
> could be from USB drivers (usb-serial, cdc-wdm, vendor-specific driver
> like sierra/hso, option) or PCI (nozomi) or platform stuff (Qualcomm
> SoC ones). You can also do AT-over-QMI if you want.
Right. The linkage here is the interesting part - for platform stuff
that might be easier (devicetree?) but not sure how we could teach that
to e.g. usb-serial, and nozomi is interesting because ... where is the
data plane even?
> I think it's worth discussing how this could be better unified but
> maybe small steps to get there are more feasible.
Fair point.
> > 4) There was a question about something like pure IP channels that
> > belong to another PDN and apparently now separate netdevs might be
> > used, but it seems to me that could just be a queue reserved on the
> > regular netdevs and then when you say ("enable video streaming on
> > wwan1 interface") that can do some magic to classify the video
> > packets (DSCP?) to another hardware queue for better QoS.
>
> Most stuff is pure IP these days (both for QMI/rmnet and MBIM at least)
> and having ethernet encapsulation is kinda pointless.
Yeah, true, not really sure why I was distinguishing this in these
terms. I think the use case really was just giving some packets higher
priority, putting them into a different *hardware* queue so the device
can see them even if the "normal" hardware queue is completely
backlogged.
Kinda a typical multi-queue TX use case.
> But anyway you'd
> need some mechanism to map that particular queue to a given channel/PDN
> created by the control channel.
Well, I was thinking that mechanism was creating the netdev, but then
*within* that some QoS seems to be desired.
> But classification is mostly done in the hardware/network because
> setting different QoS for a given PDP/EPS context is basically saying
> how much airtime the queue gets. No amount of kernel involvement is
> going to change what the network lets you transmit.
Right.
> And I honestly
> don't know how the firmware responds when its internal queues for a
> given context are full that would tell a kernel queue to stop sending
> more.
Well, at the very least it'll stop pulling packets from DMA/whatever, so
the kernel has to back off, right?
johannes
Hi Johannes,
>
> It just seemed that people do want to have a netdev (if only to see that
> their driver loaded and use ethtool to dump the firmware version), and
> then you can reassign it to some actual context later?
I can see that this is useful for developers, but really
counterproductive in production. You have a bunch of services (systemd,
NM, ConnMan, dhcpcd, etc, etc, etc) all observing the newly created
devices and trying to 'own' them. Which makes no freaking sense to do
until those devices are really usable. Just because it is a netdev,
doesn't mean it is ethernet or behaves like it.
That also leads to expectations from users that want stuff like
udev-consistent-naming to work, even though udev has no idea wtf a given
device is, when it is ready or not ready, etc. And the flip side, there
may be systems that do not use systemd/udevd, so the daemons responsible
for management of such devices cannot sanely assume anything. It is
just pure chaos.
And then there's hotplug/unplug to worry about ;)
So I would like to reiterate Marcel's point: creating unmanaged devices
should not be the default behavior.
>
> It doesn't really matter much. If you have a control channel and higher-
> level abstraction (wwan device) then having the netdev is probably more
> of a nuisance and mostly irrelevant, just might be useful for legacy
> reasons.
Which we should be trying to eradicate, not encourage ;)
>> Should you really need to account for these specially, or would some
>> kind of sysfs linkage like SET_NETDEV_DEV() be more appropriate?
>>
>> Really all you want to do is (a) identify which WWAN device a given
>> control/data channel is for and (b) perhaps tag different control/data
>> channels with attributes like primary/secondary/gps/sim/etc often
>> through USB attributes or hardcoded data on SoCs.
Userspace can also choose to do its own multiplexing, so you can't even
really assume the above is what you 'want'.
Regards,
-Denis
Hi Johannes,
>
> After all, I'm not really proposing that we put oFono or something like
> it into the kernel - far from it! I'm only proposing that we kill the
> many various ways of creating and managing the necessary netdevs (VLANs,
> sysfs, rmnet, ...) from a piece of software like oFono (or libmbim or
> whatever else).
I do like the concept of unifying this if possible. The question is, is
it actually possible :) I think Dan covered most of the aspects of what
userspace has to deal with already. But the basic issue is that there's
a heck of a lot of different ways of doing it.
>
> Apart from CAIF and phonet, oFono doesn't even try to do this though,
> afaict, so I guess it relies on the default netdev created, or some out-
> of-band configuration is still needed?
Actually it can. We can drive modems which provide only a single serial
port and run multiplexing over that. So we fully control the number of
control channels created, the number of netdevs created and even
create/destroy them on as needed basis. And these netdevs can be PPP
encapsulated or pure IP or whatever else.
Regards,
-Denis
Hi Johannes,
>> Have you actually looked at Phonet or CAIF.
>
> Phonet is just a specific protocol spoken by a specific modem (family)
> for their control plane. Not all modems are going to be speaking this.
> Same for CAIF, really. I don't really see all that much that's generic
> (enough) here. It's unfortunate that in all this time nobody ever
> bothered to even try though, and just invented all their own mechanisms
> to precisely mirror the hardware and firmware they were working with.
>
> Theoretically now, you could build a driver that still speaks CAIF or
> phonet and then translates to a specific modem, but what would the point
> be?
>
> Now, I'm looking more at the data plan than the control plane, so I
> guess you could argue I should've not just mentioned MBIM and AT
> commands, but also something like Phonet/CAIF.
actually Phonet and CAIF are the transports and endpoint connections in the devices. The modem control protocol of Phonet was ISI and for CAIF it was actually AT commands (but I think there was a plan for ISI over CAIF).
>> And netdev by default seems like repeating the same mistake we have
>> done with WiFi. Your default context in LTE cases is only available
>> when actually registered to the LTE bearer. It is pretty much
>> pointless to have a netdev if you are not registered to the network.
>
> I partially agree with this.
>
> Of course, for WiFi, that's just wrong since the control path is on the
> netdev. Without a netdev, nothing works, and so not having one by
> default just adds something pointless to the setup necessary to bring up
> anything at all. It can be argued whether not allowing to remove it is
> right, but that just detracts from the discussion at hand and there's
> also a lot of history here.
Actually we need to have that history discussion on how it was also wrong for WiFi from my point of view. I do understand how things have made sense back in the days, but these days clinging to a netdev for control point even in WiFi makes no sense. Anyway, that is a different discussion.
My point is really that a lot of other things are ongoing before a modem even gets a data plane that would be mapped to a netdev.
> I do understand (and mostly agree) that having a netdev by default
> that's not connected to anything and has no functionality until you've
> done some out-of-band (as far as the netdev is concerned) setup is
> fairly much pointless, but OTOH having a netdev is something that people
> seem to want for various reasons (discovery, ethtool, …).
That worries me even more. Discovery via RTNL is pointless and totally racy. Usage of ethtool is pointless as well since we are dealing with IP only interfaces.
I have a felling that these points are from people that don’t really understand 3GPP and want to use command line tools to handle cellular modem. Back in the days iwconfig worked as well and so did pppd, but by now we all learned that this is not going to work. Especially in the world of 3GPP it is not going to work. I do not care a bit if ethtool or any other command line tool that wants to talk directly to kernel. Unless you have a way for a telephony daemon like oFono to get its work done, any of this above is just a distraction.
>> You have to do a lot of initial modem setup before you ever get to the
>> having your default context connected. Have a look at oFono and what
>> it does to bring up the modem.
>
> Sure.
>
>>> 2) Clearly, one needs to be able to create PDN netdevs, with the PDN
>>> given to the command. Here's another advantage: If these are created
>>> on top of another abstraction, not another netdev, they can have
>>> their own queues, multiqueue RX etc. much more easily.
> [...]
>> I think you need to provide actually a lot more details on how queue
>> control inside Linux would be helpful and actually work in the first
>> place. I don’t see how Linux will be ever in charge and not the modem
>> do this all for you.
>
> I think you misunderstand. I wasn't really talking about *queue control*
> (packet dequeue etc.) although this is *also* something that could be
> interesting, since the modem can only control packets that ever made it
> to the hardware.
>
> I was more thinking of what I actually wrote - "have their own queues,
> multiqueue RX etc." - i.e. control the software layer of the queues in
> the driver, rather than having all of that managed in some stacked
> netdev like VLAN.
>
> For example, with stacked netdevs like VLANs it gets difficult (and
> awkward from a network stack perspective) to put frames for different
> connections (stacked netdevs) into different hardware queues and manage
> flow control correctly.
>
> Similarly, if one connection has maybe multiple hardware queues (say for
> a specific video stream) disentangling that when you have stacked
> netdevs is much harder than when they're all separate.
>
> (And, of course, there's little point in having the underlying netdev to
> start with since it speaks a device-specific protocol the network stack
> will never understand.)
I am not convinced that any of this will help us since it is all network specific and the Linux job is just to feed the packets into the hardware.
>>> 3) Separately, we need to have an ability to create "generalized control
>>> channels". I'm thinking there would be a general command "create
>>> control channel" with a given type (e.g. ATCMD, RPC, MBIM, GNSS) plus
>>> a list of vendor-specific channels (e.g. for tracing).
> [...]
>>> I guess such a channel should also be created by default, if it's
>>> not already created by the driver in some out-of-band way anyway (and
>>> most likely it shouldn't be, but I guess drivers might have some
>>> entirely different communication channels for AT CMDs?)
>>
>> I would just use sockets like Phonet and CAIF.
>
> It was in fact one of the options I considered, but it seems to have
> very little traction with any other userspace, and having a separate
> socket family yet again also seems a bit pointless. I guess for devices
> that do already speak Phonet or CAIF that would make sense. Possible to
> some extent, but not really useful, would be to terminate the Phonet or
> CAIF protocol inside the driver or stack, but then you end up having to
> emulate some specific firmware behaviour ...
>
> So ultimately it boils down to what protocols you want to support and
> what kind of API they typically use. I guess I don't have enough hubris
> to propose unifying the whole command set and how you talk to your
> device ;-)
>
> I suppose you could have a socket type for AT commands, but is that
> really all that useful? Or a socket type that muxes the different
> control channels you might have, which gets close to phonet/caif.
As I stated above, Phonet and CAIF where just there to connect endpoints inside the hardware / platform. The protocol running inside these pipes could have been anything. I think what is more important is to figure out on how you access the pipes.
I think that Dan already stated this, the difference is just that we have protocols that underneath accept a transport that is serialized and stream based and others where the transport has packet boundaries and is packetized. These would obviously match to SOCK_STREAM and SOCK_SEQPACKET if you had to classify them.
Now that said, it is nice to just get a file descriptor from the kernel and then drive your protocol from userspace. We have used AT commands over TCP to connect to Phonesim for example. The AT commands don’t care if it is a real serial port, an emulated one by the modem driver or even a SOCK_STREAM socket. Same applies to all the other protocols and implementation that have been written over the years.
Dan also stated the major problems that the control and data path endpoints have been served by different drivers and figuring out what character device belongs to what network interface has been a mess. Creating a character device or a TTY is far too simple and because everybody wants to run pppd on a TTY that is what led us into this mess.
So what is really needed is a way to discovery the cellular modem and its capabilities towards the Linux host. Not all modems are equal and the number of control paths might be limited in the same way as the number of contexts it can expose or its voice to the host streams.
>> Frankly I have a problem if this is designed from the hardware point
>> of view. Unless you are familiar with how 3GPP works and what a
>> telephony stack like oFono has to do, there is really no point in
>> trying to create a WWAN subsystem.
>>
>> Anything defined needs to be defined in terms of 3GPP and not what
>> current drivers have hacked together.
>
> That objection makes no sense to me. 3GPP doesn't define how the data
> plane is implemented in your device/OS. There's a data plane, it carries
> IP packets, but how you get those to your applications?
>
> After all, I'm not really proposing that we put oFono or something like
> it into the kernel - far from it! I'm only proposing that we kill the
> many various ways of creating and managing the necessary netdevs (VLANs,
> sysfs, rmnet, ...) from a piece of software like oFono (or libmbim or
> whatever else).
>
> Apart from CAIF and phonet, oFono doesn't even try to do this though,
> afaict, so I guess it relies on the default netdev created, or some out-
> of-band configuration is still needed?
As stated above, that is the problem of default netdev by drivers. Phonet and CAIF (and also our own support with 207.07 and RawIP) are superior in these regards. That is also why phones used these. The rest was then cellular data cards where the thinking was ATD+PPP.
Regards
Marcel