Hi-
I've tested this with two kinds of systems:
1. A system with no physical RDMA devices and no start-up
scripts to load these modules
2. A system with physical RDMA devices and with the start-up
scripts that load xprtrdma/svcrdma
In both cases, after doing an "rmmod rpcrdma", I can mount
a "proto=rdma" mount or start the NFS server, and the module
gets reloaded automatically.
I therefore believe it is safe to delete the code in the
rdma-core start-up scripts that manually load RPC-related
RDMA support. Either the sunrpc.ko module does this, or NFS
user space handles it. There's no need for the rdma-core
scripting.
--
Chuck Lever
On 20/05/2024 21:05, Chuck Lever III wrote:
> Hi-
>
> I've tested this with two kinds of systems:
>
> 1. A system with no physical RDMA devices and no start-up
> scripts to load these modules
>
> 2. A system with physical RDMA devices and with the start-up
> scripts that load xprtrdma/svcrdma
>
> In both cases, after doing an "rmmod rpcrdma", I can mount
> a "proto=rdma" mount or start the NFS server, and the module
> gets reloaded automatically.
>
> I therefore believe it is safe to delete the code in the
> rdma-core start-up scripts that manually load RPC-related
> RDMA support. Either the sunrpc.ko module does this, or NFS
> user space handles it. There's no need for the rdma-core
> scripting.
I didn't know that rdma-core does this... it really shouldn't, the
mount should (and does) handle it.
I also see that srp(t) and iser(t) are loaded too.. IIRC these are
loaded by their userspace counterparts as well (or at least they
should).
On Tue, May 21, 2024 at 12:04:02PM +0300, Sagi Grimberg wrote:
>
>
> On 20/05/2024 21:05, Chuck Lever III wrote:
> > Hi-
> >
> > I've tested this with two kinds of systems:
> >
> > 1. A system with no physical RDMA devices and no start-up
> > scripts to load these modules
> >
> > 2. A system with physical RDMA devices and with the start-up
> > scripts that load xprtrdma/svcrdma
> >
> > In both cases, after doing an "rmmod rpcrdma", I can mount
> > a "proto=rdma" mount or start the NFS server, and the module
> > gets reloaded automatically.
> >
> > I therefore believe it is safe to delete the code in the
> > rdma-core start-up scripts that manually load RPC-related
> > RDMA support. Either the sunrpc.ko module does this, or NFS
> > user space handles it. There's no need for the rdma-core
> > scripting.
>
> I didn't know that rdma-core does this... it really shouldn't, the
> mount should (and does) handle it.
This is new, it didn't used to do this
> I also see that srp(t) and iser(t) are loaded too.. IIRC these are
> loaded by their userspace counterparts as well (or at least they
> should).
And AFIAK, these don't have a way to autoload at all. autoload
requires the kernel to call request_module..
ipoib is also a problem, we don't have a way to autoload it either
Jason
On 21/05/2024 15:43, Jason Gunthorpe wrote:
> On Tue, May 21, 2024 at 12:04:02PM +0300, Sagi Grimberg wrote:
>>
>> On 20/05/2024 21:05, Chuck Lever III wrote:
>>> Hi-
>>>
>>> I've tested this with two kinds of systems:
>>>
>>> 1. A system with no physical RDMA devices and no start-up
>>> scripts to load these modules
>>>
>>> 2. A system with physical RDMA devices and with the start-up
>>> scripts that load xprtrdma/svcrdma
>>>
>>> In both cases, after doing an "rmmod rpcrdma", I can mount
>>> a "proto=rdma" mount or start the NFS server, and the module
>>> gets reloaded automatically.
>>>
>>> I therefore believe it is safe to delete the code in the
>>> rdma-core start-up scripts that manually load RPC-related
>>> RDMA support. Either the sunrpc.ko module does this, or NFS
>>> user space handles it. There's no need for the rdma-core
>>> scripting.
>> I didn't know that rdma-core does this... it really shouldn't, the
>> mount should (and does) handle it.
> This is new, it didn't used to do this
>
>> I also see that srp(t) and iser(t) are loaded too.. IIRC these are
>> loaded by their userspace counterparts as well (or at least they
>> should).
> And AFIAK, these don't have a way to autoload at all. autoload
> requires the kernel to call request_module..
nvme/nvmet/isert are requested by the kernel. iser is loaded by iscsiadm.
IIRC srp had a userspace daemon loading it.
On Tue, May 21, 2024 at 04:05:05PM +0300, Sagi Grimberg wrote:
>
>
> On 21/05/2024 15:43, Jason Gunthorpe wrote:
> > On Tue, May 21, 2024 at 12:04:02PM +0300, Sagi Grimberg wrote:
> > >
> > > On 20/05/2024 21:05, Chuck Lever III wrote:
> > > > Hi-
> > > >
> > > > I've tested this with two kinds of systems:
> > > >
> > > > 1. A system with no physical RDMA devices and no start-up
> > > > scripts to load these modules
> > > >
> > > > 2. A system with physical RDMA devices and with the start-up
> > > > scripts that load xprtrdma/svcrdma
> > > >
> > > > In both cases, after doing an "rmmod rpcrdma", I can mount
> > > > a "proto=rdma" mount or start the NFS server, and the module
> > > > gets reloaded automatically.
> > > >
> > > > I therefore believe it is safe to delete the code in the
> > > > rdma-core start-up scripts that manually load RPC-related
> > > > RDMA support. Either the sunrpc.ko module does this, or NFS
> > > > user space handles it. There's no need for the rdma-core
> > > > scripting.
> > > I didn't know that rdma-core does this... it really shouldn't, the
> > > mount should (and does) handle it.
> > This is new, it didn't used to do this
> >
> > > I also see that srp(t) and iser(t) are loaded too.. IIRC these are
> > > loaded by their userspace counterparts as well (or at least they
> > > should).
> > And AFIAK, these don't have a way to autoload at all. autoload
> > requires the kernel to call request_module..
>
> nvme/nvmet/isert are requested by the kernel.
How? What is the interface to trigger request_module?
> iser is loaded by iscsiadm.
Yuk :\
> IIRC srp had a userspace daemon loading it.
srp-daemon requires it already loaded AFAIK
Jason
On 21/05/2024 16:37, Jason Gunthorpe wrote:
> On Tue, May 21, 2024 at 04:05:05PM +0300, Sagi Grimberg wrote:
>>
>> On 21/05/2024 15:43, Jason Gunthorpe wrote:
>>> On Tue, May 21, 2024 at 12:04:02PM +0300, Sagi Grimberg wrote:
>>>> On 20/05/2024 21:05, Chuck Lever III wrote:
>>>>> Hi-
>>>>>
>>>>> I've tested this with two kinds of systems:
>>>>>
>>>>> 1. A system with no physical RDMA devices and no start-up
>>>>> scripts to load these modules
>>>>>
>>>>> 2. A system with physical RDMA devices and with the start-up
>>>>> scripts that load xprtrdma/svcrdma
>>>>>
>>>>> In both cases, after doing an "rmmod rpcrdma", I can mount
>>>>> a "proto=rdma" mount or start the NFS server, and the module
>>>>> gets reloaded automatically.
>>>>>
>>>>> I therefore believe it is safe to delete the code in the
>>>>> rdma-core start-up scripts that manually load RPC-related
>>>>> RDMA support. Either the sunrpc.ko module does this, or NFS
>>>>> user space handles it. There's no need for the rdma-core
>>>>> scripting.
>>>> I didn't know that rdma-core does this... it really shouldn't, the
>>>> mount should (and does) handle it.
>>> This is new, it didn't used to do this
>>>
>>>> I also see that srp(t) and iser(t) are loaded too.. IIRC these are
>>>> loaded by their userspace counterparts as well (or at least they
>>>> should).
>>> And AFIAK, these don't have a way to autoload at all. autoload
>>> requires the kernel to call request_module..
>> nvme/nvmet/isert are requested by the kernel.
> How? What is the interface to trigger request_module?
On the host, writing to the nvme-fabrics misc device a comma-separated
connection string
contains a transport string, which triggers the corresponding module to
be requested.
On the target-side, configuring a port transport, also triggers in a
similar way...
>
>> iser is loaded by iscsiadm.
> Yuk :\
Different strokes for different folks...
>
>> IIRC srp had a userspace daemon loading it.
> srp-daemon requires it already loaded AFAIK
OK, so that one is the exception...
On Tue, May 21, 2024 at 05:12:23PM +0300, Sagi Grimberg wrote:
> > > > > I also see that srp(t) and iser(t) are loaded too.. IIRC these are
> > > > > loaded by their userspace counterparts as well (or at least they
> > > > > should).
> > > > And AFIAK, these don't have a way to autoload at all. autoload
> > > > requires the kernel to call request_module..
> > > nvme/nvmet/isert are requested by the kernel.
> > How? What is the interface to trigger request_module?
>
> On the host, writing to the nvme-fabrics misc device a comma-separated
> connection string
> contains a transport string, which triggers the corresponding module to be
> requested.
But how did nvme-fabrics even get loaded to write to it's config fs in
the first place?
Jason
On 21/05/2024 18:23, Jason Gunthorpe wrote:
> On Tue, May 21, 2024 at 05:12:23PM +0300, Sagi Grimberg wrote:
>>>>>> I also see that srp(t) and iser(t) are loaded too.. IIRC these are
>>>>>> loaded by their userspace counterparts as well (or at least they
>>>>>> should).
>>>>> And AFIAK, these don't have a way to autoload at all. autoload
>>>>> requires the kernel to call request_module..
>>>> nvme/nvmet/isert are requested by the kernel.
>>> How? What is the interface to trigger request_module?
>> On the host, writing to the nvme-fabrics misc device a comma-separated
>> connection string
>> contains a transport string, which triggers the corresponding module to be
>> requested.
> But how did nvme-fabrics even get loaded to write to it's config fs in
> the first place?
Something (/etc/modules-load?) loaded it intentionally.
That something knows about a concrete intention to use nvme though...
On Tue, May 21, 2024 at 07:10:53PM +0300, Sagi Grimberg wrote:
>
>
> On 21/05/2024 18:23, Jason Gunthorpe wrote:
> > On Tue, May 21, 2024 at 05:12:23PM +0300, Sagi Grimberg wrote:
> > > > > > > I also see that srp(t) and iser(t) are loaded too.. IIRC these are
> > > > > > > loaded by their userspace counterparts as well (or at least they
> > > > > > > should).
> > > > > > And AFIAK, these don't have a way to autoload at all. autoload
> > > > > > requires the kernel to call request_module..
> > > > > nvme/nvmet/isert are requested by the kernel.
> > > > How? What is the interface to trigger request_module?
> > > On the host, writing to the nvme-fabrics misc device a comma-separated
> > > connection string
> > > contains a transport string, which triggers the corresponding module to be
> > > requested.
> > But how did nvme-fabrics even get loaded to write to it's config fs in
> > the first place?
>
> Something (/etc/modules-load?) loaded it intentionally.
> That something knows about a concrete intention to use nvme though...
This mechanism we are talking about is an add-on to /etc/modules-load
that only executes if rdma HW is present.
This is why it is a good place to load nvme-fabrics stuff, if you
don't have rdma HW then you know you don't need it.
Autoloading is the version where you do 'mount -tnfs -o=rdma' and the
kernel automatically request_module's nfs and then nfs-rdma based only
on the command line options.
I'm not sure this is even possible with configfs as the directories
you need to write into don't even exist until the module(s) are
loaded, right?
Jason
On 21/05/2024 19:37, Jason Gunthorpe wrote:
> On Tue, May 21, 2024 at 07:10:53PM +0300, Sagi Grimberg wrote:
>>
>> On 21/05/2024 18:23, Jason Gunthorpe wrote:
>>> On Tue, May 21, 2024 at 05:12:23PM +0300, Sagi Grimberg wrote:
>>>>>>>> I also see that srp(t) and iser(t) are loaded too.. IIRC these are
>>>>>>>> loaded by their userspace counterparts as well (or at least they
>>>>>>>> should).
>>>>>>> And AFIAK, these don't have a way to autoload at all. autoload
>>>>>>> requires the kernel to call request_module..
>>>>>> nvme/nvmet/isert are requested by the kernel.
>>>>> How? What is the interface to trigger request_module?
>>>> On the host, writing to the nvme-fabrics misc device a comma-separated
>>>> connection string
>>>> contains a transport string, which triggers the corresponding module to be
>>>> requested.
>>> But how did nvme-fabrics even get loaded to write to it's config fs in
>>> the first place?
>> Something (/etc/modules-load?) loaded it intentionally.
>> That something knows about a concrete intention to use nvme though...
> This mechanism we are talking about is an add-on to /etc/modules-load
> that only executes if rdma HW is present.
Still does not mean you want to use all the ulps though...
>
> This is why it is a good place to load nvme-fabrics stuff, if you
> don't have rdma HW then you know you don't need it.
Do I want to autoload nvme-fabrics if I have a nvme device? do I want
autoload nvme-tcp if I have an ethernet nic? maybe wlan nic is also a
sufficient reason?
I just don't see why the presence of an rdma device dictates that all
the ulps
autoload. Does rxe/siw count as rdma HW?
>
> Autoloading is the version where you do 'mount -tnfs -o=rdma' and the
> kernel automatically request_module's nfs and then nfs-rdma based only
> on the command line options.
>
> I'm not sure this is even possible with configfs as the directories
> you need to write into don't even exist until the module(s) are
> loaded, right?
Right. The entry-point of the subsystem needs to be loaded (nvmet is
loaded by nvmetcli),
the individual transports/drivers are auto-selected.
On Tue, May 21, 2024 at 11:30:00PM +0300, Sagi Grimberg wrote:
> I just don't see why the presence of an rdma device dictates that
> all the ulps autoload. Does rxe/siw count as rdma HW?
It doesn't do all of them, just the ones the distro decides it wants
to do, usually for boot volumes. It is a weird historical thing :|
Jason
在 2024/5/21 22:30, Sagi Grimberg 写道:
>
>
> On 21/05/2024 19:37, Jason Gunthorpe wrote:
>> On Tue, May 21, 2024 at 07:10:53PM +0300, Sagi Grimberg wrote:
>>>
>>> On 21/05/2024 18:23, Jason Gunthorpe wrote:
>>>> On Tue, May 21, 2024 at 05:12:23PM +0300, Sagi Grimberg wrote:
>>>>>>>>> I also see that srp(t) and iser(t) are loaded too.. IIRC these are
>>>>>>>>> loaded by their userspace counterparts as well (or at least they
>>>>>>>>> should).
>>>>>>>> And AFIAK, these don't have a way to autoload at all. autoload
>>>>>>>> requires the kernel to call request_module..
>>>>>>> nvme/nvmet/isert are requested by the kernel.
>>>>>> How? What is the interface to trigger request_module?
>>>>> On the host, writing to the nvme-fabrics misc device a comma-separated
>>>>> connection string
>>>>> contains a transport string, which triggers the corresponding
>>>>> module to be
>>>>> requested.
>>>> But how did nvme-fabrics even get loaded to write to it's config fs in
>>>> the first place?
>>> Something (/etc/modules-load?) loaded it intentionally.
>>> That something knows about a concrete intention to use nvme though...
>> This mechanism we are talking about is an add-on to /etc/modules-load
>> that only executes if rdma HW is present.
>
> Still does not mean you want to use all the ulps though...
>
>>
>> This is why it is a good place to load nvme-fabrics stuff, if you
>> don't have rdma HW then you know you don't need it.
>
> Do I want to autoload nvme-fabrics if I have a nvme device? do I want
> autoload nvme-tcp if I have an ethernet nic? maybe wlan nic is also a
> sufficient reason?
>
> I just don't see why the presence of an rdma device dictates that all
> the ulps
> autoload. Does rxe/siw count as rdma HW?
RXE/SIW can be auto loaded with the command "rdma link ...".
And some kernel modules, for example ib_core.ko, udp_tunnel.ko,
ip6_udp_tunnel.ko and ib_uverbs.ko, are also auto-loaded when rxe/siw
kernel modules are loaded.
RXE/SIW are emulation RDMA kernel drivers. They are based on NIC HW.
Normally all the NICs can support RXE/SIW RDMA drivers because RXE/SIW
do not require additional NIC features.
To now almost all the NIC HW can support RXE/SIW, even some virtual NICs
can also support RXE/SIW, for example, bonding, TUN and veth.
Zhu Yanjun
>
>>
>> Autoloading is the version where you do 'mount -tnfs -o=rdma' and the
>> kernel automatically request_module's nfs and then nfs-rdma based only
>> on the command line options.
>>
>> I'm not sure this is even possible with configfs as the directories
>> you need to write into don't even exist until the module(s) are
>> loaded, right?
>
> Right. The entry-point of the subsystem needs to be loaded (nvmet is
> loaded by nvmetcli),
> the individual transports/drivers are auto-selected.
On 22/05/2024 2:29, Jason Gunthorpe wrote:
> On Tue, May 21, 2024 at 11:30:00PM +0300, Sagi Grimberg wrote:
>
>> I just don't see why the presence of an rdma device dictates that
>> all the ulps autoload. Does rxe/siw count as rdma HW?
> It doesn't do all of them, just the ones the distro decides it wants
> to do, usually for boot volumes. It is a weird historical thing :|
Well, as mentioned, I find it odd. For the same reason why
all tcp flavors of the ulps discussed in this thread don't autoload if
the machine
has a nic (which means pretty much every machine).
Having said that, I don't have the same PoV as distro folks...