Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) client-ip=2620:137:e000::3:2;
Date:   Mon, 25 Sep 2023 23:18:16 +0800
From:   Dust Li <dust.li@linux.alibaba.com>
To:     Alexandra Winter <wintera@linux.ibm.com>,
        Wen Gu <guwen@linux.alibaba.com>, kgraul@linux.ibm.com,
        wenjia@linux.ibm.com, jaka@linux.ibm.com, davem@davemloft.net,
        edumazet@google.com, kuba@kernel.org, pabeni@redhat.com
Cc:     schnelle@linux.ibm.com, gbayer@linux.ibm.com, pasic@linux.ibm.com,
        alibuda@linux.alibaba.com, tonylu@linux.alibaba.com,
        linux-s390@vger.kernel.org, netdev@vger.kernel.org,
        linux-kernel@vger.kernel.org
Subject: Re: [PATCH net-next v4 09/18] net/smc: introduce SMC-D loopback
 device
Message-ID: <20230925151816.GC92403@linux.alibaba.com>
Reply-To: dust.li@linux.alibaba.com
References: <1695568613-125057-1-git-send-email-guwen@linux.alibaba.com>
 <1695568613-125057-10-git-send-email-guwen@linux.alibaba.com>
 <3febdf3e-e213-7acf-7dd4-75d177676c3e@linux.ibm.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <3febdf3e-e213-7acf-7dd4-75d177676c3e@linux.ibm.com>
Precedence: bulk

On Mon, Sep 25, 2023 at 01:50:22PM +0200, Alexandra Winter wrote:
>
>
>On 24.09.23 17:16, Wen Gu wrote:
>> This patch introduces a kind of loopback device for SMC-D. The device
>> is created when SMC module is loaded and destroyed when the SMC module
>> is unloaded. The loopback device is a kernel device used only by the
>> SMC module and is not restricted by net namespace, so it can be used
>> for local inter-process or inter-container communication.
>> 
>> Signed-off-by: Wen Gu <guwen@linux.alibaba.com>
>> ---
>>  net/smc/Kconfig        |  13 ++++
>>  net/smc/Makefile       |   2 +-
>>  net/smc/af_smc.c       |  12 +++-
>>  net/smc/smc_loopback.c | 165 +++++++++++++++++++++++++++++++++++++++++++++++++
>>  net/smc/smc_loopback.h |  33 ++++++++++
>>  5 files changed, 223 insertions(+), 2 deletions(-)
>>  create mode 100644 net/smc/smc_loopback.c
>>  create mode 100644 net/smc/smc_loopback.h
>
>
>Hello Wen Gu,
>
>thank you for adding the Kconfig, so the distributions can decide when to offer this feature.
>
>I propose you add some kind of runtime switch as well. Not every user who loads the SMC module
>may want to exploit smcd-loopback. Especially in native environements without containers.
>
>If no RoCE interfaces or no ISM interfaces exist, the respective handling is skipped in SMC.
>If loopback is always created unconditionally, there is no way to opt-out.

Hi Sandy,

After talking to Wen Gu offline, I think the real issue here might be
we don't have an abstract layer in SMC, something like net/core/dev.c

Without this, we cannot do:

1. Enable/disable those devices dynamically
   Currently, If we want to disable a SMC-R device to communicate with
   others, we need to refer to 'ip link set dev xxx down' to disable the
   netdevice, then Infiniband subsystem will notify SMC that the state of
   the IB device has changed. We cannot explicitly choose not to use some
   specific IB/RoCE devices without disable totally.
   If the loopback device need to support enable/disable itself, I
   think it might be better to enable this feature for all SMC devices.

2. Do statistics per device
   Now, we have to relay on IB/RoCE devices' hardware statistics to see
   how many packets/bytes we have sent through this device.

Both the above issues get worse when the IB/RoCE device is shared by SMC
and userspace RDMA applications. If SMC-R and userspace RDMA applications
run at the same time, we can't enable the device to run userspace RDMA
applications while block it from running SMC. For statistics, we cannot
tell how many packets/bytes were sent by SMC and how many were sent by
userspace RDMA applications.

So I think those are better to support in the SMC layer.

Best regards!
Dust