Received-SPF: pass (google.com: domain of linux-kernel+bounces-131523-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Message-ID: <7c3733b7-a3f2-45ba-9b8d-880957ed5cdf@linux.alibaba.com>
Date: Thu, 4 Apr 2024 21:44:37 +0800
Precedence: bulk
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: [RFC PATCH net-next v5 05/11] net/smc: implement DMB-related
 operations of loopback-ism
To: Niklas Schnelle <schnelle@linux.ibm.com>,
 Gerd Bayer <gbayer@linux.ibm.com>, wintera@linux.ibm.com,
 twinkler@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com,
 agordeev@linux.ibm.com, davem@davemloft.net, edumazet@google.com,
 kuba@kernel.org, pabeni@redhat.com, wenjia@linux.ibm.com, jaka@linux.ibm.com
Cc: borntraeger@linux.ibm.com, svens@linux.ibm.com,
 alibuda@linux.alibaba.com, tonylu@linux.alibaba.com,
 linux-kernel@vger.kernel.org, linux-s390@vger.kernel.org,
 netdev@vger.kernel.org
References: <20240324135522.108564-1-guwen@linux.alibaba.com>
 <20240324135522.108564-6-guwen@linux.alibaba.com>
 <9a17268d4046f99b30f3620079b5749a9ddc5cd9.camel@linux.ibm.com>
 <92b0c4b1-4844-4adf-a15a-a9323fb859e1@linux.alibaba.com>
 <6d3cfa04c9826a24f0ad8d401940af3ad02a67bc.camel@linux.ibm.com>
From: Wen Gu <guwen@linux.alibaba.com>
In-Reply-To: <6d3cfa04c9826a24f0ad8d401940af3ad02a67bc.camel@linux.ibm.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit


On 2024/4/4 19:27, Niklas Schnelle wrote:
> On Thu, 2024-04-04 at 18:20 +0800, Wen Gu wrote:
>>
>> On 2024/4/4 01:20, Gerd Bayer wrote:
>>> On Sun, 2024-03-24 at 21:55 +0800, Wen Gu wrote:
>>>
>>> When I instrumented this to see, why I still see tons of my other
>>> temporary instrumentation messages from the "ism" driver, I found that
>>> in my setup loopback-ism is used rather infrequently.
>>>
>>> I suspect this is due to how the SMC proposals are constructed in
>>> net/smc/af_smc.c and net/smc/smc_pnet.c - and later evaluated in
>>> smc_check_ism_v2_match() - where there is a first-come-first-serve
>>> selection.
>>>
>>> I wonder if one should change that to favour loopback-ism over "real"
>>> ISM devices - and how this could be achieved elegantly.
>>>
>>> Just some food for thought... Probably little you can do on x86.
>>>
>>
>> Yes, it is about the priority of available ISM devices, and now it
>> is decided by their order in the smcd_dev_list. The later registered
>> ISMv2 devices(without pnetid) will be added to the beginning of the
>> list (see smcd_register_dev()). So there is a probability that
>> loopback-ism will not be ranked first, since it is added into list
>> earlier during smc_init().
>>
>> If we have the runtime switch of loopback-ism, we can re-active the
>> loopback-ism, that make it be re-added into the beginning of the dev
>> list and be chosen first. Or a new netlink command to adjust the slot
>> order of available ISM devices in the list. As we discussed before,
>> that could be tasks in stage 1 or stage 2.
>>
>> Thanks!
> 
> Maybe when adding the ISM devices we could instead make sure that all
> ISM devices are added after loopback and loopback is added in the
> beginning. I think loopback should always be preferred and would
> consider it a bug if it isn't faster too. Between virtio-ism and ISM it
> may be less clear so maybe for stage 2 we would want a priority setting
> and then insert ordered by priority. Thoughts?
I have no objection. If we all agree, I will keep it at the beginning of the list.

Thanks!