Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <f916f306-acff-3537-1bcc-9f19c4794e81@linux.ibm.com>
Date:   Wed, 29 Jun 2022 22:29:47 +0200
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0)
 Gecko/20100101 Thunderbird/91.10.0
Subject: Re: [PATCH net-next v2] net/smc: align the connect behaviour with TCP
To:     guangguan.wang@linux.alibaba.com
Cc:     Karsten Graul <kgraul@linux.ibm.com>, liuyacan@corp.netease.com,
        davem@davemloft.net, kuba@kernel.org, linux-kernel@vger.kernel.org,
        linux-s390@vger.kernel.org, netdev@vger.kernel.org,
        pabeni@redhat.com
References: <26d43c65-1f23-5b83-6377-3327854387c4@linux.ibm.com>
 <20220524125725.951315-1-liuyacan@corp.netease.com>
 <3bb9366d-f271-a603-a280-b70ae2d59c00@linux.ibm.com>
From:   Wenjia Zhang <wenjia@linux.ibm.com>
In-Reply-To: <3bb9366d-f271-a603-a280-b70ae2d59c00@linux.ibm.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk


On 24.05.22 15:05, Karsten Graul wrote:
> On 24/05/2022 14:57, liuyacan@corp.netease.com wrote:
>>>>
>>>>
>>>> On 2022/5/23 20:24, Karsten Graul wrote:
>>>>> On 13/05/2022 04:24, Guangguan Wang wrote:
>>>>>> Connect with O_NONBLOCK will not be completed immediately
>>>>>> and returns -EINPROGRESS. It is possible to use selector/poll
>>>>>> for completion by selecting the socket for writing. After select
>>>>>> indicates writability, a second connect function call will return
>>>>>> 0 to indicate connected successfully as TCP does, but smc returns
>>>>>> -EISCONN. Use socket state for smc to indicate connect state, which
>>>>>> can help smc aligning the connect behaviour with TCP.
>>>>>>
>>>>>> Signed-off-by: Guangguan Wang <guangguan.wang@linux.alibaba.com>
>>>>>> Acked-by: Karsten Graul <kgraul@linux.ibm.com>
>>>>>> ---
>>>>>>   net/smc/af_smc.c | 50 ++++++++++++++++++++++++++++++++++++++++++++----
>>>>>>   1 file changed, 46 insertions(+), 4 deletions(-)
>>>>>>
>>>>>> diff --git a/net/smc/af_smc.c b/net/smc/af_smc.c
>>>>>> index fce16b9d6e1a..5f70642a8044 100644
>>>>>> --- a/net/smc/af_smc.c
>>>>>> +++ b/net/smc/af_smc.c
>>>>>> @@ -1544,9 +1544,29 @@ static int smc_connect(struct socket *sock, struct sockaddr *addr,
>>>>>>   		goto out_err;
>>>>>>   
>>>>>>   	lock_sock(sk);
>>>>>> +	switch (sock->state) {
>>>>>> +	default:
>>>>>> +		rc = -EINVAL;
>>>>>> +		goto out;
>>>>>> +	case SS_CONNECTED:
>>>>>> +		rc = sk->sk_state == SMC_ACTIVE ? -EISCONN : -EINVAL;
>>>>>> +		goto out;
>>>>>> +	case SS_CONNECTING:
>>>>>> +		if (sk->sk_state == SMC_ACTIVE)
>>>>>> +			goto connected;
>>>>>
>>>>> I stumbled over this when thinking about the fallback processing. If for whatever reason
>>>>> fallback==true during smc_connect(), the "if (smc->use_fallback)" below would set sock->state
>>>>> to e.g. SS_CONNECTED. But in the fallback case sk_state keeps SMC_INIT. So during the next call
>>>>> the SS_CONNECTING case above would break because sk_state in NOT SMC_ACTIVE, and we would end
>>>>> up calling kernel_connect() again. Which seems to be no problem when kernel_connect() returns
>>>>> -EISCONN and we return this to the caller. But is this how it should work, or does it work by chance?
>>>>>
>>>>
>>>> Since the sk_state keeps SMC_INIT and does not correctly indicate the state of clcsock, it should end
>>>> up calling kernel_connect() again to get the actual connection state of clcsock.
>>>>
>>>> And I'm sorry there is a problem that if sock->state==SS_CONNECTED and sk_state==SMC_INIT, further call
>>>> of smc_connect will return -EINVAL where -EISCONN is preferred.
>>>> The steps to reproduce:
>>>> 1）switch fallback before connect, such as setsockopt TCP_FASTOPEN
>>>> 2）connect with noblocking and returns -EINPROGRESS. (sock->state changes to SS_CONNECTING)
>>>> 3) end up calling connect with noblocking again and returns 0. (kernel_connect() returns 0 and sock->state changes to
>>>>     SS_CONNECTED but sk->sk_state stays SMC_INIT)
>>>> 4) call connect again, maybe by mistake, will return -EINVAL, but -EISCONN is preferred.
>>>>
>>>> What do you think about if we synchronize the sk_state to SMC_ACTIVE instead of keeping SMC_INIT when clcsock
>>>> connected successfully in fallback case described above.
>>>>
>>>> ...
>>>
>>> I start thinking that the fix in 86434744 introduced a problem. Before that fix a connect with
>>> fallback always reached __smc_connect() and on top of that function in case of fallback
>>> smc_connect_fallback() is called, which itself sets sk_state to SMC_ACTIVE.
>>>
>>> 86434744 removed that code path and I wonder what it actually fixed, because at this time the
>>> fallback check in __smc_connect() was already present.
>>>
>>> Without that "goto out;" the state would be set correctly in smc_connect_fallback(), and the
>>> socket close processing would work as expected.
>>
>> I think it is OK without that "goto out;". And I guess the purpose of "goto out;" is to avoid calling __smc_connect(),
>> because it is impossible to establish an rdma channel at this time.
> 
> Yes that was the purpose, but this disabled all the extra processing that should be done
> for fallback sockets during connect().
> 
Since Karsten's suggestion, we didn't hear from you any more. We just 
want to know:

- What do you think about the commit (86434744)? Could it be the trigger 
of the problem you met?

- Have you ever tried to just remove the following lines from 
smc_connection(), and check if your scenario could run correctly?

       if (smc->use_fallback)
               goto out;

In our opinion, we don't see the necessity of the patch, if partly 
reverting the commit (86434744) could solve the problem.