Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) client-ip=23.128.96.35;
Message-ID: <71d18598-1793-0c6c-7de6-f546befc47c1@salutedevices.com>
Date:   Thu, 30 Nov 2023 20:49:09 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.7.1
Subject: Re: [PATCH net-next v5 2/3] virtio/vsock: send credit update during
 setting SO_RCVLOWAT
Content-Language: en-US
To:     "Michael S. Tsirkin" <mst@redhat.com>
CC:     Stefano Garzarella <sgarzare@redhat.com>,
        Stefan Hajnoczi <stefanha@redhat.com>,
        "David S. Miller" <davem@davemloft.net>,
        Eric Dumazet <edumazet@google.com>,
        Jakub Kicinski <kuba@kernel.org>,
        Paolo Abeni <pabeni@redhat.com>,
        Jason Wang <jasowang@redhat.com>,
        Bobby Eshleman <bobby.eshleman@bytedance.com>,
        <kvm@vger.kernel.org>, <virtualization@lists.linux-foundation.org>,
        <netdev@vger.kernel.org>, <linux-kernel@vger.kernel.org>,
        <kernel@sberdevices.ru>, <oxffffaa@gmail.com>
References: <20231130130840.253733-1-avkrasnov@salutedevices.com>
 <20231130130840.253733-3-avkrasnov@salutedevices.com>
 <20231130084044-mutt-send-email-mst@kernel.org>
 <02de8982-ec4a-b3b2-e8e5-1bca28cfc01b@salutedevices.com>
 <20231130085445-mutt-send-email-mst@kernel.org>
 <pbkiwezwlf6dmogx7exur6tjrtcfzxyn7eqlehqxivqifbkojv@xlziiuzekon4>
 <b3fa2aaa-9fdc-30a2-4c87-53eb106900ee@salutedevices.com>
 <20231130123653-mutt-send-email-mst@kernel.org>
From:   Arseniy Krasnov <avkrasnov@salutedevices.com>
In-Reply-To: <20231130123653-mutt-send-email-mst@kernel.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 8bit
Precedence: bulk


On 30.11.2023 20:37, Michael S. Tsirkin wrote:
> On Thu, Nov 30, 2023 at 06:41:56PM +0300, Arseniy Krasnov wrote:
>>
>>
>> On 30.11.2023 17:11, Stefano Garzarella wrote:
>>> On Thu, Nov 30, 2023 at 08:58:58AM -0500, Michael S. Tsirkin wrote:
>>>> On Thu, Nov 30, 2023 at 04:43:34PM +0300, Arseniy Krasnov wrote:
>>>>>
>>>>>
>>>>> On 30.11.2023 16:42, Michael S. Tsirkin wrote:
>>>>>> On Thu, Nov 30, 2023 at 04:08:39PM +0300, Arseniy Krasnov wrote:
>>>>>>> Send credit update message when SO_RCVLOWAT is updated and it is bigger
>>>>>>> than number of bytes in rx queue. It is needed, because 'poll()' will
>>>>>>> wait until number of bytes in rx queue will be not smaller than
>>>>>>> SO_RCVLOWAT, so kick sender to send more data. Otherwise mutual hungup
>>>>>>> for tx/rx is possible: sender waits for free space and receiver is
>>>>>>> waiting data in 'poll()'.
>>>>>>>
>>>>>>> Signed-off-by: Arseniy Krasnov <avkrasnov@salutedevices.com>
>>>>>>> ---
>>>>>>>   Changelog:
>>>>>>>   v1 -> v2:
>>>>>>>    * Update commit message by removing 'This patch adds XXX' manner.
>>>>>>>    * Do not initialize 'send_update' variable - set it directly during
>>>>>>>      first usage.
>>>>>>>   v3 -> v4:
>>>>>>>    * Fit comment in 'virtio_transport_notify_set_rcvlowat()' to 80 chars.
>>>>>>>   v4 -> v5:
>>>>>>>    * Do not change callbacks order in transport structures.
>>>>>>>
>>>>>>>   drivers/vhost/vsock.c                   |  1 +
>>>>>>>   include/linux/virtio_vsock.h            |  1 +
>>>>>>>   net/vmw_vsock/virtio_transport.c        |  1 +
>>>>>>>   net/vmw_vsock/virtio_transport_common.c | 27 +++++++++++++++++++++++++
>>>>>>>   net/vmw_vsock/vsock_loopback.c          |  1 +
>>>>>>>   5 files changed, 31 insertions(+)
>>>>>>>
>>>>>>> diff --git a/drivers/vhost/vsock.c b/drivers/vhost/vsock.c
>>>>>>> index f75731396b7e..4146f80db8ac 100644
>>>>>>> --- a/drivers/vhost/vsock.c
>>>>>>> +++ b/drivers/vhost/vsock.c
>>>>>>> @@ -451,6 +451,7 @@ static struct virtio_transport vhost_transport = {
>>>>>>>           .notify_buffer_size       = virtio_transport_notify_buffer_size,
>>>>>>>
>>>>>>>           .read_skb = virtio_transport_read_skb,
>>>>>>> +        .notify_set_rcvlowat      = virtio_transport_notify_set_rcvlowat
>>>>>>>       },
>>>>>>>
>>>>>>>       .send_pkt = vhost_transport_send_pkt,
>>>>>>> diff --git a/include/linux/virtio_vsock.h b/include/linux/virtio_vsock.h
>>>>>>> index ebb3ce63d64d..c82089dee0c8 100644
>>>>>>> --- a/include/linux/virtio_vsock.h
>>>>>>> +++ b/include/linux/virtio_vsock.h
>>>>>>> @@ -256,4 +256,5 @@ void virtio_transport_put_credit(struct virtio_vsock_sock *vvs, u32 credit);
>>>>>>>   void virtio_transport_deliver_tap_pkt(struct sk_buff *skb);
>>>>>>>   int virtio_transport_purge_skbs(void *vsk, struct sk_buff_head *list);
>>>>>>>   int virtio_transport_read_skb(struct vsock_sock *vsk, skb_read_actor_t read_actor);
>>>>>>> +int virtio_transport_notify_set_rcvlowat(struct vsock_sock *vsk, int val);
>>>>>>>   #endif /* _LINUX_VIRTIO_VSOCK_H */
>>>>>>> diff --git a/net/vmw_vsock/virtio_transport.c b/net/vmw_vsock/virtio_transport.c
>>>>>>> index af5bab1acee1..8007593a3a93 100644
>>>>>>> --- a/net/vmw_vsock/virtio_transport.c
>>>>>>> +++ b/net/vmw_vsock/virtio_transport.c
>>>>>>> @@ -539,6 +539,7 @@ static struct virtio_transport virtio_transport = {
>>>>>>>           .notify_buffer_size       = virtio_transport_notify_buffer_size,
>>>>>>>
>>>>>>>           .read_skb = virtio_transport_read_skb,
>>>>>>> +        .notify_set_rcvlowat      = virtio_transport_notify_set_rcvlowat
>>>>>>>       },
>>>>>>>
>>>>>>>       .send_pkt = virtio_transport_send_pkt,
>>>>>>> diff --git a/net/vmw_vsock/virtio_transport_common.c b/net/vmw_vsock/virtio_transport_common.c
>>>>>>> index f6dc896bf44c..1cb556ad4597 100644
>>>>>>> --- a/net/vmw_vsock/virtio_transport_common.c
>>>>>>> +++ b/net/vmw_vsock/virtio_transport_common.c
>>>>>>> @@ -1684,6 +1684,33 @@ int virtio_transport_read_skb(struct vsock_sock *vsk, skb_read_actor_t recv_acto
>>>>>>>   }
>>>>>>>   EXPORT_SYMBOL_GPL(virtio_transport_read_skb);
>>>>>>>
>>>>>>> +int virtio_transport_notify_set_rcvlowat(struct vsock_sock *vsk, >> int val)
>>>>>>> +{
>>>>>>> +    struct virtio_vsock_sock *vvs = vsk->trans;
>>>>>>> +    bool send_update;
>>>>>>> +
>>>>>>> +    spin_lock_bh(&vvs->rx_lock);
>>>>>>> +
>>>>>>> +    /* If number of available bytes is less than new SO_RCVLOWAT value,
>>>>>>> +     * kick sender to send more data, because sender may sleep in >> its
>>>>>>> +     * 'send()' syscall waiting for enough space at our side.
>>>>>>> +     */
>>>>>>> +    send_update = vvs->rx_bytes < val;
>>>>>>> +
>>>>>>> +    spin_unlock_bh(&vvs->rx_lock);
>>>>>>> +
>>>>>>> +    if (send_update) {
>>>>>>> +        int err;
>>>>>>> +
>>>>>>> +        err = virtio_transport_send_credit_update(vsk);
>>>>>>> +        if (err < 0)
>>>>>>> +            return err;
>>>>>>> +    }
>>>>>>> +
>>>>>>> +    return 0;
>>>>>>> +}
>>>>>>
>>>>>>
>>>>>> I find it strange that this will send a credit update
>>>>>> even if nothing changed since this was called previously.
>>>>>> I'm not sure whether this is a problem protocol-wise,
>>>>>> but it certainly was not envisioned when the protocol was
>>>>>> built. WDYT?
>>>>>
>>>>> >From virtio spec I found:
>>>>>
>>>>> It is also valid to send a VIRTIO_VSOCK_OP_CREDIT_UPDATE packet without previously receiving a
>>>>> VIRTIO_VSOCK_OP_CREDIT_REQUEST packet. This allows communicating updates any time a change
>>>>> in buffer space occurs.
>>>>> So I guess there is no limitations to send such type of packet, e.g. it is not
>>>>> required to be a reply for some another packet. Please, correct me if im wrong.
>>>>>
>>>>> Thanks, Arseniy
>>>>
>>>>
>>>> Absolutely. My point was different - with this patch it is possible
>>>> that you are not adding any credits at all since the previous
>>>> VIRTIO_VSOCK_OP_CREDIT_UPDATE.
>>>
>>> I think the problem we're solving here is that since as an optimization we avoid sending the update for every byte we consume, but we put a threshold, then we make sure we update the peer.
>>>
>>> A credit update contains a snapshot and sending it the same as the previous one should not create any problem.
>>>
>>> My doubt now is that we only do this when we set RCVLOWAT , should we also do something when we consume bytes to avoid the optimization we have?
>>
>> @Michael, Stefano just reproduced problem during bytes reading, but there is already old fix for this, which we forget to merge:)
>> I think it must be included to this patchset.
>>
>> https://lore.kernel.org/netdev/f304eabe-d2ef-11b1-f115-6967632f0339@sberdevices.ru/
>>
>> Thanks, Arseniy
> 
> 
> I generally don't merge patches tagged as RFC.
> Repost without that tag?
> Also, it looks like a bugfix we need either way, no?

I'll repost it without RFC as part of this patchset, also i'll add test for it

Thanks, Arseniy

> 
>>>
>>> Stefano
>>>
>