Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) client-ip=23.128.96.33;
Date:   Fri, 1 Dec 2023 09:27:47 +0100
From:   Stefano Garzarella <sgarzare@redhat.com>
To:     "Michael S. Tsirkin" <mst@redhat.com>
Cc:     Arseniy Krasnov <avkrasnov@salutedevices.com>,
        Stefan Hajnoczi <stefanha@redhat.com>,
        "David S. Miller" <davem@davemloft.net>,
        Eric Dumazet <edumazet@google.com>,
        Jakub Kicinski <kuba@kernel.org>,
        Paolo Abeni <pabeni@redhat.com>,
        Jason Wang <jasowang@redhat.com>,
        Bobby Eshleman <bobby.eshleman@bytedance.com>,
        kvm@vger.kernel.org, virtualization@lists.linux-foundation.org,
        netdev@vger.kernel.org, linux-kernel@vger.kernel.org,
        kernel@sberdevices.ru, oxffffaa@gmail.com
Subject: Re: [PATCH net-next v5 2/3] virtio/vsock: send credit update during
 setting SO_RCVLOWAT
Message-ID: <smu77vmxw3ki36xhqnhtvujwswvkg5gkfwnt4vr5bnwljclseh@inbewbwkcqxs>
References: <20231130130840.253733-1-avkrasnov@salutedevices.com>
 <20231130130840.253733-3-avkrasnov@salutedevices.com>
 <20231130084044-mutt-send-email-mst@kernel.org>
 <02de8982-ec4a-b3b2-e8e5-1bca28cfc01b@salutedevices.com>
 <20231130085445-mutt-send-email-mst@kernel.org>
 <pbkiwezwlf6dmogx7exur6tjrtcfzxyn7eqlehqxivqifbkojv@xlziiuzekon4>
 <20231130123815-mutt-send-email-mst@kernel.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Disposition: inline
In-Reply-To: <20231130123815-mutt-send-email-mst@kernel.org>
Precedence: bulk

On Thu, Nov 30, 2023 at 12:40:43PM -0500, Michael S. Tsirkin wrote:
>On Thu, Nov 30, 2023 at 03:11:19PM +0100, Stefano Garzarella wrote:
>> On Thu, Nov 30, 2023 at 08:58:58AM -0500, Michael S. Tsirkin wrote:
>> > On Thu, Nov 30, 2023 at 04:43:34PM +0300, Arseniy Krasnov wrote:
>> > >
>> > >
>> > > On 30.11.2023 16:42, Michael S. Tsirkin wrote:
>> > > > On Thu, Nov 30, 2023 at 04:08:39PM +0300, Arseniy Krasnov wrote:
>> > > >> Send credit update message when SO_RCVLOWAT is updated and it is bigger
>> > > >> than number of bytes in rx queue. It is needed, because 'poll()' will
>> > > >> wait until number of bytes in rx queue will be not smaller than
>> > > >> SO_RCVLOWAT, so kick sender to send more data. Otherwise mutual hungup
>> > > >> for tx/rx is possible: sender waits for free space and receiver is
>> > > >> waiting data in 'poll()'.
>> > > >>
>> > > >> Signed-off-by: Arseniy Krasnov <avkrasnov@salutedevices.com>
>> > > >> ---
>> > > >>  Changelog:
>> > > >>  v1 -> v2:
>> > > >>   * Update commit message by removing 'This patch adds XXX' manner.
>> > > >>   * Do not initialize 'send_update' variable - set it directly during
>> > > >>     first usage.
>> > > >>  v3 -> v4:
>> > > >>   * Fit comment in 'virtio_transport_notify_set_rcvlowat()' to 80 chars.
>> > > >>  v4 -> v5:
>> > > >>   * Do not change callbacks order in transport structures.
>> > > >>
>> > > >>  drivers/vhost/vsock.c                   |  1 +
>> > > >>  include/linux/virtio_vsock.h            |  1 +
>> > > >>  net/vmw_vsock/virtio_transport.c        |  1 +
>> > > >>  net/vmw_vsock/virtio_transport_common.c | 27 +++++++++++++++++++++++++
>> > > >>  net/vmw_vsock/vsock_loopback.c          |  1 +
>> > > >>  5 files changed, 31 insertions(+)
>> > > >>
>> > > >> diff --git a/drivers/vhost/vsock.c b/drivers/vhost/vsock.c
>> > > >> index f75731396b7e..4146f80db8ac 100644
>> > > >> --- a/drivers/vhost/vsock.c
>> > > >> +++ b/drivers/vhost/vsock.c
>> > > >> @@ -451,6 +451,7 @@ static struct virtio_transport vhost_transport = {
>> > > >>  		.notify_buffer_size       = virtio_transport_notify_buffer_size,
>> > > >>
>> > > >>  		.read_skb = virtio_transport_read_skb,
>> > > >> +		.notify_set_rcvlowat      = virtio_transport_notify_set_rcvlowat
>> > > >>  	},
>> > > >>
>> > > >>  	.send_pkt = vhost_transport_send_pkt,
>> > > >> diff --git a/include/linux/virtio_vsock.h b/include/linux/virtio_vsock.h
>> > > >> index ebb3ce63d64d..c82089dee0c8 100644
>> > > >> --- a/include/linux/virtio_vsock.h
>> > > >> +++ b/include/linux/virtio_vsock.h
>> > > >> @@ -256,4 +256,5 @@ void virtio_transport_put_credit(struct virtio_vsock_sock *vvs, u32 credit);
>> > > >>  void virtio_transport_deliver_tap_pkt(struct sk_buff *skb);
>> > > >>  int virtio_transport_purge_skbs(void *vsk, struct sk_buff_head *list);
>> > > >>  int virtio_transport_read_skb(struct vsock_sock *vsk, skb_read_actor_t read_actor);
>> > > >> +int virtio_transport_notify_set_rcvlowat(struct vsock_sock *vsk, int val);
>> > > >>  #endif /* _LINUX_VIRTIO_VSOCK_H */
>> > > >> diff --git a/net/vmw_vsock/virtio_transport.c b/net/vmw_vsock/virtio_transport.c
>> > > >> index af5bab1acee1..8007593a3a93 100644
>> > > >> --- a/net/vmw_vsock/virtio_transport.c
>> > > >> +++ b/net/vmw_vsock/virtio_transport.c
>> > > >> @@ -539,6 +539,7 @@ static struct virtio_transport virtio_transport = {
>> > > >>  		.notify_buffer_size       = virtio_transport_notify_buffer_size,
>> > > >>
>> > > >>  		.read_skb = virtio_transport_read_skb,
>> > > >> +		.notify_set_rcvlowat      = virtio_transport_notify_set_rcvlowat
>> > > >>  	},
>> > > >>
>> > > >>  	.send_pkt = virtio_transport_send_pkt,
>> > > >> diff --git a/net/vmw_vsock/virtio_transport_common.c b/net/vmw_vsock/virtio_transport_common.c
>> > > >> index f6dc896bf44c..1cb556ad4597 100644
>> > > >> --- a/net/vmw_vsock/virtio_transport_common.c
>> > > >> +++ b/net/vmw_vsock/virtio_transport_common.c
>> > > >> @@ -1684,6 +1684,33 @@ int virtio_transport_read_skb(struct vsock_sock *vsk, skb_read_actor_t recv_acto
>> > > >>  }
>> > > >>  EXPORT_SYMBOL_GPL(virtio_transport_read_skb);
>> > > >>
>> > > >> +int virtio_transport_notify_set_rcvlowat(struct vsock_sock *vsk,
>> > > >> int val)
>> > > >> +{
>> > > >> +	struct virtio_vsock_sock *vvs = vsk->trans;
>> > > >> +	bool send_update;
>> > > >> +
>> > > >> +	spin_lock_bh(&vvs->rx_lock);
>> > > >> +
>> > > >> +	/* If number of available bytes is less than new SO_RCVLOWAT value,
>> > > >> +	 * kick sender to send more data, because sender may sleep in
>> > > >> its
>> > > >> +	 * 'send()' syscall waiting for enough space at our side.
>> > > >> +	 */
>> > > >> +	send_update = vvs->rx_bytes < val;
>> > > >> +
>> > > >> +	spin_unlock_bh(&vvs->rx_lock);
>> > > >> +
>> > > >> +	if (send_update) {
>> > > >> +		int err;
>> > > >> +
>> > > >> +		err = virtio_transport_send_credit_update(vsk);
>> > > >> +		if (err < 0)
>> > > >> +			return err;
>> > > >> +	}
>> > > >> +
>> > > >> +	return 0;
>> > > >> +}
>> > > >
>> > > >
>> > > > I find it strange that this will send a credit update
>> > > > even if nothing changed since this was called previously.
>> > > > I'm not sure whether this is a problem protocol-wise,
>> > > > but it certainly was not envisioned when the protocol was
>> > > > built. WDYT?
>> > >
>> > > >From virtio spec I found:
>> > >
>> > > It is also valid to send a VIRTIO_VSOCK_OP_CREDIT_UPDATE packet without previously receiving a
>> > > VIRTIO_VSOCK_OP_CREDIT_REQUEST packet. This allows communicating updates any time a change
>> > > in buffer space occurs.
>> > > So I guess there is no limitations to send such type of packet, e.g. it is not
>> > > required to be a reply for some another packet. Please, correct me if im wrong.
>> > >
>> > > Thanks, Arseniy
>> >
>> >
>> > Absolutely. My point was different - with this patch it is possible
>> > that you are not adding any credits at all since the previous
>> > VIRTIO_VSOCK_OP_CREDIT_UPDATE.
>>
>> I think the problem we're solving here is that since as an optimization we
>> avoid sending the update for every byte we consume, but we put a threshold,
>> then we make sure we update the peer.
>>
>> A credit update contains a snapshot and sending it the same as the previous
>> one should not create any problem.
>
>Well it consumes a buffer on the other side.

Sure, but we are already speculating by not updating the other side when
we consume bytes before a certain threshold. This already avoids to
consume many buffers.

Here we're only sending it once, when the user sets RCVLOWAT, so
basically I expect it won't affect performance.

>
>> My doubt now is that we only do this when we set RCVLOWAT , should we 
>> also
>> do something when we consume bytes to avoid the optimization we have?
>>
>> Stefano
>
>Isn't this why we have credit request?

Yep, but in practice we never use it. It would also consume 2 buffers,
one at the transmitter and one at the receiver.

However I agree that maybe we should start using it before we decide not
to send any more data.

To be compatible with older devices, though, I think for now we also
need to send a credit update when the bytes in the receive queue are
less than RCVLOWAT, as Arseniy proposed in the other series.

Thanks,
Stefano