Add one more condition for sending credit update during dequeue from
stream socket: when number of bytes in the rx queue is smaller than
SO_RCVLOWAT value of the socket. This is actual for non-default value
of SO_RCVLOWAT (e.g. not 1) - idea is to "kick" peer to continue data
transmission, because we need at least SO_RCVLOWAT bytes in our rx
queue to wake up user for reading data (in corner case it is also
possible to stuck both tx and rx sides, this is why 'Fixes' is used).
Fixes: b89d882dc9fc ("vsock/virtio: reduce credit update messages")
Signed-off-by: Arseniy Krasnov <[email protected]>
---
net/vmw_vsock/virtio_transport_common.c | 9 +++++++--
1 file changed, 7 insertions(+), 2 deletions(-)
diff --git a/net/vmw_vsock/virtio_transport_common.c b/net/vmw_vsock/virtio_transport_common.c
index e137d740804e..461c89882142 100644
--- a/net/vmw_vsock/virtio_transport_common.c
+++ b/net/vmw_vsock/virtio_transport_common.c
@@ -558,6 +558,7 @@ virtio_transport_stream_do_dequeue(struct vsock_sock *vsk,
struct virtio_vsock_sock *vvs = vsk->trans;
size_t bytes, total = 0;
struct sk_buff *skb;
+ bool low_rx_bytes;
int err = -EFAULT;
u32 free_space;
@@ -602,6 +603,8 @@ virtio_transport_stream_do_dequeue(struct vsock_sock *vsk,
}
free_space = vvs->buf_alloc - (vvs->fwd_cnt - vvs->last_fwd_cnt);
+ low_rx_bytes = (vvs->rx_bytes <
+ sock_rcvlowat(sk_vsock(vsk), 0, INT_MAX));
spin_unlock_bh(&vvs->rx_lock);
@@ -611,9 +614,11 @@ virtio_transport_stream_do_dequeue(struct vsock_sock *vsk,
* too high causes extra messages. Too low causes transmitter
* stalls. As stalls are in theory more expensive than extra
* messages, we set the limit to a high value. TODO: experiment
- * with different values.
+ * with different values. Also send credit update message when
+ * number of bytes in rx queue is not enough to wake up reader.
*/
- if (free_space < VIRTIO_VSOCK_MAX_PKT_BUF_SIZE)
+ if (free_space < VIRTIO_VSOCK_MAX_PKT_BUF_SIZE ||
+ low_rx_bytes)
virtio_transport_send_credit_update(vsk);
return total;
--
2.25.1
On Tue, Dec 05, 2023 at 09:48:05AM +0300, Arseniy Krasnov wrote:
>Add one more condition for sending credit update during dequeue from
>stream socket: when number of bytes in the rx queue is smaller than
>SO_RCVLOWAT value of the socket. This is actual for non-default value
>of SO_RCVLOWAT (e.g. not 1) - idea is to "kick" peer to continue data
>transmission, because we need at least SO_RCVLOWAT bytes in our rx
>queue to wake up user for reading data (in corner case it is also
>possible to stuck both tx and rx sides, this is why 'Fixes' is used).
>
>Fixes: b89d882dc9fc ("vsock/virtio: reduce credit update messages")
>Signed-off-by: Arseniy Krasnov <[email protected]>
>---
> net/vmw_vsock/virtio_transport_common.c | 9 +++++++--
> 1 file changed, 7 insertions(+), 2 deletions(-)
>
>diff --git a/net/vmw_vsock/virtio_transport_common.c b/net/vmw_vsock/virtio_transport_common.c
>index e137d740804e..461c89882142 100644
>--- a/net/vmw_vsock/virtio_transport_common.c
>+++ b/net/vmw_vsock/virtio_transport_common.c
>@@ -558,6 +558,7 @@ virtio_transport_stream_do_dequeue(struct vsock_sock *vsk,
> struct virtio_vsock_sock *vvs = vsk->trans;
> size_t bytes, total = 0;
> struct sk_buff *skb;
>+ bool low_rx_bytes;
> int err = -EFAULT;
> u32 free_space;
>
>@@ -602,6 +603,8 @@ virtio_transport_stream_do_dequeue(struct vsock_sock *vsk,
> }
>
> free_space = vvs->buf_alloc - (vvs->fwd_cnt - vvs->last_fwd_cnt);
>+ low_rx_bytes = (vvs->rx_bytes <
>+ sock_rcvlowat(sk_vsock(vsk), 0, INT_MAX));
As in the previous patch, should we avoid the update it if `fwd_cnt` and
`last_fwd_cnt` are the same?
Now I'm thinking if it is better to add that check directly in
virtio_transport_send_credit_update().
Stefano
>
> spin_unlock_bh(&vvs->rx_lock);
>
>@@ -611,9 +614,11 @@ virtio_transport_stream_do_dequeue(struct vsock_sock *vsk,
> * too high causes extra messages. Too low causes transmitter
> * stalls. As stalls are in theory more expensive than extra
> * messages, we set the limit to a high value. TODO: experiment
>- * with different values.
>+ * with different values. Also send credit update message when
>+ * number of bytes in rx queue is not enough to wake up reader.
> */
>- if (free_space < VIRTIO_VSOCK_MAX_PKT_BUF_SIZE)
>+ if (free_space < VIRTIO_VSOCK_MAX_PKT_BUF_SIZE ||
>+ low_rx_bytes)
> virtio_transport_send_credit_update(vsk);
>
> return total;
>--
>2.25.1
>
On 05.12.2023 13:54, Stefano Garzarella wrote:
> On Tue, Dec 05, 2023 at 09:48:05AM +0300, Arseniy Krasnov wrote:
>> Add one more condition for sending credit update during dequeue from
>> stream socket: when number of bytes in the rx queue is smaller than
>> SO_RCVLOWAT value of the socket. This is actual for non-default value
>> of SO_RCVLOWAT (e.g. not 1) - idea is to "kick" peer to continue data
>> transmission, because we need at least SO_RCVLOWAT bytes in our rx
>> queue to wake up user for reading data (in corner case it is also
>> possible to stuck both tx and rx sides, this is why 'Fixes' is used).
>>
>> Fixes: b89d882dc9fc ("vsock/virtio: reduce credit update messages")
>> Signed-off-by: Arseniy Krasnov <[email protected]>
>> ---
>> net/vmw_vsock/virtio_transport_common.c | 9 +++++++--
>> 1 file changed, 7 insertions(+), 2 deletions(-)
>>
>> diff --git a/net/vmw_vsock/virtio_transport_common.c b/net/vmw_vsock/virtio_transport_common.c
>> index e137d740804e..461c89882142 100644
>> --- a/net/vmw_vsock/virtio_transport_common.c
>> +++ b/net/vmw_vsock/virtio_transport_common.c
>> @@ -558,6 +558,7 @@ virtio_transport_stream_do_dequeue(struct vsock_sock *vsk,
>> struct virtio_vsock_sock *vvs = vsk->trans;
>> size_t bytes, total = 0;
>> struct sk_buff *skb;
>> + bool low_rx_bytes;
>> int err = -EFAULT;
>> u32 free_space;
>>
>> @@ -602,6 +603,8 @@ virtio_transport_stream_do_dequeue(struct vsock_sock *vsk,
>> }
>>
>> free_space = vvs->buf_alloc - (vvs->fwd_cnt - vvs->last_fwd_cnt);
>> + low_rx_bytes = (vvs->rx_bytes <
>> + sock_rcvlowat(sk_vsock(vsk), 0, INT_MAX));
>
> As in the previous patch, should we avoid the update it if `fwd_cnt` and `last_fwd_cnt` are the same?
>
> Now I'm thinking if it is better to add that check directly in virtio_transport_send_credit_update().
Good point, but I think, that it is better to keep this check here, because access to 'fwd_cnt' and 'last_fwd_cnt'
requires taking rx_lock - so I guess it is better to avoid taking this lock every time in 'virtio_transport_send_credit_update()'.
So may be we can do something like:
fwd_cnt_delta = vvs->fwd_cnt - vvs->last_fwd_cnt;
free_space = vvs->buf_alloc - fwd_cnt_delta;
and then, after lock is released:
if (fwd_cnt_delta && (free_space < VIRTIO_VSOCK_MAX_PKT_BUF_SIZE ||
low_rx_bytes))
virtio_transport_send_credit_update(vsk);
WDYT?
Also, I guess that next idea to update this optimization(in next patchset), is to make
threshold depends on vvs->buf_alloc. Because if someone changes minimum buffer size to
for example 32KB, and then sets buffer size to 32KB, then free_space will be always
non-zero, thus optimization is off now and credit update is sent on every read.
Thanks, Arseniy
>
> Stefano
>
>>
>> spin_unlock_bh(&vvs->rx_lock);
>>
>> @@ -611,9 +614,11 @@ virtio_transport_stream_do_dequeue(struct vsock_sock *vsk,
>> * too high causes extra messages. Too low causes transmitter
>> * stalls. As stalls are in theory more expensive than extra
>> * messages, we set the limit to a high value. TODO: experiment
>> - * with different values.
>> + * with different values. Also send credit update message when
>> + * number of bytes in rx queue is not enough to wake up reader.
>> */
>> - if (free_space < VIRTIO_VSOCK_MAX_PKT_BUF_SIZE)
>> + if (free_space < VIRTIO_VSOCK_MAX_PKT_BUF_SIZE ||
>> + low_rx_bytes)
>> virtio_transport_send_credit_update(vsk);
>>
>> return total;
>> --
>> 2.25.1
>>
>
On Tue, Dec 05, 2023 at 03:07:47PM +0300, Arseniy Krasnov wrote:
>
>
>On 05.12.2023 13:54, Stefano Garzarella wrote:
>> On Tue, Dec 05, 2023 at 09:48:05AM +0300, Arseniy Krasnov wrote:
>>> Add one more condition for sending credit update during dequeue from
>>> stream socket: when number of bytes in the rx queue is smaller than
>>> SO_RCVLOWAT value of the socket. This is actual for non-default value
>>> of SO_RCVLOWAT (e.g. not 1) - idea is to "kick" peer to continue data
>>> transmission, because we need at least SO_RCVLOWAT bytes in our rx
>>> queue to wake up user for reading data (in corner case it is also
>>> possible to stuck both tx and rx sides, this is why 'Fixes' is used).
>>>
>>> Fixes: b89d882dc9fc ("vsock/virtio: reduce credit update messages")
>>> Signed-off-by: Arseniy Krasnov <[email protected]>
>>> ---
>>> net/vmw_vsock/virtio_transport_common.c | 9 +++++++--
>>> 1 file changed, 7 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/net/vmw_vsock/virtio_transport_common.c b/net/vmw_vsock/virtio_transport_common.c
>>> index e137d740804e..461c89882142 100644
>>> --- a/net/vmw_vsock/virtio_transport_common.c
>>> +++ b/net/vmw_vsock/virtio_transport_common.c
>>> @@ -558,6 +558,7 @@ virtio_transport_stream_do_dequeue(struct vsock_sock *vsk,
>>> ????struct virtio_vsock_sock *vvs = vsk->trans;
>>> ????size_t bytes, total = 0;
>>> ????struct sk_buff *skb;
>>> +??? bool low_rx_bytes;
>>> ????int err = -EFAULT;
>>> ????u32 free_space;
>>>
>>> @@ -602,6 +603,8 @@ virtio_transport_stream_do_dequeue(struct vsock_sock *vsk,
>>> ????}
>>>
>>> ????free_space = vvs->buf_alloc - (vvs->fwd_cnt - vvs->last_fwd_cnt);
>>> +??? low_rx_bytes = (vvs->rx_bytes <
>>> +??????????? sock_rcvlowat(sk_vsock(vsk), 0, INT_MAX));
>>
>> As in the previous patch, should we avoid the update it if `fwd_cnt` and `last_fwd_cnt` are the same?
>>
>> Now I'm thinking if it is better to add that check directly in virtio_transport_send_credit_update().
>
>Good point, but I think, that it is better to keep this check here, because access to 'fwd_cnt' and 'last_fwd_cnt'
>requires taking rx_lock - so I guess it is better to avoid taking this lock every time in 'virtio_transport_send_credit_update()'.
Yeah, I agree.
>So may be we can do something like:
>
>
>fwd_cnt_delta = vvs->fwd_cnt - vvs->last_fwd_cnt;
>free_space = vvs->buf_alloc - fwd_cnt_delta;
Pre-existing issue, but should we handle the wrap (e.g. fwd_cnt wrapped,
but last_fwd_cnt not yet?). Maybe in that case we can foce the status
update.
>
>and then, after lock is released:
>
>if (fwd_cnt_delta && (free_space < VIRTIO_VSOCK_MAX_PKT_BUF_SIZE ||
> low_rx_bytes))
> virtio_transport_send_credit_update(vsk);
>
>WDYT?
Yep, I agree.
>
>Also, I guess that next idea to update this optimization(in next patchset), is to make
>threshold depends on vvs->buf_alloc. Because if someone changes minimum buffer size to
>for example 32KB, and then sets buffer size to 32KB, then free_space will be always
>non-zero, thus optimization is off now and credit update is sent on
>every read.
But does it make sense to allow a buffer smaller than
VIRTIO_VSOCK_MAX_PKT_BUF_SIZE?
Maybe we should fail in virtio_transport_notify_buffer_size() or use it
as minimum.
Stefano
On 05.12.2023 17:21, Stefano Garzarella wrote:
> On Tue, Dec 05, 2023 at 03:07:47PM +0300, Arseniy Krasnov wrote:
>>
>>
>> On 05.12.2023 13:54, Stefano Garzarella wrote:
>>> On Tue, Dec 05, 2023 at 09:48:05AM +0300, Arseniy Krasnov wrote:
>>>> Add one more condition for sending credit update during dequeue from
>>>> stream socket: when number of bytes in the rx queue is smaller than
>>>> SO_RCVLOWAT value of the socket. This is actual for non-default value
>>>> of SO_RCVLOWAT (e.g. not 1) - idea is to "kick" peer to continue data
>>>> transmission, because we need at least SO_RCVLOWAT bytes in our rx
>>>> queue to wake up user for reading data (in corner case it is also
>>>> possible to stuck both tx and rx sides, this is why 'Fixes' is used).
>>>>
>>>> Fixes: b89d882dc9fc ("vsock/virtio: reduce credit update messages")
>>>> Signed-off-by: Arseniy Krasnov <[email protected]>
>>>> ---
>>>> net/vmw_vsock/virtio_transport_common.c | 9 +++++++--
>>>> 1 file changed, 7 insertions(+), 2 deletions(-)
>>>>
>>>> diff --git a/net/vmw_vsock/virtio_transport_common.c b/net/vmw_vsock/virtio_transport_common.c
>>>> index e137d740804e..461c89882142 100644
>>>> --- a/net/vmw_vsock/virtio_transport_common.c
>>>> +++ b/net/vmw_vsock/virtio_transport_common.c
>>>> @@ -558,6 +558,7 @@ virtio_transport_stream_do_dequeue(struct vsock_sock *vsk,
>>>> struct virtio_vsock_sock *vvs = vsk->trans;
>>>> size_t bytes, total = 0;
>>>> struct sk_buff *skb;
>>>> + bool low_rx_bytes;
>>>> int err = -EFAULT;
>>>> u32 free_space;
>>>>
>>>> @@ -602,6 +603,8 @@ virtio_transport_stream_do_dequeue(struct vsock_sock *vsk,
>>>> }
>>>>
>>>> free_space = vvs->buf_alloc - (vvs->fwd_cnt - vvs->last_fwd_cnt);
>>>> + low_rx_bytes = (vvs->rx_bytes <
>>>> + sock_rcvlowat(sk_vsock(vsk), 0, INT_MAX));
>>>
>>> As in the previous patch, should we avoid the update it if `fwd_cnt` and `last_fwd_cnt` are the same?
>>>
>>> Now I'm thinking if it is better to add that check directly in virtio_transport_send_credit_update().
>>
>> Good point, but I think, that it is better to keep this check here, because access to 'fwd_cnt' and 'last_fwd_cnt'
>> requires taking rx_lock - so I guess it is better to avoid taking this lock every time in 'virtio_transport_send_credit_update()'.
>
> Yeah, I agree.
>
>> So may be we can do something like:
>>
>>
>> fwd_cnt_delta = vvs->fwd_cnt - vvs->last_fwd_cnt;
>> free_space = vvs->buf_alloc - fwd_cnt_delta;
>
> Pre-existing issue, but should we handle the wrap (e.g. fwd_cnt wrapped, but last_fwd_cnt not yet?). Maybe in that case we can foce the status
> update.
Agree, I'll add this logic!
>
>>
>> and then, after lock is released:
>>
>> if (fwd_cnt_delta && (free_space < VIRTIO_VSOCK_MAX_PKT_BUF_SIZE ||
>> low_rx_bytes))
>> virtio_transport_send_credit_update(vsk);
>>
>> WDYT?
>
> Yep, I agree.
>
>>
>> Also, I guess that next idea to update this optimization(in next patchset), is to make
>> threshold depends on vvs->buf_alloc. Because if someone changes minimum buffer size to
>> for example 32KB, and then sets buffer size to 32KB, then free_space will be always
>> non-zero, thus optimization is off now and credit update is sent on every read.
>
> But does it make sense to allow a buffer smaller than VIRTIO_VSOCK_MAX_PKT_BUF_SIZE?
>
> Maybe we should fail in virtio_transport_notify_buffer_size() or use it as minimum.
Yes, currently there is no limitation in this transport callback - only for maximum.
Thanks, Arseniy
>
> Stefano
>