Hello,
DESCRIPTION
This patchset fixes old problem with hungup of both rx/tx sides and adds
test for it. This happens due to non-default SO_RCVLOWAT value and
deferred credit update in virtio/vsock. Link to previous old patchset:
https://lore.kernel.org/netdev/[email protected]/
Here is what happens step by step:
TEST
INITIAL CONDITIONS
1) Vsock buffer size is 128KB.
2) Maximum packet size is also 64KB as defined in header (yes it is
hardcoded, just to remind about that value).
3) SO_RCVLOWAT is default, e.g. 1 byte.
STEPS
SENDER RECEIVER
1) sends 128KB + 1 byte in a
single buffer. 128KB will
be sent, but for 1 byte
sender will wait for free
space at peer. Sender goes
to sleep.
2) reads 64KB, credit update not sent
3) sets SO_RCVLOWAT to 64KB + 1
4) poll() -> wait forever, there is
only 64KB available to read.
So in step 4) receiver also goes to sleep, waiting for enough data or
connection shutdown message from the sender. Idea to fix it is that rx
kicks tx side to continue transmission (and may be close connection)
when rx changes number of bytes to be woken up (e.g. SO_RCVLOWAT) and
this value is bigger than number of available bytes to read.
I've added small test for this, but not sure as it uses hardcoded value
for maximum packet length, this value is defined in kernel header and
used to control deferred credit update. And as this is not available to
userspace, I can't control test parameters correctly (if one day this
define will be changed - test may become useless).
Head for this patchset is:
https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/?id=f1be1e04c76bb9c44789d3575bba4418cf0ea359
Link to v1:
https://lore.kernel.org/netdev/[email protected]/
Link to v2:
https://lore.kernel.org/netdev/[email protected]/
Link to v3:
https://lore.kernel.org/netdev/[email protected]/
Link to v4:
https://lore.kernel.org/netdev/[email protected]/
Changelog:
v1 -> v2:
* Patchset rebased and tested on new HEAD of net-next (see hash above).
* New patch is added as 0001 - it removes return from SO_RCVLOWAT set
callback in 'af_vsock.c' when transport callback is set - with that
we can set 'sk_rcvlowat' only once in 'af_vsock.c' and in future do
not copy-paste it to every transport. It was discussed in v1.
* See per-patch changelog after ---.
v2 -> v3:
* See changelog after --- in 0003 only (0001 and 0002 still same).
v3 -> v4:
* Patchset rebased and tested on new HEAD of net-next (see hash above).
* See per-patch changelog after ---.
v4 -> v5:
* Change patchset tag 'RFC' -> 'net-next'.
* See per-patch changelog after ---.
Arseniy Krasnov (3):
vsock: update SO_RCVLOWAT setting callback
virtio/vsock: send credit update during setting SO_RCVLOWAT
vsock/test: SO_RCVLOWAT + deferred credit update test
drivers/vhost/vsock.c | 1 +
include/linux/virtio_vsock.h | 1 +
include/net/af_vsock.h | 2 +-
net/vmw_vsock/af_vsock.c | 9 +-
net/vmw_vsock/hyperv_transport.c | 4 +-
net/vmw_vsock/virtio_transport.c | 1 +
net/vmw_vsock/virtio_transport_common.c | 27 +++++
net/vmw_vsock/vsock_loopback.c | 1 +
tools/testing/vsock/vsock_test.c | 142 ++++++++++++++++++++++++
9 files changed, 183 insertions(+), 5 deletions(-)
--
2.25.1
Send credit update message when SO_RCVLOWAT is updated and it is bigger
than number of bytes in rx queue. It is needed, because 'poll()' will
wait until number of bytes in rx queue will be not smaller than
SO_RCVLOWAT, so kick sender to send more data. Otherwise mutual hungup
for tx/rx is possible: sender waits for free space and receiver is
waiting data in 'poll()'.
Signed-off-by: Arseniy Krasnov <[email protected]>
---
Changelog:
v1 -> v2:
* Update commit message by removing 'This patch adds XXX' manner.
* Do not initialize 'send_update' variable - set it directly during
first usage.
v3 -> v4:
* Fit comment in 'virtio_transport_notify_set_rcvlowat()' to 80 chars.
v4 -> v5:
* Do not change callbacks order in transport structures.
drivers/vhost/vsock.c | 1 +
include/linux/virtio_vsock.h | 1 +
net/vmw_vsock/virtio_transport.c | 1 +
net/vmw_vsock/virtio_transport_common.c | 27 +++++++++++++++++++++++++
net/vmw_vsock/vsock_loopback.c | 1 +
5 files changed, 31 insertions(+)
diff --git a/drivers/vhost/vsock.c b/drivers/vhost/vsock.c
index f75731396b7e..4146f80db8ac 100644
--- a/drivers/vhost/vsock.c
+++ b/drivers/vhost/vsock.c
@@ -451,6 +451,7 @@ static struct virtio_transport vhost_transport = {
.notify_buffer_size = virtio_transport_notify_buffer_size,
.read_skb = virtio_transport_read_skb,
+ .notify_set_rcvlowat = virtio_transport_notify_set_rcvlowat
},
.send_pkt = vhost_transport_send_pkt,
diff --git a/include/linux/virtio_vsock.h b/include/linux/virtio_vsock.h
index ebb3ce63d64d..c82089dee0c8 100644
--- a/include/linux/virtio_vsock.h
+++ b/include/linux/virtio_vsock.h
@@ -256,4 +256,5 @@ void virtio_transport_put_credit(struct virtio_vsock_sock *vvs, u32 credit);
void virtio_transport_deliver_tap_pkt(struct sk_buff *skb);
int virtio_transport_purge_skbs(void *vsk, struct sk_buff_head *list);
int virtio_transport_read_skb(struct vsock_sock *vsk, skb_read_actor_t read_actor);
+int virtio_transport_notify_set_rcvlowat(struct vsock_sock *vsk, int val);
#endif /* _LINUX_VIRTIO_VSOCK_H */
diff --git a/net/vmw_vsock/virtio_transport.c b/net/vmw_vsock/virtio_transport.c
index af5bab1acee1..8007593a3a93 100644
--- a/net/vmw_vsock/virtio_transport.c
+++ b/net/vmw_vsock/virtio_transport.c
@@ -539,6 +539,7 @@ static struct virtio_transport virtio_transport = {
.notify_buffer_size = virtio_transport_notify_buffer_size,
.read_skb = virtio_transport_read_skb,
+ .notify_set_rcvlowat = virtio_transport_notify_set_rcvlowat
},
.send_pkt = virtio_transport_send_pkt,
diff --git a/net/vmw_vsock/virtio_transport_common.c b/net/vmw_vsock/virtio_transport_common.c
index f6dc896bf44c..1cb556ad4597 100644
--- a/net/vmw_vsock/virtio_transport_common.c
+++ b/net/vmw_vsock/virtio_transport_common.c
@@ -1684,6 +1684,33 @@ int virtio_transport_read_skb(struct vsock_sock *vsk, skb_read_actor_t recv_acto
}
EXPORT_SYMBOL_GPL(virtio_transport_read_skb);
+int virtio_transport_notify_set_rcvlowat(struct vsock_sock *vsk, int val)
+{
+ struct virtio_vsock_sock *vvs = vsk->trans;
+ bool send_update;
+
+ spin_lock_bh(&vvs->rx_lock);
+
+ /* If number of available bytes is less than new SO_RCVLOWAT value,
+ * kick sender to send more data, because sender may sleep in its
+ * 'send()' syscall waiting for enough space at our side.
+ */
+ send_update = vvs->rx_bytes < val;
+
+ spin_unlock_bh(&vvs->rx_lock);
+
+ if (send_update) {
+ int err;
+
+ err = virtio_transport_send_credit_update(vsk);
+ if (err < 0)
+ return err;
+ }
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(virtio_transport_notify_set_rcvlowat);
+
MODULE_LICENSE("GPL v2");
MODULE_AUTHOR("Asias He");
MODULE_DESCRIPTION("common code for virtio vsock");
diff --git a/net/vmw_vsock/vsock_loopback.c b/net/vmw_vsock/vsock_loopback.c
index 048640167411..9f4b814fbbc7 100644
--- a/net/vmw_vsock/vsock_loopback.c
+++ b/net/vmw_vsock/vsock_loopback.c
@@ -98,6 +98,7 @@ static struct virtio_transport loopback_transport = {
.notify_buffer_size = virtio_transport_notify_buffer_size,
.read_skb = virtio_transport_read_skb,
+ .notify_set_rcvlowat = virtio_transport_notify_set_rcvlowat
},
.send_pkt = vsock_loopback_send_pkt,
--
2.25.1
Do not return if transport callback for SO_RCVLOWAT is set (only in
error case). In this case we don't need to set 'sk_rcvlowat' field in
each transport - only in 'vsock_set_rcvlowat()'. Also, if 'sk_rcvlowat'
is now set only in af_vsock.c, change callback name from 'set_rcvlowat'
to 'notify_set_rcvlowat'.
Signed-off-by: Arseniy Krasnov <[email protected]>
Reviewed-by: Stefano Garzarella <[email protected]>
---
Changelog:
v3 -> v4:
* Rename 'set_rcvlowat' to 'notify_set_rcvlowat'.
* Commit message updated.
include/net/af_vsock.h | 2 +-
net/vmw_vsock/af_vsock.c | 9 +++++++--
net/vmw_vsock/hyperv_transport.c | 4 ++--
3 files changed, 10 insertions(+), 5 deletions(-)
diff --git a/include/net/af_vsock.h b/include/net/af_vsock.h
index e302c0e804d0..535701efc1e5 100644
--- a/include/net/af_vsock.h
+++ b/include/net/af_vsock.h
@@ -137,7 +137,6 @@ struct vsock_transport {
u64 (*stream_rcvhiwat)(struct vsock_sock *);
bool (*stream_is_active)(struct vsock_sock *);
bool (*stream_allow)(u32 cid, u32 port);
- int (*set_rcvlowat)(struct vsock_sock *vsk, int val);
/* SEQ_PACKET. */
ssize_t (*seqpacket_dequeue)(struct vsock_sock *vsk, struct msghdr *msg,
@@ -168,6 +167,7 @@ struct vsock_transport {
struct vsock_transport_send_notify_data *);
/* sk_lock held by the caller */
void (*notify_buffer_size)(struct vsock_sock *, u64 *);
+ int (*notify_set_rcvlowat)(struct vsock_sock *vsk, int val);
/* Shutdown. */
int (*shutdown)(struct vsock_sock *, int);
diff --git a/net/vmw_vsock/af_vsock.c b/net/vmw_vsock/af_vsock.c
index 816725af281f..54ba7316f808 100644
--- a/net/vmw_vsock/af_vsock.c
+++ b/net/vmw_vsock/af_vsock.c
@@ -2264,8 +2264,13 @@ static int vsock_set_rcvlowat(struct sock *sk, int val)
transport = vsk->transport;
- if (transport && transport->set_rcvlowat)
- return transport->set_rcvlowat(vsk, val);
+ if (transport && transport->notify_set_rcvlowat) {
+ int err;
+
+ err = transport->notify_set_rcvlowat(vsk, val);
+ if (err)
+ return err;
+ }
WRITE_ONCE(sk->sk_rcvlowat, val ? : 1);
return 0;
diff --git a/net/vmw_vsock/hyperv_transport.c b/net/vmw_vsock/hyperv_transport.c
index 7cb1a9d2cdb4..e2157e387217 100644
--- a/net/vmw_vsock/hyperv_transport.c
+++ b/net/vmw_vsock/hyperv_transport.c
@@ -816,7 +816,7 @@ int hvs_notify_send_post_enqueue(struct vsock_sock *vsk, ssize_t written,
}
static
-int hvs_set_rcvlowat(struct vsock_sock *vsk, int val)
+int hvs_notify_set_rcvlowat(struct vsock_sock *vsk, int val)
{
return -EOPNOTSUPP;
}
@@ -856,7 +856,7 @@ static struct vsock_transport hvs_transport = {
.notify_send_pre_enqueue = hvs_notify_send_pre_enqueue,
.notify_send_post_enqueue = hvs_notify_send_post_enqueue,
- .set_rcvlowat = hvs_set_rcvlowat
+ .notify_set_rcvlowat = hvs_notify_set_rcvlowat
};
static bool hvs_check_transport(struct vsock_sock *vsk)
--
2.25.1
Test which checks, that updating SO_RCVLOWAT value also sends credit
update message. Otherwise mutual hungup may happen when receiver didn't
send credit update and then calls 'poll()' with non default SO_RCVLOWAT
value (e.g. waiting enough bytes to read), while sender waits for free
space at receiver's side. Important thing is that this test relies on
kernel's define for maximum packet size for virtio transport and this
value is not exported to user: VIRTIO_VSOCK_MAX_PKT_BUF_SIZE (this
define is used to control moment when to send credit update message).
If this value or its usage will be changed in kernel - this test may
become useless/broken.
Signed-off-by: Arseniy Krasnov <[email protected]>
---
Changelog:
v1 -> v2:
* Update commit message by removing 'This patch adds XXX' manner.
* Update commit message by adding details about dependency for this
test from kernel internal define VIRTIO_VSOCK_MAX_PKT_BUF_SIZE.
* Add comment for this dependency in 'vsock_test.c' where this define
is duplicated.
v2 -> v3:
* Replace synchronization based on control TCP socket with vsock
data socket - this is needed to allow sender transmit data only
when new buffer size of receiver is visible to sender. Otherwise
there is race and test fails sometimes.
v3 -> v4:
* Replace 'recv_buf()' to 'recv(MSG_DONTWAIT)' in last read operation
in server part. This is needed to ensure that 'poll()' wake up us
when number of bytes ready to read is equal to SO_RCVLOWAT value.
v4 -> v5:
* Use 'recv_buf(MSG_DONTWAIT)' instead of 'recv(MSG_DONTWAIT)'.
tools/testing/vsock/vsock_test.c | 142 +++++++++++++++++++++++++++++++
1 file changed, 142 insertions(+)
diff --git a/tools/testing/vsock/vsock_test.c b/tools/testing/vsock/vsock_test.c
index 01fa816868bc..d66bc4987026 100644
--- a/tools/testing/vsock/vsock_test.c
+++ b/tools/testing/vsock/vsock_test.c
@@ -1232,6 +1232,143 @@ static void test_double_bind_connect_client(const struct test_opts *opts)
}
}
+#define RCVLOWAT_CREDIT_UPD_BUF_SIZE (1024 * 128)
+/* This define is the same as in 'include/linux/virtio_vsock.h':
+ * it is used to decide when to send credit update message during
+ * reading from rx queue of a socket. Value and its usage in
+ * kernel is important for this test.
+ */
+#define VIRTIO_VSOCK_MAX_PKT_BUF_SIZE (1024 * 64)
+
+static void test_stream_rcvlowat_def_cred_upd_client(const struct test_opts *opts)
+{
+ size_t buf_size;
+ void *buf;
+ int fd;
+
+ fd = vsock_stream_connect(opts->peer_cid, 1234);
+ if (fd < 0) {
+ perror("connect");
+ exit(EXIT_FAILURE);
+ }
+
+ /* Send 1 byte more than peer's buffer size. */
+ buf_size = RCVLOWAT_CREDIT_UPD_BUF_SIZE + 1;
+
+ buf = malloc(buf_size);
+ if (!buf) {
+ perror("malloc");
+ exit(EXIT_FAILURE);
+ }
+
+ /* Wait until peer sets needed buffer size. */
+ recv_byte(fd, 1, 0);
+
+ if (send(fd, buf, buf_size, 0) != buf_size) {
+ perror("send failed");
+ exit(EXIT_FAILURE);
+ }
+
+ free(buf);
+ close(fd);
+}
+
+static void test_stream_rcvlowat_def_cred_upd_server(const struct test_opts *opts)
+{
+ size_t recv_buf_size;
+ struct pollfd fds;
+ size_t buf_size;
+ void *buf;
+ int fd;
+
+ fd = vsock_stream_accept(VMADDR_CID_ANY, 1234, NULL);
+ if (fd < 0) {
+ perror("accept");
+ exit(EXIT_FAILURE);
+ }
+
+ buf_size = RCVLOWAT_CREDIT_UPD_BUF_SIZE;
+
+ if (setsockopt(fd, AF_VSOCK, SO_VM_SOCKETS_BUFFER_SIZE,
+ &buf_size, sizeof(buf_size))) {
+ perror("setsockopt(SO_VM_SOCKETS_BUFFER_SIZE)");
+ exit(EXIT_FAILURE);
+ }
+
+ /* Send one dummy byte here, because 'setsockopt()' above also
+ * sends special packet which tells sender to update our buffer
+ * size. This 'send_byte()' will serialize such packet with data
+ * reads in a loop below. Sender starts transmission only when
+ * it receives this single byte.
+ */
+ send_byte(fd, 1, 0);
+
+ buf = malloc(buf_size);
+ if (!buf) {
+ perror("malloc");
+ exit(EXIT_FAILURE);
+ }
+
+ /* Wait until there will be 128KB of data in rx queue. */
+ while (1) {
+ ssize_t res;
+
+ res = recv(fd, buf, buf_size, MSG_PEEK);
+ if (res == buf_size)
+ break;
+
+ if (res <= 0) {
+ fprintf(stderr, "unexpected 'recv()' return: %zi\n", res);
+ exit(EXIT_FAILURE);
+ }
+ }
+
+ /* There is 128KB of data in the socket's rx queue,
+ * dequeue first 64KB, credit update is not sent.
+ */
+ recv_buf_size = VIRTIO_VSOCK_MAX_PKT_BUF_SIZE;
+ recv_buf(fd, buf, recv_buf_size, 0, recv_buf_size);
+ recv_buf_size++;
+
+ /* Updating SO_RCVLOWAT will send credit update. */
+ if (setsockopt(fd, SOL_SOCKET, SO_RCVLOWAT,
+ &recv_buf_size, sizeof(recv_buf_size))) {
+ perror("setsockopt(SO_RCVLOWAT)");
+ exit(EXIT_FAILURE);
+ }
+
+ memset(&fds, 0, sizeof(fds));
+ fds.fd = fd;
+ fds.events = POLLIN | POLLRDNORM | POLLERR |
+ POLLRDHUP | POLLHUP;
+
+ /* This 'poll()' will return once we receive last byte
+ * sent by client.
+ */
+ if (poll(&fds, 1, -1) < 0) {
+ perror("poll");
+ exit(EXIT_FAILURE);
+ }
+
+ if (fds.revents & POLLERR) {
+ fprintf(stderr, "'poll()' error\n");
+ exit(EXIT_FAILURE);
+ }
+
+ if (fds.revents & (POLLIN | POLLRDNORM)) {
+ recv_buf(fd, buf, recv_buf_size, MSG_DONTWAIT, recv_buf_size);
+ } else {
+ /* These flags must be set, as there is at
+ * least 64KB of data ready to read.
+ */
+ fprintf(stderr, "POLLIN | POLLRDNORM expected\n");
+ exit(EXIT_FAILURE);
+ }
+
+ free(buf);
+ close(fd);
+}
+
static struct test_case test_cases[] = {
{
.name = "SOCK_STREAM connection reset",
@@ -1342,6 +1479,11 @@ static struct test_case test_cases[] = {
.run_client = test_double_bind_connect_client,
.run_server = test_double_bind_connect_server,
},
+ {
+ .name = "SOCK_STREAM virtio SO_RCVLOWAT + deferred cred update",
+ .run_client = test_stream_rcvlowat_def_cred_upd_client,
+ .run_server = test_stream_rcvlowat_def_cred_upd_server,
+ },
{},
};
--
2.25.1
On Thu, Nov 30, 2023 at 04:08:39PM +0300, Arseniy Krasnov wrote:
>Send credit update message when SO_RCVLOWAT is updated and it is bigger
>than number of bytes in rx queue. It is needed, because 'poll()' will
>wait until number of bytes in rx queue will be not smaller than
>SO_RCVLOWAT, so kick sender to send more data. Otherwise mutual hungup
>for tx/rx is possible: sender waits for free space and receiver is
>waiting data in 'poll()'.
>
>Signed-off-by: Arseniy Krasnov <[email protected]>
>---
> Changelog:
> v1 -> v2:
> * Update commit message by removing 'This patch adds XXX' manner.
> * Do not initialize 'send_update' variable - set it directly during
> first usage.
> v3 -> v4:
> * Fit comment in 'virtio_transport_notify_set_rcvlowat()' to 80 chars.
> v4 -> v5:
> * Do not change callbacks order in transport structures.
>
> drivers/vhost/vsock.c | 1 +
> include/linux/virtio_vsock.h | 1 +
> net/vmw_vsock/virtio_transport.c | 1 +
> net/vmw_vsock/virtio_transport_common.c | 27 +++++++++++++++++++++++++
> net/vmw_vsock/vsock_loopback.c | 1 +
> 5 files changed, 31 insertions(+)
>
>diff --git a/drivers/vhost/vsock.c b/drivers/vhost/vsock.c
>index f75731396b7e..4146f80db8ac 100644
>--- a/drivers/vhost/vsock.c
>+++ b/drivers/vhost/vsock.c
>@@ -451,6 +451,7 @@ static struct virtio_transport vhost_transport = {
> .notify_buffer_size = virtio_transport_notify_buffer_size,
>
> .read_skb = virtio_transport_read_skb,
>+ .notify_set_rcvlowat = virtio_transport_notify_set_rcvlowat
As we discussed in chat, better the order of the previous version, but
leaving the line of `.read_skb` untouched (with the final comma).
With that fixed in all transports, feel free to add:
Reviewed-by: Stefano Garzarella <[email protected]>
> },
>
> .send_pkt = vhost_transport_send_pkt,
>diff --git a/include/linux/virtio_vsock.h b/include/linux/virtio_vsock.h
>index ebb3ce63d64d..c82089dee0c8 100644
>--- a/include/linux/virtio_vsock.h
>+++ b/include/linux/virtio_vsock.h
>@@ -256,4 +256,5 @@ void virtio_transport_put_credit(struct virtio_vsock_sock *vvs, u32 credit);
> void virtio_transport_deliver_tap_pkt(struct sk_buff *skb);
> int virtio_transport_purge_skbs(void *vsk, struct sk_buff_head *list);
> int virtio_transport_read_skb(struct vsock_sock *vsk, skb_read_actor_t read_actor);
>+int virtio_transport_notify_set_rcvlowat(struct vsock_sock *vsk, int val);
> #endif /* _LINUX_VIRTIO_VSOCK_H */
>diff --git a/net/vmw_vsock/virtio_transport.c b/net/vmw_vsock/virtio_transport.c
>index af5bab1acee1..8007593a3a93 100644
>--- a/net/vmw_vsock/virtio_transport.c
>+++ b/net/vmw_vsock/virtio_transport.c
>@@ -539,6 +539,7 @@ static struct virtio_transport virtio_transport = {
> .notify_buffer_size = virtio_transport_notify_buffer_size,
>
> .read_skb = virtio_transport_read_skb,
>+ .notify_set_rcvlowat = virtio_transport_notify_set_rcvlowat
> },
>
> .send_pkt = virtio_transport_send_pkt,
>diff --git a/net/vmw_vsock/virtio_transport_common.c b/net/vmw_vsock/virtio_transport_common.c
>index f6dc896bf44c..1cb556ad4597 100644
>--- a/net/vmw_vsock/virtio_transport_common.c
>+++ b/net/vmw_vsock/virtio_transport_common.c
>@@ -1684,6 +1684,33 @@ int virtio_transport_read_skb(struct vsock_sock *vsk, skb_read_actor_t recv_acto
> }
> EXPORT_SYMBOL_GPL(virtio_transport_read_skb);
>
>+int virtio_transport_notify_set_rcvlowat(struct vsock_sock *vsk, int val)
>+{
>+ struct virtio_vsock_sock *vvs = vsk->trans;
>+ bool send_update;
>+
>+ spin_lock_bh(&vvs->rx_lock);
>+
>+ /* If number of available bytes is less than new SO_RCVLOWAT value,
>+ * kick sender to send more data, because sender may sleep in its
>+ * 'send()' syscall waiting for enough space at our side.
>+ */
>+ send_update = vvs->rx_bytes < val;
>+
>+ spin_unlock_bh(&vvs->rx_lock);
>+
>+ if (send_update) {
>+ int err;
>+
>+ err = virtio_transport_send_credit_update(vsk);
>+ if (err < 0)
>+ return err;
>+ }
>+
>+ return 0;
>+}
>+EXPORT_SYMBOL_GPL(virtio_transport_notify_set_rcvlowat);
>+
> MODULE_LICENSE("GPL v2");
> MODULE_AUTHOR("Asias He");
> MODULE_DESCRIPTION("common code for virtio vsock");
>diff --git a/net/vmw_vsock/vsock_loopback.c b/net/vmw_vsock/vsock_loopback.c
>index 048640167411..9f4b814fbbc7 100644
>--- a/net/vmw_vsock/vsock_loopback.c
>+++ b/net/vmw_vsock/vsock_loopback.c
>@@ -98,6 +98,7 @@ static struct virtio_transport loopback_transport = {
> .notify_buffer_size = virtio_transport_notify_buffer_size,
>
> .read_skb = virtio_transport_read_skb,
>+ .notify_set_rcvlowat = virtio_transport_notify_set_rcvlowat
> },
>
> .send_pkt = vsock_loopback_send_pkt,
>--
>2.25.1
>
On Thu, Nov 30, 2023 at 04:08:40PM +0300, Arseniy Krasnov wrote:
>Test which checks, that updating SO_RCVLOWAT value also sends credit
>update message. Otherwise mutual hungup may happen when receiver didn't
>send credit update and then calls 'poll()' with non default SO_RCVLOWAT
>value (e.g. waiting enough bytes to read), while sender waits for free
>space at receiver's side. Important thing is that this test relies on
>kernel's define for maximum packet size for virtio transport and this
>value is not exported to user: VIRTIO_VSOCK_MAX_PKT_BUF_SIZE (this
>define is used to control moment when to send credit update message).
>If this value or its usage will be changed in kernel - this test may
>become useless/broken.
>
>Signed-off-by: Arseniy Krasnov <[email protected]>
>---
> Changelog:
> v1 -> v2:
> * Update commit message by removing 'This patch adds XXX' manner.
> * Update commit message by adding details about dependency for this
> test from kernel internal define VIRTIO_VSOCK_MAX_PKT_BUF_SIZE.
> * Add comment for this dependency in 'vsock_test.c' where this define
> is duplicated.
> v2 -> v3:
> * Replace synchronization based on control TCP socket with vsock
> data socket - this is needed to allow sender transmit data only
> when new buffer size of receiver is visible to sender. Otherwise
> there is race and test fails sometimes.
> v3 -> v4:
> * Replace 'recv_buf()' to 'recv(MSG_DONTWAIT)' in last read operation
> in server part. This is needed to ensure that 'poll()' wake up us
> when number of bytes ready to read is equal to SO_RCVLOWAT value.
> v4 -> v5:
> * Use 'recv_buf(MSG_DONTWAIT)' instead of 'recv(MSG_DONTWAIT)'.
>
> tools/testing/vsock/vsock_test.c | 142 +++++++++++++++++++++++++++++++
> 1 file changed, 142 insertions(+)
>
>diff --git a/tools/testing/vsock/vsock_test.c b/tools/testing/vsock/vsock_test.c
>index 01fa816868bc..d66bc4987026 100644
>--- a/tools/testing/vsock/vsock_test.c
>+++ b/tools/testing/vsock/vsock_test.c
>@@ -1232,6 +1232,143 @@ static void test_double_bind_connect_client(const struct test_opts *opts)
> }
> }
>
>+#define RCVLOWAT_CREDIT_UPD_BUF_SIZE (1024 * 128)
>+/* This define is the same as in 'include/linux/virtio_vsock.h':
>+ * it is used to decide when to send credit update message during
>+ * reading from rx queue of a socket. Value and its usage in
>+ * kernel is important for this test.
>+ */
>+#define VIRTIO_VSOCK_MAX_PKT_BUF_SIZE (1024 * 64)
>+
>+static void test_stream_rcvlowat_def_cred_upd_client(const struct test_opts *opts)
>+{
>+ size_t buf_size;
>+ void *buf;
>+ int fd;
>+
>+ fd = vsock_stream_connect(opts->peer_cid, 1234);
>+ if (fd < 0) {
>+ perror("connect");
>+ exit(EXIT_FAILURE);
>+ }
>+
>+ /* Send 1 byte more than peer's buffer size. */
>+ buf_size = RCVLOWAT_CREDIT_UPD_BUF_SIZE + 1;
>+
>+ buf = malloc(buf_size);
>+ if (!buf) {
>+ perror("malloc");
>+ exit(EXIT_FAILURE);
>+ }
>+
>+ /* Wait until peer sets needed buffer size. */
>+ recv_byte(fd, 1, 0);
>+
>+ if (send(fd, buf, buf_size, 0) != buf_size) {
>+ perror("send failed");
>+ exit(EXIT_FAILURE);
>+ }
>+
>+ free(buf);
>+ close(fd);
>+}
>+
>+static void test_stream_rcvlowat_def_cred_upd_server(const struct test_opts *opts)
>+{
>+ size_t recv_buf_size;
>+ struct pollfd fds;
>+ size_t buf_size;
>+ void *buf;
>+ int fd;
>+
>+ fd = vsock_stream_accept(VMADDR_CID_ANY, 1234, NULL);
>+ if (fd < 0) {
>+ perror("accept");
>+ exit(EXIT_FAILURE);
>+ }
>+
>+ buf_size = RCVLOWAT_CREDIT_UPD_BUF_SIZE;
>+
>+ if (setsockopt(fd, AF_VSOCK, SO_VM_SOCKETS_BUFFER_SIZE,
>+ &buf_size, sizeof(buf_size))) {
>+ perror("setsockopt(SO_VM_SOCKETS_BUFFER_SIZE)");
>+ exit(EXIT_FAILURE);
>+ }
>+
>+ /* Send one dummy byte here, because 'setsockopt()' above also
>+ * sends special packet which tells sender to update our buffer
>+ * size. This 'send_byte()' will serialize such packet with data
>+ * reads in a loop below. Sender starts transmission only when
>+ * it receives this single byte.
>+ */
>+ send_byte(fd, 1, 0);
>+
>+ buf = malloc(buf_size);
>+ if (!buf) {
>+ perror("malloc");
>+ exit(EXIT_FAILURE);
>+ }
>+
>+ /* Wait until there will be 128KB of data in rx queue. */
>+ while (1) {
>+ ssize_t res;
>+
>+ res = recv(fd, buf, buf_size, MSG_PEEK);
>+ if (res == buf_size)
>+ break;
>+
>+ if (res <= 0) {
>+ fprintf(stderr, "unexpected 'recv()' return: %zi\n", res);
>+ exit(EXIT_FAILURE);
>+ }
>+ }
>+
>+ /* There is 128KB of data in the socket's rx queue,
>+ * dequeue first 64KB, credit update is not sent.
>+ */
>+ recv_buf_size = VIRTIO_VSOCK_MAX_PKT_BUF_SIZE;
>+ recv_buf(fd, buf, recv_buf_size, 0, recv_buf_size);
>+ recv_buf_size++;
>+
>+ /* Updating SO_RCVLOWAT will send credit update. */
>+ if (setsockopt(fd, SOL_SOCKET, SO_RCVLOWAT,
>+ &recv_buf_size, sizeof(recv_buf_size))) {
>+ perror("setsockopt(SO_RCVLOWAT)");
>+ exit(EXIT_FAILURE);
>+ }
>+
>+ memset(&fds, 0, sizeof(fds));
>+ fds.fd = fd;
>+ fds.events = POLLIN | POLLRDNORM | POLLERR |
>+ POLLRDHUP | POLLHUP;
>+
>+ /* This 'poll()' will return once we receive last byte
>+ * sent by client.
>+ */
>+ if (poll(&fds, 1, -1) < 0) {
>+ perror("poll");
>+ exit(EXIT_FAILURE);
>+ }
>+
>+ if (fds.revents & POLLERR) {
>+ fprintf(stderr, "'poll()' error\n");
>+ exit(EXIT_FAILURE);
>+ }
>+
>+ if (fds.revents & (POLLIN | POLLRDNORM)) {
I would add a comment here, like this:
We expect poll to wake up only after receiving all the
bytes requested with SO_RCVLOWAT, so we don't want the
read to block waiting for more.
The rest LGTM.
Thanks,
Stefano
>+ recv_buf(fd, buf, recv_buf_size, MSG_DONTWAIT, recv_buf_size);
>+ } else {
>+ /* These flags must be set, as there is at
>+ * least 64KB of data ready to read.
>+ */
>+ fprintf(stderr, "POLLIN | POLLRDNORM expected\n");
>+ exit(EXIT_FAILURE);
>+ }
>+
>+ free(buf);
>+ close(fd);
>+}
>+
> static struct test_case test_cases[] = {
> {
> .name = "SOCK_STREAM connection reset",
>@@ -1342,6 +1479,11 @@ static struct test_case test_cases[] = {
> .run_client = test_double_bind_connect_client,
> .run_server = test_double_bind_connect_server,
> },
>+ {
>+ .name = "SOCK_STREAM virtio SO_RCVLOWAT + deferred cred update",
>+ .run_client = test_stream_rcvlowat_def_cred_upd_client,
>+ .run_server = test_stream_rcvlowat_def_cred_upd_server,
>+ },
> {},
> };
>
>--
>2.25.1
>
On Thu, Nov 30, 2023 at 04:08:39PM +0300, Arseniy Krasnov wrote:
> Send credit update message when SO_RCVLOWAT is updated and it is bigger
> than number of bytes in rx queue. It is needed, because 'poll()' will
> wait until number of bytes in rx queue will be not smaller than
> SO_RCVLOWAT, so kick sender to send more data. Otherwise mutual hungup
> for tx/rx is possible: sender waits for free space and receiver is
> waiting data in 'poll()'.
>
> Signed-off-by: Arseniy Krasnov <[email protected]>
> ---
> Changelog:
> v1 -> v2:
> * Update commit message by removing 'This patch adds XXX' manner.
> * Do not initialize 'send_update' variable - set it directly during
> first usage.
> v3 -> v4:
> * Fit comment in 'virtio_transport_notify_set_rcvlowat()' to 80 chars.
> v4 -> v5:
> * Do not change callbacks order in transport structures.
>
> drivers/vhost/vsock.c | 1 +
> include/linux/virtio_vsock.h | 1 +
> net/vmw_vsock/virtio_transport.c | 1 +
> net/vmw_vsock/virtio_transport_common.c | 27 +++++++++++++++++++++++++
> net/vmw_vsock/vsock_loopback.c | 1 +
> 5 files changed, 31 insertions(+)
>
> diff --git a/drivers/vhost/vsock.c b/drivers/vhost/vsock.c
> index f75731396b7e..4146f80db8ac 100644
> --- a/drivers/vhost/vsock.c
> +++ b/drivers/vhost/vsock.c
> @@ -451,6 +451,7 @@ static struct virtio_transport vhost_transport = {
> .notify_buffer_size = virtio_transport_notify_buffer_size,
>
> .read_skb = virtio_transport_read_skb,
> + .notify_set_rcvlowat = virtio_transport_notify_set_rcvlowat
> },
>
> .send_pkt = vhost_transport_send_pkt,
> diff --git a/include/linux/virtio_vsock.h b/include/linux/virtio_vsock.h
> index ebb3ce63d64d..c82089dee0c8 100644
> --- a/include/linux/virtio_vsock.h
> +++ b/include/linux/virtio_vsock.h
> @@ -256,4 +256,5 @@ void virtio_transport_put_credit(struct virtio_vsock_sock *vvs, u32 credit);
> void virtio_transport_deliver_tap_pkt(struct sk_buff *skb);
> int virtio_transport_purge_skbs(void *vsk, struct sk_buff_head *list);
> int virtio_transport_read_skb(struct vsock_sock *vsk, skb_read_actor_t read_actor);
> +int virtio_transport_notify_set_rcvlowat(struct vsock_sock *vsk, int val);
> #endif /* _LINUX_VIRTIO_VSOCK_H */
> diff --git a/net/vmw_vsock/virtio_transport.c b/net/vmw_vsock/virtio_transport.c
> index af5bab1acee1..8007593a3a93 100644
> --- a/net/vmw_vsock/virtio_transport.c
> +++ b/net/vmw_vsock/virtio_transport.c
> @@ -539,6 +539,7 @@ static struct virtio_transport virtio_transport = {
> .notify_buffer_size = virtio_transport_notify_buffer_size,
>
> .read_skb = virtio_transport_read_skb,
> + .notify_set_rcvlowat = virtio_transport_notify_set_rcvlowat
> },
>
> .send_pkt = virtio_transport_send_pkt,
> diff --git a/net/vmw_vsock/virtio_transport_common.c b/net/vmw_vsock/virtio_transport_common.c
> index f6dc896bf44c..1cb556ad4597 100644
> --- a/net/vmw_vsock/virtio_transport_common.c
> +++ b/net/vmw_vsock/virtio_transport_common.c
> @@ -1684,6 +1684,33 @@ int virtio_transport_read_skb(struct vsock_sock *vsk, skb_read_actor_t recv_acto
> }
> EXPORT_SYMBOL_GPL(virtio_transport_read_skb);
>
> +int virtio_transport_notify_set_rcvlowat(struct vsock_sock *vsk, int val)
> +{
> + struct virtio_vsock_sock *vvs = vsk->trans;
> + bool send_update;
> +
> + spin_lock_bh(&vvs->rx_lock);
> +
> + /* If number of available bytes is less than new SO_RCVLOWAT value,
> + * kick sender to send more data, because sender may sleep in its
> + * 'send()' syscall waiting for enough space at our side.
> + */
> + send_update = vvs->rx_bytes < val;
> +
> + spin_unlock_bh(&vvs->rx_lock);
> +
> + if (send_update) {
> + int err;
> +
> + err = virtio_transport_send_credit_update(vsk);
> + if (err < 0)
> + return err;
> + }
> +
> + return 0;
> +}
I find it strange that this will send a credit update
even if nothing changed since this was called previously.
I'm not sure whether this is a problem protocol-wise,
but it certainly was not envisioned when the protocol was
built. WDYT?
> +EXPORT_SYMBOL_GPL(virtio_transport_notify_set_rcvlowat);
> +
> MODULE_LICENSE("GPL v2");
> MODULE_AUTHOR("Asias He");
> MODULE_DESCRIPTION("common code for virtio vsock");
> diff --git a/net/vmw_vsock/vsock_loopback.c b/net/vmw_vsock/vsock_loopback.c
> index 048640167411..9f4b814fbbc7 100644
> --- a/net/vmw_vsock/vsock_loopback.c
> +++ b/net/vmw_vsock/vsock_loopback.c
> @@ -98,6 +98,7 @@ static struct virtio_transport loopback_transport = {
> .notify_buffer_size = virtio_transport_notify_buffer_size,
>
> .read_skb = virtio_transport_read_skb,
> + .notify_set_rcvlowat = virtio_transport_notify_set_rcvlowat
> },
>
> .send_pkt = vsock_loopback_send_pkt,
> --
> 2.25.1
On 30.11.2023 16:42, Michael S. Tsirkin wrote:
> On Thu, Nov 30, 2023 at 04:08:39PM +0300, Arseniy Krasnov wrote:
>> Send credit update message when SO_RCVLOWAT is updated and it is bigger
>> than number of bytes in rx queue. It is needed, because 'poll()' will
>> wait until number of bytes in rx queue will be not smaller than
>> SO_RCVLOWAT, so kick sender to send more data. Otherwise mutual hungup
>> for tx/rx is possible: sender waits for free space and receiver is
>> waiting data in 'poll()'.
>>
>> Signed-off-by: Arseniy Krasnov <[email protected]>
>> ---
>> Changelog:
>> v1 -> v2:
>> * Update commit message by removing 'This patch adds XXX' manner.
>> * Do not initialize 'send_update' variable - set it directly during
>> first usage.
>> v3 -> v4:
>> * Fit comment in 'virtio_transport_notify_set_rcvlowat()' to 80 chars.
>> v4 -> v5:
>> * Do not change callbacks order in transport structures.
>>
>> drivers/vhost/vsock.c | 1 +
>> include/linux/virtio_vsock.h | 1 +
>> net/vmw_vsock/virtio_transport.c | 1 +
>> net/vmw_vsock/virtio_transport_common.c | 27 +++++++++++++++++++++++++
>> net/vmw_vsock/vsock_loopback.c | 1 +
>> 5 files changed, 31 insertions(+)
>>
>> diff --git a/drivers/vhost/vsock.c b/drivers/vhost/vsock.c
>> index f75731396b7e..4146f80db8ac 100644
>> --- a/drivers/vhost/vsock.c
>> +++ b/drivers/vhost/vsock.c
>> @@ -451,6 +451,7 @@ static struct virtio_transport vhost_transport = {
>> .notify_buffer_size = virtio_transport_notify_buffer_size,
>>
>> .read_skb = virtio_transport_read_skb,
>> + .notify_set_rcvlowat = virtio_transport_notify_set_rcvlowat
>> },
>>
>> .send_pkt = vhost_transport_send_pkt,
>> diff --git a/include/linux/virtio_vsock.h b/include/linux/virtio_vsock.h
>> index ebb3ce63d64d..c82089dee0c8 100644
>> --- a/include/linux/virtio_vsock.h
>> +++ b/include/linux/virtio_vsock.h
>> @@ -256,4 +256,5 @@ void virtio_transport_put_credit(struct virtio_vsock_sock *vvs, u32 credit);
>> void virtio_transport_deliver_tap_pkt(struct sk_buff *skb);
>> int virtio_transport_purge_skbs(void *vsk, struct sk_buff_head *list);
>> int virtio_transport_read_skb(struct vsock_sock *vsk, skb_read_actor_t read_actor);
>> +int virtio_transport_notify_set_rcvlowat(struct vsock_sock *vsk, int val);
>> #endif /* _LINUX_VIRTIO_VSOCK_H */
>> diff --git a/net/vmw_vsock/virtio_transport.c b/net/vmw_vsock/virtio_transport.c
>> index af5bab1acee1..8007593a3a93 100644
>> --- a/net/vmw_vsock/virtio_transport.c
>> +++ b/net/vmw_vsock/virtio_transport.c
>> @@ -539,6 +539,7 @@ static struct virtio_transport virtio_transport = {
>> .notify_buffer_size = virtio_transport_notify_buffer_size,
>>
>> .read_skb = virtio_transport_read_skb,
>> + .notify_set_rcvlowat = virtio_transport_notify_set_rcvlowat
>> },
>>
>> .send_pkt = virtio_transport_send_pkt,
>> diff --git a/net/vmw_vsock/virtio_transport_common.c b/net/vmw_vsock/virtio_transport_common.c
>> index f6dc896bf44c..1cb556ad4597 100644
>> --- a/net/vmw_vsock/virtio_transport_common.c
>> +++ b/net/vmw_vsock/virtio_transport_common.c
>> @@ -1684,6 +1684,33 @@ int virtio_transport_read_skb(struct vsock_sock *vsk, skb_read_actor_t recv_acto
>> }
>> EXPORT_SYMBOL_GPL(virtio_transport_read_skb);
>>
>> +int virtio_transport_notify_set_rcvlowat(struct vsock_sock *vsk, int val)
>> +{
>> + struct virtio_vsock_sock *vvs = vsk->trans;
>> + bool send_update;
>> +
>> + spin_lock_bh(&vvs->rx_lock);
>> +
>> + /* If number of available bytes is less than new SO_RCVLOWAT value,
>> + * kick sender to send more data, because sender may sleep in its
>> + * 'send()' syscall waiting for enough space at our side.
>> + */
>> + send_update = vvs->rx_bytes < val;
>> +
>> + spin_unlock_bh(&vvs->rx_lock);
>> +
>> + if (send_update) {
>> + int err;
>> +
>> + err = virtio_transport_send_credit_update(vsk);
>> + if (err < 0)
>> + return err;
>> + }
>> +
>> + return 0;
>> +}
>
>
> I find it strange that this will send a credit update
> even if nothing changed since this was called previously.
> I'm not sure whether this is a problem protocol-wise,
> but it certainly was not envisioned when the protocol was
> built. WDYT?
From virtio spec I found:
It is also valid to send a VIRTIO_VSOCK_OP_CREDIT_UPDATE packet without previously receiving a
VIRTIO_VSOCK_OP_CREDIT_REQUEST packet. This allows communicating updates any time a change
in buffer space occurs.
So I guess there is no limitations to send such type of packet, e.g. it is not
required to be a reply for some another packet. Please, correct me if im wrong.
Thanks, Arseniy
>
>
>> +EXPORT_SYMBOL_GPL(virtio_transport_notify_set_rcvlowat);
>> +
>> MODULE_LICENSE("GPL v2");
>> MODULE_AUTHOR("Asias He");
>> MODULE_DESCRIPTION("common code for virtio vsock");
>> diff --git a/net/vmw_vsock/vsock_loopback.c b/net/vmw_vsock/vsock_loopback.c
>> index 048640167411..9f4b814fbbc7 100644
>> --- a/net/vmw_vsock/vsock_loopback.c
>> +++ b/net/vmw_vsock/vsock_loopback.c
>> @@ -98,6 +98,7 @@ static struct virtio_transport loopback_transport = {
>> .notify_buffer_size = virtio_transport_notify_buffer_size,
>>
>> .read_skb = virtio_transport_read_skb,
>> + .notify_set_rcvlowat = virtio_transport_notify_set_rcvlowat
>> },
>>
>> .send_pkt = vsock_loopback_send_pkt,
>> --
>> 2.25.1
>
On Thu, Nov 30, 2023 at 04:43:34PM +0300, Arseniy Krasnov wrote:
>
>
> On 30.11.2023 16:42, Michael S. Tsirkin wrote:
> > On Thu, Nov 30, 2023 at 04:08:39PM +0300, Arseniy Krasnov wrote:
> >> Send credit update message when SO_RCVLOWAT is updated and it is bigger
> >> than number of bytes in rx queue. It is needed, because 'poll()' will
> >> wait until number of bytes in rx queue will be not smaller than
> >> SO_RCVLOWAT, so kick sender to send more data. Otherwise mutual hungup
> >> for tx/rx is possible: sender waits for free space and receiver is
> >> waiting data in 'poll()'.
> >>
> >> Signed-off-by: Arseniy Krasnov <[email protected]>
> >> ---
> >> Changelog:
> >> v1 -> v2:
> >> * Update commit message by removing 'This patch adds XXX' manner.
> >> * Do not initialize 'send_update' variable - set it directly during
> >> first usage.
> >> v3 -> v4:
> >> * Fit comment in 'virtio_transport_notify_set_rcvlowat()' to 80 chars.
> >> v4 -> v5:
> >> * Do not change callbacks order in transport structures.
> >>
> >> drivers/vhost/vsock.c | 1 +
> >> include/linux/virtio_vsock.h | 1 +
> >> net/vmw_vsock/virtio_transport.c | 1 +
> >> net/vmw_vsock/virtio_transport_common.c | 27 +++++++++++++++++++++++++
> >> net/vmw_vsock/vsock_loopback.c | 1 +
> >> 5 files changed, 31 insertions(+)
> >>
> >> diff --git a/drivers/vhost/vsock.c b/drivers/vhost/vsock.c
> >> index f75731396b7e..4146f80db8ac 100644
> >> --- a/drivers/vhost/vsock.c
> >> +++ b/drivers/vhost/vsock.c
> >> @@ -451,6 +451,7 @@ static struct virtio_transport vhost_transport = {
> >> .notify_buffer_size = virtio_transport_notify_buffer_size,
> >>
> >> .read_skb = virtio_transport_read_skb,
> >> + .notify_set_rcvlowat = virtio_transport_notify_set_rcvlowat
> >> },
> >>
> >> .send_pkt = vhost_transport_send_pkt,
> >> diff --git a/include/linux/virtio_vsock.h b/include/linux/virtio_vsock.h
> >> index ebb3ce63d64d..c82089dee0c8 100644
> >> --- a/include/linux/virtio_vsock.h
> >> +++ b/include/linux/virtio_vsock.h
> >> @@ -256,4 +256,5 @@ void virtio_transport_put_credit(struct virtio_vsock_sock *vvs, u32 credit);
> >> void virtio_transport_deliver_tap_pkt(struct sk_buff *skb);
> >> int virtio_transport_purge_skbs(void *vsk, struct sk_buff_head *list);
> >> int virtio_transport_read_skb(struct vsock_sock *vsk, skb_read_actor_t read_actor);
> >> +int virtio_transport_notify_set_rcvlowat(struct vsock_sock *vsk, int val);
> >> #endif /* _LINUX_VIRTIO_VSOCK_H */
> >> diff --git a/net/vmw_vsock/virtio_transport.c b/net/vmw_vsock/virtio_transport.c
> >> index af5bab1acee1..8007593a3a93 100644
> >> --- a/net/vmw_vsock/virtio_transport.c
> >> +++ b/net/vmw_vsock/virtio_transport.c
> >> @@ -539,6 +539,7 @@ static struct virtio_transport virtio_transport = {
> >> .notify_buffer_size = virtio_transport_notify_buffer_size,
> >>
> >> .read_skb = virtio_transport_read_skb,
> >> + .notify_set_rcvlowat = virtio_transport_notify_set_rcvlowat
> >> },
> >>
> >> .send_pkt = virtio_transport_send_pkt,
> >> diff --git a/net/vmw_vsock/virtio_transport_common.c b/net/vmw_vsock/virtio_transport_common.c
> >> index f6dc896bf44c..1cb556ad4597 100644
> >> --- a/net/vmw_vsock/virtio_transport_common.c
> >> +++ b/net/vmw_vsock/virtio_transport_common.c
> >> @@ -1684,6 +1684,33 @@ int virtio_transport_read_skb(struct vsock_sock *vsk, skb_read_actor_t recv_acto
> >> }
> >> EXPORT_SYMBOL_GPL(virtio_transport_read_skb);
> >>
> >> +int virtio_transport_notify_set_rcvlowat(struct vsock_sock *vsk, int val)
> >> +{
> >> + struct virtio_vsock_sock *vvs = vsk->trans;
> >> + bool send_update;
> >> +
> >> + spin_lock_bh(&vvs->rx_lock);
> >> +
> >> + /* If number of available bytes is less than new SO_RCVLOWAT value,
> >> + * kick sender to send more data, because sender may sleep in its
> >> + * 'send()' syscall waiting for enough space at our side.
> >> + */
> >> + send_update = vvs->rx_bytes < val;
> >> +
> >> + spin_unlock_bh(&vvs->rx_lock);
> >> +
> >> + if (send_update) {
> >> + int err;
> >> +
> >> + err = virtio_transport_send_credit_update(vsk);
> >> + if (err < 0)
> >> + return err;
> >> + }
> >> +
> >> + return 0;
> >> +}
> >
> >
> > I find it strange that this will send a credit update
> > even if nothing changed since this was called previously.
> > I'm not sure whether this is a problem protocol-wise,
> > but it certainly was not envisioned when the protocol was
> > built. WDYT?
>
> >From virtio spec I found:
>
> It is also valid to send a VIRTIO_VSOCK_OP_CREDIT_UPDATE packet without previously receiving a
> VIRTIO_VSOCK_OP_CREDIT_REQUEST packet. This allows communicating updates any time a change
> in buffer space occurs.
> So I guess there is no limitations to send such type of packet, e.g. it is not
> required to be a reply for some another packet. Please, correct me if im wrong.
>
> Thanks, Arseniy
Absolutely. My point was different - with this patch it is possible
that you are not adding any credits at all since the previous
VIRTIO_VSOCK_OP_CREDIT_UPDATE.
>
> >
> >
> >> +EXPORT_SYMBOL_GPL(virtio_transport_notify_set_rcvlowat);
> >> +
> >> MODULE_LICENSE("GPL v2");
> >> MODULE_AUTHOR("Asias He");
> >> MODULE_DESCRIPTION("common code for virtio vsock");
> >> diff --git a/net/vmw_vsock/vsock_loopback.c b/net/vmw_vsock/vsock_loopback.c
> >> index 048640167411..9f4b814fbbc7 100644
> >> --- a/net/vmw_vsock/vsock_loopback.c
> >> +++ b/net/vmw_vsock/vsock_loopback.c
> >> @@ -98,6 +98,7 @@ static struct virtio_transport loopback_transport = {
> >> .notify_buffer_size = virtio_transport_notify_buffer_size,
> >>
> >> .read_skb = virtio_transport_read_skb,
> >> + .notify_set_rcvlowat = virtio_transport_notify_set_rcvlowat
> >> },
> >>
> >> .send_pkt = vsock_loopback_send_pkt,
> >> --
> >> 2.25.1
> >
On 30.11.2023 16:58, Michael S. Tsirkin wrote:
> On Thu, Nov 30, 2023 at 04:43:34PM +0300, Arseniy Krasnov wrote:
>>
>>
>> On 30.11.2023 16:42, Michael S. Tsirkin wrote:
>>> On Thu, Nov 30, 2023 at 04:08:39PM +0300, Arseniy Krasnov wrote:
>>>> Send credit update message when SO_RCVLOWAT is updated and it is bigger
>>>> than number of bytes in rx queue. It is needed, because 'poll()' will
>>>> wait until number of bytes in rx queue will be not smaller than
>>>> SO_RCVLOWAT, so kick sender to send more data. Otherwise mutual hungup
>>>> for tx/rx is possible: sender waits for free space and receiver is
>>>> waiting data in 'poll()'.
>>>>
>>>> Signed-off-by: Arseniy Krasnov <[email protected]>
>>>> ---
>>>> Changelog:
>>>> v1 -> v2:
>>>> * Update commit message by removing 'This patch adds XXX' manner.
>>>> * Do not initialize 'send_update' variable - set it directly during
>>>> first usage.
>>>> v3 -> v4:
>>>> * Fit comment in 'virtio_transport_notify_set_rcvlowat()' to 80 chars.
>>>> v4 -> v5:
>>>> * Do not change callbacks order in transport structures.
>>>>
>>>> drivers/vhost/vsock.c | 1 +
>>>> include/linux/virtio_vsock.h | 1 +
>>>> net/vmw_vsock/virtio_transport.c | 1 +
>>>> net/vmw_vsock/virtio_transport_common.c | 27 +++++++++++++++++++++++++
>>>> net/vmw_vsock/vsock_loopback.c | 1 +
>>>> 5 files changed, 31 insertions(+)
>>>>
>>>> diff --git a/drivers/vhost/vsock.c b/drivers/vhost/vsock.c
>>>> index f75731396b7e..4146f80db8ac 100644
>>>> --- a/drivers/vhost/vsock.c
>>>> +++ b/drivers/vhost/vsock.c
>>>> @@ -451,6 +451,7 @@ static struct virtio_transport vhost_transport = {
>>>> .notify_buffer_size = virtio_transport_notify_buffer_size,
>>>>
>>>> .read_skb = virtio_transport_read_skb,
>>>> + .notify_set_rcvlowat = virtio_transport_notify_set_rcvlowat
>>>> },
>>>>
>>>> .send_pkt = vhost_transport_send_pkt,
>>>> diff --git a/include/linux/virtio_vsock.h b/include/linux/virtio_vsock.h
>>>> index ebb3ce63d64d..c82089dee0c8 100644
>>>> --- a/include/linux/virtio_vsock.h
>>>> +++ b/include/linux/virtio_vsock.h
>>>> @@ -256,4 +256,5 @@ void virtio_transport_put_credit(struct virtio_vsock_sock *vvs, u32 credit);
>>>> void virtio_transport_deliver_tap_pkt(struct sk_buff *skb);
>>>> int virtio_transport_purge_skbs(void *vsk, struct sk_buff_head *list);
>>>> int virtio_transport_read_skb(struct vsock_sock *vsk, skb_read_actor_t read_actor);
>>>> +int virtio_transport_notify_set_rcvlowat(struct vsock_sock *vsk, int val);
>>>> #endif /* _LINUX_VIRTIO_VSOCK_H */
>>>> diff --git a/net/vmw_vsock/virtio_transport.c b/net/vmw_vsock/virtio_transport.c
>>>> index af5bab1acee1..8007593a3a93 100644
>>>> --- a/net/vmw_vsock/virtio_transport.c
>>>> +++ b/net/vmw_vsock/virtio_transport.c
>>>> @@ -539,6 +539,7 @@ static struct virtio_transport virtio_transport = {
>>>> .notify_buffer_size = virtio_transport_notify_buffer_size,
>>>>
>>>> .read_skb = virtio_transport_read_skb,
>>>> + .notify_set_rcvlowat = virtio_transport_notify_set_rcvlowat
>>>> },
>>>>
>>>> .send_pkt = virtio_transport_send_pkt,
>>>> diff --git a/net/vmw_vsock/virtio_transport_common.c b/net/vmw_vsock/virtio_transport_common.c
>>>> index f6dc896bf44c..1cb556ad4597 100644
>>>> --- a/net/vmw_vsock/virtio_transport_common.c
>>>> +++ b/net/vmw_vsock/virtio_transport_common.c
>>>> @@ -1684,6 +1684,33 @@ int virtio_transport_read_skb(struct vsock_sock *vsk, skb_read_actor_t recv_acto
>>>> }
>>>> EXPORT_SYMBOL_GPL(virtio_transport_read_skb);
>>>>
>>>> +int virtio_transport_notify_set_rcvlowat(struct vsock_sock *vsk, int val)
>>>> +{
>>>> + struct virtio_vsock_sock *vvs = vsk->trans;
>>>> + bool send_update;
>>>> +
>>>> + spin_lock_bh(&vvs->rx_lock);
>>>> +
>>>> + /* If number of available bytes is less than new SO_RCVLOWAT value,
>>>> + * kick sender to send more data, because sender may sleep in its
>>>> + * 'send()' syscall waiting for enough space at our side.
>>>> + */
>>>> + send_update = vvs->rx_bytes < val;
>>>> +
>>>> + spin_unlock_bh(&vvs->rx_lock);
>>>> +
>>>> + if (send_update) {
>>>> + int err;
>>>> +
>>>> + err = virtio_transport_send_credit_update(vsk);
>>>> + if (err < 0)
>>>> + return err;
>>>> + }
>>>> +
>>>> + return 0;
>>>> +}
>>>
>>>
>>> I find it strange that this will send a credit update
>>> even if nothing changed since this was called previously.
>>> I'm not sure whether this is a problem protocol-wise,
>>> but it certainly was not envisioned when the protocol was
>>> built. WDYT?
>>
>> >From virtio spec I found:
>>
>> It is also valid to send a VIRTIO_VSOCK_OP_CREDIT_UPDATE packet without previously receiving a
>> VIRTIO_VSOCK_OP_CREDIT_REQUEST packet. This allows communicating updates any time a change
>> in buffer space occurs.
>> So I guess there is no limitations to send such type of packet, e.g. it is not
>> required to be a reply for some another packet. Please, correct me if im wrong.
>>
>> Thanks, Arseniy
>
>
> Absolutely. My point was different - with this patch it is possible
> that you are not adding any credits at all since the previous
> VIRTIO_VSOCK_OP_CREDIT_UPDATE.
I see, @Stefano, what do you think ?
Thanks, Arseniy
>
>>
>>>
>>>
>>>> +EXPORT_SYMBOL_GPL(virtio_transport_notify_set_rcvlowat);
>>>> +
>>>> MODULE_LICENSE("GPL v2");
>>>> MODULE_AUTHOR("Asias He");
>>>> MODULE_DESCRIPTION("common code for virtio vsock");
>>>> diff --git a/net/vmw_vsock/vsock_loopback.c b/net/vmw_vsock/vsock_loopback.c
>>>> index 048640167411..9f4b814fbbc7 100644
>>>> --- a/net/vmw_vsock/vsock_loopback.c
>>>> +++ b/net/vmw_vsock/vsock_loopback.c
>>>> @@ -98,6 +98,7 @@ static struct virtio_transport loopback_transport = {
>>>> .notify_buffer_size = virtio_transport_notify_buffer_size,
>>>>
>>>> .read_skb = virtio_transport_read_skb,
>>>> + .notify_set_rcvlowat = virtio_transport_notify_set_rcvlowat
>>>> },
>>>>
>>>> .send_pkt = vsock_loopback_send_pkt,
>>>> --
>>>> 2.25.1
>>>
>
On Thu, Nov 30, 2023 at 08:58:58AM -0500, Michael S. Tsirkin wrote:
>On Thu, Nov 30, 2023 at 04:43:34PM +0300, Arseniy Krasnov wrote:
>>
>>
>> On 30.11.2023 16:42, Michael S. Tsirkin wrote:
>> > On Thu, Nov 30, 2023 at 04:08:39PM +0300, Arseniy Krasnov wrote:
>> >> Send credit update message when SO_RCVLOWAT is updated and it is bigger
>> >> than number of bytes in rx queue. It is needed, because 'poll()' will
>> >> wait until number of bytes in rx queue will be not smaller than
>> >> SO_RCVLOWAT, so kick sender to send more data. Otherwise mutual hungup
>> >> for tx/rx is possible: sender waits for free space and receiver is
>> >> waiting data in 'poll()'.
>> >>
>> >> Signed-off-by: Arseniy Krasnov <[email protected]>
>> >> ---
>> >> Changelog:
>> >> v1 -> v2:
>> >> * Update commit message by removing 'This patch adds XXX' manner.
>> >> * Do not initialize 'send_update' variable - set it directly during
>> >> first usage.
>> >> v3 -> v4:
>> >> * Fit comment in 'virtio_transport_notify_set_rcvlowat()' to 80 chars.
>> >> v4 -> v5:
>> >> * Do not change callbacks order in transport structures.
>> >>
>> >> drivers/vhost/vsock.c | 1 +
>> >> include/linux/virtio_vsock.h | 1 +
>> >> net/vmw_vsock/virtio_transport.c | 1 +
>> >> net/vmw_vsock/virtio_transport_common.c | 27 +++++++++++++++++++++++++
>> >> net/vmw_vsock/vsock_loopback.c | 1 +
>> >> 5 files changed, 31 insertions(+)
>> >>
>> >> diff --git a/drivers/vhost/vsock.c b/drivers/vhost/vsock.c
>> >> index f75731396b7e..4146f80db8ac 100644
>> >> --- a/drivers/vhost/vsock.c
>> >> +++ b/drivers/vhost/vsock.c
>> >> @@ -451,6 +451,7 @@ static struct virtio_transport vhost_transport = {
>> >> .notify_buffer_size = virtio_transport_notify_buffer_size,
>> >>
>> >> .read_skb = virtio_transport_read_skb,
>> >> + .notify_set_rcvlowat = virtio_transport_notify_set_rcvlowat
>> >> },
>> >>
>> >> .send_pkt = vhost_transport_send_pkt,
>> >> diff --git a/include/linux/virtio_vsock.h b/include/linux/virtio_vsock.h
>> >> index ebb3ce63d64d..c82089dee0c8 100644
>> >> --- a/include/linux/virtio_vsock.h
>> >> +++ b/include/linux/virtio_vsock.h
>> >> @@ -256,4 +256,5 @@ void virtio_transport_put_credit(struct virtio_vsock_sock *vvs, u32 credit);
>> >> void virtio_transport_deliver_tap_pkt(struct sk_buff *skb);
>> >> int virtio_transport_purge_skbs(void *vsk, struct sk_buff_head *list);
>> >> int virtio_transport_read_skb(struct vsock_sock *vsk, skb_read_actor_t read_actor);
>> >> +int virtio_transport_notify_set_rcvlowat(struct vsock_sock *vsk, int val);
>> >> #endif /* _LINUX_VIRTIO_VSOCK_H */
>> >> diff --git a/net/vmw_vsock/virtio_transport.c b/net/vmw_vsock/virtio_transport.c
>> >> index af5bab1acee1..8007593a3a93 100644
>> >> --- a/net/vmw_vsock/virtio_transport.c
>> >> +++ b/net/vmw_vsock/virtio_transport.c
>> >> @@ -539,6 +539,7 @@ static struct virtio_transport virtio_transport = {
>> >> .notify_buffer_size = virtio_transport_notify_buffer_size,
>> >>
>> >> .read_skb = virtio_transport_read_skb,
>> >> + .notify_set_rcvlowat = virtio_transport_notify_set_rcvlowat
>> >> },
>> >>
>> >> .send_pkt = virtio_transport_send_pkt,
>> >> diff --git a/net/vmw_vsock/virtio_transport_common.c b/net/vmw_vsock/virtio_transport_common.c
>> >> index f6dc896bf44c..1cb556ad4597 100644
>> >> --- a/net/vmw_vsock/virtio_transport_common.c
>> >> +++ b/net/vmw_vsock/virtio_transport_common.c
>> >> @@ -1684,6 +1684,33 @@ int virtio_transport_read_skb(struct vsock_sock *vsk, skb_read_actor_t recv_acto
>> >> }
>> >> EXPORT_SYMBOL_GPL(virtio_transport_read_skb);
>> >>
>> >> +int virtio_transport_notify_set_rcvlowat(struct vsock_sock *vsk,
>> >> int val)
>> >> +{
>> >> + struct virtio_vsock_sock *vvs = vsk->trans;
>> >> + bool send_update;
>> >> +
>> >> + spin_lock_bh(&vvs->rx_lock);
>> >> +
>> >> + /* If number of available bytes is less than new SO_RCVLOWAT value,
>> >> + * kick sender to send more data, because sender may sleep in
>> >> its
>> >> + * 'send()' syscall waiting for enough space at our side.
>> >> + */
>> >> + send_update = vvs->rx_bytes < val;
>> >> +
>> >> + spin_unlock_bh(&vvs->rx_lock);
>> >> +
>> >> + if (send_update) {
>> >> + int err;
>> >> +
>> >> + err = virtio_transport_send_credit_update(vsk);
>> >> + if (err < 0)
>> >> + return err;
>> >> + }
>> >> +
>> >> + return 0;
>> >> +}
>> >
>> >
>> > I find it strange that this will send a credit update
>> > even if nothing changed since this was called previously.
>> > I'm not sure whether this is a problem protocol-wise,
>> > but it certainly was not envisioned when the protocol was
>> > built. WDYT?
>>
>> >From virtio spec I found:
>>
>> It is also valid to send a VIRTIO_VSOCK_OP_CREDIT_UPDATE packet without previously receiving a
>> VIRTIO_VSOCK_OP_CREDIT_REQUEST packet. This allows communicating updates any time a change
>> in buffer space occurs.
>> So I guess there is no limitations to send such type of packet, e.g. it is not
>> required to be a reply for some another packet. Please, correct me if im wrong.
>>
>> Thanks, Arseniy
>
>
>Absolutely. My point was different - with this patch it is possible
>that you are not adding any credits at all since the previous
>VIRTIO_VSOCK_OP_CREDIT_UPDATE.
I think the problem we're solving here is that since as an optimization
we avoid sending the update for every byte we consume, but we put a
threshold, then we make sure we update the peer.
A credit update contains a snapshot and sending it the same as the
previous one should not create any problem.
My doubt now is that we only do this when we set RCVLOWAT , should we
also do something when we consume bytes to avoid the optimization we
have?
Stefano
On 30.11.2023 17:11, Stefano Garzarella wrote:
> On Thu, Nov 30, 2023 at 08:58:58AM -0500, Michael S. Tsirkin wrote:
>> On Thu, Nov 30, 2023 at 04:43:34PM +0300, Arseniy Krasnov wrote:
>>>
>>>
>>> On 30.11.2023 16:42, Michael S. Tsirkin wrote:
>>> > On Thu, Nov 30, 2023 at 04:08:39PM +0300, Arseniy Krasnov wrote:
>>> >> Send credit update message when SO_RCVLOWAT is updated and it is bigger
>>> >> than number of bytes in rx queue. It is needed, because 'poll()' will
>>> >> wait until number of bytes in rx queue will be not smaller than
>>> >> SO_RCVLOWAT, so kick sender to send more data. Otherwise mutual hungup
>>> >> for tx/rx is possible: sender waits for free space and receiver is
>>> >> waiting data in 'poll()'.
>>> >>
>>> >> Signed-off-by: Arseniy Krasnov <[email protected]>
>>> >> ---
>>> >> Changelog:
>>> >> v1 -> v2:
>>> >> * Update commit message by removing 'This patch adds XXX' manner.
>>> >> * Do not initialize 'send_update' variable - set it directly during
>>> >> first usage.
>>> >> v3 -> v4:
>>> >> * Fit comment in 'virtio_transport_notify_set_rcvlowat()' to 80 chars.
>>> >> v4 -> v5:
>>> >> * Do not change callbacks order in transport structures.
>>> >>
>>> >> drivers/vhost/vsock.c | 1 +
>>> >> include/linux/virtio_vsock.h | 1 +
>>> >> net/vmw_vsock/virtio_transport.c | 1 +
>>> >> net/vmw_vsock/virtio_transport_common.c | 27 +++++++++++++++++++++++++
>>> >> net/vmw_vsock/vsock_loopback.c | 1 +
>>> >> 5 files changed, 31 insertions(+)
>>> >>
>>> >> diff --git a/drivers/vhost/vsock.c b/drivers/vhost/vsock.c
>>> >> index f75731396b7e..4146f80db8ac 100644
>>> >> --- a/drivers/vhost/vsock.c
>>> >> +++ b/drivers/vhost/vsock.c
>>> >> @@ -451,6 +451,7 @@ static struct virtio_transport vhost_transport = {
>>> >> .notify_buffer_size = virtio_transport_notify_buffer_size,
>>> >>
>>> >> .read_skb = virtio_transport_read_skb,
>>> >> + .notify_set_rcvlowat = virtio_transport_notify_set_rcvlowat
>>> >> },
>>> >>
>>> >> .send_pkt = vhost_transport_send_pkt,
>>> >> diff --git a/include/linux/virtio_vsock.h b/include/linux/virtio_vsock.h
>>> >> index ebb3ce63d64d..c82089dee0c8 100644
>>> >> --- a/include/linux/virtio_vsock.h
>>> >> +++ b/include/linux/virtio_vsock.h
>>> >> @@ -256,4 +256,5 @@ void virtio_transport_put_credit(struct virtio_vsock_sock *vvs, u32 credit);
>>> >> void virtio_transport_deliver_tap_pkt(struct sk_buff *skb);
>>> >> int virtio_transport_purge_skbs(void *vsk, struct sk_buff_head *list);
>>> >> int virtio_transport_read_skb(struct vsock_sock *vsk, skb_read_actor_t read_actor);
>>> >> +int virtio_transport_notify_set_rcvlowat(struct vsock_sock *vsk, int val);
>>> >> #endif /* _LINUX_VIRTIO_VSOCK_H */
>>> >> diff --git a/net/vmw_vsock/virtio_transport.c b/net/vmw_vsock/virtio_transport.c
>>> >> index af5bab1acee1..8007593a3a93 100644
>>> >> --- a/net/vmw_vsock/virtio_transport.c
>>> >> +++ b/net/vmw_vsock/virtio_transport.c
>>> >> @@ -539,6 +539,7 @@ static struct virtio_transport virtio_transport = {
>>> >> .notify_buffer_size = virtio_transport_notify_buffer_size,
>>> >>
>>> >> .read_skb = virtio_transport_read_skb,
>>> >> + .notify_set_rcvlowat = virtio_transport_notify_set_rcvlowat
>>> >> },
>>> >>
>>> >> .send_pkt = virtio_transport_send_pkt,
>>> >> diff --git a/net/vmw_vsock/virtio_transport_common.c b/net/vmw_vsock/virtio_transport_common.c
>>> >> index f6dc896bf44c..1cb556ad4597 100644
>>> >> --- a/net/vmw_vsock/virtio_transport_common.c
>>> >> +++ b/net/vmw_vsock/virtio_transport_common.c
>>> >> @@ -1684,6 +1684,33 @@ int virtio_transport_read_skb(struct vsock_sock *vsk, skb_read_actor_t recv_acto
>>> >> }
>>> >> EXPORT_SYMBOL_GPL(virtio_transport_read_skb);
>>> >>
>>> >> +int virtio_transport_notify_set_rcvlowat(struct vsock_sock *vsk, >> int val)
>>> >> +{
>>> >> + struct virtio_vsock_sock *vvs = vsk->trans;
>>> >> + bool send_update;
>>> >> +
>>> >> + spin_lock_bh(&vvs->rx_lock);
>>> >> +
>>> >> + /* If number of available bytes is less than new SO_RCVLOWAT value,
>>> >> + * kick sender to send more data, because sender may sleep in >> its
>>> >> + * 'send()' syscall waiting for enough space at our side.
>>> >> + */
>>> >> + send_update = vvs->rx_bytes < val;
>>> >> +
>>> >> + spin_unlock_bh(&vvs->rx_lock);
>>> >> +
>>> >> + if (send_update) {
>>> >> + int err;
>>> >> +
>>> >> + err = virtio_transport_send_credit_update(vsk);
>>> >> + if (err < 0)
>>> >> + return err;
>>> >> + }
>>> >> +
>>> >> + return 0;
>>> >> +}
>>> >
>>> >
>>> > I find it strange that this will send a credit update
>>> > even if nothing changed since this was called previously.
>>> > I'm not sure whether this is a problem protocol-wise,
>>> > but it certainly was not envisioned when the protocol was
>>> > built. WDYT?
>>>
>>> >From virtio spec I found:
>>>
>>> It is also valid to send a VIRTIO_VSOCK_OP_CREDIT_UPDATE packet without previously receiving a
>>> VIRTIO_VSOCK_OP_CREDIT_REQUEST packet. This allows communicating updates any time a change
>>> in buffer space occurs.
>>> So I guess there is no limitations to send such type of packet, e.g. it is not
>>> required to be a reply for some another packet. Please, correct me if im wrong.
>>>
>>> Thanks, Arseniy
>>
>>
>> Absolutely. My point was different - with this patch it is possible
>> that you are not adding any credits at all since the previous
>> VIRTIO_VSOCK_OP_CREDIT_UPDATE.
>
> I think the problem we're solving here is that since as an optimization we avoid sending the update for every byte we consume, but we put a threshold, then we make sure we update the peer.
>
> A credit update contains a snapshot and sending it the same as the previous one should not create any problem.
>
> My doubt now is that we only do this when we set RCVLOWAT , should we also do something when we consume bytes to avoid the optimization we have?
@Michael, Stefano just reproduced problem during bytes reading, but there is already old fix for this, which we forget to merge:)
I think it must be included to this patchset.
https://lore.kernel.org/netdev/[email protected]/
Thanks, Arseniy
>
> Stefano
>
On Thu, Nov 30, 2023 at 06:41:56PM +0300, Arseniy Krasnov wrote:
>
>
> On 30.11.2023 17:11, Stefano Garzarella wrote:
> > On Thu, Nov 30, 2023 at 08:58:58AM -0500, Michael S. Tsirkin wrote:
> >> On Thu, Nov 30, 2023 at 04:43:34PM +0300, Arseniy Krasnov wrote:
> >>>
> >>>
> >>> On 30.11.2023 16:42, Michael S. Tsirkin wrote:
> >>> > On Thu, Nov 30, 2023 at 04:08:39PM +0300, Arseniy Krasnov wrote:
> >>> >> Send credit update message when SO_RCVLOWAT is updated and it is bigger
> >>> >> than number of bytes in rx queue. It is needed, because 'poll()' will
> >>> >> wait until number of bytes in rx queue will be not smaller than
> >>> >> SO_RCVLOWAT, so kick sender to send more data. Otherwise mutual hungup
> >>> >> for tx/rx is possible: sender waits for free space and receiver is
> >>> >> waiting data in 'poll()'.
> >>> >>
> >>> >> Signed-off-by: Arseniy Krasnov <[email protected]>
> >>> >> ---
> >>> >>? Changelog:
> >>> >>? v1 -> v2:
> >>> >>?? * Update commit message by removing 'This patch adds XXX' manner.
> >>> >>?? * Do not initialize 'send_update' variable - set it directly during
> >>> >>???? first usage.
> >>> >>? v3 -> v4:
> >>> >>?? * Fit comment in 'virtio_transport_notify_set_rcvlowat()' to 80 chars.
> >>> >>? v4 -> v5:
> >>> >>?? * Do not change callbacks order in transport structures.
> >>> >>
> >>> >>? drivers/vhost/vsock.c?????????????????? |? 1 +
> >>> >>? include/linux/virtio_vsock.h??????????? |? 1 +
> >>> >>? net/vmw_vsock/virtio_transport.c??????? |? 1 +
> >>> >>? net/vmw_vsock/virtio_transport_common.c | 27 +++++++++++++++++++++++++
> >>> >>? net/vmw_vsock/vsock_loopback.c????????? |? 1 +
> >>> >>? 5 files changed, 31 insertions(+)
> >>> >>
> >>> >> diff --git a/drivers/vhost/vsock.c b/drivers/vhost/vsock.c
> >>> >> index f75731396b7e..4146f80db8ac 100644
> >>> >> --- a/drivers/vhost/vsock.c
> >>> >> +++ b/drivers/vhost/vsock.c
> >>> >> @@ -451,6 +451,7 @@ static struct virtio_transport vhost_transport = {
> >>> >>????????? .notify_buffer_size?????? = virtio_transport_notify_buffer_size,
> >>> >>
> >>> >>????????? .read_skb = virtio_transport_read_skb,
> >>> >> +??????? .notify_set_rcvlowat????? = virtio_transport_notify_set_rcvlowat
> >>> >>????? },
> >>> >>
> >>> >>????? .send_pkt = vhost_transport_send_pkt,
> >>> >> diff --git a/include/linux/virtio_vsock.h b/include/linux/virtio_vsock.h
> >>> >> index ebb3ce63d64d..c82089dee0c8 100644
> >>> >> --- a/include/linux/virtio_vsock.h
> >>> >> +++ b/include/linux/virtio_vsock.h
> >>> >> @@ -256,4 +256,5 @@ void virtio_transport_put_credit(struct virtio_vsock_sock *vvs, u32 credit);
> >>> >>? void virtio_transport_deliver_tap_pkt(struct sk_buff *skb);
> >>> >>? int virtio_transport_purge_skbs(void *vsk, struct sk_buff_head *list);
> >>> >>? int virtio_transport_read_skb(struct vsock_sock *vsk, skb_read_actor_t read_actor);
> >>> >> +int virtio_transport_notify_set_rcvlowat(struct vsock_sock *vsk, int val);
> >>> >>? #endif /* _LINUX_VIRTIO_VSOCK_H */
> >>> >> diff --git a/net/vmw_vsock/virtio_transport.c b/net/vmw_vsock/virtio_transport.c
> >>> >> index af5bab1acee1..8007593a3a93 100644
> >>> >> --- a/net/vmw_vsock/virtio_transport.c
> >>> >> +++ b/net/vmw_vsock/virtio_transport.c
> >>> >> @@ -539,6 +539,7 @@ static struct virtio_transport virtio_transport = {
> >>> >>????????? .notify_buffer_size?????? = virtio_transport_notify_buffer_size,
> >>> >>
> >>> >>????????? .read_skb = virtio_transport_read_skb,
> >>> >> +??????? .notify_set_rcvlowat????? = virtio_transport_notify_set_rcvlowat
> >>> >>????? },
> >>> >>
> >>> >>????? .send_pkt = virtio_transport_send_pkt,
> >>> >> diff --git a/net/vmw_vsock/virtio_transport_common.c b/net/vmw_vsock/virtio_transport_common.c
> >>> >> index f6dc896bf44c..1cb556ad4597 100644
> >>> >> --- a/net/vmw_vsock/virtio_transport_common.c
> >>> >> +++ b/net/vmw_vsock/virtio_transport_common.c
> >>> >> @@ -1684,6 +1684,33 @@ int virtio_transport_read_skb(struct vsock_sock *vsk, skb_read_actor_t recv_acto
> >>> >>? }
> >>> >>? EXPORT_SYMBOL_GPL(virtio_transport_read_skb);
> >>> >>
> >>> >> +int virtio_transport_notify_set_rcvlowat(struct vsock_sock *vsk, >> int val)
> >>> >> +{
> >>> >> +??? struct virtio_vsock_sock *vvs = vsk->trans;
> >>> >> +??? bool send_update;
> >>> >> +
> >>> >> +??? spin_lock_bh(&vvs->rx_lock);
> >>> >> +
> >>> >> +??? /* If number of available bytes is less than new SO_RCVLOWAT value,
> >>> >> +???? * kick sender to send more data, because sender may sleep in >> its
> >>> >> +???? * 'send()' syscall waiting for enough space at our side.
> >>> >> +???? */
> >>> >> +??? send_update = vvs->rx_bytes < val;
> >>> >> +
> >>> >> +??? spin_unlock_bh(&vvs->rx_lock);
> >>> >> +
> >>> >> +??? if (send_update) {
> >>> >> +??????? int err;
> >>> >> +
> >>> >> +??????? err = virtio_transport_send_credit_update(vsk);
> >>> >> +??????? if (err < 0)
> >>> >> +??????????? return err;
> >>> >> +??? }
> >>> >> +
> >>> >> +??? return 0;
> >>> >> +}
> >>> >
> >>> >
> >>> > I find it strange that this will send a credit update
> >>> > even if nothing changed since this was called previously.
> >>> > I'm not sure whether this is a problem protocol-wise,
> >>> > but it certainly was not envisioned when the protocol was
> >>> > built. WDYT?
> >>>
> >>> >From virtio spec I found:
> >>>
> >>> It is also valid to send a VIRTIO_VSOCK_OP_CREDIT_UPDATE packet without previously receiving a
> >>> VIRTIO_VSOCK_OP_CREDIT_REQUEST packet. This allows communicating updates any time a change
> >>> in buffer space occurs.
> >>> So I guess there is no limitations to send such type of packet, e.g. it is not
> >>> required to be a reply for some another packet. Please, correct me if im wrong.
> >>>
> >>> Thanks, Arseniy
> >>
> >>
> >> Absolutely. My point was different - with this patch it is possible
> >> that you are not adding any credits at all since the previous
> >> VIRTIO_VSOCK_OP_CREDIT_UPDATE.
> >
> > I think the problem we're solving here is that since as an optimization we avoid sending the update for every byte we consume, but we put a threshold, then we make sure we update the peer.
> >
> > A credit update contains a snapshot and sending it the same as the previous one should not create any problem.
> >
> > My doubt now is that we only do this when we set RCVLOWAT , should we also do something when we consume bytes to avoid the optimization we have?
>
> @Michael, Stefano just reproduced problem during bytes reading, but there is already old fix for this, which we forget to merge:)
> I think it must be included to this patchset.
>
> https://lore.kernel.org/netdev/[email protected]/
>
> Thanks, Arseniy
I generally don't merge patches tagged as RFC.
Repost without that tag?
Also, it looks like a bugfix we need either way, no?
> >
> > Stefano
> >
On Thu, Nov 30, 2023 at 03:11:19PM +0100, Stefano Garzarella wrote:
> On Thu, Nov 30, 2023 at 08:58:58AM -0500, Michael S. Tsirkin wrote:
> > On Thu, Nov 30, 2023 at 04:43:34PM +0300, Arseniy Krasnov wrote:
> > >
> > >
> > > On 30.11.2023 16:42, Michael S. Tsirkin wrote:
> > > > On Thu, Nov 30, 2023 at 04:08:39PM +0300, Arseniy Krasnov wrote:
> > > >> Send credit update message when SO_RCVLOWAT is updated and it is bigger
> > > >> than number of bytes in rx queue. It is needed, because 'poll()' will
> > > >> wait until number of bytes in rx queue will be not smaller than
> > > >> SO_RCVLOWAT, so kick sender to send more data. Otherwise mutual hungup
> > > >> for tx/rx is possible: sender waits for free space and receiver is
> > > >> waiting data in 'poll()'.
> > > >>
> > > >> Signed-off-by: Arseniy Krasnov <[email protected]>
> > > >> ---
> > > >> Changelog:
> > > >> v1 -> v2:
> > > >> * Update commit message by removing 'This patch adds XXX' manner.
> > > >> * Do not initialize 'send_update' variable - set it directly during
> > > >> first usage.
> > > >> v3 -> v4:
> > > >> * Fit comment in 'virtio_transport_notify_set_rcvlowat()' to 80 chars.
> > > >> v4 -> v5:
> > > >> * Do not change callbacks order in transport structures.
> > > >>
> > > >> drivers/vhost/vsock.c | 1 +
> > > >> include/linux/virtio_vsock.h | 1 +
> > > >> net/vmw_vsock/virtio_transport.c | 1 +
> > > >> net/vmw_vsock/virtio_transport_common.c | 27 +++++++++++++++++++++++++
> > > >> net/vmw_vsock/vsock_loopback.c | 1 +
> > > >> 5 files changed, 31 insertions(+)
> > > >>
> > > >> diff --git a/drivers/vhost/vsock.c b/drivers/vhost/vsock.c
> > > >> index f75731396b7e..4146f80db8ac 100644
> > > >> --- a/drivers/vhost/vsock.c
> > > >> +++ b/drivers/vhost/vsock.c
> > > >> @@ -451,6 +451,7 @@ static struct virtio_transport vhost_transport = {
> > > >> .notify_buffer_size = virtio_transport_notify_buffer_size,
> > > >>
> > > >> .read_skb = virtio_transport_read_skb,
> > > >> + .notify_set_rcvlowat = virtio_transport_notify_set_rcvlowat
> > > >> },
> > > >>
> > > >> .send_pkt = vhost_transport_send_pkt,
> > > >> diff --git a/include/linux/virtio_vsock.h b/include/linux/virtio_vsock.h
> > > >> index ebb3ce63d64d..c82089dee0c8 100644
> > > >> --- a/include/linux/virtio_vsock.h
> > > >> +++ b/include/linux/virtio_vsock.h
> > > >> @@ -256,4 +256,5 @@ void virtio_transport_put_credit(struct virtio_vsock_sock *vvs, u32 credit);
> > > >> void virtio_transport_deliver_tap_pkt(struct sk_buff *skb);
> > > >> int virtio_transport_purge_skbs(void *vsk, struct sk_buff_head *list);
> > > >> int virtio_transport_read_skb(struct vsock_sock *vsk, skb_read_actor_t read_actor);
> > > >> +int virtio_transport_notify_set_rcvlowat(struct vsock_sock *vsk, int val);
> > > >> #endif /* _LINUX_VIRTIO_VSOCK_H */
> > > >> diff --git a/net/vmw_vsock/virtio_transport.c b/net/vmw_vsock/virtio_transport.c
> > > >> index af5bab1acee1..8007593a3a93 100644
> > > >> --- a/net/vmw_vsock/virtio_transport.c
> > > >> +++ b/net/vmw_vsock/virtio_transport.c
> > > >> @@ -539,6 +539,7 @@ static struct virtio_transport virtio_transport = {
> > > >> .notify_buffer_size = virtio_transport_notify_buffer_size,
> > > >>
> > > >> .read_skb = virtio_transport_read_skb,
> > > >> + .notify_set_rcvlowat = virtio_transport_notify_set_rcvlowat
> > > >> },
> > > >>
> > > >> .send_pkt = virtio_transport_send_pkt,
> > > >> diff --git a/net/vmw_vsock/virtio_transport_common.c b/net/vmw_vsock/virtio_transport_common.c
> > > >> index f6dc896bf44c..1cb556ad4597 100644
> > > >> --- a/net/vmw_vsock/virtio_transport_common.c
> > > >> +++ b/net/vmw_vsock/virtio_transport_common.c
> > > >> @@ -1684,6 +1684,33 @@ int virtio_transport_read_skb(struct vsock_sock *vsk, skb_read_actor_t recv_acto
> > > >> }
> > > >> EXPORT_SYMBOL_GPL(virtio_transport_read_skb);
> > > >>
> > > >> +int virtio_transport_notify_set_rcvlowat(struct vsock_sock *vsk,
> > > >> int val)
> > > >> +{
> > > >> + struct virtio_vsock_sock *vvs = vsk->trans;
> > > >> + bool send_update;
> > > >> +
> > > >> + spin_lock_bh(&vvs->rx_lock);
> > > >> +
> > > >> + /* If number of available bytes is less than new SO_RCVLOWAT value,
> > > >> + * kick sender to send more data, because sender may sleep in
> > > >> its
> > > >> + * 'send()' syscall waiting for enough space at our side.
> > > >> + */
> > > >> + send_update = vvs->rx_bytes < val;
> > > >> +
> > > >> + spin_unlock_bh(&vvs->rx_lock);
> > > >> +
> > > >> + if (send_update) {
> > > >> + int err;
> > > >> +
> > > >> + err = virtio_transport_send_credit_update(vsk);
> > > >> + if (err < 0)
> > > >> + return err;
> > > >> + }
> > > >> +
> > > >> + return 0;
> > > >> +}
> > > >
> > > >
> > > > I find it strange that this will send a credit update
> > > > even if nothing changed since this was called previously.
> > > > I'm not sure whether this is a problem protocol-wise,
> > > > but it certainly was not envisioned when the protocol was
> > > > built. WDYT?
> > >
> > > >From virtio spec I found:
> > >
> > > It is also valid to send a VIRTIO_VSOCK_OP_CREDIT_UPDATE packet without previously receiving a
> > > VIRTIO_VSOCK_OP_CREDIT_REQUEST packet. This allows communicating updates any time a change
> > > in buffer space occurs.
> > > So I guess there is no limitations to send such type of packet, e.g. it is not
> > > required to be a reply for some another packet. Please, correct me if im wrong.
> > >
> > > Thanks, Arseniy
> >
> >
> > Absolutely. My point was different - with this patch it is possible
> > that you are not adding any credits at all since the previous
> > VIRTIO_VSOCK_OP_CREDIT_UPDATE.
>
> I think the problem we're solving here is that since as an optimization we
> avoid sending the update for every byte we consume, but we put a threshold,
> then we make sure we update the peer.
>
> A credit update contains a snapshot and sending it the same as the previous
> one should not create any problem.
Well it consumes a buffer on the other side.
> My doubt now is that we only do this when we set RCVLOWAT , should we also
> do something when we consume bytes to avoid the optimization we have?
>
> Stefano
Isn't this why we have credit request?
--
MST
On 30.11.2023 20:37, Michael S. Tsirkin wrote:
> On Thu, Nov 30, 2023 at 06:41:56PM +0300, Arseniy Krasnov wrote:
>>
>>
>> On 30.11.2023 17:11, Stefano Garzarella wrote:
>>> On Thu, Nov 30, 2023 at 08:58:58AM -0500, Michael S. Tsirkin wrote:
>>>> On Thu, Nov 30, 2023 at 04:43:34PM +0300, Arseniy Krasnov wrote:
>>>>>
>>>>>
>>>>> On 30.11.2023 16:42, Michael S. Tsirkin wrote:
>>>>>> On Thu, Nov 30, 2023 at 04:08:39PM +0300, Arseniy Krasnov wrote:
>>>>>>> Send credit update message when SO_RCVLOWAT is updated and it is bigger
>>>>>>> than number of bytes in rx queue. It is needed, because 'poll()' will
>>>>>>> wait until number of bytes in rx queue will be not smaller than
>>>>>>> SO_RCVLOWAT, so kick sender to send more data. Otherwise mutual hungup
>>>>>>> for tx/rx is possible: sender waits for free space and receiver is
>>>>>>> waiting data in 'poll()'.
>>>>>>>
>>>>>>> Signed-off-by: Arseniy Krasnov <[email protected]>
>>>>>>> ---
>>>>>>> Changelog:
>>>>>>> v1 -> v2:
>>>>>>> * Update commit message by removing 'This patch adds XXX' manner.
>>>>>>> * Do not initialize 'send_update' variable - set it directly during
>>>>>>> first usage.
>>>>>>> v3 -> v4:
>>>>>>> * Fit comment in 'virtio_transport_notify_set_rcvlowat()' to 80 chars.
>>>>>>> v4 -> v5:
>>>>>>> * Do not change callbacks order in transport structures.
>>>>>>>
>>>>>>> drivers/vhost/vsock.c | 1 +
>>>>>>> include/linux/virtio_vsock.h | 1 +
>>>>>>> net/vmw_vsock/virtio_transport.c | 1 +
>>>>>>> net/vmw_vsock/virtio_transport_common.c | 27 +++++++++++++++++++++++++
>>>>>>> net/vmw_vsock/vsock_loopback.c | 1 +
>>>>>>> 5 files changed, 31 insertions(+)
>>>>>>>
>>>>>>> diff --git a/drivers/vhost/vsock.c b/drivers/vhost/vsock.c
>>>>>>> index f75731396b7e..4146f80db8ac 100644
>>>>>>> --- a/drivers/vhost/vsock.c
>>>>>>> +++ b/drivers/vhost/vsock.c
>>>>>>> @@ -451,6 +451,7 @@ static struct virtio_transport vhost_transport = {
>>>>>>> .notify_buffer_size = virtio_transport_notify_buffer_size,
>>>>>>>
>>>>>>> .read_skb = virtio_transport_read_skb,
>>>>>>> + .notify_set_rcvlowat = virtio_transport_notify_set_rcvlowat
>>>>>>> },
>>>>>>>
>>>>>>> .send_pkt = vhost_transport_send_pkt,
>>>>>>> diff --git a/include/linux/virtio_vsock.h b/include/linux/virtio_vsock.h
>>>>>>> index ebb3ce63d64d..c82089dee0c8 100644
>>>>>>> --- a/include/linux/virtio_vsock.h
>>>>>>> +++ b/include/linux/virtio_vsock.h
>>>>>>> @@ -256,4 +256,5 @@ void virtio_transport_put_credit(struct virtio_vsock_sock *vvs, u32 credit);
>>>>>>> void virtio_transport_deliver_tap_pkt(struct sk_buff *skb);
>>>>>>> int virtio_transport_purge_skbs(void *vsk, struct sk_buff_head *list);
>>>>>>> int virtio_transport_read_skb(struct vsock_sock *vsk, skb_read_actor_t read_actor);
>>>>>>> +int virtio_transport_notify_set_rcvlowat(struct vsock_sock *vsk, int val);
>>>>>>> #endif /* _LINUX_VIRTIO_VSOCK_H */
>>>>>>> diff --git a/net/vmw_vsock/virtio_transport.c b/net/vmw_vsock/virtio_transport.c
>>>>>>> index af5bab1acee1..8007593a3a93 100644
>>>>>>> --- a/net/vmw_vsock/virtio_transport.c
>>>>>>> +++ b/net/vmw_vsock/virtio_transport.c
>>>>>>> @@ -539,6 +539,7 @@ static struct virtio_transport virtio_transport = {
>>>>>>> .notify_buffer_size = virtio_transport_notify_buffer_size,
>>>>>>>
>>>>>>> .read_skb = virtio_transport_read_skb,
>>>>>>> + .notify_set_rcvlowat = virtio_transport_notify_set_rcvlowat
>>>>>>> },
>>>>>>>
>>>>>>> .send_pkt = virtio_transport_send_pkt,
>>>>>>> diff --git a/net/vmw_vsock/virtio_transport_common.c b/net/vmw_vsock/virtio_transport_common.c
>>>>>>> index f6dc896bf44c..1cb556ad4597 100644
>>>>>>> --- a/net/vmw_vsock/virtio_transport_common.c
>>>>>>> +++ b/net/vmw_vsock/virtio_transport_common.c
>>>>>>> @@ -1684,6 +1684,33 @@ int virtio_transport_read_skb(struct vsock_sock *vsk, skb_read_actor_t recv_acto
>>>>>>> }
>>>>>>> EXPORT_SYMBOL_GPL(virtio_transport_read_skb);
>>>>>>>
>>>>>>> +int virtio_transport_notify_set_rcvlowat(struct vsock_sock *vsk, >> int val)
>>>>>>> +{
>>>>>>> + struct virtio_vsock_sock *vvs = vsk->trans;
>>>>>>> + bool send_update;
>>>>>>> +
>>>>>>> + spin_lock_bh(&vvs->rx_lock);
>>>>>>> +
>>>>>>> + /* If number of available bytes is less than new SO_RCVLOWAT value,
>>>>>>> + * kick sender to send more data, because sender may sleep in >> its
>>>>>>> + * 'send()' syscall waiting for enough space at our side.
>>>>>>> + */
>>>>>>> + send_update = vvs->rx_bytes < val;
>>>>>>> +
>>>>>>> + spin_unlock_bh(&vvs->rx_lock);
>>>>>>> +
>>>>>>> + if (send_update) {
>>>>>>> + int err;
>>>>>>> +
>>>>>>> + err = virtio_transport_send_credit_update(vsk);
>>>>>>> + if (err < 0)
>>>>>>> + return err;
>>>>>>> + }
>>>>>>> +
>>>>>>> + return 0;
>>>>>>> +}
>>>>>>
>>>>>>
>>>>>> I find it strange that this will send a credit update
>>>>>> even if nothing changed since this was called previously.
>>>>>> I'm not sure whether this is a problem protocol-wise,
>>>>>> but it certainly was not envisioned when the protocol was
>>>>>> built. WDYT?
>>>>>
>>>>> >From virtio spec I found:
>>>>>
>>>>> It is also valid to send a VIRTIO_VSOCK_OP_CREDIT_UPDATE packet without previously receiving a
>>>>> VIRTIO_VSOCK_OP_CREDIT_REQUEST packet. This allows communicating updates any time a change
>>>>> in buffer space occurs.
>>>>> So I guess there is no limitations to send such type of packet, e.g. it is not
>>>>> required to be a reply for some another packet. Please, correct me if im wrong.
>>>>>
>>>>> Thanks, Arseniy
>>>>
>>>>
>>>> Absolutely. My point was different - with this patch it is possible
>>>> that you are not adding any credits at all since the previous
>>>> VIRTIO_VSOCK_OP_CREDIT_UPDATE.
>>>
>>> I think the problem we're solving here is that since as an optimization we avoid sending the update for every byte we consume, but we put a threshold, then we make sure we update the peer.
>>>
>>> A credit update contains a snapshot and sending it the same as the previous one should not create any problem.
>>>
>>> My doubt now is that we only do this when we set RCVLOWAT , should we also do something when we consume bytes to avoid the optimization we have?
>>
>> @Michael, Stefano just reproduced problem during bytes reading, but there is already old fix for this, which we forget to merge:)
>> I think it must be included to this patchset.
>>
>> https://lore.kernel.org/netdev/[email protected]/
>>
>> Thanks, Arseniy
>
>
> I generally don't merge patches tagged as RFC.
> Repost without that tag?
> Also, it looks like a bugfix we need either way, no?
I'll repost it without RFC as part of this patchset, also i'll add test for it
Thanks, Arseniy
>
>>>
>>> Stefano
>>>
>
On Thu, Nov 30, 2023 at 12:40:43PM -0500, Michael S. Tsirkin wrote:
>On Thu, Nov 30, 2023 at 03:11:19PM +0100, Stefano Garzarella wrote:
>> On Thu, Nov 30, 2023 at 08:58:58AM -0500, Michael S. Tsirkin wrote:
>> > On Thu, Nov 30, 2023 at 04:43:34PM +0300, Arseniy Krasnov wrote:
>> > >
>> > >
>> > > On 30.11.2023 16:42, Michael S. Tsirkin wrote:
>> > > > On Thu, Nov 30, 2023 at 04:08:39PM +0300, Arseniy Krasnov wrote:
>> > > >> Send credit update message when SO_RCVLOWAT is updated and it is bigger
>> > > >> than number of bytes in rx queue. It is needed, because 'poll()' will
>> > > >> wait until number of bytes in rx queue will be not smaller than
>> > > >> SO_RCVLOWAT, so kick sender to send more data. Otherwise mutual hungup
>> > > >> for tx/rx is possible: sender waits for free space and receiver is
>> > > >> waiting data in 'poll()'.
>> > > >>
>> > > >> Signed-off-by: Arseniy Krasnov <[email protected]>
>> > > >> ---
>> > > >> Changelog:
>> > > >> v1 -> v2:
>> > > >> * Update commit message by removing 'This patch adds XXX' manner.
>> > > >> * Do not initialize 'send_update' variable - set it directly during
>> > > >> first usage.
>> > > >> v3 -> v4:
>> > > >> * Fit comment in 'virtio_transport_notify_set_rcvlowat()' to 80 chars.
>> > > >> v4 -> v5:
>> > > >> * Do not change callbacks order in transport structures.
>> > > >>
>> > > >> drivers/vhost/vsock.c | 1 +
>> > > >> include/linux/virtio_vsock.h | 1 +
>> > > >> net/vmw_vsock/virtio_transport.c | 1 +
>> > > >> net/vmw_vsock/virtio_transport_common.c | 27 +++++++++++++++++++++++++
>> > > >> net/vmw_vsock/vsock_loopback.c | 1 +
>> > > >> 5 files changed, 31 insertions(+)
>> > > >>
>> > > >> diff --git a/drivers/vhost/vsock.c b/drivers/vhost/vsock.c
>> > > >> index f75731396b7e..4146f80db8ac 100644
>> > > >> --- a/drivers/vhost/vsock.c
>> > > >> +++ b/drivers/vhost/vsock.c
>> > > >> @@ -451,6 +451,7 @@ static struct virtio_transport vhost_transport = {
>> > > >> .notify_buffer_size = virtio_transport_notify_buffer_size,
>> > > >>
>> > > >> .read_skb = virtio_transport_read_skb,
>> > > >> + .notify_set_rcvlowat = virtio_transport_notify_set_rcvlowat
>> > > >> },
>> > > >>
>> > > >> .send_pkt = vhost_transport_send_pkt,
>> > > >> diff --git a/include/linux/virtio_vsock.h b/include/linux/virtio_vsock.h
>> > > >> index ebb3ce63d64d..c82089dee0c8 100644
>> > > >> --- a/include/linux/virtio_vsock.h
>> > > >> +++ b/include/linux/virtio_vsock.h
>> > > >> @@ -256,4 +256,5 @@ void virtio_transport_put_credit(struct virtio_vsock_sock *vvs, u32 credit);
>> > > >> void virtio_transport_deliver_tap_pkt(struct sk_buff *skb);
>> > > >> int virtio_transport_purge_skbs(void *vsk, struct sk_buff_head *list);
>> > > >> int virtio_transport_read_skb(struct vsock_sock *vsk, skb_read_actor_t read_actor);
>> > > >> +int virtio_transport_notify_set_rcvlowat(struct vsock_sock *vsk, int val);
>> > > >> #endif /* _LINUX_VIRTIO_VSOCK_H */
>> > > >> diff --git a/net/vmw_vsock/virtio_transport.c b/net/vmw_vsock/virtio_transport.c
>> > > >> index af5bab1acee1..8007593a3a93 100644
>> > > >> --- a/net/vmw_vsock/virtio_transport.c
>> > > >> +++ b/net/vmw_vsock/virtio_transport.c
>> > > >> @@ -539,6 +539,7 @@ static struct virtio_transport virtio_transport = {
>> > > >> .notify_buffer_size = virtio_transport_notify_buffer_size,
>> > > >>
>> > > >> .read_skb = virtio_transport_read_skb,
>> > > >> + .notify_set_rcvlowat = virtio_transport_notify_set_rcvlowat
>> > > >> },
>> > > >>
>> > > >> .send_pkt = virtio_transport_send_pkt,
>> > > >> diff --git a/net/vmw_vsock/virtio_transport_common.c b/net/vmw_vsock/virtio_transport_common.c
>> > > >> index f6dc896bf44c..1cb556ad4597 100644
>> > > >> --- a/net/vmw_vsock/virtio_transport_common.c
>> > > >> +++ b/net/vmw_vsock/virtio_transport_common.c
>> > > >> @@ -1684,6 +1684,33 @@ int virtio_transport_read_skb(struct vsock_sock *vsk, skb_read_actor_t recv_acto
>> > > >> }
>> > > >> EXPORT_SYMBOL_GPL(virtio_transport_read_skb);
>> > > >>
>> > > >> +int virtio_transport_notify_set_rcvlowat(struct vsock_sock *vsk,
>> > > >> int val)
>> > > >> +{
>> > > >> + struct virtio_vsock_sock *vvs = vsk->trans;
>> > > >> + bool send_update;
>> > > >> +
>> > > >> + spin_lock_bh(&vvs->rx_lock);
>> > > >> +
>> > > >> + /* If number of available bytes is less than new SO_RCVLOWAT value,
>> > > >> + * kick sender to send more data, because sender may sleep in
>> > > >> its
>> > > >> + * 'send()' syscall waiting for enough space at our side.
>> > > >> + */
>> > > >> + send_update = vvs->rx_bytes < val;
>> > > >> +
>> > > >> + spin_unlock_bh(&vvs->rx_lock);
>> > > >> +
>> > > >> + if (send_update) {
>> > > >> + int err;
>> > > >> +
>> > > >> + err = virtio_transport_send_credit_update(vsk);
>> > > >> + if (err < 0)
>> > > >> + return err;
>> > > >> + }
>> > > >> +
>> > > >> + return 0;
>> > > >> +}
>> > > >
>> > > >
>> > > > I find it strange that this will send a credit update
>> > > > even if nothing changed since this was called previously.
>> > > > I'm not sure whether this is a problem protocol-wise,
>> > > > but it certainly was not envisioned when the protocol was
>> > > > built. WDYT?
>> > >
>> > > >From virtio spec I found:
>> > >
>> > > It is also valid to send a VIRTIO_VSOCK_OP_CREDIT_UPDATE packet without previously receiving a
>> > > VIRTIO_VSOCK_OP_CREDIT_REQUEST packet. This allows communicating updates any time a change
>> > > in buffer space occurs.
>> > > So I guess there is no limitations to send such type of packet, e.g. it is not
>> > > required to be a reply for some another packet. Please, correct me if im wrong.
>> > >
>> > > Thanks, Arseniy
>> >
>> >
>> > Absolutely. My point was different - with this patch it is possible
>> > that you are not adding any credits at all since the previous
>> > VIRTIO_VSOCK_OP_CREDIT_UPDATE.
>>
>> I think the problem we're solving here is that since as an optimization we
>> avoid sending the update for every byte we consume, but we put a threshold,
>> then we make sure we update the peer.
>>
>> A credit update contains a snapshot and sending it the same as the previous
>> one should not create any problem.
>
>Well it consumes a buffer on the other side.
Sure, but we are already speculating by not updating the other side when
we consume bytes before a certain threshold. This already avoids to
consume many buffers.
Here we're only sending it once, when the user sets RCVLOWAT, so
basically I expect it won't affect performance.
>
>> My doubt now is that we only do this when we set RCVLOWAT , should we
>> also
>> do something when we consume bytes to avoid the optimization we have?
>>
>> Stefano
>
>Isn't this why we have credit request?
Yep, but in practice we never use it. It would also consume 2 buffers,
one at the transmitter and one at the receiver.
However I agree that maybe we should start using it before we decide not
to send any more data.
To be compatible with older devices, though, I think for now we also
need to send a credit update when the bytes in the receive queue are
less than RCVLOWAT, as Arseniy proposed in the other series.
Thanks,
Stefano
On 01.12.2023 11:27, Stefano Garzarella wrote:
> On Thu, Nov 30, 2023 at 12:40:43PM -0500, Michael S. Tsirkin wrote:
>> On Thu, Nov 30, 2023 at 03:11:19PM +0100, Stefano Garzarella wrote:
>>> On Thu, Nov 30, 2023 at 08:58:58AM -0500, Michael S. Tsirkin wrote:
>>> > On Thu, Nov 30, 2023 at 04:43:34PM +0300, Arseniy Krasnov wrote:
>>> > >
>>> > >
>>> > > On 30.11.2023 16:42, Michael S. Tsirkin wrote:
>>> > > > On Thu, Nov 30, 2023 at 04:08:39PM +0300, Arseniy Krasnov wrote:
>>> > > >> Send credit update message when SO_RCVLOWAT is updated and it is bigger
>>> > > >> than number of bytes in rx queue. It is needed, because 'poll()' will
>>> > > >> wait until number of bytes in rx queue will be not smaller than
>>> > > >> SO_RCVLOWAT, so kick sender to send more data. Otherwise mutual hungup
>>> > > >> for tx/rx is possible: sender waits for free space and receiver is
>>> > > >> waiting data in 'poll()'.
>>> > > >>
>>> > > >> Signed-off-by: Arseniy Krasnov <[email protected]>
>>> > > >> ---
>>> > > >> Changelog:
>>> > > >> v1 -> v2:
>>> > > >> * Update commit message by removing 'This patch adds XXX' manner.
>>> > > >> * Do not initialize 'send_update' variable - set it directly during
>>> > > >> first usage.
>>> > > >> v3 -> v4:
>>> > > >> * Fit comment in 'virtio_transport_notify_set_rcvlowat()' to 80 chars.
>>> > > >> v4 -> v5:
>>> > > >> * Do not change callbacks order in transport structures.
>>> > > >>
>>> > > >> drivers/vhost/vsock.c | 1 +
>>> > > >> include/linux/virtio_vsock.h | 1 +
>>> > > >> net/vmw_vsock/virtio_transport.c | 1 +
>>> > > >> net/vmw_vsock/virtio_transport_common.c | 27 +++++++++++++++++++++++++
>>> > > >> net/vmw_vsock/vsock_loopback.c | 1 +
>>> > > >> 5 files changed, 31 insertions(+)
>>> > > >>
>>> > > >> diff --git a/drivers/vhost/vsock.c b/drivers/vhost/vsock.c
>>> > > >> index f75731396b7e..4146f80db8ac 100644
>>> > > >> --- a/drivers/vhost/vsock.c
>>> > > >> +++ b/drivers/vhost/vsock.c
>>> > > >> @@ -451,6 +451,7 @@ static struct virtio_transport vhost_transport = {
>>> > > >> .notify_buffer_size = virtio_transport_notify_buffer_size,
>>> > > >>
>>> > > >> .read_skb = virtio_transport_read_skb,
>>> > > >> + .notify_set_rcvlowat = virtio_transport_notify_set_rcvlowat
>>> > > >> },
>>> > > >>
>>> > > >> .send_pkt = vhost_transport_send_pkt,
>>> > > >> diff --git a/include/linux/virtio_vsock.h b/include/linux/virtio_vsock.h
>>> > > >> index ebb3ce63d64d..c82089dee0c8 100644
>>> > > >> --- a/include/linux/virtio_vsock.h
>>> > > >> +++ b/include/linux/virtio_vsock.h
>>> > > >> @@ -256,4 +256,5 @@ void virtio_transport_put_credit(struct virtio_vsock_sock *vvs, u32 credit);
>>> > > >> void virtio_transport_deliver_tap_pkt(struct sk_buff *skb);
>>> > > >> int virtio_transport_purge_skbs(void *vsk, struct sk_buff_head *list);
>>> > > >> int virtio_transport_read_skb(struct vsock_sock *vsk, skb_read_actor_t read_actor);
>>> > > >> +int virtio_transport_notify_set_rcvlowat(struct vsock_sock *vsk, int val);
>>> > > >> #endif /* _LINUX_VIRTIO_VSOCK_H */
>>> > > >> diff --git a/net/vmw_vsock/virtio_transport.c b/net/vmw_vsock/virtio_transport.c
>>> > > >> index af5bab1acee1..8007593a3a93 100644
>>> > > >> --- a/net/vmw_vsock/virtio_transport.c
>>> > > >> +++ b/net/vmw_vsock/virtio_transport.c
>>> > > >> @@ -539,6 +539,7 @@ static struct virtio_transport virtio_transport = {
>>> > > >> .notify_buffer_size = virtio_transport_notify_buffer_size,
>>> > > >>
>>> > > >> .read_skb = virtio_transport_read_skb,
>>> > > >> + .notify_set_rcvlowat = virtio_transport_notify_set_rcvlowat
>>> > > >> },
>>> > > >>
>>> > > >> .send_pkt = virtio_transport_send_pkt,
>>> > > >> diff --git a/net/vmw_vsock/virtio_transport_common.c b/net/vmw_vsock/virtio_transport_common.c
>>> > > >> index f6dc896bf44c..1cb556ad4597 100644
>>> > > >> --- a/net/vmw_vsock/virtio_transport_common.c
>>> > > >> +++ b/net/vmw_vsock/virtio_transport_common.c
>>> > > >> @@ -1684,6 +1684,33 @@ int virtio_transport_read_skb(struct vsock_sock *vsk, skb_read_actor_t recv_acto
>>> > > >> }
>>> > > >> EXPORT_SYMBOL_GPL(virtio_transport_read_skb);
>>> > > >>
>>> > > >> +int virtio_transport_notify_set_rcvlowat(struct vsock_sock *vsk,
>>> > > >> int val)
>>> > > >> +{
>>> > > >> + struct virtio_vsock_sock *vvs = vsk->trans;
>>> > > >> + bool send_update;
>>> > > >> +
>>> > > >> + spin_lock_bh(&vvs->rx_lock);
>>> > > >> +
>>> > > >> + /* If number of available bytes is less than new SO_RCVLOWAT value,
>>> > > >> + * kick sender to send more data, because sender may sleep in
>>> > > >> its
>>> > > >> + * 'send()' syscall waiting for enough space at our side.
>>> > > >> + */
>>> > > >> + send_update = vvs->rx_bytes < val;
>>> > > >> +
>>> > > >> + spin_unlock_bh(&vvs->rx_lock);
>>> > > >> +
>>> > > >> + if (send_update) {
>>> > > >> + int err;
>>> > > >> +
>>> > > >> + err = virtio_transport_send_credit_update(vsk);
>>> > > >> + if (err < 0)
>>> > > >> + return err;
>>> > > >> + }
>>> > > >> +
>>> > > >> + return 0;
>>> > > >> +}
>>> > > >
>>> > > >
>>> > > > I find it strange that this will send a credit update
>>> > > > even if nothing changed since this was called previously.
>>> > > > I'm not sure whether this is a problem protocol-wise,
>>> > > > but it certainly was not envisioned when the protocol was
>>> > > > built. WDYT?
>>> > >
>>> > > >From virtio spec I found:
>>> > >
>>> > > It is also valid to send a VIRTIO_VSOCK_OP_CREDIT_UPDATE packet without previously receiving a
>>> > > VIRTIO_VSOCK_OP_CREDIT_REQUEST packet. This allows communicating updates any time a change
>>> > > in buffer space occurs.
>>> > > So I guess there is no limitations to send such type of packet, e.g. it is not
>>> > > required to be a reply for some another packet. Please, correct me if im wrong.
>>> > >
>>> > > Thanks, Arseniy
>>> >
>>> >
>>> > Absolutely. My point was different - with this patch it is possible
>>> > that you are not adding any credits at all since the previous
>>> > VIRTIO_VSOCK_OP_CREDIT_UPDATE.
>>>
>>> I think the problem we're solving here is that since as an optimization we
>>> avoid sending the update for every byte we consume, but we put a threshold,
>>> then we make sure we update the peer.
>>>
>>> A credit update contains a snapshot and sending it the same as the previous
>>> one should not create any problem.
>>
>> Well it consumes a buffer on the other side.
>
> Sure, but we are already speculating by not updating the other side when
> we consume bytes before a certain threshold. This already avoids to
> consume many buffers.
>
> Here we're only sending it once, when the user sets RCVLOWAT, so
> basically I expect it won't affect performance.
Moreover I think in practice setting RCVLOWAT is rare case, while this patch
fixes real problem I guess
>
>>
>>> My doubt now is that we only do this when we set RCVLOWAT , should we also
>>> do something when we consume bytes to avoid the optimization we have?
>>>
>>> Stefano
>>
>> Isn't this why we have credit request?
>
> Yep, but in practice we never use it. It would also consume 2 buffers,
> one at the transmitter and one at the receiver.
>
> However I agree that maybe we should start using it before we decide not
> to send any more data.
>
> To be compatible with older devices, though, I think for now we also
> need to send a credit update when the bytes in the receive queue are
> less than RCVLOWAT, as Arseniy proposed in the other series.
Looks like (in theory of course), that credit request is considered to be
paired with credit update. While current usage of credit update is something
like ACK packet in TCP, e.g. telling peer that we are ready to receive more
data.
Thanks, Arseniy
>
> Thanks,
> Stefano
>
On Fri, Dec 01, 2023 at 11:35:56AM +0300, Arseniy Krasnov wrote:
>
>
>On 01.12.2023 11:27, Stefano Garzarella wrote:
>> On Thu, Nov 30, 2023 at 12:40:43PM -0500, Michael S. Tsirkin wrote:
>>> On Thu, Nov 30, 2023 at 03:11:19PM +0100, Stefano Garzarella wrote:
>>>> On Thu, Nov 30, 2023 at 08:58:58AM -0500, Michael S. Tsirkin wrote:
>>>> > On Thu, Nov 30, 2023 at 04:43:34PM +0300, Arseniy Krasnov wrote:
>>>> > >
>>>> > >
>>>> > > On 30.11.2023 16:42, Michael S. Tsirkin wrote:
>>>> > > > On Thu, Nov 30, 2023 at 04:08:39PM +0300, Arseniy Krasnov wrote:
>>>> > > >> Send credit update message when SO_RCVLOWAT is updated and it is bigger
>>>> > > >> than number of bytes in rx queue. It is needed, because 'poll()' will
>>>> > > >> wait until number of bytes in rx queue will be not smaller than
>>>> > > >> SO_RCVLOWAT, so kick sender to send more data. Otherwise mutual hungup
>>>> > > >> for tx/rx is possible: sender waits for free space and receiver is
>>>> > > >> waiting data in 'poll()'.
>>>> > > >>
>>>> > > >> Signed-off-by: Arseniy Krasnov <[email protected]>
>>>> > > >> ---
>>>> > > >>? Changelog:
>>>> > > >>? v1 -> v2:
>>>> > > >>?? * Update commit message by removing 'This patch adds XXX' manner.
>>>> > > >>?? * Do not initialize 'send_update' variable - set it directly during
>>>> > > >>???? first usage.
>>>> > > >>? v3 -> v4:
>>>> > > >>?? * Fit comment in 'virtio_transport_notify_set_rcvlowat()' to 80 chars.
>>>> > > >>? v4 -> v5:
>>>> > > >>?? * Do not change callbacks order in transport structures.
>>>> > > >>
>>>> > > >>? drivers/vhost/vsock.c?????????????????? |? 1 +
>>>> > > >>? include/linux/virtio_vsock.h??????????? |? 1 +
>>>> > > >>? net/vmw_vsock/virtio_transport.c??????? |? 1 +
>>>> > > >>? net/vmw_vsock/virtio_transport_common.c | 27 +++++++++++++++++++++++++
>>>> > > >>? net/vmw_vsock/vsock_loopback.c????????? |? 1 +
>>>> > > >>? 5 files changed, 31 insertions(+)
>>>> > > >>
>>>> > > >> diff --git a/drivers/vhost/vsock.c b/drivers/vhost/vsock.c
>>>> > > >> index f75731396b7e..4146f80db8ac 100644
>>>> > > >> --- a/drivers/vhost/vsock.c
>>>> > > >> +++ b/drivers/vhost/vsock.c
>>>> > > >> @@ -451,6 +451,7 @@ static struct virtio_transport vhost_transport = {
>>>> > > >>????????? .notify_buffer_size?????? = virtio_transport_notify_buffer_size,
>>>> > > >>
>>>> > > >>????????? .read_skb = virtio_transport_read_skb,
>>>> > > >> +??????? .notify_set_rcvlowat????? = virtio_transport_notify_set_rcvlowat
>>>> > > >>????? },
>>>> > > >>
>>>> > > >>????? .send_pkt = vhost_transport_send_pkt,
>>>> > > >> diff --git a/include/linux/virtio_vsock.h b/include/linux/virtio_vsock.h
>>>> > > >> index ebb3ce63d64d..c82089dee0c8 100644
>>>> > > >> --- a/include/linux/virtio_vsock.h
>>>> > > >> +++ b/include/linux/virtio_vsock.h
>>>> > > >> @@ -256,4 +256,5 @@ void virtio_transport_put_credit(struct virtio_vsock_sock *vvs, u32 credit);
>>>> > > >>? void virtio_transport_deliver_tap_pkt(struct sk_buff *skb);
>>>> > > >>? int virtio_transport_purge_skbs(void *vsk, struct sk_buff_head *list);
>>>> > > >>? int virtio_transport_read_skb(struct vsock_sock *vsk, skb_read_actor_t read_actor);
>>>> > > >> +int virtio_transport_notify_set_rcvlowat(struct vsock_sock *vsk, int val);
>>>> > > >>? #endif /* _LINUX_VIRTIO_VSOCK_H */
>>>> > > >> diff --git a/net/vmw_vsock/virtio_transport.c b/net/vmw_vsock/virtio_transport.c
>>>> > > >> index af5bab1acee1..8007593a3a93 100644
>>>> > > >> --- a/net/vmw_vsock/virtio_transport.c
>>>> > > >> +++ b/net/vmw_vsock/virtio_transport.c
>>>> > > >> @@ -539,6 +539,7 @@ static struct virtio_transport virtio_transport = {
>>>> > > >>????????? .notify_buffer_size?????? = virtio_transport_notify_buffer_size,
>>>> > > >>
>>>> > > >>????????? .read_skb = virtio_transport_read_skb,
>>>> > > >> +??????? .notify_set_rcvlowat????? = virtio_transport_notify_set_rcvlowat
>>>> > > >>????? },
>>>> > > >>
>>>> > > >>????? .send_pkt = virtio_transport_send_pkt,
>>>> > > >> diff --git a/net/vmw_vsock/virtio_transport_common.c b/net/vmw_vsock/virtio_transport_common.c
>>>> > > >> index f6dc896bf44c..1cb556ad4597 100644
>>>> > > >> --- a/net/vmw_vsock/virtio_transport_common.c
>>>> > > >> +++ b/net/vmw_vsock/virtio_transport_common.c
>>>> > > >> @@ -1684,6 +1684,33 @@ int virtio_transport_read_skb(struct vsock_sock *vsk, skb_read_actor_t recv_acto
>>>> > > >>? }
>>>> > > >>? EXPORT_SYMBOL_GPL(virtio_transport_read_skb);
>>>> > > >>
>>>> > > >> +int virtio_transport_notify_set_rcvlowat(struct vsock_sock *vsk,
>>>> > > >> int val)
>>>> > > >> +{
>>>> > > >> +??? struct virtio_vsock_sock *vvs = vsk->trans;
>>>> > > >> +??? bool send_update;
>>>> > > >> +
>>>> > > >> +??? spin_lock_bh(&vvs->rx_lock);
>>>> > > >> +
>>>> > > >> +??? /* If number of available bytes is less than new SO_RCVLOWAT value,
>>>> > > >> +???? * kick sender to send more data, because sender may sleep in
>>>> > > >> its
>>>> > > >> +???? * 'send()' syscall waiting for enough space at our side.
>>>> > > >> +???? */
>>>> > > >> +??? send_update = vvs->rx_bytes < val;
>>>> > > >> +
>>>> > > >> +??? spin_unlock_bh(&vvs->rx_lock);
>>>> > > >> +
>>>> > > >> +??? if (send_update) {
>>>> > > >> +??????? int err;
>>>> > > >> +
>>>> > > >> +??????? err = virtio_transport_send_credit_update(vsk);
>>>> > > >> +??????? if (err < 0)
>>>> > > >> +??????????? return err;
>>>> > > >> +??? }
>>>> > > >> +
>>>> > > >> +??? return 0;
>>>> > > >> +}
>>>> > > >
>>>> > > >
>>>> > > > I find it strange that this will send a credit update
>>>> > > > even if nothing changed since this was called previously.
>>>> > > > I'm not sure whether this is a problem protocol-wise,
>>>> > > > but it certainly was not envisioned when the protocol was
>>>> > > > built. WDYT?
>>>> > >
>>>> > > >From virtio spec I found:
>>>> > >
>>>> > > It is also valid to send a VIRTIO_VSOCK_OP_CREDIT_UPDATE packet without previously receiving a
>>>> > > VIRTIO_VSOCK_OP_CREDIT_REQUEST packet. This allows communicating updates any time a change
>>>> > > in buffer space occurs.
>>>> > > So I guess there is no limitations to send such type of packet, e.g. it is not
>>>> > > required to be a reply for some another packet. Please, correct me if im wrong.
>>>> > >
>>>> > > Thanks, Arseniy
>>>> >
>>>> >
>>>> > Absolutely. My point was different - with this patch it is possible
>>>> > that you are not adding any credits at all since the previous
>>>> > VIRTIO_VSOCK_OP_CREDIT_UPDATE.
>>>>
>>>> I think the problem we're solving here is that since as an optimization we
>>>> avoid sending the update for every byte we consume, but we put a threshold,
>>>> then we make sure we update the peer.
>>>>
>>>> A credit update contains a snapshot and sending it the same as the previous
>>>> one should not create any problem.
>>>
>>> Well it consumes a buffer on the other side.
>>
>> Sure, but we are already speculating by not updating the other side when
>> we consume bytes before a certain threshold. This already avoids to
>> consume many buffers.
>>
>> Here we're only sending it once, when the user sets RCVLOWAT, so
>> basically I expect it won't affect performance.
>
>Moreover I think in practice setting RCVLOWAT is rare case, while this patch
>fixes real problem I guess
>
>
>>
>>>
>>>> My doubt now is that we only do this when we set RCVLOWAT , should we also
>>>> do something when we consume bytes to avoid the optimization we have?
>>>>
>>>> Stefano
>>>
>>> Isn't this why we have credit request?
>>
>> Yep, but in practice we never use it. It would also consume 2 buffers,
>> one at the transmitter and one at the receiver.
>>
>> However I agree that maybe we should start using it before we decide not
>> to send any more data.
>>
>> To be compatible with older devices, though, I think for now we also
>> need to send a credit update when the bytes in the receive queue are
>> less than RCVLOWAT, as Arseniy proposed in the other series.
>
>Looks like (in theory of course), that credit request is considered to be
>paired with credit update. While current usage of credit update is something
>like ACK packet in TCP, e.g. telling peer that we are ready to receive more
>data.
I don't honestly know what the original author's choice was, but I think
we reduce latency this way.
Effectively though, if we never send any credit update when we consume
bytes and always leave it up to the transmitter to ask for an update
before transmission, we save even more buffer than the optimization we
have, but maybe the latency would grow a lot.
Stefano
On 01.12.2023 12:48, Stefano Garzarella wrote:
> On Fri, Dec 01, 2023 at 11:35:56AM +0300, Arseniy Krasnov wrote:
>>
>>
>> On 01.12.2023 11:27, Stefano Garzarella wrote:
>>> On Thu, Nov 30, 2023 at 12:40:43PM -0500, Michael S. Tsirkin wrote:
>>>> On Thu, Nov 30, 2023 at 03:11:19PM +0100, Stefano Garzarella wrote:
>>>>> On Thu, Nov 30, 2023 at 08:58:58AM -0500, Michael S. Tsirkin wrote:
>>>>> > On Thu, Nov 30, 2023 at 04:43:34PM +0300, Arseniy Krasnov wrote:
>>>>> > >
>>>>> > >
>>>>> > > On 30.11.2023 16:42, Michael S. Tsirkin wrote:
>>>>> > > > On Thu, Nov 30, 2023 at 04:08:39PM +0300, Arseniy Krasnov wrote:
>>>>> > > >> Send credit update message when SO_RCVLOWAT is updated and it is bigger
>>>>> > > >> than number of bytes in rx queue. It is needed, because 'poll()' will
>>>>> > > >> wait until number of bytes in rx queue will be not smaller than
>>>>> > > >> SO_RCVLOWAT, so kick sender to send more data. Otherwise mutual hungup
>>>>> > > >> for tx/rx is possible: sender waits for free space and receiver is
>>>>> > > >> waiting data in 'poll()'.
>>>>> > > >>
>>>>> > > >> Signed-off-by: Arseniy Krasnov <[email protected]>
>>>>> > > >> ---
>>>>> > > >> Changelog:
>>>>> > > >> v1 -> v2:
>>>>> > > >> * Update commit message by removing 'This patch adds XXX' manner.
>>>>> > > >> * Do not initialize 'send_update' variable - set it directly during
>>>>> > > >> first usage.
>>>>> > > >> v3 -> v4:
>>>>> > > >> * Fit comment in 'virtio_transport_notify_set_rcvlowat()' to 80 chars.
>>>>> > > >> v4 -> v5:
>>>>> > > >> * Do not change callbacks order in transport structures.
>>>>> > > >>
>>>>> > > >> drivers/vhost/vsock.c | 1 +
>>>>> > > >> include/linux/virtio_vsock.h | 1 +
>>>>> > > >> net/vmw_vsock/virtio_transport.c | 1 +
>>>>> > > >> net/vmw_vsock/virtio_transport_common.c | 27 +++++++++++++++++++++++++
>>>>> > > >> net/vmw_vsock/vsock_loopback.c | 1 +
>>>>> > > >> 5 files changed, 31 insertions(+)
>>>>> > > >>
>>>>> > > >> diff --git a/drivers/vhost/vsock.c b/drivers/vhost/vsock.c
>>>>> > > >> index f75731396b7e..4146f80db8ac 100644
>>>>> > > >> --- a/drivers/vhost/vsock.c
>>>>> > > >> +++ b/drivers/vhost/vsock.c
>>>>> > > >> @@ -451,6 +451,7 @@ static struct virtio_transport vhost_transport = {
>>>>> > > >> .notify_buffer_size = virtio_transport_notify_buffer_size,
>>>>> > > >>
>>>>> > > >> .read_skb = virtio_transport_read_skb,
>>>>> > > >> + .notify_set_rcvlowat = virtio_transport_notify_set_rcvlowat
>>>>> > > >> },
>>>>> > > >>
>>>>> > > >> .send_pkt = vhost_transport_send_pkt,
>>>>> > > >> diff --git a/include/linux/virtio_vsock.h b/include/linux/virtio_vsock.h
>>>>> > > >> index ebb3ce63d64d..c82089dee0c8 100644
>>>>> > > >> --- a/include/linux/virtio_vsock.h
>>>>> > > >> +++ b/include/linux/virtio_vsock.h
>>>>> > > >> @@ -256,4 +256,5 @@ void virtio_transport_put_credit(struct virtio_vsock_sock *vvs, u32 credit);
>>>>> > > >> void virtio_transport_deliver_tap_pkt(struct sk_buff *skb);
>>>>> > > >> int virtio_transport_purge_skbs(void *vsk, struct sk_buff_head *list);
>>>>> > > >> int virtio_transport_read_skb(struct vsock_sock *vsk, skb_read_actor_t read_actor);
>>>>> > > >> +int virtio_transport_notify_set_rcvlowat(struct vsock_sock *vsk, int val);
>>>>> > > >> #endif /* _LINUX_VIRTIO_VSOCK_H */
>>>>> > > >> diff --git a/net/vmw_vsock/virtio_transport.c b/net/vmw_vsock/virtio_transport.c
>>>>> > > >> index af5bab1acee1..8007593a3a93 100644
>>>>> > > >> --- a/net/vmw_vsock/virtio_transport.c
>>>>> > > >> +++ b/net/vmw_vsock/virtio_transport.c
>>>>> > > >> @@ -539,6 +539,7 @@ static struct virtio_transport virtio_transport = {
>>>>> > > >> .notify_buffer_size = virtio_transport_notify_buffer_size,
>>>>> > > >>
>>>>> > > >> .read_skb = virtio_transport_read_skb,
>>>>> > > >> + .notify_set_rcvlowat = virtio_transport_notify_set_rcvlowat
>>>>> > > >> },
>>>>> > > >>
>>>>> > > >> .send_pkt = virtio_transport_send_pkt,
>>>>> > > >> diff --git a/net/vmw_vsock/virtio_transport_common.c b/net/vmw_vsock/virtio_transport_common.c
>>>>> > > >> index f6dc896bf44c..1cb556ad4597 100644
>>>>> > > >> --- a/net/vmw_vsock/virtio_transport_common.c
>>>>> > > >> +++ b/net/vmw_vsock/virtio_transport_common.c
>>>>> > > >> @@ -1684,6 +1684,33 @@ int virtio_transport_read_skb(struct vsock_sock *vsk, skb_read_actor_t recv_acto
>>>>> > > >> }
>>>>> > > >> EXPORT_SYMBOL_GPL(virtio_transport_read_skb);
>>>>> > > >>
>>>>> > > >> +int virtio_transport_notify_set_rcvlowat(struct vsock_sock *vsk,
>>>>> > > >> int val)
>>>>> > > >> +{
>>>>> > > >> + struct virtio_vsock_sock *vvs = vsk->trans;
>>>>> > > >> + bool send_update;
>>>>> > > >> +
>>>>> > > >> + spin_lock_bh(&vvs->rx_lock);
>>>>> > > >> +
>>>>> > > >> + /* If number of available bytes is less than new SO_RCVLOWAT value,
>>>>> > > >> + * kick sender to send more data, because sender may sleep in
>>>>> > > >> its
>>>>> > > >> + * 'send()' syscall waiting for enough space at our side.
>>>>> > > >> + */
>>>>> > > >> + send_update = vvs->rx_bytes < val;
>>>>> > > >> +
>>>>> > > >> + spin_unlock_bh(&vvs->rx_lock);
>>>>> > > >> +
>>>>> > > >> + if (send_update) {
>>>>> > > >> + int err;
>>>>> > > >> +
>>>>> > > >> + err = virtio_transport_send_credit_update(vsk);
>>>>> > > >> + if (err < 0)
>>>>> > > >> + return err;
>>>>> > > >> + }
>>>>> > > >> +
>>>>> > > >> + return 0;
>>>>> > > >> +}
>>>>> > > >
>>>>> > > >
>>>>> > > > I find it strange that this will send a credit update
>>>>> > > > even if nothing changed since this was called previously.
>>>>> > > > I'm not sure whether this is a problem protocol-wise,
>>>>> > > > but it certainly was not envisioned when the protocol was
>>>>> > > > built. WDYT?
>>>>> > >
>>>>> > > >From virtio spec I found:
>>>>> > >
>>>>> > > It is also valid to send a VIRTIO_VSOCK_OP_CREDIT_UPDATE packet without previously receiving a
>>>>> > > VIRTIO_VSOCK_OP_CREDIT_REQUEST packet. This allows communicating updates any time a change
>>>>> > > in buffer space occurs.
>>>>> > > So I guess there is no limitations to send such type of packet, e.g. it is not
>>>>> > > required to be a reply for some another packet. Please, correct me if im wrong.
>>>>> > >
>>>>> > > Thanks, Arseniy
>>>>> >
>>>>> >
>>>>> > Absolutely. My point was different - with this patch it is possible
>>>>> > that you are not adding any credits at all since the previous
>>>>> > VIRTIO_VSOCK_OP_CREDIT_UPDATE.
>>>>>
>>>>> I think the problem we're solving here is that since as an optimization we
>>>>> avoid sending the update for every byte we consume, but we put a threshold,
>>>>> then we make sure we update the peer.
>>>>>
>>>>> A credit update contains a snapshot and sending it the same as the previous
>>>>> one should not create any problem.
>>>>
>>>> Well it consumes a buffer on the other side.
>>>
>>> Sure, but we are already speculating by not updating the other side when
>>> we consume bytes before a certain threshold. This already avoids to
>>> consume many buffers.
>>>
>>> Here we're only sending it once, when the user sets RCVLOWAT, so
>>> basically I expect it won't affect performance.
>>
>> Moreover I think in practice setting RCVLOWAT is rare case, while this patch
>> fixes real problem I guess
>>
>>
>>>
>>>>
>>>>> My doubt now is that we only do this when we set RCVLOWAT , should we also
>>>>> do something when we consume bytes to avoid the optimization we have?
>>>>>
>>>>> Stefano
>>>>
>>>> Isn't this why we have credit request?
>>>
>>> Yep, but in practice we never use it. It would also consume 2 buffers,
>>> one at the transmitter and one at the receiver.
>>>
>>> However I agree that maybe we should start using it before we decide not
>>> to send any more data.
>>>
>>> To be compatible with older devices, though, I think for now we also
>>> need to send a credit update when the bytes in the receive queue are
>>> less than RCVLOWAT, as Arseniy proposed in the other series.
>>
>> Looks like (in theory of course), that credit request is considered to be
>> paired with credit update. While current usage of credit update is something
>> like ACK packet in TCP, e.g. telling peer that we are ready to receive more
>> data.
>
> I don't honestly know what the original author's choice was, but I think we reduce latency this way.
Ah I see,ok
>
> Effectively though, if we never send any credit update when we consume bytes and always leave it up to the transmitter to ask for an update before transmission, we save even more buffer than the optimization we have, but maybe the latency would grow a lot.
I think:
1) Way where sender must request current credit status before sending packet requires rework of kernel part, and for me this approach is not
so clear than current simple implementation (send RW, reply with CREDIT_UPDATE if needed).
2) In theory yes, we need one more buffer for such CREDIT_UPDATE, but in practice I don't know how big is this trouble.
Thanks, Arseniy
>
> Stefano
>
On Fri, Dec 01, 2023 at 01:40:41PM +0300, Arseniy Krasnov wrote:
>
>
> On 01.12.2023 12:48, Stefano Garzarella wrote:
> > On Fri, Dec 01, 2023 at 11:35:56AM +0300, Arseniy Krasnov wrote:
> >>
> >>
> >> On 01.12.2023 11:27, Stefano Garzarella wrote:
> >>> On Thu, Nov 30, 2023 at 12:40:43PM -0500, Michael S. Tsirkin wrote:
> >>>> On Thu, Nov 30, 2023 at 03:11:19PM +0100, Stefano Garzarella wrote:
> >>>>> On Thu, Nov 30, 2023 at 08:58:58AM -0500, Michael S. Tsirkin wrote:
> >>>>> > On Thu, Nov 30, 2023 at 04:43:34PM +0300, Arseniy Krasnov wrote:
> >>>>> > >
> >>>>> > >
> >>>>> > > On 30.11.2023 16:42, Michael S. Tsirkin wrote:
> >>>>> > > > On Thu, Nov 30, 2023 at 04:08:39PM +0300, Arseniy Krasnov wrote:
> >>>>> > > >> Send credit update message when SO_RCVLOWAT is updated and it is bigger
> >>>>> > > >> than number of bytes in rx queue. It is needed, because 'poll()' will
> >>>>> > > >> wait until number of bytes in rx queue will be not smaller than
> >>>>> > > >> SO_RCVLOWAT, so kick sender to send more data. Otherwise mutual hungup
> >>>>> > > >> for tx/rx is possible: sender waits for free space and receiver is
> >>>>> > > >> waiting data in 'poll()'.
> >>>>> > > >>
> >>>>> > > >> Signed-off-by: Arseniy Krasnov <[email protected]>
> >>>>> > > >> ---
> >>>>> > > >>? Changelog:
> >>>>> > > >>? v1 -> v2:
> >>>>> > > >>?? * Update commit message by removing 'This patch adds XXX' manner.
> >>>>> > > >>?? * Do not initialize 'send_update' variable - set it directly during
> >>>>> > > >>???? first usage.
> >>>>> > > >>? v3 -> v4:
> >>>>> > > >>?? * Fit comment in 'virtio_transport_notify_set_rcvlowat()' to 80 chars.
> >>>>> > > >>? v4 -> v5:
> >>>>> > > >>?? * Do not change callbacks order in transport structures.
> >>>>> > > >>
> >>>>> > > >>? drivers/vhost/vsock.c?????????????????? |? 1 +
> >>>>> > > >>? include/linux/virtio_vsock.h??????????? |? 1 +
> >>>>> > > >>? net/vmw_vsock/virtio_transport.c??????? |? 1 +
> >>>>> > > >>? net/vmw_vsock/virtio_transport_common.c | 27 +++++++++++++++++++++++++
> >>>>> > > >>? net/vmw_vsock/vsock_loopback.c????????? |? 1 +
> >>>>> > > >>? 5 files changed, 31 insertions(+)
> >>>>> > > >>
> >>>>> > > >> diff --git a/drivers/vhost/vsock.c b/drivers/vhost/vsock.c
> >>>>> > > >> index f75731396b7e..4146f80db8ac 100644
> >>>>> > > >> --- a/drivers/vhost/vsock.c
> >>>>> > > >> +++ b/drivers/vhost/vsock.c
> >>>>> > > >> @@ -451,6 +451,7 @@ static struct virtio_transport vhost_transport = {
> >>>>> > > >>????????? .notify_buffer_size?????? = virtio_transport_notify_buffer_size,
> >>>>> > > >>
> >>>>> > > >>????????? .read_skb = virtio_transport_read_skb,
> >>>>> > > >> +??????? .notify_set_rcvlowat????? = virtio_transport_notify_set_rcvlowat
> >>>>> > > >>????? },
> >>>>> > > >>
> >>>>> > > >>????? .send_pkt = vhost_transport_send_pkt,
> >>>>> > > >> diff --git a/include/linux/virtio_vsock.h b/include/linux/virtio_vsock.h
> >>>>> > > >> index ebb3ce63d64d..c82089dee0c8 100644
> >>>>> > > >> --- a/include/linux/virtio_vsock.h
> >>>>> > > >> +++ b/include/linux/virtio_vsock.h
> >>>>> > > >> @@ -256,4 +256,5 @@ void virtio_transport_put_credit(struct virtio_vsock_sock *vvs, u32 credit);
> >>>>> > > >>? void virtio_transport_deliver_tap_pkt(struct sk_buff *skb);
> >>>>> > > >>? int virtio_transport_purge_skbs(void *vsk, struct sk_buff_head *list);
> >>>>> > > >>? int virtio_transport_read_skb(struct vsock_sock *vsk, skb_read_actor_t read_actor);
> >>>>> > > >> +int virtio_transport_notify_set_rcvlowat(struct vsock_sock *vsk, int val);
> >>>>> > > >>? #endif /* _LINUX_VIRTIO_VSOCK_H */
> >>>>> > > >> diff --git a/net/vmw_vsock/virtio_transport.c b/net/vmw_vsock/virtio_transport.c
> >>>>> > > >> index af5bab1acee1..8007593a3a93 100644
> >>>>> > > >> --- a/net/vmw_vsock/virtio_transport.c
> >>>>> > > >> +++ b/net/vmw_vsock/virtio_transport.c
> >>>>> > > >> @@ -539,6 +539,7 @@ static struct virtio_transport virtio_transport = {
> >>>>> > > >>????????? .notify_buffer_size?????? = virtio_transport_notify_buffer_size,
> >>>>> > > >>
> >>>>> > > >>????????? .read_skb = virtio_transport_read_skb,
> >>>>> > > >> +??????? .notify_set_rcvlowat????? = virtio_transport_notify_set_rcvlowat
> >>>>> > > >>????? },
> >>>>> > > >>
> >>>>> > > >>????? .send_pkt = virtio_transport_send_pkt,
> >>>>> > > >> diff --git a/net/vmw_vsock/virtio_transport_common.c b/net/vmw_vsock/virtio_transport_common.c
> >>>>> > > >> index f6dc896bf44c..1cb556ad4597 100644
> >>>>> > > >> --- a/net/vmw_vsock/virtio_transport_common.c
> >>>>> > > >> +++ b/net/vmw_vsock/virtio_transport_common.c
> >>>>> > > >> @@ -1684,6 +1684,33 @@ int virtio_transport_read_skb(struct vsock_sock *vsk, skb_read_actor_t recv_acto
> >>>>> > > >>? }
> >>>>> > > >>? EXPORT_SYMBOL_GPL(virtio_transport_read_skb);
> >>>>> > > >>
> >>>>> > > >> +int virtio_transport_notify_set_rcvlowat(struct vsock_sock *vsk,
> >>>>> > > >> int val)
> >>>>> > > >> +{
> >>>>> > > >> +??? struct virtio_vsock_sock *vvs = vsk->trans;
> >>>>> > > >> +??? bool send_update;
> >>>>> > > >> +
> >>>>> > > >> +??? spin_lock_bh(&vvs->rx_lock);
> >>>>> > > >> +
> >>>>> > > >> +??? /* If number of available bytes is less than new SO_RCVLOWAT value,
> >>>>> > > >> +???? * kick sender to send more data, because sender may sleep in
> >>>>> > > >> its
> >>>>> > > >> +???? * 'send()' syscall waiting for enough space at our side.
> >>>>> > > >> +???? */
> >>>>> > > >> +??? send_update = vvs->rx_bytes < val;
> >>>>> > > >> +
> >>>>> > > >> +??? spin_unlock_bh(&vvs->rx_lock);
> >>>>> > > >> +
> >>>>> > > >> +??? if (send_update) {
> >>>>> > > >> +??????? int err;
> >>>>> > > >> +
> >>>>> > > >> +??????? err = virtio_transport_send_credit_update(vsk);
> >>>>> > > >> +??????? if (err < 0)
> >>>>> > > >> +??????????? return err;
> >>>>> > > >> +??? }
> >>>>> > > >> +
> >>>>> > > >> +??? return 0;
> >>>>> > > >> +}
> >>>>> > > >
> >>>>> > > >
> >>>>> > > > I find it strange that this will send a credit update
> >>>>> > > > even if nothing changed since this was called previously.
> >>>>> > > > I'm not sure whether this is a problem protocol-wise,
> >>>>> > > > but it certainly was not envisioned when the protocol was
> >>>>> > > > built. WDYT?
> >>>>> > >
> >>>>> > > >From virtio spec I found:
> >>>>> > >
> >>>>> > > It is also valid to send a VIRTIO_VSOCK_OP_CREDIT_UPDATE packet without previously receiving a
> >>>>> > > VIRTIO_VSOCK_OP_CREDIT_REQUEST packet. This allows communicating updates any time a change
> >>>>> > > in buffer space occurs.
> >>>>> > > So I guess there is no limitations to send such type of packet, e.g. it is not
> >>>>> > > required to be a reply for some another packet. Please, correct me if im wrong.
> >>>>> > >
> >>>>> > > Thanks, Arseniy
> >>>>> >
> >>>>> >
> >>>>> > Absolutely. My point was different - with this patch it is possible
> >>>>> > that you are not adding any credits at all since the previous
> >>>>> > VIRTIO_VSOCK_OP_CREDIT_UPDATE.
> >>>>>
> >>>>> I think the problem we're solving here is that since as an optimization we
> >>>>> avoid sending the update for every byte we consume, but we put a threshold,
> >>>>> then we make sure we update the peer.
> >>>>>
> >>>>> A credit update contains a snapshot and sending it the same as the previous
> >>>>> one should not create any problem.
> >>>>
> >>>> Well it consumes a buffer on the other side.
> >>>
> >>> Sure, but we are already speculating by not updating the other side when
> >>> we consume bytes before a certain threshold. This already avoids to
> >>> consume many buffers.
> >>>
> >>> Here we're only sending it once, when the user sets RCVLOWAT, so
> >>> basically I expect it won't affect performance.
> >>
> >> Moreover I think in practice setting RCVLOWAT is rare case, while this patch
> >> fixes real problem I guess
> >>
> >>
> >>>
> >>>>
> >>>>> My doubt now is that we only do this when we set RCVLOWAT , should we also
> >>>>> do something when we consume bytes to avoid the optimization we have?
> >>>>>
> >>>>> Stefano
> >>>>
> >>>> Isn't this why we have credit request?
> >>>
> >>> Yep, but in practice we never use it. It would also consume 2 buffers,
> >>> one at the transmitter and one at the receiver.
> >>>
> >>> However I agree that maybe we should start using it before we decide not
> >>> to send any more data.
> >>>
> >>> To be compatible with older devices, though, I think for now we also
> >>> need to send a credit update when the bytes in the receive queue are
> >>> less than RCVLOWAT, as Arseniy proposed in the other series.
> >>
> >> Looks like (in theory of course), that credit request is considered to be
> >> paired with credit update. While current usage of credit update is something
> >> like ACK packet in TCP, e.g. telling peer that we are ready to receive more
> >> data.
> >
> > I don't honestly know what the original author's choice was, but I think we reduce latency this way.
>
> Ah I see,ok
>
> >
> > Effectively though, if we never send any credit update when we consume bytes and always leave it up to the transmitter to ask for an update before transmission, we save even more buffer than the optimization we have, but maybe the latency would grow a lot.
>
> I think:
> 1) Way where sender must request current credit status before sending packet requires rework of kernel part, and for me this approach is not
> so clear than current simple implementation (send RW, reply with CREDIT_UPDATE if needed).
> 2) In theory yes, we need one more buffer for such CREDIT_UPDATE, but in practice I don't know how big is this trouble.
>
> Thanks, Arseniy
I just worry that yes, normal users will only call RCVLOWAT once,
but a bad user might call it many times causing a ton of
credit updates. This is why I feel it's prudent to at least
keep track of last credit update and if nothing changed
do not resend it on a repeated RCVLOWAT.
> >
> > Stefano
> >
On 02.12.2023 23:22, Michael S. Tsirkin wrote:
> On Fri, Dec 01, 2023 at 01:40:41PM +0300, Arseniy Krasnov wrote:
>>
>>
>> On 01.12.2023 12:48, Stefano Garzarella wrote:
>>> On Fri, Dec 01, 2023 at 11:35:56AM +0300, Arseniy Krasnov wrote:
>>>>
>>>>
>>>> On 01.12.2023 11:27, Stefano Garzarella wrote:
>>>>> On Thu, Nov 30, 2023 at 12:40:43PM -0500, Michael S. Tsirkin wrote:
>>>>>> On Thu, Nov 30, 2023 at 03:11:19PM +0100, Stefano Garzarella wrote:
>>>>>>> On Thu, Nov 30, 2023 at 08:58:58AM -0500, Michael S. Tsirkin wrote:
>>>>>>>> On Thu, Nov 30, 2023 at 04:43:34PM +0300, Arseniy Krasnov wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 30.11.2023 16:42, Michael S. Tsirkin wrote:
>>>>>>>>>> On Thu, Nov 30, 2023 at 04:08:39PM +0300, Arseniy Krasnov wrote:
>>>>>>>>>>> Send credit update message when SO_RCVLOWAT is updated and it is bigger
>>>>>>>>>>> than number of bytes in rx queue. It is needed, because 'poll()' will
>>>>>>>>>>> wait until number of bytes in rx queue will be not smaller than
>>>>>>>>>>> SO_RCVLOWAT, so kick sender to send more data. Otherwise mutual hungup
>>>>>>>>>>> for tx/rx is possible: sender waits for free space and receiver is
>>>>>>>>>>> waiting data in 'poll()'.
>>>>>>>>>>>
>>>>>>>>>>> Signed-off-by: Arseniy Krasnov <[email protected]>
>>>>>>>>>>> ---
>>>>>>>>>>> Changelog:
>>>>>>>>>>> v1 -> v2:
>>>>>>>>>>> * Update commit message by removing 'This patch adds XXX' manner.
>>>>>>>>>>> * Do not initialize 'send_update' variable - set it directly during
>>>>>>>>>>> first usage.
>>>>>>>>>>> v3 -> v4:
>>>>>>>>>>> * Fit comment in 'virtio_transport_notify_set_rcvlowat()' to 80 chars.
>>>>>>>>>>> v4 -> v5:
>>>>>>>>>>> * Do not change callbacks order in transport structures.
>>>>>>>>>>>
>>>>>>>>>>> drivers/vhost/vsock.c | 1 +
>>>>>>>>>>> include/linux/virtio_vsock.h | 1 +
>>>>>>>>>>> net/vmw_vsock/virtio_transport.c | 1 +
>>>>>>>>>>> net/vmw_vsock/virtio_transport_common.c | 27 +++++++++++++++++++++++++
>>>>>>>>>>> net/vmw_vsock/vsock_loopback.c | 1 +
>>>>>>>>>>> 5 files changed, 31 insertions(+)
>>>>>>>>>>>
>>>>>>>>>>> diff --git a/drivers/vhost/vsock.c b/drivers/vhost/vsock.c
>>>>>>>>>>> index f75731396b7e..4146f80db8ac 100644
>>>>>>>>>>> --- a/drivers/vhost/vsock.c
>>>>>>>>>>> +++ b/drivers/vhost/vsock.c
>>>>>>>>>>> @@ -451,6 +451,7 @@ static struct virtio_transport vhost_transport = {
>>>>>>>>>>> .notify_buffer_size = virtio_transport_notify_buffer_size,
>>>>>>>>>>>
>>>>>>>>>>> .read_skb = virtio_transport_read_skb,
>>>>>>>>>>> + .notify_set_rcvlowat = virtio_transport_notify_set_rcvlowat
>>>>>>>>>>> },
>>>>>>>>>>>
>>>>>>>>>>> .send_pkt = vhost_transport_send_pkt,
>>>>>>>>>>> diff --git a/include/linux/virtio_vsock.h b/include/linux/virtio_vsock.h
>>>>>>>>>>> index ebb3ce63d64d..c82089dee0c8 100644
>>>>>>>>>>> --- a/include/linux/virtio_vsock.h
>>>>>>>>>>> +++ b/include/linux/virtio_vsock.h
>>>>>>>>>>> @@ -256,4 +256,5 @@ void virtio_transport_put_credit(struct virtio_vsock_sock *vvs, u32 credit);
>>>>>>>>>>> void virtio_transport_deliver_tap_pkt(struct sk_buff *skb);
>>>>>>>>>>> int virtio_transport_purge_skbs(void *vsk, struct sk_buff_head *list);
>>>>>>>>>>> int virtio_transport_read_skb(struct vsock_sock *vsk, skb_read_actor_t read_actor);
>>>>>>>>>>> +int virtio_transport_notify_set_rcvlowat(struct vsock_sock *vsk, int val);
>>>>>>>>>>> #endif /* _LINUX_VIRTIO_VSOCK_H */
>>>>>>>>>>> diff --git a/net/vmw_vsock/virtio_transport.c b/net/vmw_vsock/virtio_transport.c
>>>>>>>>>>> index af5bab1acee1..8007593a3a93 100644
>>>>>>>>>>> --- a/net/vmw_vsock/virtio_transport.c
>>>>>>>>>>> +++ b/net/vmw_vsock/virtio_transport.c
>>>>>>>>>>> @@ -539,6 +539,7 @@ static struct virtio_transport virtio_transport = {
>>>>>>>>>>> .notify_buffer_size = virtio_transport_notify_buffer_size,
>>>>>>>>>>>
>>>>>>>>>>> .read_skb = virtio_transport_read_skb,
>>>>>>>>>>> + .notify_set_rcvlowat = virtio_transport_notify_set_rcvlowat
>>>>>>>>>>> },
>>>>>>>>>>>
>>>>>>>>>>> .send_pkt = virtio_transport_send_pkt,
>>>>>>>>>>> diff --git a/net/vmw_vsock/virtio_transport_common.c b/net/vmw_vsock/virtio_transport_common.c
>>>>>>>>>>> index f6dc896bf44c..1cb556ad4597 100644
>>>>>>>>>>> --- a/net/vmw_vsock/virtio_transport_common.c
>>>>>>>>>>> +++ b/net/vmw_vsock/virtio_transport_common.c
>>>>>>>>>>> @@ -1684,6 +1684,33 @@ int virtio_transport_read_skb(struct vsock_sock *vsk, skb_read_actor_t recv_acto
>>>>>>>>>>> }
>>>>>>>>>>> EXPORT_SYMBOL_GPL(virtio_transport_read_skb);
>>>>>>>>>>>
>>>>>>>>>>> +int virtio_transport_notify_set_rcvlowat(struct vsock_sock *vsk,
>>>>>>>>>>> int val)
>>>>>>>>>>> +{
>>>>>>>>>>> + struct virtio_vsock_sock *vvs = vsk->trans;
>>>>>>>>>>> + bool send_update;
>>>>>>>>>>> +
>>>>>>>>>>> + spin_lock_bh(&vvs->rx_lock);
>>>>>>>>>>> +
>>>>>>>>>>> + /* If number of available bytes is less than new SO_RCVLOWAT value,
>>>>>>>>>>> + * kick sender to send more data, because sender may sleep in
>>>>>>>>>>> its
>>>>>>>>>>> + * 'send()' syscall waiting for enough space at our side.
>>>>>>>>>>> + */
>>>>>>>>>>> + send_update = vvs->rx_bytes < val;
>>>>>>>>>>> +
>>>>>>>>>>> + spin_unlock_bh(&vvs->rx_lock);
>>>>>>>>>>> +
>>>>>>>>>>> + if (send_update) {
>>>>>>>>>>> + int err;
>>>>>>>>>>> +
>>>>>>>>>>> + err = virtio_transport_send_credit_update(vsk);
>>>>>>>>>>> + if (err < 0)
>>>>>>>>>>> + return err;
>>>>>>>>>>> + }
>>>>>>>>>>> +
>>>>>>>>>>> + return 0;
>>>>>>>>>>> +}
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I find it strange that this will send a credit update
>>>>>>>>>> even if nothing changed since this was called previously.
>>>>>>>>>> I'm not sure whether this is a problem protocol-wise,
>>>>>>>>>> but it certainly was not envisioned when the protocol was
>>>>>>>>>> built. WDYT?
>>>>>>>>>
>>>>>>>>> >From virtio spec I found:
>>>>>>>>>
>>>>>>>>> It is also valid to send a VIRTIO_VSOCK_OP_CREDIT_UPDATE packet without previously receiving a
>>>>>>>>> VIRTIO_VSOCK_OP_CREDIT_REQUEST packet. This allows communicating updates any time a change
>>>>>>>>> in buffer space occurs.
>>>>>>>>> So I guess there is no limitations to send such type of packet, e.g. it is not
>>>>>>>>> required to be a reply for some another packet. Please, correct me if im wrong.
>>>>>>>>>
>>>>>>>>> Thanks, Arseniy
>>>>>>>>
>>>>>>>>
>>>>>>>> Absolutely. My point was different - with this patch it is possible
>>>>>>>> that you are not adding any credits at all since the previous
>>>>>>>> VIRTIO_VSOCK_OP_CREDIT_UPDATE.
>>>>>>>
>>>>>>> I think the problem we're solving here is that since as an optimization we
>>>>>>> avoid sending the update for every byte we consume, but we put a threshold,
>>>>>>> then we make sure we update the peer.
>>>>>>>
>>>>>>> A credit update contains a snapshot and sending it the same as the previous
>>>>>>> one should not create any problem.
>>>>>>
>>>>>> Well it consumes a buffer on the other side.
>>>>>
>>>>> Sure, but we are already speculating by not updating the other side when
>>>>> we consume bytes before a certain threshold. This already avoids to
>>>>> consume many buffers.
>>>>>
>>>>> Here we're only sending it once, when the user sets RCVLOWAT, so
>>>>> basically I expect it won't affect performance.
>>>>
>>>> Moreover I think in practice setting RCVLOWAT is rare case, while this patch
>>>> fixes real problem I guess
>>>>
>>>>
>>>>>
>>>>>>
>>>>>>> My doubt now is that we only do this when we set RCVLOWAT , should we also
>>>>>>> do something when we consume bytes to avoid the optimization we have?
>>>>>>>
>>>>>>> Stefano
>>>>>>
>>>>>> Isn't this why we have credit request?
>>>>>
>>>>> Yep, but in practice we never use it. It would also consume 2 buffers,
>>>>> one at the transmitter and one at the receiver.
>>>>>
>>>>> However I agree that maybe we should start using it before we decide not
>>>>> to send any more data.
>>>>>
>>>>> To be compatible with older devices, though, I think for now we also
>>>>> need to send a credit update when the bytes in the receive queue are
>>>>> less than RCVLOWAT, as Arseniy proposed in the other series.
>>>>
>>>> Looks like (in theory of course), that credit request is considered to be
>>>> paired with credit update. While current usage of credit update is something
>>>> like ACK packet in TCP, e.g. telling peer that we are ready to receive more
>>>> data.
>>>
>>> I don't honestly know what the original author's choice was, but I think we reduce latency this way.
>>
>> Ah I see,ok
>>
>>>
>>> Effectively though, if we never send any credit update when we consume bytes and always leave it up to the transmitter to ask for an update before transmission, we save even more buffer than the optimization we have, but maybe the latency would grow a lot.
>>
>> I think:
>> 1) Way where sender must request current credit status before sending packet requires rework of kernel part, and for me this approach is not
>> so clear than current simple implementation (send RW, reply with CREDIT_UPDATE if needed).
>> 2) In theory yes, we need one more buffer for such CREDIT_UPDATE, but in practice I don't know how big is this trouble.
>>
>> Thanks, Arseniy
>
> I just worry that yes, normal users will only call RCVLOWAT once,
> but a bad user might call it many times causing a ton of
> credit updates. This is why I feel it's prudent to at least
> keep track of last credit update and if nothing changed
> do not resend it on a repeated RCVLOWAT.
I see, agree, I'll add this check in the next version!
Thanks, Arseniy
>
>
>
>
>>>
>>> Stefano
>>>
>
On Sat, Dec 02, 2023 at 03:22:39PM -0500, Michael S. Tsirkin wrote:
>On Fri, Dec 01, 2023 at 01:40:41PM +0300, Arseniy Krasnov wrote:
>>
>>
>> On 01.12.2023 12:48, Stefano Garzarella wrote:
>> > On Fri, Dec 01, 2023 at 11:35:56AM +0300, Arseniy Krasnov wrote:
>> >>
>> >>
>> >> On 01.12.2023 11:27, Stefano Garzarella wrote:
>> >>> On Thu, Nov 30, 2023 at 12:40:43PM -0500, Michael S. Tsirkin wrote:
>> >>>> On Thu, Nov 30, 2023 at 03:11:19PM +0100, Stefano Garzarella wrote:
>> >>>>> On Thu, Nov 30, 2023 at 08:58:58AM -0500, Michael S. Tsirkin wrote:
>> >>>>> > On Thu, Nov 30, 2023 at 04:43:34PM +0300, Arseniy Krasnov wrote:
>> >>>>> > >
>> >>>>> > >
>> >>>>> > > On 30.11.2023 16:42, Michael S. Tsirkin wrote:
>> >>>>> > > > On Thu, Nov 30, 2023 at 04:08:39PM +0300, Arseniy Krasnov wrote:
>> >>>>> > > >> Send credit update message when SO_RCVLOWAT is updated and it is bigger
>> >>>>> > > >> than number of bytes in rx queue. It is needed, because 'poll()' will
>> >>>>> > > >> wait until number of bytes in rx queue will be not smaller than
>> >>>>> > > >> SO_RCVLOWAT, so kick sender to send more data. Otherwise mutual hungup
>> >>>>> > > >> for tx/rx is possible: sender waits for free space and receiver is
>> >>>>> > > >> waiting data in 'poll()'.
>> >>>>> > > >>
>> >>>>> > > >> Signed-off-by: Arseniy Krasnov <[email protected]>
>> >>>>> > > >> ---
>> >>>>> > > >>? Changelog:
>> >>>>> > > >>? v1 -> v2:
>> >>>>> > > >>?? * Update commit message by removing 'This patch adds XXX' manner.
>> >>>>> > > >>?? * Do not initialize 'send_update' variable - set it directly during
>> >>>>> > > >>???? first usage.
>> >>>>> > > >>? v3 -> v4:
>> >>>>> > > >>?? * Fit comment in 'virtio_transport_notify_set_rcvlowat()' to 80 chars.
>> >>>>> > > >>? v4 -> v5:
>> >>>>> > > >>?? * Do not change callbacks order in transport structures.
>> >>>>> > > >>
>> >>>>> > > >>? drivers/vhost/vsock.c?????????????????? |? 1 +
>> >>>>> > > >>? include/linux/virtio_vsock.h??????????? |? 1 +
>> >>>>> > > >>? net/vmw_vsock/virtio_transport.c??????? |? 1 +
>> >>>>> > > >>? net/vmw_vsock/virtio_transport_common.c | 27 +++++++++++++++++++++++++
>> >>>>> > > >>? net/vmw_vsock/vsock_loopback.c????????? |? 1 +
>> >>>>> > > >>? 5 files changed, 31 insertions(+)
>> >>>>> > > >>
>> >>>>> > > >> diff --git a/drivers/vhost/vsock.c b/drivers/vhost/vsock.c
>> >>>>> > > >> index f75731396b7e..4146f80db8ac 100644
>> >>>>> > > >> --- a/drivers/vhost/vsock.c
>> >>>>> > > >> +++ b/drivers/vhost/vsock.c
>> >>>>> > > >> @@ -451,6 +451,7 @@ static struct virtio_transport vhost_transport = {
>> >>>>> > > >>????????? .notify_buffer_size?????? = virtio_transport_notify_buffer_size,
>> >>>>> > > >>
>> >>>>> > > >>????????? .read_skb = virtio_transport_read_skb,
>> >>>>> > > >> +??????? .notify_set_rcvlowat????? = virtio_transport_notify_set_rcvlowat
>> >>>>> > > >>????? },
>> >>>>> > > >>
>> >>>>> > > >>????? .send_pkt = vhost_transport_send_pkt,
>> >>>>> > > >> diff --git a/include/linux/virtio_vsock.h b/include/linux/virtio_vsock.h
>> >>>>> > > >> index ebb3ce63d64d..c82089dee0c8 100644
>> >>>>> > > >> --- a/include/linux/virtio_vsock.h
>> >>>>> > > >> +++ b/include/linux/virtio_vsock.h
>> >>>>> > > >> @@ -256,4 +256,5 @@ void virtio_transport_put_credit(struct virtio_vsock_sock *vvs, u32 credit);
>> >>>>> > > >>? void virtio_transport_deliver_tap_pkt(struct sk_buff *skb);
>> >>>>> > > >>? int virtio_transport_purge_skbs(void *vsk, struct sk_buff_head *list);
>> >>>>> > > >>? int virtio_transport_read_skb(struct vsock_sock *vsk, skb_read_actor_t read_actor);
>> >>>>> > > >> +int virtio_transport_notify_set_rcvlowat(struct vsock_sock *vsk, int val);
>> >>>>> > > >>? #endif /* _LINUX_VIRTIO_VSOCK_H */
>> >>>>> > > >> diff --git a/net/vmw_vsock/virtio_transport.c b/net/vmw_vsock/virtio_transport.c
>> >>>>> > > >> index af5bab1acee1..8007593a3a93 100644
>> >>>>> > > >> --- a/net/vmw_vsock/virtio_transport.c
>> >>>>> > > >> +++ b/net/vmw_vsock/virtio_transport.c
>> >>>>> > > >> @@ -539,6 +539,7 @@ static struct virtio_transport virtio_transport = {
>> >>>>> > > >>????????? .notify_buffer_size?????? = virtio_transport_notify_buffer_size,
>> >>>>> > > >>
>> >>>>> > > >>????????? .read_skb = virtio_transport_read_skb,
>> >>>>> > > >> +??????? .notify_set_rcvlowat????? = virtio_transport_notify_set_rcvlowat
>> >>>>> > > >>????? },
>> >>>>> > > >>
>> >>>>> > > >>????? .send_pkt = virtio_transport_send_pkt,
>> >>>>> > > >> diff --git a/net/vmw_vsock/virtio_transport_common.c b/net/vmw_vsock/virtio_transport_common.c
>> >>>>> > > >> index f6dc896bf44c..1cb556ad4597 100644
>> >>>>> > > >> --- a/net/vmw_vsock/virtio_transport_common.c
>> >>>>> > > >> +++ b/net/vmw_vsock/virtio_transport_common.c
>> >>>>> > > >> @@ -1684,6 +1684,33 @@ int virtio_transport_read_skb(struct vsock_sock *vsk, skb_read_actor_t recv_acto
>> >>>>> > > >>? }
>> >>>>> > > >>? EXPORT_SYMBOL_GPL(virtio_transport_read_skb);
>> >>>>> > > >>
>> >>>>> > > >> +int virtio_transport_notify_set_rcvlowat(struct vsock_sock *vsk,
>> >>>>> > > >> int val)
>> >>>>> > > >> +{
>> >>>>> > > >> +??? struct virtio_vsock_sock *vvs = vsk->trans;
>> >>>>> > > >> +??? bool send_update;
>> >>>>> > > >> +
>> >>>>> > > >> +??? spin_lock_bh(&vvs->rx_lock);
>> >>>>> > > >> +
>> >>>>> > > >> +??? /* If number of available bytes is less than new SO_RCVLOWAT value,
>> >>>>> > > >> +???? * kick sender to send more data, because sender may sleep in
>> >>>>> > > >> its
>> >>>>> > > >> +???? * 'send()' syscall waiting for enough space at our side.
>> >>>>> > > >> +???? */
>> >>>>> > > >> +??? send_update = vvs->rx_bytes < val;
>> >>>>> > > >> +
>> >>>>> > > >> +??? spin_unlock_bh(&vvs->rx_lock);
>> >>>>> > > >> +
>> >>>>> > > >> +??? if (send_update) {
>> >>>>> > > >> +??????? int err;
>> >>>>> > > >> +
>> >>>>> > > >> +??????? err = virtio_transport_send_credit_update(vsk);
>> >>>>> > > >> +??????? if (err < 0)
>> >>>>> > > >> +??????????? return err;
>> >>>>> > > >> +??? }
>> >>>>> > > >> +
>> >>>>> > > >> +??? return 0;
>> >>>>> > > >> +}
>> >>>>> > > >
>> >>>>> > > >
>> >>>>> > > > I find it strange that this will send a credit update
>> >>>>> > > > even if nothing changed since this was called previously.
>> >>>>> > > > I'm not sure whether this is a problem protocol-wise,
>> >>>>> > > > but it certainly was not envisioned when the protocol was
>> >>>>> > > > built. WDYT?
>> >>>>> > >
>> >>>>> > > >From virtio spec I found:
>> >>>>> > >
>> >>>>> > > It is also valid to send a VIRTIO_VSOCK_OP_CREDIT_UPDATE packet without previously receiving a
>> >>>>> > > VIRTIO_VSOCK_OP_CREDIT_REQUEST packet. This allows communicating updates any time a change
>> >>>>> > > in buffer space occurs.
>> >>>>> > > So I guess there is no limitations to send such type of packet, e.g. it is not
>> >>>>> > > required to be a reply for some another packet. Please, correct me if im wrong.
>> >>>>> > >
>> >>>>> > > Thanks, Arseniy
>> >>>>> >
>> >>>>> >
>> >>>>> > Absolutely. My point was different - with this patch it is possible
>> >>>>> > that you are not adding any credits at all since the previous
>> >>>>> > VIRTIO_VSOCK_OP_CREDIT_UPDATE.
>> >>>>>
>> >>>>> I think the problem we're solving here is that since as an optimization we
>> >>>>> avoid sending the update for every byte we consume, but we put a threshold,
>> >>>>> then we make sure we update the peer.
>> >>>>>
>> >>>>> A credit update contains a snapshot and sending it the same as the previous
>> >>>>> one should not create any problem.
>> >>>>
>> >>>> Well it consumes a buffer on the other side.
>> >>>
>> >>> Sure, but we are already speculating by not updating the other side when
>> >>> we consume bytes before a certain threshold. This already avoids to
>> >>> consume many buffers.
>> >>>
>> >>> Here we're only sending it once, when the user sets RCVLOWAT, so
>> >>> basically I expect it won't affect performance.
>> >>
>> >> Moreover I think in practice setting RCVLOWAT is rare case, while this patch
>> >> fixes real problem I guess
>> >>
>> >>
>> >>>
>> >>>>
>> >>>>> My doubt now is that we only do this when we set RCVLOWAT , should we also
>> >>>>> do something when we consume bytes to avoid the optimization we have?
>> >>>>>
>> >>>>> Stefano
>> >>>>
>> >>>> Isn't this why we have credit request?
>> >>>
>> >>> Yep, but in practice we never use it. It would also consume 2 buffers,
>> >>> one at the transmitter and one at the receiver.
>> >>>
>> >>> However I agree that maybe we should start using it before we decide not
>> >>> to send any more data.
>> >>>
>> >>> To be compatible with older devices, though, I think for now we also
>> >>> need to send a credit update when the bytes in the receive queue are
>> >>> less than RCVLOWAT, as Arseniy proposed in the other series.
>> >>
>> >> Looks like (in theory of course), that credit request is considered to be
>> >> paired with credit update. While current usage of credit update is something
>> >> like ACK packet in TCP, e.g. telling peer that we are ready to receive more
>> >> data.
>> >
>> > I don't honestly know what the original author's choice was, but I think we reduce latency this way.
>>
>> Ah I see,ok
>>
>> >
>> > Effectively though, if we never send any credit update when we consume bytes and always leave it up to the transmitter to ask for an update before transmission, we save even more buffer than the optimization we have, but maybe the latency would grow a lot.
>>
>> I think:
>> 1) Way where sender must request current credit status before sending packet requires rework of kernel part, and for me this approach is not
>> so clear than current simple implementation (send RW, reply with CREDIT_UPDATE if needed).
>> 2) In theory yes, we need one more buffer for such CREDIT_UPDATE, but in practice I don't know how big is this trouble.
>>
>> Thanks, Arseniy
>
>I just worry that yes, normal users will only call RCVLOWAT once,
>but a bad user might call it many times causing a ton of
>credit updates. This is why I feel it's prudent to at least
>keep track of last credit update and if nothing changed
>do not resend it on a repeated RCVLOWAT.
Okay, now I get you!
Yeah, absolutely, we already have `last_fwd_cnt` in `struct
virtio_vsock_sock` to do that check.
Thanks,
Stefano