2014-11-20 04:42:30

by Dexuan Cui

Subject: [PATCH v2] tools: hv: ignore ENOBUFS and ENOMEM in the KVP daemon

Under high memory pressure and very high KVP R/W test pressure, the netlink
recvfrom() may transiently return ENOBUFS to the daemon -- we found this
during a 2-week stress test.

We'd better not terminate the daemon on the failure, because a typical KVP
user will re-try the R/W and hopefully it will succeed next time.

We can also ignore the errors on sending.

Cc: Vitaly Kuznetsov <[email protected]>
Cc: K. Y. Srinivasan <[email protected]>
Signed-off-by: Dexuan Cui <[email protected]>
---

v2: I also ignore the errors on sending, as Vitaly suggested.

tools/hv/hv_kvp_daemon.c | 14 ++++++++++++++
1 file changed, 14 insertions(+)

diff --git a/tools/hv/hv_kvp_daemon.c b/tools/hv/hv_kvp_daemon.c
index 22b0764..6a6432a 100644
--- a/tools/hv/hv_kvp_daemon.c
+++ b/tools/hv/hv_kvp_daemon.c
@@ -1559,8 +1559,15 @@ int main(int argc, char *argv[])
addr_p, &addr_l);

if (len < 0) {
+ int saved_errno = errno;
syslog(LOG_ERR, "recvfrom failed; pid:%u error:%d %s",
addr.nl_pid, errno, strerror(errno));
+
+ if (saved_errno == ENOBUFS) {
+ syslog(LOG_ERR, "receive error: ignored");
+ continue;
+ }
+
close(fd);
return -1;
}
@@ -1763,8 +1770,15 @@ kvp_done:

len = netlink_send(fd, incoming_cn_msg);
if (len < 0) {
+ int saved_errno = errno;
syslog(LOG_ERR, "net_link send failed; error: %d %s", errno,
strerror(errno));
+
+ if (saved_errno == ENOMEM || saved_errno == ENOBUFS) {
+ syslog(LOG_ERR, "send error: ignored");
+ continue;
+ }
+
exit(EXIT_FAILURE);
}
}
--
1.9.1

2014-11-20 09:37:57

by Vitaly Kuznetsov

Subject: Re: [PATCH v2] tools: hv: ignore ENOBUFS and ENOMEM in the KVP daemon

Dexuan Cui <[email protected]> writes:

> Under high memory pressure and very high KVP R/W test pressure, the netlink
> recvfrom() may transiently return ENOBUFS to the daemon -- we found this
> during a 2-week stress test.
>
> We'd better not terminate the daemon on the failure, because a typical KVP
> user will re-try the R/W and hopefully it will succeed next time.
>
> We can also ignore the errors on sending.
>
> Cc: Vitaly Kuznetsov <[email protected]>
> Cc: K. Y. Srinivasan <[email protected]>
> Signed-off-by: Dexuan Cui <[email protected]>
> ---
>
> v2: I also ignore the errors on sending, as Vitaly suggested.

Thanks,

Reviewed-by: Vitaly Kuznetsov <[email protected]>

>
> tools/hv/hv_kvp_daemon.c | 14 ++++++++++++++
> 1 file changed, 14 insertions(+)
>
> diff --git a/tools/hv/hv_kvp_daemon.c b/tools/hv/hv_kvp_daemon.c
> index 22b0764..6a6432a 100644
> --- a/tools/hv/hv_kvp_daemon.c
> +++ b/tools/hv/hv_kvp_daemon.c
> @@ -1559,8 +1559,15 @@ int main(int argc, char *argv[])
> addr_p, &addr_l);
>
> if (len < 0) {
> + int saved_errno = errno;
> syslog(LOG_ERR, "recvfrom failed; pid:%u error:%d %s",
> addr.nl_pid, errno, strerror(errno));
> +
> + if (saved_errno == ENOBUFS) {
> + syslog(LOG_ERR, "receive error: ignored");
> + continue;
> + }
> +
> close(fd);
> return -1;
> }
> @@ -1763,8 +1770,15 @@ kvp_done:
>
> len = netlink_send(fd, incoming_cn_msg);
> if (len < 0) {
> + int saved_errno = errno;
> syslog(LOG_ERR, "net_link send failed; error: %d %s", errno,
> strerror(errno));
> +
> + if (saved_errno == ENOMEM || saved_errno == ENOBUFS) {
> + syslog(LOG_ERR, "send error: ignored");
> + continue;
> + }
> +
> exit(EXIT_FAILURE);
> }
> }

--
Vitaly