Hi,
When the guest runs under critial memory pressure, the guest becomss
too slow, even sshd turns D state(uninterruptible) on memory
allocation. We can't login this VM to do any work on trouble shooting.
Guest kernel log via virtual TTY(on host side) only provides a few
necessary log after OOM. More detail memory statistics are required,
then we can know explicit memory events and estimate the pressure.
I'm going to introduce several VM counters for virtio balloon:
- oom-kill
- alloc-stall
- scan-async
- scan-direct
- reclaim-async
- reclaim-direct
Then we have a metric to analyze the memory performance:
[also describe this metric in patch
'virtio_balloon: introduce memory scan/reclaim info']
y: counter increases
n: counter does not changes
h: the rate of counter change is high
l: the rate of counter change is low
OOM: VIRTIO_BALLOON_S_OOM_KILL
STALL: VIRTIO_BALLOON_S_ALLOC_STALL
ASCAN: VIRTIO_BALLOON_S_SCAN_ASYNC
DSCAN: VIRTIO_BALLOON_S_SCAN_DIRECT
ARCLM: VIRTIO_BALLOON_S_RECLAIM_ASYNC
DRCLM: VIRTIO_BALLOON_S_RECLAIM_DIRECT
- OOM[y], STALL[*], ASCAN[*], DSCAN[*], ARCLM[*], DRCLM[*]:
the guest runs under really critial memory pressure
- OOM[n], STALL[h], ASCAN[*], DSCAN[l], ARCLM[*], DRCLM[l]:
the memory allocation stalls due to cgroup, not the global memory
pressure.
- OOM[n], STALL[h], ASCAN[*], DSCAN[h], ARCLM[*], DRCLM[h]:
the memory allocation stalls due to global memory pressure. The
performance gets hurt a lot. A high ratio between DRCLM/DSCAN shows
quite effective memory reclaiming.
- OOM[n], STALL[h], ASCAN[*], DSCAN[h], ARCLM[*], DRCLM[l]:
the memory allocation stalls due to global memory pressure.
the ratio between DRCLM/DSCAN gets low, the guest OS is thrashing
heavily, the serious case leads poor performance and difficult
trouble shooting. Ex, sshd may block on memory allocation when
accepting new connections, a user can't login a VM by ssh command.
- OOM[n], STALL[n], ASCAN[h], DSCAN[n], ARCLM[l], DRCLM[n]:
the low ratio between ARCLM/ASCAN shows that the guest tries to
reclaim more memory, but it can't. Once more memory is required in
future, it will struggle to reclaim memory.
zhenwei pi (3):
virtio_balloon: introduce oom-kill invocations
virtio_balloon: introduce memory allocation stall counter
virtio_balloon: introduce memory scan/reclaim info
drivers/virtio/virtio_balloon.c | 30 ++++++++++++++++++++++++++++-
include/uapi/linux/virtio_balloon.h | 16 +++++++++++++--
2 files changed, 43 insertions(+), 3 deletions(-)
--
2.34.1
When the guest OS runs under critical memory pressure, the guest
starts to kill processes. A guest monitor agent may scan 'oom_kill'
from /proc/vmstat, and reports the OOM KILL event. However, the agent
may be killed and we will loss this critical event(and the later
events).
For now we can also grep for magic words in guest kernel log from host
side. Rather than this unstable way, virtio balloon reports OOM-KILL
invocations instead.
Signed-off-by: zhenwei pi <[email protected]>
---
drivers/virtio/virtio_balloon.c | 2 ++
include/uapi/linux/virtio_balloon.h | 6 ++++--
2 files changed, 6 insertions(+), 2 deletions(-)
diff --git a/drivers/virtio/virtio_balloon.c b/drivers/virtio/virtio_balloon.c
index 1f5b3dd31fcf..fd8daa742734 100644
--- a/drivers/virtio/virtio_balloon.c
+++ b/drivers/virtio/virtio_balloon.c
@@ -352,6 +352,8 @@ static unsigned int update_balloon_stats(struct virtio_balloon *vb)
pages_to_bytes(available));
update_stat(vb, idx++, VIRTIO_BALLOON_S_CACHES,
pages_to_bytes(caches));
+ update_stat(vb, idx++, VIRTIO_BALLOON_S_OOM_KILL,
+ events[OOM_KILL]);
return idx;
}
diff --git a/include/uapi/linux/virtio_balloon.h b/include/uapi/linux/virtio_balloon.h
index ddaa45e723c4..cde5547e64a7 100644
--- a/include/uapi/linux/virtio_balloon.h
+++ b/include/uapi/linux/virtio_balloon.h
@@ -71,7 +71,8 @@ struct virtio_balloon_config {
#define VIRTIO_BALLOON_S_CACHES 7 /* Disk caches */
#define VIRTIO_BALLOON_S_HTLB_PGALLOC 8 /* Hugetlb page allocations */
#define VIRTIO_BALLOON_S_HTLB_PGFAIL 9 /* Hugetlb page allocation failures */
-#define VIRTIO_BALLOON_S_NR 10
+#define VIRTIO_BALLOON_S_OOM_KILL 10 /* OOM killer invocations */
+#define VIRTIO_BALLOON_S_NR 11
#define VIRTIO_BALLOON_S_NAMES_WITH_PREFIX(VIRTIO_BALLOON_S_NAMES_prefix) { \
VIRTIO_BALLOON_S_NAMES_prefix "swap-in", \
@@ -83,7 +84,8 @@ struct virtio_balloon_config {
VIRTIO_BALLOON_S_NAMES_prefix "available-memory", \
VIRTIO_BALLOON_S_NAMES_prefix "disk-caches", \
VIRTIO_BALLOON_S_NAMES_prefix "hugetlb-allocations", \
- VIRTIO_BALLOON_S_NAMES_prefix "hugetlb-failures" \
+ VIRTIO_BALLOON_S_NAMES_prefix "hugetlb-failures", \
+ VIRTIO_BALLOON_S_NAMES_prefix "oom-kill" \
}
#define VIRTIO_BALLOON_S_NAMES VIRTIO_BALLOON_S_NAMES_WITH_PREFIX("")
--
2.34.1
Expose memory scan/reclaim information to the host side via virtio
balloon device.
Now we have a metric to analyze the memory performance:
y: counter increases
n: counter does not changes
h: the rate of counter change is high
l: the rate of counter change is low
OOM: VIRTIO_BALLOON_S_OOM_KILL
STALL: VIRTIO_BALLOON_S_ALLOC_STALL
ASCAN: VIRTIO_BALLOON_S_SCAN_ASYNC
DSCAN: VIRTIO_BALLOON_S_SCAN_DIRECT
ARCLM: VIRTIO_BALLOON_S_RECLAIM_ASYNC
DRCLM: VIRTIO_BALLOON_S_RECLAIM_DIRECT
- OOM[y], STALL[*], ASCAN[*], DSCAN[*], ARCLM[*], DRCLM[*]:
the guest runs under really critial memory pressure
- OOM[n], STALL[h], ASCAN[*], DSCAN[l], ARCLM[*], DRCLM[l]:
the memory allocation stalls due to cgroup, not the global memory
pressure.
- OOM[n], STALL[h], ASCAN[*], DSCAN[h], ARCLM[*], DRCLM[h]:
the memory allocation stalls due to global memory pressure. The
performance gets hurt a lot. A high ratio between DRCLM/DSCAN shows
quite effective memory reclaiming.
- OOM[n], STALL[h], ASCAN[*], DSCAN[h], ARCLM[*], DRCLM[l]:
the memory allocation stalls due to global memory pressure.
the ratio between DRCLM/DSCAN gets low, the guest OS is thrashing
heavily, the serious case leads poor performance and difficult
trouble shooting. Ex, sshd may block on memory allocation when
accepting new connections, a user can't login a VM by ssh command.
- OOM[n], STALL[n], ASCAN[h], DSCAN[n], ARCLM[l], DRCLM[n]:
the low ratio between ARCLM/ASCAN shows that the guest tries to
reclaim more memory, but it can't. Once more memory is required in
future, it will struggle to reclaim memory.
Signed-off-by: zhenwei pi <[email protected]>
---
drivers/virtio/virtio_balloon.c | 9 +++++++++
include/uapi/linux/virtio_balloon.h | 12 ++++++++++--
2 files changed, 19 insertions(+), 2 deletions(-)
diff --git a/drivers/virtio/virtio_balloon.c b/drivers/virtio/virtio_balloon.c
index 4b9c9569f6e5..7b86514e99d4 100644
--- a/drivers/virtio/virtio_balloon.c
+++ b/drivers/virtio/virtio_balloon.c
@@ -372,6 +372,15 @@ static unsigned int update_balloon_stats(struct virtio_balloon *vb)
stall += events[ALLOCSTALL_MOVABLE];
update_stat(vb, idx++, VIRTIO_BALLOON_S_ALLOC_STALL, stall);
+ update_stat(vb, idx++, VIRTIO_BALLOON_S_SCAN_ASYNC,
+ pages_to_bytes(events[PGSCAN_KSWAPD]));
+ update_stat(vb, idx++, VIRTIO_BALLOON_S_SCAN_DIRECT,
+ pages_to_bytes(events[PGSCAN_DIRECT]));
+ update_stat(vb, idx++, VIRTIO_BALLOON_S_RECLAIM_ASYNC,
+ pages_to_bytes(events[PGSTEAL_KSWAPD]));
+ update_stat(vb, idx++, VIRTIO_BALLOON_S_RECLAIM_DIRECT,
+ pages_to_bytes(events[PGSTEAL_DIRECT]));
+
return idx;
}
diff --git a/include/uapi/linux/virtio_balloon.h b/include/uapi/linux/virtio_balloon.h
index 13d0c32ba27c..0875a9cccb01 100644
--- a/include/uapi/linux/virtio_balloon.h
+++ b/include/uapi/linux/virtio_balloon.h
@@ -73,7 +73,11 @@ struct virtio_balloon_config {
#define VIRTIO_BALLOON_S_HTLB_PGFAIL 9 /* Hugetlb page allocation failures */
#define VIRTIO_BALLOON_S_OOM_KILL 10 /* OOM killer invocations */
#define VIRTIO_BALLOON_S_ALLOC_STALL 11 /* Stall count of memory allocatoin */
-#define VIRTIO_BALLOON_S_NR 12
+#define VIRTIO_BALLOON_S_SCAN_ASYNC 12 /* Amount of memory scanned asynchronously */
+#define VIRTIO_BALLOON_S_SCAN_DIRECT 13 /* Amount of memory scanned directly */
+#define VIRTIO_BALLOON_S_RECLAIM_ASYNC 14 /* Amount of memory reclaimed asynchronously */
+#define VIRTIO_BALLOON_S_RECLAIM_DIRECT 15 /* Amount of memory reclaimed directly */
+#define VIRTIO_BALLOON_S_NR 16
#define VIRTIO_BALLOON_S_NAMES_WITH_PREFIX(VIRTIO_BALLOON_S_NAMES_prefix) { \
VIRTIO_BALLOON_S_NAMES_prefix "swap-in", \
@@ -87,7 +91,11 @@ struct virtio_balloon_config {
VIRTIO_BALLOON_S_NAMES_prefix "hugetlb-allocations", \
VIRTIO_BALLOON_S_NAMES_prefix "hugetlb-failures", \
VIRTIO_BALLOON_S_NAMES_prefix "oom-kill", \
- VIRTIO_BALLOON_S_NAMES_prefix "alloc-stall" \
+ VIRTIO_BALLOON_S_NAMES_prefix "alloc-stall", \
+ VIRTIO_BALLOON_S_NAMES_prefix "scan-async", \
+ VIRTIO_BALLOON_S_NAMES_prefix "scan-direct", \
+ VIRTIO_BALLOON_S_NAMES_prefix "reclaim-async", \
+ VIRTIO_BALLOON_S_NAMES_prefix "reclaim-direct" \
}
#define VIRTIO_BALLOON_S_NAMES VIRTIO_BALLOON_S_NAMES_WITH_PREFIX("")
--
2.34.1
Memory allocation stall counter represents the performance/latency of
memory allocation, expose this counter to the host side by virtio
balloon device via out-of-bound way.
Signed-off-by: zhenwei pi <[email protected]>
---
drivers/virtio/virtio_balloon.c | 19 ++++++++++++++++++-
include/uapi/linux/virtio_balloon.h | 6 ++++--
2 files changed, 22 insertions(+), 3 deletions(-)
diff --git a/drivers/virtio/virtio_balloon.c b/drivers/virtio/virtio_balloon.c
index fd8daa742734..4b9c9569f6e5 100644
--- a/drivers/virtio/virtio_balloon.c
+++ b/drivers/virtio/virtio_balloon.c
@@ -321,7 +321,7 @@ static unsigned int update_balloon_stats(struct virtio_balloon *vb)
unsigned long events[NR_VM_EVENT_ITEMS];
struct sysinfo i;
unsigned int idx = 0;
- long available;
+ long available, stall = 0;
unsigned long caches;
all_vm_events(events);
@@ -355,6 +355,23 @@ static unsigned int update_balloon_stats(struct virtio_balloon *vb)
update_stat(vb, idx++, VIRTIO_BALLOON_S_OOM_KILL,
events[OOM_KILL]);
+ /* sum all the stall event */
+#ifdef CONFIG_ZONE_DMA
+ stall += events[ALLOCSTALL_DMA];
+#endif
+#ifdef CONFIG_ZONE_DMA32
+ stall += events[ALLOCSTALL_DMA32];
+#endif
+#ifdef CONFIG_HIGHMEM
+ stall += events[ALLOCSTALL_HIGH];
+#endif
+#ifdef CONFIG_ZONE_DEVICE
+ stall += events[ALLOCSTALL_DEVICE];
+#endif
+ stall += events[ALLOCSTALL_NORMAL];
+ stall += events[ALLOCSTALL_MOVABLE];
+ update_stat(vb, idx++, VIRTIO_BALLOON_S_ALLOC_STALL, stall);
+
return idx;
}
diff --git a/include/uapi/linux/virtio_balloon.h b/include/uapi/linux/virtio_balloon.h
index cde5547e64a7..13d0c32ba27c 100644
--- a/include/uapi/linux/virtio_balloon.h
+++ b/include/uapi/linux/virtio_balloon.h
@@ -72,7 +72,8 @@ struct virtio_balloon_config {
#define VIRTIO_BALLOON_S_HTLB_PGALLOC 8 /* Hugetlb page allocations */
#define VIRTIO_BALLOON_S_HTLB_PGFAIL 9 /* Hugetlb page allocation failures */
#define VIRTIO_BALLOON_S_OOM_KILL 10 /* OOM killer invocations */
-#define VIRTIO_BALLOON_S_NR 11
+#define VIRTIO_BALLOON_S_ALLOC_STALL 11 /* Stall count of memory allocatoin */
+#define VIRTIO_BALLOON_S_NR 12
#define VIRTIO_BALLOON_S_NAMES_WITH_PREFIX(VIRTIO_BALLOON_S_NAMES_prefix) { \
VIRTIO_BALLOON_S_NAMES_prefix "swap-in", \
@@ -85,7 +86,8 @@ struct virtio_balloon_config {
VIRTIO_BALLOON_S_NAMES_prefix "disk-caches", \
VIRTIO_BALLOON_S_NAMES_prefix "hugetlb-allocations", \
VIRTIO_BALLOON_S_NAMES_prefix "hugetlb-failures", \
- VIRTIO_BALLOON_S_NAMES_prefix "oom-kill" \
+ VIRTIO_BALLOON_S_NAMES_prefix "oom-kill", \
+ VIRTIO_BALLOON_S_NAMES_prefix "alloc-stall" \
}
#define VIRTIO_BALLOON_S_NAMES VIRTIO_BALLOON_S_NAMES_WITH_PREFIX("")
--
2.34.1
On 15.04.24 10:41, zhenwei pi wrote:
> Hi,
>
> When the guest runs under critial memory pressure, the guest becomss
> too slow, even sshd turns D state(uninterruptible) on memory
> allocation. We can't login this VM to do any work on trouble shooting.
>
> Guest kernel log via virtual TTY(on host side) only provides a few
> necessary log after OOM. More detail memory statistics are required,
> then we can know explicit memory events and estimate the pressure.
>
> I'm going to introduce several VM counters for virtio balloon:
> - oom-kill
> - alloc-stall
> - scan-async
> - scan-direct
> - reclaim-async
> - reclaim-direct
IIUC, we're only exposing events that are already getting provided via
all_vm_events(), correct?
In that case, I don't really see a major issue. Some considerations:
(1) These new events are fairly Linux specific.
PSWPIN and friends are fairly generic, but HGTLB is also already fairly
Linux specific already. OOM-kills don't really exist on Windows, for
example. We'll have to be careful of properly describing what the
semantics are.
(2) How should we handle if Linux ever stops supporting a certain event
(e.g., major reclaim rework). I assume, simply return nothing like we
currently would for VIRTIO_BALLOON_S_HTLB_PGALLOC without
CONFIG_HUGETLB_PAGE.
--
Cheers,
David / dhildenb
On 15.04.24 10:41, zhenwei pi wrote:
> When the guest OS runs under critical memory pressure, the guest
> starts to kill processes. A guest monitor agent may scan 'oom_kill'
> from /proc/vmstat, and reports the OOM KILL event. However, the agent
> may be killed and we will loss this critical event(and the later
> events).
>
> For now we can also grep for magic words in guest kernel log from host
> side. Rather than this unstable way, virtio balloon reports OOM-KILL
> invocations instead.
>
> Signed-off-by: zhenwei pi <[email protected]>
> ---
> drivers/virtio/virtio_balloon.c | 2 ++
> include/uapi/linux/virtio_balloon.h | 6 ++++--
> 2 files changed, 6 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/virtio/virtio_balloon.c b/drivers/virtio/virtio_balloon.c
> index 1f5b3dd31fcf..fd8daa742734 100644
> --- a/drivers/virtio/virtio_balloon.c
> +++ b/drivers/virtio/virtio_balloon.c
> @@ -352,6 +352,8 @@ static unsigned int update_balloon_stats(struct virtio_balloon *vb)
> pages_to_bytes(available));
> update_stat(vb, idx++, VIRTIO_BALLOON_S_CACHES,
> pages_to_bytes(caches));
> + update_stat(vb, idx++, VIRTIO_BALLOON_S_OOM_KILL,
> + events[OOM_KILL]);
>
> return idx;
> }
> diff --git a/include/uapi/linux/virtio_balloon.h b/include/uapi/linux/virtio_balloon.h
> index ddaa45e723c4..cde5547e64a7 100644
> --- a/include/uapi/linux/virtio_balloon.h
> +++ b/include/uapi/linux/virtio_balloon.h
> @@ -71,7 +71,8 @@ struct virtio_balloon_config {
> #define VIRTIO_BALLOON_S_CACHES 7 /* Disk caches */
> #define VIRTIO_BALLOON_S_HTLB_PGALLOC 8 /* Hugetlb page allocations */
> #define VIRTIO_BALLOON_S_HTLB_PGFAIL 9 /* Hugetlb page allocation failures */
> -#define VIRTIO_BALLOON_S_NR 10
> +#define VIRTIO_BALLOON_S_OOM_KILL 10 /* OOM killer invocations */
> +#define VIRTIO_BALLOON_S_NR 11
>
> #define VIRTIO_BALLOON_S_NAMES_WITH_PREFIX(VIRTIO_BALLOON_S_NAMES_prefix) { \
> VIRTIO_BALLOON_S_NAMES_prefix "swap-in", \
> @@ -83,7 +84,8 @@ struct virtio_balloon_config {
> VIRTIO_BALLOON_S_NAMES_prefix "available-memory", \
> VIRTIO_BALLOON_S_NAMES_prefix "disk-caches", \
> VIRTIO_BALLOON_S_NAMES_prefix "hugetlb-allocations", \
> - VIRTIO_BALLOON_S_NAMES_prefix "hugetlb-failures" \
> + VIRTIO_BALLOON_S_NAMES_prefix "hugetlb-failures", \
> + VIRTIO_BALLOON_S_NAMES_prefix "oom-kill" \
"oom-kills"
--
Cheers,
David / dhildenb
On 15.04.24 10:41, zhenwei pi wrote:
> Memory allocation stall counter represents the performance/latency of
> memory allocation, expose this counter to the host side by virtio
> balloon device via out-of-bound way.
>
> Signed-off-by: zhenwei pi <[email protected]>
> ---
> drivers/virtio/virtio_balloon.c | 19 ++++++++++++++++++-
> include/uapi/linux/virtio_balloon.h | 6 ++++--
> 2 files changed, 22 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/virtio/virtio_balloon.c b/drivers/virtio/virtio_balloon.c
> index fd8daa742734..4b9c9569f6e5 100644
> --- a/drivers/virtio/virtio_balloon.c
> +++ b/drivers/virtio/virtio_balloon.c
> @@ -321,7 +321,7 @@ static unsigned int update_balloon_stats(struct virtio_balloon *vb)
> unsigned long events[NR_VM_EVENT_ITEMS];
> struct sysinfo i;
> unsigned int idx = 0;
> - long available;
> + long available, stall = 0;
> unsigned long caches;
>
> all_vm_events(events);
> @@ -355,6 +355,23 @@ static unsigned int update_balloon_stats(struct virtio_balloon *vb)
> update_stat(vb, idx++, VIRTIO_BALLOON_S_OOM_KILL,
> events[OOM_KILL]);
>
> + /* sum all the stall event */
> +#ifdef CONFIG_ZONE_DMA
> + stall += events[ALLOCSTALL_DMA];
> +#endif
> +#ifdef CONFIG_ZONE_DMA32
> + stall += events[ALLOCSTALL_DMA32];
> +#endif
> +#ifdef CONFIG_HIGHMEM
> + stall += events[ALLOCSTALL_HIGH];
> +#endif
> +#ifdef CONFIG_ZONE_DEVICE
> + stall += events[ALLOCSTALL_DEVICE];
> +#endif
> + stall += events[ALLOCSTALL_NORMAL];
> + stall += events[ALLOCSTALL_MOVABLE];
> + update_stat(vb, idx++, VIRTIO_BALLOON_S_ALLOC_STALL, stall);
> +
> return idx;
> }
>
> diff --git a/include/uapi/linux/virtio_balloon.h b/include/uapi/linux/virtio_balloon.h
> index cde5547e64a7..13d0c32ba27c 100644
> --- a/include/uapi/linux/virtio_balloon.h
> +++ b/include/uapi/linux/virtio_balloon.h
> @@ -72,7 +72,8 @@ struct virtio_balloon_config {
> #define VIRTIO_BALLOON_S_HTLB_PGALLOC 8 /* Hugetlb page allocations */
> #define VIRTIO_BALLOON_S_HTLB_PGFAIL 9 /* Hugetlb page allocation failures */
> #define VIRTIO_BALLOON_S_OOM_KILL 10 /* OOM killer invocations */
> -#define VIRTIO_BALLOON_S_NR 11
> +#define VIRTIO_BALLOON_S_ALLOC_STALL 11 /* Stall count of memory allocatoin */
> +#define VIRTIO_BALLOON_S_NR 12
>
> #define VIRTIO_BALLOON_S_NAMES_WITH_PREFIX(VIRTIO_BALLOON_S_NAMES_prefix) { \
> VIRTIO_BALLOON_S_NAMES_prefix "swap-in", \
> @@ -85,7 +86,8 @@ struct virtio_balloon_config {
> VIRTIO_BALLOON_S_NAMES_prefix "disk-caches", \
> VIRTIO_BALLOON_S_NAMES_prefix "hugetlb-allocations", \
> VIRTIO_BALLOON_S_NAMES_prefix "hugetlb-failures", \
> - VIRTIO_BALLOON_S_NAMES_prefix "oom-kill" \
> + VIRTIO_BALLOON_S_NAMES_prefix "oom-kill", \
> + VIRTIO_BALLOON_S_NAMES_prefix "alloc-stall" \
"alloc-stalls"
--
Cheers,
David / dhildenb
On 15.04.24 10:41, zhenwei pi wrote:
> Expose memory scan/reclaim information to the host side via virtio
> balloon device.
>
> Now we have a metric to analyze the memory performance:
>
> y: counter increases
> n: counter does not changes
> h: the rate of counter change is high
> l: the rate of counter change is low
>
> OOM: VIRTIO_BALLOON_S_OOM_KILL
> STALL: VIRTIO_BALLOON_S_ALLOC_STALL
> ASCAN: VIRTIO_BALLOON_S_SCAN_ASYNC
> DSCAN: VIRTIO_BALLOON_S_SCAN_DIRECT
> ARCLM: VIRTIO_BALLOON_S_RECLAIM_ASYNC
> DRCLM: VIRTIO_BALLOON_S_RECLAIM_DIRECT
>
> - OOM[y], STALL[*], ASCAN[*], DSCAN[*], ARCLM[*], DRCLM[*]:
> the guest runs under really critial memory pressure
>
> - OOM[n], STALL[h], ASCAN[*], DSCAN[l], ARCLM[*], DRCLM[l]:
> the memory allocation stalls due to cgroup, not the global memory
> pressure.
>
> - OOM[n], STALL[h], ASCAN[*], DSCAN[h], ARCLM[*], DRCLM[h]:
> the memory allocation stalls due to global memory pressure. The
> performance gets hurt a lot. A high ratio between DRCLM/DSCAN shows
> quite effective memory reclaiming.
>
> - OOM[n], STALL[h], ASCAN[*], DSCAN[h], ARCLM[*], DRCLM[l]:
> the memory allocation stalls due to global memory pressure.
> the ratio between DRCLM/DSCAN gets low, the guest OS is thrashing
> heavily, the serious case leads poor performance and difficult
> trouble shooting. Ex, sshd may block on memory allocation when
> accepting new connections, a user can't login a VM by ssh command.
>
> - OOM[n], STALL[n], ASCAN[h], DSCAN[n], ARCLM[l], DRCLM[n]:
> the low ratio between ARCLM/ASCAN shows that the guest tries to
> reclaim more memory, but it can't. Once more memory is required in
> future, it will struggle to reclaim memory.
>
> Signed-off-by: zhenwei pi <[email protected]>
> ---
> drivers/virtio/virtio_balloon.c | 9 +++++++++
> include/uapi/linux/virtio_balloon.h | 12 ++++++++++--
> 2 files changed, 19 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/virtio/virtio_balloon.c b/drivers/virtio/virtio_balloon.c
> index 4b9c9569f6e5..7b86514e99d4 100644
> --- a/drivers/virtio/virtio_balloon.c
> +++ b/drivers/virtio/virtio_balloon.c
> @@ -372,6 +372,15 @@ static unsigned int update_balloon_stats(struct virtio_balloon *vb)
> stall += events[ALLOCSTALL_MOVABLE];
> update_stat(vb, idx++, VIRTIO_BALLOON_S_ALLOC_STALL, stall);
>
> + update_stat(vb, idx++, VIRTIO_BALLOON_S_SCAN_ASYNC,
> + pages_to_bytes(events[PGSCAN_KSWAPD]));
> + update_stat(vb, idx++, VIRTIO_BALLOON_S_SCAN_DIRECT,
> + pages_to_bytes(events[PGSCAN_DIRECT]));
> + update_stat(vb, idx++, VIRTIO_BALLOON_S_RECLAIM_ASYNC,
> + pages_to_bytes(events[PGSTEAL_KSWAPD]));
> + update_stat(vb, idx++, VIRTIO_BALLOON_S_RECLAIM_DIRECT,
> + pages_to_bytes(events[PGSTEAL_DIRECT]));
> +
> return idx;
> }
>
> diff --git a/include/uapi/linux/virtio_balloon.h b/include/uapi/linux/virtio_balloon.h
> index 13d0c32ba27c..0875a9cccb01 100644
> --- a/include/uapi/linux/virtio_balloon.h
> +++ b/include/uapi/linux/virtio_balloon.h
> @@ -73,7 +73,11 @@ struct virtio_balloon_config {
> #define VIRTIO_BALLOON_S_HTLB_PGFAIL 9 /* Hugetlb page allocation failures */
> #define VIRTIO_BALLOON_S_OOM_KILL 10 /* OOM killer invocations */
> #define VIRTIO_BALLOON_S_ALLOC_STALL 11 /* Stall count of memory allocatoin */
> -#define VIRTIO_BALLOON_S_NR 12
> +#define VIRTIO_BALLOON_S_SCAN_ASYNC 12 /* Amount of memory scanned asynchronously */
> +#define VIRTIO_BALLOON_S_SCAN_DIRECT 13 /* Amount of memory scanned directly */
> +#define VIRTIO_BALLOON_S_RECLAIM_ASYNC 14 /* Amount of memory reclaimed asynchronously */
> +#define VIRTIO_BALLOON_S_RECLAIM_DIRECT 15 /* Amount of memory reclaimed directly */
Should these be the other way around:
ASYN_SCAN
..
ASYNC_RECLAIM
so we can get ...
> +#define VIRTIO_BALLOON_S_NR 16
>
> #define VIRTIO_BALLOON_S_NAMES_WITH_PREFIX(VIRTIO_BALLOON_S_NAMES_prefix) { \
> VIRTIO_BALLOON_S_NAMES_prefix "swap-in", \
> @@ -87,7 +91,11 @@ struct virtio_balloon_config {
> VIRTIO_BALLOON_S_NAMES_prefix "hugetlb-allocations", \
> VIRTIO_BALLOON_S_NAMES_prefix "hugetlb-failures", \
> VIRTIO_BALLOON_S_NAMES_prefix "oom-kill", \
> - VIRTIO_BALLOON_S_NAMES_prefix "alloc-stall" \
> + VIRTIO_BALLOON_S_NAMES_prefix "alloc-stall", \
> + VIRTIO_BALLOON_S_NAMES_prefix "scan-async", \
> + VIRTIO_BALLOON_S_NAMES_prefix "scan-direct", \
> + VIRTIO_BALLOON_S_NAMES_prefix "reclaim-async", \
> + VIRTIO_BALLOON_S_NAMES_prefix "reclaim-direct" \
..
"async-scans", "async-reclaims" ...
--
Cheers,
David / dhildenb
On 4/15/24 23:01, David Hildenbrand wrote:
> On 15.04.24 10:41, zhenwei pi wrote:
>> Hi,
>>
>> When the guest runs under critial memory pressure, the guest becomss
>> too slow, even sshd turns D state(uninterruptible) on memory
>> allocation. We can't login this VM to do any work on trouble shooting.
>>
>> Guest kernel log via virtual TTY(on host side) only provides a few
>> necessary log after OOM. More detail memory statistics are required,
>> then we can know explicit memory events and estimate the pressure.
>>
>> I'm going to introduce several VM counters for virtio balloon:
>> - oom-kill
>> - alloc-stall
>> - scan-async
>> - scan-direct
>> - reclaim-async
>> - reclaim-direct
>
> IIUC, we're only exposing events that are already getting provided via
> all_vm_events(), correct?
>
Yes, all of these counters come from all_vm_events(). The 'alloc-stall'
is summary of several classes of alloc-stall. please see '[RFC 2/3]
virtio_balloon: introduce memory allocation stall counter'.
> In that case, I don't really see a major issue. Some considerations:
>
> (1) These new events are fairly Linux specific.
>
> PSWPIN and friends are fairly generic, but HGTLB is also already fairly
> Linux specific already. OOM-kills don't really exist on Windows, for
> example. We'll have to be careful of properly describing what the
> semantics are.
>
I also notice FreeBSD supports virtio balloon for a long time, 'OOM
kill' is used on FreeBSD too.(LINK:
https://klarasystems.com/articles/exploring-swap-on-freebsd/)
> (2) How should we handle if Linux ever stops supporting a certain event
> (e.g., major reclaim rework). I assume, simply return nothing like we
> currently would for VIRTIO_BALLOON_S_HTLB_PGALLOC without
> CONFIG_HUGETLB_PAGE.
>
Luckily, virtio balloon stats schema is tag-value style. This way would
be safe enough.
Suggestions in patch [1-3] are good, I'll fix them in the next version
if this series is acceptable.
--
zhenwei pi