The softlockup still occurs in get_swap_pages() under memory pressure.
64 CPU cores, 64GB memory, and 28 zram devices, the disksize of each
zram device is 50MB with same priority as si. Use the stress-ng tool
to increase memory pressure, causing the system to oom frequently.
The plist_for_each_entry_safe() loops in get_swap_pages() could reach
tens of thousands of times to find available space (extreme case:
cond_resched() is not called in scan_swap_map_slots()). Let's add
cond_resched() into get_swap_pages() when failed to find available
space to avoid softlockup.
Signed-off-by: Longlong Xia <[email protected]>
---
mm/swapfile.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/mm/swapfile.c b/mm/swapfile.c
index 908a529bca12..4fa440e87cd6 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -1100,6 +1100,7 @@ int get_swap_pages(int n_goal, swp_entry_t swp_entries[], int entry_size)
goto check_out;
pr_debug("scan_swap_map of si %d failed to find offset\n",
si->type);
+ cond_resched();
spin_lock(&swap_avail_lock);
nextsi:
--
2.25.1
Longlong Xia <[email protected]> writes:
> The softlockup still occurs in get_swap_pages() under memory pressure.
> 64 CPU cores, 64GB memory, and 28 zram devices, the disksize of each
> zram device is 50MB with same priority as si. Use the stress-ng tool
> to increase memory pressure, causing the system to oom frequently.
>
> The plist_for_each_entry_safe() loops in get_swap_pages() could reach
> tens of thousands of times to find available space (extreme case:
> cond_resched() is not called in scan_swap_map_slots()). Let's add
> cond_resched() into get_swap_pages() when failed to find available
> space to avoid softlockup.
>
> Signed-off-by: Longlong Xia <[email protected]>
Looks good to me. Thanks!
Reviewed-by: "Huang, Ying" <[email protected]>
> ---
> mm/swapfile.c | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/mm/swapfile.c b/mm/swapfile.c
> index 908a529bca12..4fa440e87cd6 100644
> --- a/mm/swapfile.c
> +++ b/mm/swapfile.c
> @@ -1100,6 +1100,7 @@ int get_swap_pages(int n_goal, swp_entry_t swp_entries[], int entry_size)
> goto check_out;
> pr_debug("scan_swap_map of si %d failed to find offset\n",
> si->type);
> + cond_resched();
>
> spin_lock(&swap_avail_lock);
> nextsi:
On Sat, 28 Jan 2023 09:47:57 +0000 Longlong Xia <[email protected]> wrote:
> The softlockup still occurs in get_swap_pages() under memory pressure.
> 64 CPU cores, 64GB memory, and 28 zram devices, the disksize of each
> zram device is 50MB with same priority as si. Use the stress-ng tool
> to increase memory pressure, causing the system to oom frequently.
>
> The plist_for_each_entry_safe() loops in get_swap_pages() could reach
> tens of thousands of times to find available space (extreme case:
> cond_resched() is not called in scan_swap_map_slots()). Let's add
> cond_resched() into get_swap_pages() when failed to find available
> space to avoid softlockup.
>
> ...
>
> --- a/mm/swapfile.c
> +++ b/mm/swapfile.c
> @@ -1100,6 +1100,7 @@ int get_swap_pages(int n_goal, swp_entry_t swp_entries[], int entry_size)
> goto check_out;
> pr_debug("scan_swap_map of si %d failed to find offset\n",
> si->type);
> + cond_resched();
>
> spin_lock(&swap_avail_lock);
> nextsi:
This must be pretty rare? My googling for "scan_swap_map of si %d
failed to find offset" turns up zero reports, but I guess few people
enable pr_debug.
I wonder if we should remove that pr_debug(). I mean, it's known that
this happens, what value does the printk add?
I'm thinking this fix should be backported into -stable kernels.
Andrew Morton <[email protected]> writes:
> On Sat, 28 Jan 2023 09:47:57 +0000 Longlong Xia <[email protected]> wrote:
>
>> The softlockup still occurs in get_swap_pages() under memory pressure.
>> 64 CPU cores, 64GB memory, and 28 zram devices, the disksize of each
>> zram device is 50MB with same priority as si. Use the stress-ng tool
>> to increase memory pressure, causing the system to oom frequently.
>>
>> The plist_for_each_entry_safe() loops in get_swap_pages() could reach
>> tens of thousands of times to find available space (extreme case:
>> cond_resched() is not called in scan_swap_map_slots()). Let's add
>> cond_resched() into get_swap_pages() when failed to find available
>> space to avoid softlockup.
>>
>> ...
>>
>> --- a/mm/swapfile.c
>> +++ b/mm/swapfile.c
>> @@ -1100,6 +1100,7 @@ int get_swap_pages(int n_goal, swp_entry_t swp_entries[], int entry_size)
>> goto check_out;
>> pr_debug("scan_swap_map of si %d failed to find offset\n",
>> si->type);
>> + cond_resched();
>>
>> spin_lock(&swap_avail_lock);
>> nextsi:
>
> This must be pretty rare? My googling for "scan_swap_map of si %d
> failed to find offset" turns up zero reports, but I guess few people
> enable pr_debug.
>
> I wonder if we should remove that pr_debug(). I mean, it's known that
> this happens, what value does the printk add?
Sounds reasonable to me. And if we want to debug, we can use bpf too.
> I'm thinking this fix should be backported into -stable kernels.
Best Regards,
Huang, Ying
It's known that get_swap_pages() may fail to find available space
under some extreme case, but pr_debug() provides useless information.
Let's remove it.
Signed-off-by: Longlong Xia <[email protected]>
---
mm/swapfile.c | 2 --
1 file changed, 2 deletions(-)
diff --git a/mm/swapfile.c b/mm/swapfile.c
index 6202a6668a63..99143875d6f0 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -1098,8 +1098,6 @@ int get_swap_pages(int n_goal, swp_entry_t swp_entries[], int entry_size)
spin_unlock(&si->lock);
if (n_ret || size == SWAPFILE_CLUSTER)
goto check_out;
- pr_debug("scan_swap_map of si %d failed to find offset\n",
- si->type);
cond_resched();
spin_lock(&swap_avail_lock);
--
2.25.1
Longlong Xia <[email protected]> writes:
> It's known that get_swap_pages() may fail to find available space
> under some extreme case, but pr_debug() provides useless information.
> Let's remove it.
>
> Signed-off-by: Longlong Xia <[email protected]>
Thanks!
Reviewed-by: "Huang, Ying" <[email protected]>
> ---
> mm/swapfile.c | 2 --
> 1 file changed, 2 deletions(-)
>
> diff --git a/mm/swapfile.c b/mm/swapfile.c
> index 6202a6668a63..99143875d6f0 100644
> --- a/mm/swapfile.c
> +++ b/mm/swapfile.c
> @@ -1098,8 +1098,6 @@ int get_swap_pages(int n_goal, swp_entry_t swp_entries[], int entry_size)
> spin_unlock(&si->lock);
> if (n_ret || size == SWAPFILE_CLUSTER)
> goto check_out;
> - pr_debug("scan_swap_map of si %d failed to find offset\n",
> - si->type);
> cond_resched();
>
> spin_lock(&swap_avail_lock);