Reverting the patchset "Allocate memmap from hotadded memory (per device)" [1] from today's linux-next fixed a crash while online/offline memory sections.
[1] https://lore.kernel.org/linux-mm/[email protected]/
[ 162.327720][ T1694] kernel BUG at include/linux/mm.h:1383!
[ 162.333695][ T1694] Internal error: Oops - BUG: 0 [#1] SMP
[ 162.339181][ T1694] Modules linked in: loop processor efivarfs ip_tables x_tables ext4 mbcache jbd2 dm_mod igb nvme i2c_algo_bit mlx5_core i2c_core nvme_core firmware_class
[ 162.354604][ T1694] CPU: 13 PID: 1694 Comm: ranbug Not tainted 5.12.0-next-20210524+ #4
[ 162.362601][ T1694] Hardware name: MiTAC RAPTOR EV-883832-X3-0001/RAPTOR, BIOS 1.6 06/28/2020
[ 162.371116][ T1694] pstate: 60000005 (nZCv daif -PAN -UAO -TCO BTYPE=--)
[ 162.377811][ T1694] pc : memory_subsys_offline+0x1f8/0x250
[ 162.383295][ T1694] lr : memory_subsys_offline+0x1f8/0x250
[ 162.388773][ T1694] sp : ffff80002458f8e0
[ 162.392773][ T1694] x29: ffff80002458f8e0 x28: ffff800010914d30 x27: 0000000000000000
[ 162.400602][ T1694] x26: 0000000000002000 x25: 1fffe00002550401 x24: ffff000012a82008
[ 162.408431][ T1694] x23: fffffc0000000000 x22: 0000000000008000 x21: 0000000000000001
[ 162.416259][ T1694] x20: ffffffffffffffff x19: ffff000012a82018 x18: ffff0008527b6a70
[ 162.424086][ T1694] x17: 0000000000000000 x16: 0000000000000007 x15: 00000000000000c8
[ 162.431914][ T1694] x14: 0000000000000000 x13: ffff800011c6eea4 x12: ffff60136ceb8574
[ 162.439742][ T1694] x11: 1fffe0136ceb8573 x10: ffff60136ceb8573 x9 : dfff800000000000
[ 162.447570][ T1694] x8 : ffff009b675c2b9b x7 : 0000000000000001 x6 : ffff009b675c2b98
[ 162.455398][ T1694] x5 : 00009fec93147a8d x4 : ffff009b675c2b98 x3 : 1fffe0010a4f6c09
[ 162.463226][ T1694] x2 : 0000000000000000 x1 : 0000000000000000 x0 : 0000000000000034
[ 162.471054][ T1694] Call trace:
[ 162.474186][ T1694] memory_subsys_offline+0x1f8/0x250
[ 162.479318][ T1694] device_offline+0x154/0x1d8
[ 162.483844][ T1694] online_store+0xa4/0x118
[ 162.488107][ T1694] dev_attr_store+0x44/0x78
[ 162.492457][ T1694] sysfs_kf_write+0xe8/0x138
[ 162.496896][ T1694] kernfs_fop_write_iter+0x26c/0x3d0
[ 162.502028][ T1694] new_sync_write+0x2bc/0x4f8
[ 162.506552][ T1694] vfs_write+0x718/0xc88
[ 162.510643][ T1694] ksys_write+0xf8/0x1e0
[ 162.514732][ T1694] __arm64_sys_write+0x74/0xa8
[ 162.519342][ T1694] invoke_syscall.constprop.0+0x78/0x1e8
[ 162.524824][ T1694] do_el0_svc+0xe4/0x298
[ 162.528914][ T1694] el0_svc+0x20/0x30
[ 162.532658][ T1694] el0_sync_handler+0xb0/0xb8
[ 162.537181][ T1694] el0_sync+0x178/0x180
[ 162.541187][ T1694] Code: f00033e1 91318021 91090021 97e38d8b (d4210000)
[ 162.547968][ T1694] ---[ end trace 2a1964462a219f20 ]---
[ 162.553273][ T1694] Kernel panic - not syncing: Oops - BUG: Fatal exception
[ 162.560250][ T1694] SMP: stopping secondary CPUs
[ 162.564871][ T1694] Kernel Offset: disabled
[ 162.569045][ T1694] CPU features: 0x00000251,20000846
[ 162.574089][ T1694] Memory Limit: none
[ 162.577849][ T1694] ---[ end Kernel panic - not syncing: Oops - BUG: Fatal exception ]---
Occasionally, it failed like this,
[ 119.319732][ T1387] Unable to handle kernel paging request at virtual address dfff800000000000
[ 119.328575][ T1387] Mem abort info:
[ 119.332073][ T1387] ESR = 0x96000004
[ 119.335879][ T1387] EC = 0x25: DABT (current EL), IL = 32 bits
[ 119.341895][ T1387] SET = 0, FnV = 0
[ 119.345697][ T1387] EA = 0, S1PTW = 0
[ 119.349542][ T1387] Data abort info:
[ 119.353121][ T1387] ISV = 0, ISS = 0x00000004
[ 119.358387][ T1387] CM = 0, WnR = 0
[ 119.362055][ T1387] [dfff800000000000] address between user and kernel address ranges
[ 119.370166][ T1387] Internal error: Oops: 96000004 [#1] SMP
[ 119.375739][ T1387] Modules linked in: loop processor efivarfs ip_tables x_tables ext4 mbcache jbd2 dm_mod igb i2c_algo_bit nvme mlx5_core i2c_core nvme_core firmware_class
[ 119.391164][ T1387] CPU: 2 PID: 1387 Comm: ranbug Not tainted 5.13.0-rc3+ #16
[ 119.398294][ T1387] Hardware name: MiTAC RAPTOR EV-883832-X3-0001/RAPTOR, BIOS 1.6 06/28/2020
[ 119.406810][ T1387] pstate: 20000005 (nzCv daif -PAN -UAO -TCO BTYPE=--)
[ 119.413504][ T1387] pc : hex_dump_to_buffer+0x3ac/0x9f0
[ 119.418728][ T1387] lr : print_hex_dump+0x10c/0x1d8
[ 119.423600][ T1387] sp : ffff800021b4f3a0
[ 119.427600][ T1387] x29: ffff800021b4f3a0 x28: dfff800000000000 x27: ffff800021b4f500
[ 119.435429][ T1387] x26: 0000000000000083 x25: fffffffffffffffe x24: 0000000000000008
[ 119.443257][ T1387] x23: 0000000000000045 x22: 0000000000000083 x21: 0000000000000020
[ 119.451084][ T1387] x20: 0000000000000000 x19: ffff800021b4f500 x18: ffff0008650187e8
[ 119.458912][ T1387] x17: 0000000000000000 x16: 0000000000000007 x15: 00000000000000c8
[ 119.466740][ T1387] x14: 0000000000000000 x13: ffff800011c6eea4 x12: 000000000000f1f1
[ 119.474568][ T1387] x11: 0000000000000005 x10: ffff8000112eec80 x9 : 00000000f3f3f3f3
[ 119.482396][ T1387] x8 : 0000000000000010 x7 : 0000000000000000 x6 : fffffffffffffffe
[ 119.490224][ T1387] x5 : 0000000000000083 x4 : 0000000000000000 x3 : 1fffffffffffffff
[ 119.498051][ T1387] x2 : ffff8000112eecc0 x1 : 0000000000000006 x0 : 0000000000000000
[ 119.505879][ T1387] Call trace:
[ 119.509011][ T1387] hex_dump_to_buffer+0x3ac/0x9f0
[ 119.513884][ T1387] print_hex_dump+0x10c/0x1d8
[ 119.518407][ T1387] __dump_page+0x8c8/0xba8
[ 119.522672][ T1387] dump_page+0x20/0x50
[ 119.526587][ T1387] memory_subsys_offline+0x1f8/0x250
[ 119.531722][ T1387] device_offline+0x154/0x1d8
[ 119.536247][ T1387] online_store+0xa4/0x118
[ 119.540511][ T1387] dev_attr_store+0x44/0x78
[ 119.544860][ T1387] sysfs_kf_write+0xe8/0x138
[ 119.549298][ T1387] kernfs_fop_write_iter+0x26c/0x3d0
[ 119.554429][ T1387] new_sync_write+0x2bc/0x4f8
[ 119.558953][ T1387] vfs_write+0x718/0xc88
[ 119.563042][ T1387] ksys_write+0xf8/0x1e0
[ 119.567130][ T1387] __arm64_sys_write+0x74/0xa8
[ 119.571740][ T1387] invoke_syscall.constprop.0+0x78/0x1e8
[ 119.577222][ T1387] do_el0_svc+0xe4/0x298
[ 119.581311][ T1387] el0_svc+0x20/0x30
[ 119.585054][ T1387] el0_sync_handler+0xb0/0xb8
[ 119.589578][ T1387] el0_sync+0x178/0x180
[ 119.593583][ T1387] Code: 1200080b 120008c1 d343fc00 38fc6864 (38fc6803)
[ 119.600364][ T1387] ---[ end trace b03c958bc1accb66 ]---
[ 119.605669][ T1387] Kernel panic - not syncing: Oops: Fatal exception
[ 119.612124][ T1387] SMP: stopping secondary CPUs
[ 119.616747][ T1387] Kernel Offset: disabled
[ 119.620923][ T1387] CPU features: 0x00000251,20000846
[ 119.625968][ T1387] Memory Limit: none
[ 119.629728][ T1387] ---[ end Kernel panic - not syncing: Oops: Fatal exception ]---
> -----Original Message-----
> From: David Hildenbrand <[email protected]>
> Sent: Tuesday, May 25, 2021 11:40 AM
> To: Qian Cai (QUIC) <[email protected]>; Oscar Salvador
> <[email protected]>
> Cc: Andrew Morton <[email protected]>; Anshuman Khandual
> <[email protected]>; Vlastimil Babka <[email protected]>;
> Michal Hocko <[email protected]>; Linux Memory Management List
> <[email protected]>; Linux Kernel Mailing List <linux-
> [email protected]>
> Subject: Re: Arm64 crash while online/offline memory sections
>
> On 25.05.21 17:36, Qian Cai (QUIC) wrote:
> > Reverting the patchset "Allocate memmap from hotadded memory (per
> device)" [1] from today's linux-next fixed a crash while online/offline
> memory sections.
>
> Do we know which patch in particular is problematic?
I don't know yet. It could be messy to dive into a patchset which usually has less clear boundaries, but I'll try to dig a bit more.
> Do we know which patch in particular is problematic?
Okay, the winner is "mm,memory_hotplug: Allocate memmap from the added memory range".
https://lore.kernel.org/linux-mm/[email protected]/
That and the rest of the series are the minimal I have to revert to fix the crash.
On 25.05.21 17:36, Qian Cai (QUIC) wrote:
> Reverting the patchset "Allocate memmap from hotadded memory (per device)" [1] from today's linux-next fixed a crash while online/offline memory sections.
Do we know which patch in particular is problematic?
--
Thanks,
David / dhildenb
On Tue, May 25, 2021 at 05:57:34PM +0000, Qian Cai (QUIC) wrote:
> > Do we know which patch in particular is problematic?
>
> Okay, the winner is "mm,memory_hotplug: Allocate memmap from the added memory range".
>
> https://lore.kernel.org/linux-mm/[email protected]/
Ok, which means that is irrelevant to having it enabled, as the latter
patch of that series actualy enables it for arm64.
Can you work out where exactly the crash happens?
I will have a look into it tomorrow.
Thanks for reporting.
--
Oscar Salvador
SUSE L3
On 25.05.21 20:00, Oscar Salvador wrote:
> On Tue, May 25, 2021 at 05:57:34PM +0000, Qian Cai (QUIC) wrote:
>>> Do we know which patch in particular is problematic?
>>
>> Okay, the winner is "mm,memory_hotplug: Allocate memmap from the added memory range".
>>
>> https://lore.kernel.org/linux-mm/[email protected]/
>
> Ok, which means that is irrelevant to having it enabled, as the latter
> patch of that series actualy enables it for arm64.
> Can you work out where exactly the crash happens?
>
> I will have a look into it tomorrow.
>
> Thanks for reporting.
>
I assume the following will work:
diff --git a/drivers/base/memory.c b/drivers/base/memory.c
index b31b3af5c490..6e661d106e96 100644
--- a/drivers/base/memory.c
+++ b/drivers/base/memory.c
@@ -218,14 +218,15 @@ static int memory_block_offline(struct memory_block *mem)
struct zone *zone;
int ret;
- zone = page_zone(pfn_to_page(start_pfn));
-
/*
* Unaccount before offlining, such that unpopulated zone and kthreads
* can properly be torn down in offline_pages().
*/
- if (nr_vmemmap_pages)
+ if (nr_vmemmap_pages) {
+ /* Hotplugged memory has no holes. */
+ zone = page_zone(pfn_to_page(start_pfn));
adjust_present_page_count(zone, -nr_vmemmap_pages);
+ }
ret = offline_pages(start_pfn + nr_vmemmap_pages,
nr_pages - nr_vmemmap_pages);
We must not touch pfn_to_page(start_pfn) if it might be a memory hole.
offline_pages() will make sure there are no holes, but that's too late.
--
Thanks,
David / dhildenb
> -----Original Message-----
> From: David Hildenbrand <[email protected]>
> Sent: Tuesday, May 25, 2021 2:12 PM
> I assume the following will work:
>
> diff --git a/drivers/base/memory.c b/drivers/base/memory.c
> index b31b3af5c490..6e661d106e96 100644
> --- a/drivers/base/memory.c
> +++ b/drivers/base/memory.c
> @@ -218,14 +218,15 @@ static int memory_block_offline(struct memory_block *mem)
> struct zone *zone;
> int ret;
>
> - zone = page_zone(pfn_to_page(start_pfn));
> -
> /*
> * Unaccount before offlining, such that unpopulated zone and kthreads
> * can properly be torn down in offline_pages().
> */
> - if (nr_vmemmap_pages)
> + if (nr_vmemmap_pages) {
> + /* Hotplugged memory has no holes. */
> + zone = page_zone(pfn_to_page(start_pfn));
> adjust_present_page_count(zone, -nr_vmemmap_pages);
> + }
>
> ret = offline_pages(start_pfn + nr_vmemmap_pages,
> nr_pages - nr_vmemmap_pages);
>
>
> We must not touch pfn_to_page(start_pfn) if it might be a memory hole.
> offline_pages() will make sure there are no holes, but that's too late.
Good catch, David. This patch works well.
On Tue, May 25, 2021 at 08:12:22PM +0200, David Hildenbrand wrote:
> diff --git a/drivers/base/memory.c b/drivers/base/memory.c
> index b31b3af5c490..6e661d106e96 100644
> --- a/drivers/base/memory.c
> +++ b/drivers/base/memory.c
> @@ -218,14 +218,15 @@ static int memory_block_offline(struct memory_block *mem)
> struct zone *zone;
> int ret;
> - zone = page_zone(pfn_to_page(start_pfn));
> -
> /*
> * Unaccount before offlining, such that unpopulated zone and kthreads
> * can properly be torn down in offline_pages().
> */
> - if (nr_vmemmap_pages)
> + if (nr_vmemmap_pages) {
> + /* Hotplugged memory has no holes. */
> + zone = page_zone(pfn_to_page(start_pfn));
> adjust_present_page_count(zone, -nr_vmemmap_pages);
> + }
> ret = offline_pages(start_pfn + nr_vmemmap_pages,
> nr_pages - nr_vmemmap_pages);
>
>
> We must not touch pfn_to_page(start_pfn) if it might be a memory hole.
> offline_pages() will make sure there are no holes, but that's too late.
Yeah, definitely. I somehow didn't think about holes when writing that.
Thanks for catching it David. Would you be so kind to send a patch?
Thanks!
--
Oscar Salvador
SUSE L3