It's possible that early_init_fdt_scan_reserved_mem() allocates memory
from memblock for dynamic reserved memory in `/reserved-memory` node.
Any fixed reservation must be done before that to avoid potential
conflicts.
Reserve the DTB in memblock just after early scanning it.
Signed-off-by: Woody Zhang <[email protected]>
---
arch/riscv/kernel/setup.c | 10 ++++++++++
arch/riscv/mm/init.c | 9 ---------
2 files changed, 10 insertions(+), 9 deletions(-)
diff --git a/arch/riscv/kernel/setup.c b/arch/riscv/kernel/setup.c
index 36b026057503..c147fa8da929 100644
--- a/arch/riscv/kernel/setup.c
+++ b/arch/riscv/kernel/setup.c
@@ -16,6 +16,7 @@
#include <linux/console.h>
#include <linux/screen_info.h>
#include <linux/of_fdt.h>
+#include <linux/libfdt.h>
#include <linux/sched/task.h>
#include <linux/smp.h>
#include <linux/efi.h>
@@ -256,6 +257,15 @@ static void __init parse_dtb(void)
pr_err("No DTB passed to the kernel\n");
}
+ /*
+ * If DTB is built in, no need to reserve its memblock.
+ * Otherwise, do reserve it but avoid using
+ * early_init_fdt_reserve_self() since __pa() does
+ * not work for DTB pointers that are fixmap addresses
+ */
+ if (!IS_ENABLED(CONFIG_BUILTIN_DTB))
+ memblock_reserve(dtb_early_pa, fdt_totalsize(dtb_early_va));
+
#ifdef CONFIG_CMDLINE_FORCE
strscpy(boot_command_line, CONFIG_CMDLINE, COMMAND_LINE_SIZE);
pr_info("Forcing kernel command line to: %s\n", boot_command_line);
diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c
index c6bb966e4123..f8c9a79acd94 100644
--- a/arch/riscv/mm/init.c
+++ b/arch/riscv/mm/init.c
@@ -254,15 +254,6 @@ static void __init setup_bootmem(void)
*/
early_init_fdt_scan_reserved_mem();
- /*
- * If DTB is built in, no need to reserve its memblock.
- * Otherwise, do reserve it but avoid using
- * early_init_fdt_reserve_self() since __pa() does
- * not work for DTB pointers that are fixmap addresses
- */
- if (!IS_ENABLED(CONFIG_BUILTIN_DTB))
- memblock_reserve(dtb_early_pa, fdt_totalsize(dtb_early_va));
-
dma_contiguous_reserve(dma32_phys_limit);
if (IS_ENABLED(CONFIG_64BIT))
hugetlb_cma_reserve(PUD_SHIFT - PAGE_SHIFT);
--
2.39.2
+CC Alex, you should take a look at this patch.
On Wed, Jun 07, 2023 at 09:35:19PM +0800, Woody Zhang wrote:
> It's possible that early_init_fdt_scan_reserved_mem() allocates memory
> from memblock for dynamic reserved memory in `/reserved-memory` node.
> Any fixed reservation must be done before that to avoid potential
> conflicts.
>
> Reserve the DTB in memblock just after early scanning it.
The rationale makes sense to me, I am just wondering what compelling
reason there is to move it away from the memblock_reserve()s for the
initd and vmlinux? Moving it above early_init_fdt_scan_reserved_mem()
should be the sufficient minimum & would keep things together.
Cheers,
Conor.
>
> Signed-off-by: Woody Zhang <[email protected]>
> ---
> arch/riscv/kernel/setup.c | 10 ++++++++++
> arch/riscv/mm/init.c | 9 ---------
> 2 files changed, 10 insertions(+), 9 deletions(-)
>
> diff --git a/arch/riscv/kernel/setup.c b/arch/riscv/kernel/setup.c
> index 36b026057503..c147fa8da929 100644
> --- a/arch/riscv/kernel/setup.c
> +++ b/arch/riscv/kernel/setup.c
> @@ -16,6 +16,7 @@
> #include <linux/console.h>
> #include <linux/screen_info.h>
> #include <linux/of_fdt.h>
> +#include <linux/libfdt.h>
> #include <linux/sched/task.h>
> #include <linux/smp.h>
> #include <linux/efi.h>
> @@ -256,6 +257,15 @@ static void __init parse_dtb(void)
> pr_err("No DTB passed to the kernel\n");
> }
>
> + /*
> + * If DTB is built in, no need to reserve its memblock.
> + * Otherwise, do reserve it but avoid using
> + * early_init_fdt_reserve_self() since __pa() does
> + * not work for DTB pointers that are fixmap addresses
> + */
> + if (!IS_ENABLED(CONFIG_BUILTIN_DTB))
> + memblock_reserve(dtb_early_pa, fdt_totalsize(dtb_early_va));
> +
> #ifdef CONFIG_CMDLINE_FORCE
> strscpy(boot_command_line, CONFIG_CMDLINE, COMMAND_LINE_SIZE);
> pr_info("Forcing kernel command line to: %s\n", boot_command_line);
> diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c
> index c6bb966e4123..f8c9a79acd94 100644
> --- a/arch/riscv/mm/init.c
> +++ b/arch/riscv/mm/init.c
> @@ -254,15 +254,6 @@ static void __init setup_bootmem(void)
> */
> early_init_fdt_scan_reserved_mem();
>
> - /*
> - * If DTB is built in, no need to reserve its memblock.
> - * Otherwise, do reserve it but avoid using
> - * early_init_fdt_reserve_self() since __pa() does
> - * not work for DTB pointers that are fixmap addresses
> - */
> - if (!IS_ENABLED(CONFIG_BUILTIN_DTB))
> - memblock_reserve(dtb_early_pa, fdt_totalsize(dtb_early_va));
> -
> dma_contiguous_reserve(dma32_phys_limit);
> if (IS_ENABLED(CONFIG_64BIT))
> hugetlb_cma_reserve(PUD_SHIFT - PAGE_SHIFT);
> --
> 2.39.2
>
On Thu, Jun 08, 2023 at 06:17:22AM +0800, Woody Zhang wrote:
> Hi, Conor
>
> On Wed, Jun 07, 2023 at 07:17:28PM +0100, Conor Dooley wrote:
> >+CC Alex, you should take a look at this patch.
> >
> >On Wed, Jun 07, 2023 at 09:35:19PM +0800, Woody Zhang wrote:
> >> It's possible that early_init_fdt_scan_reserved_mem() allocates memory
> >> from memblock for dynamic reserved memory in `/reserved-memory` node.
> >> Any fixed reservation must be done before that to avoid potential
> >> conflicts.
> >>
> >> Reserve the DTB in memblock just after early scanning it.
> >
> >The rationale makes sense to me, I am just wondering what compelling
> >reason there is to move it away from the memblock_reserve()s for the
> >initd and vmlinux? Moving it above early_init_fdt_scan_reserved_mem()
> >should be the sufficient minimum & would keep things together.
>
> IMO, moving it to parse_dtb() is more reasonable as early scanning and
> reservation are both subject to DTB. It can also lower the risk to
> mess up the sequence in the future. BTW, it's also invoked in
> setup_machine_fdt() in arm64.
I'm fine with the change either way, so:
Reviewed-by: Conor Dooley <[email protected]>
Mostly wanted to know whether you'd considered the minimal change.
Cheers,
Conor.
Hi, Conor
On Wed, Jun 07, 2023 at 07:17:28PM +0100, Conor Dooley wrote:
>+CC Alex, you should take a look at this patch.
>
>On Wed, Jun 07, 2023 at 09:35:19PM +0800, Woody Zhang wrote:
>> It's possible that early_init_fdt_scan_reserved_mem() allocates memory
>> from memblock for dynamic reserved memory in `/reserved-memory` node.
>> Any fixed reservation must be done before that to avoid potential
>> conflicts.
>>
>> Reserve the DTB in memblock just after early scanning it.
>
>The rationale makes sense to me, I am just wondering what compelling
>reason there is to move it away from the memblock_reserve()s for the
>initd and vmlinux? Moving it above early_init_fdt_scan_reserved_mem()
>should be the sufficient minimum & would keep things together.
IMO, moving it to parse_dtb() is more reasonable as early scanning and
reservation are both subject to DTB. It can also lower the risk to
mess up the sequence in the future. BTW, it's also invoked in
setup_machine_fdt() in arm64.
Thanks,
Woody
>
>Cheers,
>Conor.
>
>>
>> Signed-off-by: Woody Zhang <[email protected]>
>> ---
>> arch/riscv/kernel/setup.c | 10 ++++++++++
>> arch/riscv/mm/init.c | 9 ---------
>> 2 files changed, 10 insertions(+), 9 deletions(-)
>>
>> diff --git a/arch/riscv/kernel/setup.c b/arch/riscv/kernel/setup.c
>> index 36b026057503..c147fa8da929 100644
>> --- a/arch/riscv/kernel/setup.c
>> +++ b/arch/riscv/kernel/setup.c
>> @@ -16,6 +16,7 @@
>> #include <linux/console.h>
>> #include <linux/screen_info.h>
>> #include <linux/of_fdt.h>
>> +#include <linux/libfdt.h>
>> #include <linux/sched/task.h>
>> #include <linux/smp.h>
>> #include <linux/efi.h>
>> @@ -256,6 +257,15 @@ static void __init parse_dtb(void)
>> pr_err("No DTB passed to the kernel\n");
>> }
>>
>> + /*
>> + * If DTB is built in, no need to reserve its memblock.
>> + * Otherwise, do reserve it but avoid using
>> + * early_init_fdt_reserve_self() since __pa() does
>> + * not work for DTB pointers that are fixmap addresses
>> + */
>> + if (!IS_ENABLED(CONFIG_BUILTIN_DTB))
>> + memblock_reserve(dtb_early_pa, fdt_totalsize(dtb_early_va));
>> +
>> #ifdef CONFIG_CMDLINE_FORCE
>> strscpy(boot_command_line, CONFIG_CMDLINE, COMMAND_LINE_SIZE);
>> pr_info("Forcing kernel command line to: %s\n", boot_command_line);
>> diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c
>> index c6bb966e4123..f8c9a79acd94 100644
>> --- a/arch/riscv/mm/init.c
>> +++ b/arch/riscv/mm/init.c
>> @@ -254,15 +254,6 @@ static void __init setup_bootmem(void)
>> */
>> early_init_fdt_scan_reserved_mem();
>>
>> - /*
>> - * If DTB is built in, no need to reserve its memblock.
>> - * Otherwise, do reserve it but avoid using
>> - * early_init_fdt_reserve_self() since __pa() does
>> - * not work for DTB pointers that are fixmap addresses
>> - */
>> - if (!IS_ENABLED(CONFIG_BUILTIN_DTB))
>> - memblock_reserve(dtb_early_pa, fdt_totalsize(dtb_early_va));
>> -
>> dma_contiguous_reserve(dma32_phys_limit);
>> if (IS_ENABLED(CONFIG_64BIT))
>> hugetlb_cma_reserve(PUD_SHIFT - PAGE_SHIFT);
>> --
>> 2.39.2
>>
On Wed, Jun 7, 2023 at 8:17 PM Conor Dooley <[email protected]> wrote:
>
> +CC Alex, you should take a look at this patch.
>
> On Wed, Jun 07, 2023 at 09:35:19PM +0800, Woody Zhang wrote:
> > It's possible that early_init_fdt_scan_reserved_mem() allocates memory
> > from memblock for dynamic reserved memory in `/reserved-memory` node.
> > Any fixed reservation must be done before that to avoid potential
> > conflicts.
> >
> > Reserve the DTB in memblock just after early scanning it.
>
> The rationale makes sense to me, I am just wondering what compelling
> reason there is to move it away from the memblock_reserve()s for the
> initd and vmlinux? Moving it above early_init_fdt_scan_reserved_mem()
> should be the sufficient minimum & would keep things together.
>
> Cheers,
> Conor.
Thanks Conor.
So the patch looks good to me.
But I find this fragile:
- we do not check memblock_reserve() return value to make sure the
reservation really happened (and quickly looking at the code, I'm not
even sure it returns an error if the region was already allocated).
- we have to make sure no memblock allocation happens before setup_bootmem().
- we also have to check that no fixed memblock_reserve() happens after.
The last 2 points may sound natural, but we'll have to take great care
when adding some code around here. I'm working on an "early boot
document" and I'll add something about that, but a runtime thing would
be way better IMO.
You can add:
Reviewed-by: Alexandre Ghiti <[email protected]>
Thanks,
Alex
>
> >
> > Signed-off-by: Woody Zhang <[email protected]>
> > ---
> > arch/riscv/kernel/setup.c | 10 ++++++++++
> > arch/riscv/mm/init.c | 9 ---------
> > 2 files changed, 10 insertions(+), 9 deletions(-)
> >
> > diff --git a/arch/riscv/kernel/setup.c b/arch/riscv/kernel/setup.c
> > index 36b026057503..c147fa8da929 100644
> > --- a/arch/riscv/kernel/setup.c
> > +++ b/arch/riscv/kernel/setup.c
> > @@ -16,6 +16,7 @@
> > #include <linux/console.h>
> > #include <linux/screen_info.h>
> > #include <linux/of_fdt.h>
> > +#include <linux/libfdt.h>
> > #include <linux/sched/task.h>
> > #include <linux/smp.h>
> > #include <linux/efi.h>
> > @@ -256,6 +257,15 @@ static void __init parse_dtb(void)
> > pr_err("No DTB passed to the kernel\n");
> > }
> >
> > + /*
> > + * If DTB is built in, no need to reserve its memblock.
> > + * Otherwise, do reserve it but avoid using
> > + * early_init_fdt_reserve_self() since __pa() does
> > + * not work for DTB pointers that are fixmap addresses
> > + */
> > + if (!IS_ENABLED(CONFIG_BUILTIN_DTB))
> > + memblock_reserve(dtb_early_pa, fdt_totalsize(dtb_early_va));
> > +
> > #ifdef CONFIG_CMDLINE_FORCE
> > strscpy(boot_command_line, CONFIG_CMDLINE, COMMAND_LINE_SIZE);
> > pr_info("Forcing kernel command line to: %s\n", boot_command_line);
> > diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c
> > index c6bb966e4123..f8c9a79acd94 100644
> > --- a/arch/riscv/mm/init.c
> > +++ b/arch/riscv/mm/init.c
> > @@ -254,15 +254,6 @@ static void __init setup_bootmem(void)
> > */
> > early_init_fdt_scan_reserved_mem();
> >
> > - /*
> > - * If DTB is built in, no need to reserve its memblock.
> > - * Otherwise, do reserve it but avoid using
> > - * early_init_fdt_reserve_self() since __pa() does
> > - * not work for DTB pointers that are fixmap addresses
> > - */
> > - if (!IS_ENABLED(CONFIG_BUILTIN_DTB))
> > - memblock_reserve(dtb_early_pa, fdt_totalsize(dtb_early_va));
> > -
> > dma_contiguous_reserve(dma32_phys_limit);
> > if (IS_ENABLED(CONFIG_64BIT))
> > hugetlb_cma_reserve(PUD_SHIFT - PAGE_SHIFT);
> > --
> > 2.39.2
> >
On Thu, Jun 08, 2023 at 09:49:44AM +0200, Alexandre Ghiti wrote:
> On Wed, Jun 7, 2023 at 8:17 PM Conor Dooley <[email protected]> wrote:
> >
> > +CC Alex, you should take a look at this patch.
> >
> > On Wed, Jun 07, 2023 at 09:35:19PM +0800, Woody Zhang wrote:
> > > It's possible that early_init_fdt_scan_reserved_mem() allocates memory
> > > from memblock for dynamic reserved memory in `/reserved-memory` node.
> > > Any fixed reservation must be done before that to avoid potential
> > > conflicts.
> > >
> > > Reserve the DTB in memblock just after early scanning it.
> >
> > The rationale makes sense to me, I am just wondering what compelling
> > reason there is to move it away from the memblock_reserve()s for the
> > initd and vmlinux? Moving it above early_init_fdt_scan_reserved_mem()
> > should be the sufficient minimum & would keep things together.
> Thanks Conor.
>
> So the patch looks good to me.
>
> But I find this fragile:
>
> - we do not check memblock_reserve() return value to make sure the
> reservation really happened (and quickly looking at the code, I'm not
> even sure it returns an error if the region was already allocated).
> - we have to make sure no memblock allocation happens before setup_bootmem().
> - we also have to check that no fixed memblock_reserve() happens after.
>
> The last 2 points may sound natural, but we'll have to take great care
> when adding some code around here. I'm working on an "early boot
> document" and I'll add something about that, but a runtime thing would
> be way better IMO.
>
> You can add:
>
> Reviewed-by: Alexandre Ghiti <[email protected]>
btw Alex/Woody, what is the appropriate Fixes tag here?
Cheers,
Conor.
On Sat, Jun 10, 2023 at 04:41:17PM +0100, Conor Dooley wrote:
>On Thu, Jun 08, 2023 at 09:49:44AM +0200, Alexandre Ghiti wrote:
>> On Wed, Jun 7, 2023 at 8:17 PM Conor Dooley <[email protected]> wrote:
>> >
>> > +CC Alex, you should take a look at this patch.
>> >
>> > On Wed, Jun 07, 2023 at 09:35:19PM +0800, Woody Zhang wrote:
>> > > It's possible that early_init_fdt_scan_reserved_mem() allocates memory
>> > > from memblock for dynamic reserved memory in `/reserved-memory` node.
>> > > Any fixed reservation must be done before that to avoid potential
>> > > conflicts.
>> > >
>> > > Reserve the DTB in memblock just after early scanning it.
>> >
>> > The rationale makes sense to me, I am just wondering what compelling
>> > reason there is to move it away from the memblock_reserve()s for the
>> > initd and vmlinux? Moving it above early_init_fdt_scan_reserved_mem()
>> > should be the sufficient minimum & would keep things together.
>
>> Thanks Conor.
>>
>> So the patch looks good to me.
>>
>> But I find this fragile:
>>
>> - we do not check memblock_reserve() return value to make sure the
>> reservation really happened (and quickly looking at the code, I'm not
>> even sure it returns an error if the region was already allocated).
>> - we have to make sure no memblock allocation happens before setup_bootmem().
>> - we also have to check that no fixed memblock_reserve() happens after.
>>
>> The last 2 points may sound natural, but we'll have to take great care
>> when adding some code around here. I'm working on an "early boot
>> document" and I'll add something about that, but a runtime thing would
>> be way better IMO.
>>
>> You can add:
>>
>> Reviewed-by: Alexandre Ghiti <[email protected]>
>
>btw Alex/Woody, what is the appropriate Fixes tag here?
In ef69d2559fe9 ("riscv: Move early dtb mapping into the fixmap region"),
Alex move early_init_fdt_scan_reserved_mem to setup_bootmem() to prevent
memory allocations before of reservations. But it should not be put before
DTB reservation.
Woody
On Sun, Jun 11, 2023 at 1:27 AM Woody Zhang <[email protected]> wrote:
>
> On Sat, Jun 10, 2023 at 04:41:17PM +0100, Conor Dooley wrote:
> >On Thu, Jun 08, 2023 at 09:49:44AM +0200, Alexandre Ghiti wrote:
> >> On Wed, Jun 7, 2023 at 8:17 PM Conor Dooley <[email protected]> wrote:
> >> >
> >> > +CC Alex, you should take a look at this patch.
> >> >
> >> > On Wed, Jun 07, 2023 at 09:35:19PM +0800, Woody Zhang wrote:
> >> > > It's possible that early_init_fdt_scan_reserved_mem() allocates memory
> >> > > from memblock for dynamic reserved memory in `/reserved-memory` node.
> >> > > Any fixed reservation must be done before that to avoid potential
> >> > > conflicts.
> >> > >
> >> > > Reserve the DTB in memblock just after early scanning it.
> >> >
> >> > The rationale makes sense to me, I am just wondering what compelling
> >> > reason there is to move it away from the memblock_reserve()s for the
> >> > initd and vmlinux? Moving it above early_init_fdt_scan_reserved_mem()
> >> > should be the sufficient minimum & would keep things together.
> >
> >> Thanks Conor.
> >>
> >> So the patch looks good to me.
> >>
> >> But I find this fragile:
> >>
> >> - we do not check memblock_reserve() return value to make sure the
> >> reservation really happened (and quickly looking at the code, I'm not
> >> even sure it returns an error if the region was already allocated).
> >> - we have to make sure no memblock allocation happens before setup_bootmem().
> >> - we also have to check that no fixed memblock_reserve() happens after.
> >>
> >> The last 2 points may sound natural, but we'll have to take great care
> >> when adding some code around here. I'm working on an "early boot
> >> document" and I'll add something about that, but a runtime thing would
> >> be way better IMO.
> >>
> >> You can add:
> >>
> >> Reviewed-by: Alexandre Ghiti <[email protected]>
> >
> >btw Alex/Woody, what is the appropriate Fixes tag here?
>
> In ef69d2559fe9 ("riscv: Move early dtb mapping into the fixmap region"),
> Alex move early_init_fdt_scan_reserved_mem to setup_bootmem() to prevent
> memory allocations before of reservations. But it should not be put before
> DTB reservation.
>
Yep, that's the culprit, let's add the proper tag:
Fixes: ef69d2559fe9 ("riscv: Move early dtb mapping into the fixmap region")
Thanks!
Alex
>
> Woody
>
On Wed, Jun 07, 2023 at 11:23:31PM +0100, Conor Dooley wrote:
> On Thu, Jun 08, 2023 at 06:17:22AM +0800, Woody Zhang wrote:
> > Hi, Conor
> >
> > On Wed, Jun 07, 2023 at 07:17:28PM +0100, Conor Dooley wrote:
> > >+CC Alex, you should take a look at this patch.
> > >
> > >On Wed, Jun 07, 2023 at 09:35:19PM +0800, Woody Zhang wrote:
> > >> It's possible that early_init_fdt_scan_reserved_mem() allocates memory
> > >> from memblock for dynamic reserved memory in `/reserved-memory` node.
> > >> Any fixed reservation must be done before that to avoid potential
> > >> conflicts.
> > >>
> > >> Reserve the DTB in memblock just after early scanning it.
> > >
> > >The rationale makes sense to me, I am just wondering what compelling
> > >reason there is to move it away from the memblock_reserve()s for the
> > >initd and vmlinux? Moving it above early_init_fdt_scan_reserved_mem()
> > >should be the sufficient minimum & would keep things together.
> >
> > IMO, moving it to parse_dtb() is more reasonable as early scanning and
> > reservation are both subject to DTB. It can also lower the risk to
> > mess up the sequence in the future. BTW, it's also invoked in
> > setup_machine_fdt() in arm64.
>
> I'm fine with the change either way, so:
> Reviewed-by: Conor Dooley <[email protected]>
> Mostly wanted to know whether you'd considered the minimal change.
What ever happened to this patch?