LinuxLists.cc - [PATCH 1/2] arm64: mem-model: add flatmem model for arm64

[permalink] [raw]

Subject: [PATCH 2/2] arm64: mm: make pfn always valid with flat memory

Make the pfn always valid when using flat memory.
If the reserved memory is not align to memblock-size,
there will be holes in zone.

This patch makes the memory in buddy always in the
array of mem-map.

Signed-off-by: Chen Feng <[email protected]>
Signed-off-by: Fu Jun <[email protected]>
---
arch/arm64/mm/init.c | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
index ea989d8..0e1d5b7 100644
--- a/arch/arm64/mm/init.c
+++ b/arch/arm64/mm/init.c
@@ -306,7 +306,8 @@ static void __init free_unused_memmap(void)
struct memblock_region *reg;

for_each_memblock(memory, reg) {
- start = __phys_to_pfn(reg->base);
+ start = round_down(__phys_to_pfn(reg->base),
+ MAX_ORDER_NR_PAGES);

#ifdef CONFIG_SPARSEMEM
/*
@@ -327,8 +328,8 @@ static void __init free_unused_memmap(void)
* memmap entries are valid from the bank end aligned to
* MAX_ORDER_NR_PAGES.
*/
- prev_end = ALIGN(__phys_to_pfn(reg->base + reg->size),
- MAX_ORDER_NR_PAGES);
+ prev_end = round_up(__phys_to_pfn(reg->base + reg->size),
+ MAX_ORDER_NR_PAGES);
}

#ifdef CONFIG_SPARSEMEM
--
1.9.1

2016-04-07 07:40:40

[permalink] [raw]

Subject: Re: [PATCH 2/2] arm64: mm: make pfn always valid with flat memory

add Mel Gorman

On 2016/4/5 16:22, Chen Feng wrote:
> Make the pfn always valid when using flat memory.
> If the reserved memory is not align to memblock-size,
> there will be holes in zone.
>
> This patch makes the memory in buddy always in the
> array of mem-map.
>
> Signed-off-by: Chen Feng <[email protected]>
> Signed-off-by: Fu Jun <[email protected]>
> ---
> arch/arm64/mm/init.c | 7 ++++---
> 1 file changed, 4 insertions(+), 3 deletions(-)
>
> diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
> index ea989d8..0e1d5b7 100644
> --- a/arch/arm64/mm/init.c
> +++ b/arch/arm64/mm/init.c
> @@ -306,7 +306,8 @@ static void __init free_unused_memmap(void)
> struct memblock_region *reg;
>
> for_each_memblock(memory, reg) {
> - start = __phys_to_pfn(reg->base);
> + start = round_down(__phys_to_pfn(reg->base),
> + MAX_ORDER_NR_PAGES);
>
> #ifdef CONFIG_SPARSEMEM
> /*
> @@ -327,8 +328,8 @@ static void __init free_unused_memmap(void)
> * memmap entries are valid from the bank end aligned to
> * MAX_ORDER_NR_PAGES.
> */
> - prev_end = ALIGN(__phys_to_pfn(reg->base + reg->size),
> - MAX_ORDER_NR_PAGES);
> + prev_end = round_up(__phys_to_pfn(reg->base + reg->size),
> + MAX_ORDER_NR_PAGES);
> }
>
> #ifdef CONFIG_SPARSEMEM
>

2016-04-07 07:40:52

[permalink] [raw]

Subject: Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64

add Mel Gorman

On 2016/4/5 16:22, Chen Feng wrote:
> We can reduce the memory allocated at mem-map
> by flatmem.
>
> currently, the default memory-model in arm64 is
> sparse memory. The mem-map array is not freed in
> this scene. If the physical address is too long,
> it will reserved too much memory for the mem-map
> array.
>
> Signed-off-by: Chen Feng <[email protected]>
> Signed-off-by: Fu Jun <[email protected]>
> ---
> arch/arm64/Kconfig | 3 +++
> 1 file changed, 3 insertions(+)
>
> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
> index 4f43622..c18930d 100644
> --- a/arch/arm64/Kconfig
> +++ b/arch/arm64/Kconfig
> @@ -559,6 +559,9 @@ config ARCH_SPARSEMEM_ENABLE
> def_bool y
> select SPARSEMEM_VMEMMAP_ENABLE
>
> +config ARCH_FLATMEM_ENABLE
> + def_bool y
> +
> config ARCH_SPARSEMEM_DEFAULT
> def_bool ARCH_SPARSEMEM_ENABLE
>
>

2016-04-07 14:21:56

by Will Deacon

[permalink] [raw]

Subject: Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64

On Tue, Apr 05, 2016 at 04:22:51PM +0800, Chen Feng wrote:
> We can reduce the memory allocated at mem-map
> by flatmem.
>
> currently, the default memory-model in arm64 is
> sparse memory. The mem-map array is not freed in
> this scene. If the physical address is too long,
> it will reserved too much memory for the mem-map
> array.

Can you elaborate a bit more on this, please? We use the vmemmap, so any
spaces between memory banks only burns up virtual space. What exactly is
the problem you're seeing that makes you want to use flatmem (which is
probably unsuitable for the majority of arm64 machines).

Will

2016-04-11 02:53:18

[permalink] [raw]

Subject: Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64

Hi will,
Thanks for review.

On 2016/4/7 22:21, Will Deacon wrote:
> On Tue, Apr 05, 2016 at 04:22:51PM +0800, Chen Feng wrote:
>> We can reduce the memory allocated at mem-map
>> by flatmem.
>>
>> currently, the default memory-model in arm64 is
>> sparse memory. The mem-map array is not freed in
>> this scene. If the physical address is too long,
>> it will reserved too much memory for the mem-map
>> array.
>
> Can you elaborate a bit more on this, please? We use the vmemmap, so any
> spaces between memory banks only burns up virtual space. What exactly is
> the problem you're seeing that makes you want to use flatmem (which is
> probably unsuitable for the majority of arm64 machines).
>
The root cause we want to use flat-mem is the mam_map alloced in sparse-mem
is not freed.

take a look at here:
arm64/mm/init.c
void __init mem_init(void)
{
#ifndef CONFIG_SPARSEMEM_VMEMMAP
free_unused_memmap();
#endif
}

Memory layout (3GB)

0 1.5G 2G 3.5G 4G
| | | | |
+--------------+------+---------------+--------------+
| MEM | hole | MEM | IO (regs) |
+--------------+------+---------------+--------------+

Memory layout (4GB)

0 3.5G 4G 4.5G
| | | |
+-------------------------------------+--------------+-------+
| MEM | IO (regs) | MEM |
+-------------------------------------+--------------+-------+

Currently, the sparse memory section is 1GB.

3GB ddr: the 1.5 ~2G and 3.5 ~ 4G are holes.
3GB ddr: the 3.5 ~ 4G and 4.5 ~ 5G are holes.

This will alloc 1G/4K * (struct page) memory for mem_map array.

We want to use flat-mem to reduce the alloced mem_map.

I don't know why you tell us the flatmem is unsuitable for the
majority of arm64 machines. Can tell us the reason of it?

And we are not going to limit the memdel in arm64, we just want to
make the flat-mem is an optional item in arm64.

puck,

> Will
>
> .
>

2016-04-11 07:35:47

[permalink] [raw]

Subject: Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64

On 11 April 2016 at 04:49, Chen Feng <[email protected]> wrote:
> Hi will,
> Thanks for review.
>
> On 2016/4/7 22:21, Will Deacon wrote:
>> On Tue, Apr 05, 2016 at 04:22:51PM +0800, Chen Feng wrote:
>>> We can reduce the memory allocated at mem-map
>>> by flatmem.
>>>
>>> currently, the default memory-model in arm64 is
>>> sparse memory. The mem-map array is not freed in
>>> this scene. If the physical address is too long,
>>> it will reserved too much memory for the mem-map
>>> array.
>>
>> Can you elaborate a bit more on this, please? We use the vmemmap, so any
>> spaces between memory banks only burns up virtual space. What exactly is
>> the problem you're seeing that makes you want to use flatmem (which is
>> probably unsuitable for the majority of arm64 machines).
>>
> The root cause we want to use flat-mem is the mam_map alloced in sparse-mem
> is not freed.
>
> take a look at here:
> arm64/mm/init.c
> void __init mem_init(void)
> {
> #ifndef CONFIG_SPARSEMEM_VMEMMAP
> free_unused_memmap();
> #endif
> }
>
> Memory layout (3GB)
>
> 0 1.5G 2G 3.5G 4G
> | | | | |
> +--------------+------+---------------+--------------+
> | MEM | hole | MEM | IO (regs) |
> +--------------+------+---------------+--------------+
>
>
> Memory layout (4GB)
>
> 0 3.5G 4G 4.5G
> | | | |
> +-------------------------------------+--------------+-------+
> | MEM | IO (regs) | MEM |
> +-------------------------------------+--------------+-------+
>
> Currently, the sparse memory section is 1GB.
>
> 3GB ddr: the 1.5 ~2G and 3.5 ~ 4G are holes.
> 3GB ddr: the 3.5 ~ 4G and 4.5 ~ 5G are holes.
>
> This will alloc 1G/4K * (struct page) memory for mem_map array.
>

No, this is incorrect. Sparsemem vmemmap only allocates struct pages
for memory regions that are actually populated.

For instance, on the Foundation model with 4 GB of memory, you may see
something like this in the boot log

[ 0.000000] vmemmap : 0xffffffbdc0000000 - 0xffffffbfc0000000
( 8 GB maximum)
[ 0.000000] 0xffffffbdc0000000 - 0xffffffbde2000000
( 544 MB actual)

but in reality, only the following regions have been allocated

---[ vmemmap start ]---
0xffffffbdc0000000-0xffffffbdc2000000 32M RW NX SHD AF
BLK UXN MEM/NORMAL
0xffffffbde0000000-0xffffffbde2000000 32M RW NX SHD AF
BLK UXN MEM/NORMAL
---[ vmemmap end ]---

so only 64 MB is used to back 4 GB of RAM with struct pages, which is
minimal. Moving to flatmem will not reduce the memory footprint at
all.

2016-04-11 07:57:08

[permalink] [raw]

Subject: Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64

Hi Ard,

On 2016/4/11 15:35, Ard Biesheuvel wrote:
> On 11 April 2016 at 04:49, Chen Feng <[email protected]> wrote:
>> Hi will,
>> Thanks for review.
>>
>> On 2016/4/7 22:21, Will Deacon wrote:
>>> On Tue, Apr 05, 2016 at 04:22:51PM +0800, Chen Feng wrote:
>>>> We can reduce the memory allocated at mem-map
>>>> by flatmem.
>>>>
>>>> currently, the default memory-model in arm64 is
>>>> sparse memory. The mem-map array is not freed in
>>>> this scene. If the physical address is too long,
>>>> it will reserved too much memory for the mem-map
>>>> array.
>>>
>>> Can you elaborate a bit more on this, please? We use the vmemmap, so any
>>> spaces between memory banks only burns up virtual space. What exactly is
>>> the problem you're seeing that makes you want to use flatmem (which is
>>> probably unsuitable for the majority of arm64 machines).
>>>
>> The root cause we want to use flat-mem is the mam_map alloced in sparse-mem
>> is not freed.
>>
>> take a look at here:
>> arm64/mm/init.c
>> void __init mem_init(void)
>> {
>> #ifndef CONFIG_SPARSEMEM_VMEMMAP
>> free_unused_memmap();
>> #endif
>> }
>>
>> Memory layout (3GB)
>>
>> 0 1.5G 2G 3.5G 4G
>> | | | | |
>> +--------------+------+---------------+--------------+
>> | MEM | hole | MEM | IO (regs) |
>> +--------------+------+---------------+--------------+
>>
>>
>> Memory layout (4GB)
>>
>> 0 3.5G 4G 4.5G
>> | | | |
>> +-------------------------------------+--------------+-------+
>> | MEM | IO (regs) | MEM |
>> +-------------------------------------+--------------+-------+
>>
>> Currently, the sparse memory section is 1GB.
>>
>> 3GB ddr: the 1.5 ~2G and 3.5 ~ 4G are holes.
>> 3GB ddr: the 3.5 ~ 4G and 4.5 ~ 5G are holes.
>>
>> This will alloc 1G/4K * (struct page) memory for mem_map array.
>>
>
> No, this is incorrect. Sparsemem vmemmap only allocates struct pages
> for memory regions that are actually populated.
>
> For instance, on the Foundation model with 4 GB of memory, you may see
> something like this in the boot log
>
> [ 0.000000] vmemmap : 0xffffffbdc0000000 - 0xffffffbfc0000000
> ( 8 GB maximum)
> [ 0.000000] 0xffffffbdc0000000 - 0xffffffbde2000000
> ( 544 MB actual)
>
> but in reality, only the following regions have been allocated
>
> ---[ vmemmap start ]---
> 0xffffffbdc0000000-0xffffffbdc2000000 32M RW NX SHD AF
> BLK UXN MEM/NORMAL
> 0xffffffbde0000000-0xffffffbde2000000 32M RW NX SHD AF
> BLK UXN MEM/NORMAL
> ---[ vmemmap end ]---
>
> so only 64 MB is used to back 4 GB of RAM with struct pages, which is
> minimal. Moving to flatmem will not reduce the memory footprint at
> all.

Yes,but the populate is section, which is 1GB. Take a look at the above
memory layout.

The section 1G ~ 2G is a section. But 1.5G ~ 2G is a hole.

The section 3G ~ 4G is a section. But 3.5G ~ 4G is a hole.
>> 0 1.5G 2G 3.5G 4G
>> | | | | |
>> +--------------+------+---------------+--------------+
>> | MEM | hole | MEM | IO (regs) |
>> +--------------+------+---------------+--------------+
The hole in 1.5G ~ 2G is also allocated mem-map array. And also with the 3.5G ~ 4G.

We want free the the mem-map array. With flat-mem we can work with this scene very well.

Thanks,

>
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to [email protected]. For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
>
> .
>

2016-04-11 08:00:17

[permalink] [raw]

Subject: Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64

On 11 April 2016 at 09:55, Chen Feng <[email protected]> wrote:
> Hi Ard,
>
> On 2016/4/11 15:35, Ard Biesheuvel wrote:
>> On 11 April 2016 at 04:49, Chen Feng <[email protected]> wrote:
>>> Hi will,
>>> Thanks for review.
>>>
>>> On 2016/4/7 22:21, Will Deacon wrote:
>>>> On Tue, Apr 05, 2016 at 04:22:51PM +0800, Chen Feng wrote:
>>>>> We can reduce the memory allocated at mem-map
>>>>> by flatmem.
>>>>>
>>>>> currently, the default memory-model in arm64 is
>>>>> sparse memory. The mem-map array is not freed in
>>>>> this scene. If the physical address is too long,
>>>>> it will reserved too much memory for the mem-map
>>>>> array.
>>>>
>>>> Can you elaborate a bit more on this, please? We use the vmemmap, so any
>>>> spaces between memory banks only burns up virtual space. What exactly is
>>>> the problem you're seeing that makes you want to use flatmem (which is
>>>> probably unsuitable for the majority of arm64 machines).
>>>>
>>> The root cause we want to use flat-mem is the mam_map alloced in sparse-mem
>>> is not freed.
>>>
>>> take a look at here:
>>> arm64/mm/init.c
>>> void __init mem_init(void)
>>> {
>>> #ifndef CONFIG_SPARSEMEM_VMEMMAP
>>> free_unused_memmap();
>>> #endif
>>> }
>>>
>>> Memory layout (3GB)
>>>
>>> 0 1.5G 2G 3.5G 4G
>>> | | | | |
>>> +--------------+------+---------------+--------------+
>>> | MEM | hole | MEM | IO (regs) |
>>> +--------------+------+---------------+--------------+
>>>
>>>
>>> Memory layout (4GB)
>>>
>>> 0 3.5G 4G 4.5G
>>> | | | |
>>> +-------------------------------------+--------------+-------+
>>> | MEM | IO (regs) | MEM |
>>> +-------------------------------------+--------------+-------+
>>>
>>> Currently, the sparse memory section is 1GB.
>>>
>>> 3GB ddr: the 1.5 ~2G and 3.5 ~ 4G are holes.
>>> 3GB ddr: the 3.5 ~ 4G and 4.5 ~ 5G are holes.
>>>
>>> This will alloc 1G/4K * (struct page) memory for mem_map array.
>>>
>>
>> No, this is incorrect. Sparsemem vmemmap only allocates struct pages
>> for memory regions that are actually populated.
>>
>> For instance, on the Foundation model with 4 GB of memory, you may see
>> something like this in the boot log
>>
>> [ 0.000000] vmemmap : 0xffffffbdc0000000 - 0xffffffbfc0000000
>> ( 8 GB maximum)
>> [ 0.000000] 0xffffffbdc0000000 - 0xffffffbde2000000
>> ( 544 MB actual)
>>
>> but in reality, only the following regions have been allocated
>>
>> ---[ vmemmap start ]---
>> 0xffffffbdc0000000-0xffffffbdc2000000 32M RW NX SHD AF
>> BLK UXN MEM/NORMAL
>> 0xffffffbde0000000-0xffffffbde2000000 32M RW NX SHD AF
>> BLK UXN MEM/NORMAL
>> ---[ vmemmap end ]---
>>
>> so only 64 MB is used to back 4 GB of RAM with struct pages, which is
>> minimal. Moving to flatmem will not reduce the memory footprint at
>> all.
>
> Yes,but the populate is section, which is 1GB. Take a look at the above
> memory layout.
>
> The section 1G ~ 2G is a section. But 1.5G ~ 2G is a hole.
>
> The section 3G ~ 4G is a section. But 3.5G ~ 4G is a hole.
>>> 0 1.5G 2G 3.5G 4G
>>> | | | | |
>>> +--------------+------+---------------+--------------+
>>> | MEM | hole | MEM | IO (regs) |
>>> +--------------+------+---------------+--------------+
> The hole in 1.5G ~ 2G is also allocated mem-map array. And also with the 3.5G ~ 4G.
>

No, it is not. It may be covered by a section, but that does not mean
sparsemem vmemmap will actually allocate backing for it. The
granularity used by sparsemem vmemmap on a 4k pages kernel is 128 MB,
due to the fact that the backing is performed at PMD granularity.

Please, could you share the contents of the vmemmap section in
/sys/kernel/debug/kernel_page_tables of your system running with
sparsemem vmemmap enabled? You will need to set CONFIG_ARM64_PTDUMP=y

2016-04-11 10:04:25

[permalink] [raw]

Subject: Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64

Hi Ard,

On 2016/4/11 16:00, Ard Biesheuvel wrote:
> On 11 April 2016 at 09:55, Chen Feng <[email protected]> wrote:
>> Hi Ard,
>>
>> On 2016/4/11 15:35, Ard Biesheuvel wrote:
>>> On 11 April 2016 at 04:49, Chen Feng <[email protected]> wrote:
>>>> Hi will,
>>>> Thanks for review.
>>>>
>>>> On 2016/4/7 22:21, Will Deacon wrote:
>>>>> On Tue, Apr 05, 2016 at 04:22:51PM +0800, Chen Feng wrote:
>>>>>> We can reduce the memory allocated at mem-map
>>>>>> by flatmem.
>>>>>>
>>>>>> currently, the default memory-model in arm64 is
>>>>>> sparse memory. The mem-map array is not freed in
>>>>>> this scene. If the physical address is too long,
>>>>>> it will reserved too much memory for the mem-map
>>>>>> array.
>>>>>
>>>>> Can you elaborate a bit more on this, please? We use the vmemmap, so any
>>>>> spaces between memory banks only burns up virtual space. What exactly is
>>>>> the problem you're seeing that makes you want to use flatmem (which is
>>>>> probably unsuitable for the majority of arm64 machines).
>>>>>
>>>> The root cause we want to use flat-mem is the mam_map alloced in sparse-mem
>>>> is not freed.
>>>>
>>>> take a look at here:
>>>> arm64/mm/init.c
>>>> void __init mem_init(void)
>>>> {
>>>> #ifndef CONFIG_SPARSEMEM_VMEMMAP
>>>> free_unused_memmap();
>>>> #endif
>>>> }
>>>>
>>>> Memory layout (3GB)
>>>>
>>>> 0 1.5G 2G 3.5G 4G
>>>> | | | | |
>>>> +--------------+------+---------------+--------------+
>>>> | MEM | hole | MEM | IO (regs) |
>>>> +--------------+------+---------------+--------------+
>>>>
>>>>
>>>> Memory layout (4GB)
>>>>
>>>> 0 3.5G 4G 4.5G
>>>> | | | |
>>>> +-------------------------------------+--------------+-------+
>>>> | MEM | IO (regs) | MEM |
>>>> +-------------------------------------+--------------+-------+
>>>>
>>>> Currently, the sparse memory section is 1GB.
>>>>
>>>> 3GB ddr: the 1.5 ~2G and 3.5 ~ 4G are holes.
>>>> 3GB ddr: the 3.5 ~ 4G and 4.5 ~ 5G are holes.
>>>>
>>>> This will alloc 1G/4K * (struct page) memory for mem_map array.
>>>>
>>>
>>> No, this is incorrect. Sparsemem vmemmap only allocates struct pages
>>> for memory regions that are actually populated.
>>>
>>> For instance, on the Foundation model with 4 GB of memory, you may see
>>> something like this in the boot log
>>>
>>> [ 0.000000] vmemmap : 0xffffffbdc0000000 - 0xffffffbfc0000000
>>> ( 8 GB maximum)
>>> [ 0.000000] 0xffffffbdc0000000 - 0xffffffbde2000000
>>> ( 544 MB actual)
>>>
>>> but in reality, only the following regions have been allocated
>>>
>>> ---[ vmemmap start ]---
>>> 0xffffffbdc0000000-0xffffffbdc2000000 32M RW NX SHD AF
>>> BLK UXN MEM/NORMAL
>>> 0xffffffbde0000000-0xffffffbde2000000 32M RW NX SHD AF
>>> BLK UXN MEM/NORMAL
>>> ---[ vmemmap end ]---
>>>
>>> so only 64 MB is used to back 4 GB of RAM with struct pages, which is
>>> minimal. Moving to flatmem will not reduce the memory footprint at
>>> all.
>>
>> Yes,but the populate is section, which is 1GB. Take a look at the above
>> memory layout.
>>
>> The section 1G ~ 2G is a section. But 1.5G ~ 2G is a hole.
>>
>> The section 3G ~ 4G is a section. But 3.5G ~ 4G is a hole.
>>>> 0 1.5G 2G 3.5G 4G
>>>> | | | | |
>>>> +--------------+------+---------------+--------------+
>>>> | MEM | hole | MEM | IO (regs) |
>>>> +--------------+------+---------------+--------------+
>> The hole in 1.5G ~ 2G is also allocated mem-map array. And also with the 3.5G ~ 4G.
>>
>
> No, it is not. It may be covered by a section, but that does not mean
> sparsemem vmemmap will actually allocate backing for it. The
> granularity used by sparsemem vmemmap on a 4k pages kernel is 128 MB,
> due to the fact that the backing is performed at PMD granularity.
>
> Please, could you share the contents of the vmemmap section in
> /sys/kernel/debug/kernel_page_tables of your system running with
> sparsemem vmemmap enabled? You will need to set CONFIG_ARM64_PTDUMP=y
>

Please see the pg-tables below.

With sparse and vmemmap enable.

---[ vmemmap start ]---
0xffffffbdc0200000-0xffffffbdc4800000 70M RW NX SHD AF UXN MEM/NORMAL
---[ vmemmap end ]---

The board is 4GB, and the memap is 70MB
1G memory --- 14MB mem_map array.
So the 4GB has 5 sections, which used 5 * 14MB memory.

> .
>

2016-04-11 10:31:56

[permalink] [raw]

Subject: Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64

On 11 April 2016 at 11:59, Chen Feng <[email protected]> wrote:
> Hi Ard,
>
> On 2016/4/11 16:00, Ard Biesheuvel wrote:
>> On 11 April 2016 at 09:55, Chen Feng <[email protected]> wrote:
>>> Hi Ard,
>>>
>>> On 2016/4/11 15:35, Ard Biesheuvel wrote:
>>>> On 11 April 2016 at 04:49, Chen Feng <[email protected]> wrote:
>>>>> Hi will,
>>>>> Thanks for review.
>>>>>
>>>>> On 2016/4/7 22:21, Will Deacon wrote:
>>>>>> On Tue, Apr 05, 2016 at 04:22:51PM +0800, Chen Feng wrote:
>>>>>>> We can reduce the memory allocated at mem-map
>>>>>>> by flatmem.
>>>>>>>
>>>>>>> currently, the default memory-model in arm64 is
>>>>>>> sparse memory. The mem-map array is not freed in
>>>>>>> this scene. If the physical address is too long,
>>>>>>> it will reserved too much memory for the mem-map
>>>>>>> array.
>>>>>>
>>>>>> Can you elaborate a bit more on this, please? We use the vmemmap, so any
>>>>>> spaces between memory banks only burns up virtual space. What exactly is
>>>>>> the problem you're seeing that makes you want to use flatmem (which is
>>>>>> probably unsuitable for the majority of arm64 machines).
>>>>>>
>>>>> The root cause we want to use flat-mem is the mam_map alloced in sparse-mem
>>>>> is not freed.
>>>>>
>>>>> take a look at here:
>>>>> arm64/mm/init.c
>>>>> void __init mem_init(void)
>>>>> {
>>>>> #ifndef CONFIG_SPARSEMEM_VMEMMAP
>>>>> free_unused_memmap();
>>>>> #endif
>>>>> }
>>>>>
>>>>> Memory layout (3GB)
>>>>>
>>>>> 0 1.5G 2G 3.5G 4G
>>>>> | | | | |
>>>>> +--------------+------+---------------+--------------+
>>>>> | MEM | hole | MEM | IO (regs) |
>>>>> +--------------+------+---------------+--------------+
>>>>>
>>>>>
>>>>> Memory layout (4GB)
>>>>>
>>>>> 0 3.5G 4G 4.5G
>>>>> | | | |
>>>>> +-------------------------------------+--------------+-------+
>>>>> | MEM | IO (regs) | MEM |
>>>>> +-------------------------------------+--------------+-------+
>>>>>
>>>>> Currently, the sparse memory section is 1GB.
>>>>>
>>>>> 3GB ddr: the 1.5 ~2G and 3.5 ~ 4G are holes.
>>>>> 3GB ddr: the 3.5 ~ 4G and 4.5 ~ 5G are holes.
>>>>>
>>>>> This will alloc 1G/4K * (struct page) memory for mem_map array.
>>>>>
>>>>
>>>> No, this is incorrect. Sparsemem vmemmap only allocates struct pages
>>>> for memory regions that are actually populated.
>>>>
>>>> For instance, on the Foundation model with 4 GB of memory, you may see
>>>> something like this in the boot log
>>>>
>>>> [ 0.000000] vmemmap : 0xffffffbdc0000000 - 0xffffffbfc0000000
>>>> ( 8 GB maximum)
>>>> [ 0.000000] 0xffffffbdc0000000 - 0xffffffbde2000000
>>>> ( 544 MB actual)
>>>>
>>>> but in reality, only the following regions have been allocated
>>>>
>>>> ---[ vmemmap start ]---
>>>> 0xffffffbdc0000000-0xffffffbdc2000000 32M RW NX SHD AF
>>>> BLK UXN MEM/NORMAL
>>>> 0xffffffbde0000000-0xffffffbde2000000 32M RW NX SHD AF
>>>> BLK UXN MEM/NORMAL
>>>> ---[ vmemmap end ]---
>>>>
>>>> so only 64 MB is used to back 4 GB of RAM with struct pages, which is
>>>> minimal. Moving to flatmem will not reduce the memory footprint at
>>>> all.
>>>
>>> Yes,but the populate is section, which is 1GB. Take a look at the above
>>> memory layout.
>>>
>>> The section 1G ~ 2G is a section. But 1.5G ~ 2G is a hole.
>>>
>>> The section 3G ~ 4G is a section. But 3.5G ~ 4G is a hole.
>>>>> 0 1.5G 2G 3.5G 4G
>>>>> | | | | |
>>>>> +--------------+------+---------------+--------------+
>>>>> | MEM | hole | MEM | IO (regs) |
>>>>> +--------------+------+---------------+--------------+
>>> The hole in 1.5G ~ 2G is also allocated mem-map array. And also with the 3.5G ~ 4G.
>>>
>>
>> No, it is not. It may be covered by a section, but that does not mean
>> sparsemem vmemmap will actually allocate backing for it. The
>> granularity used by sparsemem vmemmap on a 4k pages kernel is 128 MB,
>> due to the fact that the backing is performed at PMD granularity.
>>
>> Please, could you share the contents of the vmemmap section in
>> /sys/kernel/debug/kernel_page_tables of your system running with
>> sparsemem vmemmap enabled? You will need to set CONFIG_ARM64_PTDUMP=y
>>
>
> Please see the pg-tables below.
>
>
> With sparse and vmemmap enable.
>
> ---[ vmemmap start ]---
> 0xffffffbdc0200000-0xffffffbdc4800000 70M RW NX SHD AF UXN MEM/NORMAL
> ---[ vmemmap end ]---
>

OK, I see what you mean now. Sorry for taking so long to catch up.

> The board is 4GB, and the memap is 70MB
> 1G memory --- 14MB mem_map array.

No, this is incorrect. 1 GB corresponds with 16 MB worth of struct
pages assuming sizeof(struct page) == 64

So you are losing 6 MB to rounding here, which I agree is significant.
I wonder if it makes sense to use a lower value for SECTION_SIZE_BITS
on 4k pages kernels, but perhaps we're better off asking the opinion
of the other cc'ees.

Thanks,
Ard.

2016-04-11 10:40:25

by Will Deacon

[permalink] [raw]

Subject: Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64

On Mon, Apr 11, 2016 at 12:31:53PM +0200, Ard Biesheuvel wrote:
> On 11 April 2016 at 11:59, Chen Feng <[email protected]> wrote:
> > Please see the pg-tables below.
> >
> >
> > With sparse and vmemmap enable.
> >
> > ---[ vmemmap start ]---
> > 0xffffffbdc0200000-0xffffffbdc4800000 70M RW NX SHD AF UXN MEM/NORMAL
> > ---[ vmemmap end ]---
> >
>
> OK, I see what you mean now. Sorry for taking so long to catch up.
>
> > The board is 4GB, and the memap is 70MB
> > 1G memory --- 14MB mem_map array.
>
> No, this is incorrect. 1 GB corresponds with 16 MB worth of struct
> pages assuming sizeof(struct page) == 64
>
> So you are losing 6 MB to rounding here, which I agree is significant.
> I wonder if it makes sense to use a lower value for SECTION_SIZE_BITS
> on 4k pages kernels, but perhaps we're better off asking the opinion
> of the other cc'ees.

You need to be really careful making SECTION_SIZE_BITS smaller because
it has a direct correlation on the use of page->flags and you can end up
running out of bits fairly easily.

Will

2016-04-11 10:49:26

[permalink] [raw]

Subject: Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64

On 2016/4/11 17:59, Chen Feng wrote:
> Hi Ard,
>
> On 2016/4/11 16:00, Ard Biesheuvel wrote:
>> On 11 April 2016 at 09:55, Chen Feng <[email protected]> wrote:
>>> Hi Ard,
>>>
>>> On 2016/4/11 15:35, Ard Biesheuvel wrote:
>>>> On 11 April 2016 at 04:49, Chen Feng <[email protected]> wrote:
>>>>> Hi will,
>>>>> Thanks for review.
>>>>>
>>>>> On 2016/4/7 22:21, Will Deacon wrote:
>>>>>> On Tue, Apr 05, 2016 at 04:22:51PM +0800, Chen Feng wrote:
>>>>>>> We can reduce the memory allocated at mem-map
>>>>>>> by flatmem.
>>>>>>>
>>>>>>> currently, the default memory-model in arm64 is
>>>>>>> sparse memory. The mem-map array is not freed in
>>>>>>> this scene. If the physical address is too long,
>>>>>>> it will reserved too much memory for the mem-map
>>>>>>> array.
>>>>>>
>>>>>> Can you elaborate a bit more on this, please? We use the vmemmap, so any
>>>>>> spaces between memory banks only burns up virtual space. What exactly is
>>>>>> the problem you're seeing that makes you want to use flatmem (which is
>>>>>> probably unsuitable for the majority of arm64 machines).
>>>>>>
>>>>> The root cause we want to use flat-mem is the mam_map alloced in sparse-mem
>>>>> is not freed.
>>>>>
>>>>> take a look at here:
>>>>> arm64/mm/init.c
>>>>> void __init mem_init(void)
>>>>> {
>>>>> #ifndef CONFIG_SPARSEMEM_VMEMMAP
>>>>> free_unused_memmap();
>>>>> #endif
>>>>> }
>>>>>
>>>>> Memory layout (3GB)
>>>>>
>>>>> 0 1.5G 2G 3.5G 4G
>>>>> | | | | |
>>>>> +--------------+------+---------------+--------------+
>>>>> | MEM | hole | MEM | IO (regs) |
>>>>> +--------------+------+---------------+--------------+
>>>>>
>>>>>
>>>>> Memory layout (4GB)
>>>>>
>>>>> 0 3.5G 4G 4.5G
>>>>> | | | |
>>>>> +-------------------------------------+--------------+-------+
>>>>> | MEM | IO (regs) | MEM |
>>>>> +-------------------------------------+--------------+-------+
>>>>>
>>>>> Currently, the sparse memory section is 1GB.
>>>>>
>>>>> 3GB ddr: the 1.5 ~2G and 3.5 ~ 4G are holes.
>>>>> 3GB ddr: the 3.5 ~ 4G and 4.5 ~ 5G are holes.
>>>>>
>>>>> This will alloc 1G/4K * (struct page) memory for mem_map array.
>>>>>
>>>>
>>>> No, this is incorrect. Sparsemem vmemmap only allocates struct pages
>>>> for memory regions that are actually populated.
>>>>
>>>> For instance, on the Foundation model with 4 GB of memory, you may see
>>>> something like this in the boot log
>>>>
>>>> [ 0.000000] vmemmap : 0xffffffbdc0000000 - 0xffffffbfc0000000
>>>> ( 8 GB maximum)
>>>> [ 0.000000] 0xffffffbdc0000000 - 0xffffffbde2000000
>>>> ( 544 MB actual)
>>>>
>>>> but in reality, only the following regions have been allocated
>>>>
>>>> ---[ vmemmap start ]---
>>>> 0xffffffbdc0000000-0xffffffbdc2000000 32M RW NX SHD AF
>>>> BLK UXN MEM/NORMAL
>>>> 0xffffffbde0000000-0xffffffbde2000000 32M RW NX SHD AF
>>>> BLK UXN MEM/NORMAL
>>>> ---[ vmemmap end ]---
>>>>
>>>> so only 64 MB is used to back 4 GB of RAM with struct pages, which is
>>>> minimal. Moving to flatmem will not reduce the memory footprint at
>>>> all.
>>>
>>> Yes,but the populate is section, which is 1GB. Take a look at the above
>>> memory layout.
>>>
>>> The section 1G ~ 2G is a section. But 1.5G ~ 2G is a hole.
>>>
>>> The section 3G ~ 4G is a section. But 3.5G ~ 4G is a hole.
>>>>> 0 1.5G 2G 3.5G 4G
>>>>> | | | | |
>>>>> +--------------+------+---------------+--------------+
>>>>> | MEM | hole | MEM | IO (regs) |
>>>>> +--------------+------+---------------+--------------+
>>> The hole in 1.5G ~ 2G is also allocated mem-map array. And also with the 3.5G ~ 4G.
>>>
>>
>> No, it is not. It may be covered by a section, but that does not mean
>> sparsemem vmemmap will actually allocate backing for it. The
>> granularity used by sparsemem vmemmap on a 4k pages kernel is 128 MB,
>> due to the fact that the backing is performed at PMD granularity.
>>
>> Please, could you share the contents of the vmemmap section in
>> /sys/kernel/debug/kernel_page_tables of your system running with
>> sparsemem vmemmap enabled? You will need to set CONFIG_ARM64_PTDUMP=y
>>
>
> Please see the pg-tables below.
>
>
> With sparse and vmemmap enable.
>
> ---[ vmemmap start ]---
> 0xffffffbdc0200000-0xffffffbdc4800000 70M RW NX SHD AF UXN MEM/NORMAL
> ---[ vmemmap end ]---
>
>
> The board is 4GB, and the memap is 70MB
> 1G memory --- 14MB mem_map array.
> So the 4GB has 5 sections, which used 5 * 14MB memory.
>
>
Sorry, 1G memory is 16GB
5 sections is 5 * 16 = 80MB

1G / 4K * (struct page) 64B = 16MB

I don't know why the vmemap dump in pg-tables is 70MB.

I add hack code in vmemmap_populate sparse_mem_map_populate.

here is the log:
sparse_mem_map_populate 188 start ffffffbdc0000000 end ffffffbdc1000000 PAGES_PER_SECTION 40000 nid 0
vmemmap_populate 549 size 200000 total 200000 addr ffffffbdc0000000
vmemmap_populate 549 size 200000 total 400000 addr ffffffbdc0200000
vmemmap_populate 549 size 200000 total 600000 addr ffffffbdc0400000
vmemmap_populate 549 size 200000 total 800000 addr ffffffbdc0600000
vmemmap_populate 549 size 200000 total a00000 addr ffffffbdc0800000
vmemmap_populate 549 size 200000 total c00000 addr ffffffbdc0a00000
vmemmap_populate 549 size 200000 total e00000 addr ffffffbdc0c00000
vmemmap_populate 549 size 200000 total 1000000 addr ffffffbdc0e00000
sparse_mem_map_populate 188 start ffffffbdc1000000 end ffffffbdc2000000 PAGES_PER_SECTION 40000 nid 0
...
sparse_mem_map_populate 188 start ffffffbdc2000000 end ffffffbdc3000000 PAGES_PER_SECTION 40000 nid 0
sparse_mem_map_populate 188 start ffffffbdc3000000 end ffffffbdc4000000 PAGES_PER_SECTION 40000 nid 0
sparse_mem_map_populate 188 start ffffffbdc4000000 end ffffffbdc5000000 PAGES_PER_SECTION 40000 nid 0

With 4GB memory, it allocated 2MB * 8 * 5 = 80MB.
> 0 3.5G 4G 4.5G
> | | | |
> +-------------------------------------+--------------+-------+
> | MEM | IO (regs) | MEM |
> +-------------------------------------+--------------+-------+

4GB memory ,5 sections. 80MB mem_map allocated.

>
>
>
>
>> .
>>

2016-04-11 10:58:06

[permalink] [raw]

Subject: Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64

Hi Will,

On 2016/4/11 18:40, Will Deacon wrote:
> On Mon, Apr 11, 2016 at 12:31:53PM +0200, Ard Biesheuvel wrote:
>> On 11 April 2016 at 11:59, Chen Feng <[email protected]> wrote:
>>> Please see the pg-tables below.
>>>
>>>
>>> With sparse and vmemmap enable.
>>>
>>> ---[ vmemmap start ]---
>>> 0xffffffbdc0200000-0xffffffbdc4800000 70M RW NX SHD AF UXN MEM/NORMAL
>>> ---[ vmemmap end ]---
>>>
>>
>> OK, I see what you mean now. Sorry for taking so long to catch up.
>>
>>> The board is 4GB, and the memap is 70MB
>>> 1G memory --- 14MB mem_map array.
>>
>> No, this is incorrect. 1 GB corresponds with 16 MB worth of struct
>> pages assuming sizeof(struct page) == 64
>>
>> So you are losing 6 MB to rounding here, which I agree is significant.
>> I wonder if it makes sense to use a lower value for SECTION_SIZE_BITS
>> on 4k pages kernels, but perhaps we're better off asking the opinion
>> of the other cc'ees.
>
> You need to be really careful making SECTION_SIZE_BITS smaller because
> it has a direct correlation on the use of page->flags and you can end up
> running out of bits fairly easily.

Yes, making SECTION_SIZE_BITS smaller can solve the current situation.

But if the phys-addr is 64GB, but only 4GB ddr is the valid address. And the

holes are not always 512MB.

But, can you tell us why *smaller SIZE makes running out of bits fairly easily*?

And how about the flat-mem model?

>
> Will
>
> .
>

2016-04-11 11:02:10

[permalink] [raw]

Subject: Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64

On 11 April 2016 at 12:48, Chen Feng <[email protected]> wrote:
>
>
> On 2016/4/11 17:59, Chen Feng wrote:
>> Hi Ard,
>>
>> On 2016/4/11 16:00, Ard Biesheuvel wrote:
>>> On 11 April 2016 at 09:55, Chen Feng <[email protected]> wrote:
>>>> Hi Ard,
>>>>
>>>> On 2016/4/11 15:35, Ard Biesheuvel wrote:
>>>>> On 11 April 2016 at 04:49, Chen Feng <[email protected]> wrote:
>>>>>> Hi will,
>>>>>> Thanks for review.
>>>>>>
>>>>>> On 2016/4/7 22:21, Will Deacon wrote:
>>>>>>> On Tue, Apr 05, 2016 at 04:22:51PM +0800, Chen Feng wrote:
>>>>>>>> We can reduce the memory allocated at mem-map
>>>>>>>> by flatmem.
>>>>>>>>
>>>>>>>> currently, the default memory-model in arm64 is
>>>>>>>> sparse memory. The mem-map array is not freed in
>>>>>>>> this scene. If the physical address is too long,
>>>>>>>> it will reserved too much memory for the mem-map
>>>>>>>> array.
>>>>>>>
>>>>>>> Can you elaborate a bit more on this, please? We use the vmemmap, so any
>>>>>>> spaces between memory banks only burns up virtual space. What exactly is
>>>>>>> the problem you're seeing that makes you want to use flatmem (which is
>>>>>>> probably unsuitable for the majority of arm64 machines).
>>>>>>>
>>>>>> The root cause we want to use flat-mem is the mam_map alloced in sparse-mem
>>>>>> is not freed.
>>>>>>
>>>>>> take a look at here:
>>>>>> arm64/mm/init.c
>>>>>> void __init mem_init(void)
>>>>>> {
>>>>>> #ifndef CONFIG_SPARSEMEM_VMEMMAP
>>>>>> free_unused_memmap();
>>>>>> #endif
>>>>>> }
>>>>>>
>>>>>> Memory layout (3GB)
>>>>>>
>>>>>> 0 1.5G 2G 3.5G 4G
>>>>>> | | | | |
>>>>>> +--------------+------+---------------+--------------+
>>>>>> | MEM | hole | MEM | IO (regs) |
>>>>>> +--------------+------+---------------+--------------+
>>>>>>
>>>>>>
>>>>>> Memory layout (4GB)
>>>>>>
>>>>>> 0 3.5G 4G 4.5G
>>>>>> | | | |
>>>>>> +-------------------------------------+--------------+-------+
>>>>>> | MEM | IO (regs) | MEM |
>>>>>> +-------------------------------------+--------------+-------+
>>>>>>
>>>>>> Currently, the sparse memory section is 1GB.
>>>>>>
>>>>>> 3GB ddr: the 1.5 ~2G and 3.5 ~ 4G are holes.
>>>>>> 3GB ddr: the 3.5 ~ 4G and 4.5 ~ 5G are holes.
>>>>>>
>>>>>> This will alloc 1G/4K * (struct page) memory for mem_map array.
>>>>>>
>>>>>
>>>>> No, this is incorrect. Sparsemem vmemmap only allocates struct pages
>>>>> for memory regions that are actually populated.
>>>>>
>>>>> For instance, on the Foundation model with 4 GB of memory, you may see
>>>>> something like this in the boot log
>>>>>
>>>>> [ 0.000000] vmemmap : 0xffffffbdc0000000 - 0xffffffbfc0000000
>>>>> ( 8 GB maximum)
>>>>> [ 0.000000] 0xffffffbdc0000000 - 0xffffffbde2000000
>>>>> ( 544 MB actual)
>>>>>
>>>>> but in reality, only the following regions have been allocated
>>>>>
>>>>> ---[ vmemmap start ]---
>>>>> 0xffffffbdc0000000-0xffffffbdc2000000 32M RW NX SHD AF
>>>>> BLK UXN MEM/NORMAL
>>>>> 0xffffffbde0000000-0xffffffbde2000000 32M RW NX SHD AF
>>>>> BLK UXN MEM/NORMAL
>>>>> ---[ vmemmap end ]---
>>>>>
>>>>> so only 64 MB is used to back 4 GB of RAM with struct pages, which is
>>>>> minimal. Moving to flatmem will not reduce the memory footprint at
>>>>> all.
>>>>
>>>> Yes,but the populate is section, which is 1GB. Take a look at the above
>>>> memory layout.
>>>>
>>>> The section 1G ~ 2G is a section. But 1.5G ~ 2G is a hole.
>>>>
>>>> The section 3G ~ 4G is a section. But 3.5G ~ 4G is a hole.
>>>>>> 0 1.5G 2G 3.5G 4G
>>>>>> | | | | |
>>>>>> +--------------+------+---------------+--------------+
>>>>>> | MEM | hole | MEM | IO (regs) |
>>>>>> +--------------+------+---------------+--------------+
>>>> The hole in 1.5G ~ 2G is also allocated mem-map array. And also with the 3.5G ~ 4G.
>>>>
>>>
>>> No, it is not. It may be covered by a section, but that does not mean
>>> sparsemem vmemmap will actually allocate backing for it. The
>>> granularity used by sparsemem vmemmap on a 4k pages kernel is 128 MB,
>>> due to the fact that the backing is performed at PMD granularity.
>>>
>>> Please, could you share the contents of the vmemmap section in
>>> /sys/kernel/debug/kernel_page_tables of your system running with
>>> sparsemem vmemmap enabled? You will need to set CONFIG_ARM64_PTDUMP=y
>>>
>>
>> Please see the pg-tables below.
>>
>>
>> With sparse and vmemmap enable.
>>
>> ---[ vmemmap start ]---
>> 0xffffffbdc0200000-0xffffffbdc4800000 70M RW NX SHD AF UXN MEM/NORMAL
>> ---[ vmemmap end ]---
>>
>>
>> The board is 4GB, and the memap is 70MB
>> 1G memory --- 14MB mem_map array.
>> So the 4GB has 5 sections, which used 5 * 14MB memory.
>>
>>
> Sorry, 1G memory is 16GB
> 5 sections is 5 * 16 = 80MB
>
> 1G / 4K * (struct page) 64B = 16MB
>
> I don't know why the vmemap dump in pg-tables is 70MB.
>

It may be the PTDUMP code that emits the vmemmap start marker
incorrectly. Could you please double check?

> I add hack code in vmemmap_populate sparse_mem_map_populate.
>
> here is the log:
> sparse_mem_map_populate 188 start ffffffbdc0000000 end ffffffbdc1000000 PAGES_PER_SECTION 40000 nid 0
> vmemmap_populate 549 size 200000 total 200000 addr ffffffbdc0000000
> vmemmap_populate 549 size 200000 total 400000 addr ffffffbdc0200000
> vmemmap_populate 549 size 200000 total 600000 addr ffffffbdc0400000
> vmemmap_populate 549 size 200000 total 800000 addr ffffffbdc0600000
> vmemmap_populate 549 size 200000 total a00000 addr ffffffbdc0800000
> vmemmap_populate 549 size 200000 total c00000 addr ffffffbdc0a00000
> vmemmap_populate 549 size 200000 total e00000 addr ffffffbdc0c00000
> vmemmap_populate 549 size 200000 total 1000000 addr ffffffbdc0e00000
> sparse_mem_map_populate 188 start ffffffbdc1000000 end ffffffbdc2000000 PAGES_PER_SECTION 40000 nid 0
> ...
> sparse_mem_map_populate 188 start ffffffbdc2000000 end ffffffbdc3000000 PAGES_PER_SECTION 40000 nid 0
> sparse_mem_map_populate 188 start ffffffbdc3000000 end ffffffbdc4000000 PAGES_PER_SECTION 40000 nid 0
> sparse_mem_map_populate 188 start ffffffbdc4000000 end ffffffbdc5000000 PAGES_PER_SECTION 40000 nid 0
>
>
> With 4GB memory, it allocated 2MB * 8 * 5 = 80MB.
>> 0 3.5G 4G 4.5G
>> | | | |
>> +-------------------------------------+--------------+-------+
>> | MEM | IO (regs) | MEM |
>> +-------------------------------------+--------------+-------+
>
> 4GB memory ,5 sections. 80MB mem_map allocated.
>

I suppose using

#define SECTION_SIZE_BITS 29

in arch/arm64/include/asm/sparsemem.h would get rid of the overhead
completely in this particular case. Could you confirm, please?

@Will: is the rationale for the default value of 30 for
SECTION_SIZE_BITS documented anywhere? Compared to other
architectures, it seems on the high side, but I did notice that 64k
granule kernels require at least 28 in order not to trigger the
following assertion

include/linux/mmzone.h:1029:2: error: #error Allocator MAX_ORDER
exceeds SECTION_SIZE

2016-04-11 11:17:55

by Xishi Qiu

[permalink] [raw]

Subject: Re: [PATCH 2/2] arm64: mm: make pfn always valid with flat memory

On 2016/4/5 16:22, Chen Feng wrote:

> Make the pfn always valid when using flat memory.
> If the reserved memory is not align to memblock-size,
> there will be holes in zone.
>
> This patch makes the memory in buddy always in the
> array of mem-map.
>
> Signed-off-by: Chen Feng <[email protected]>
> Signed-off-by: Fu Jun <[email protected]>
> ---
> arch/arm64/mm/init.c | 7 ++++---
> 1 file changed, 4 insertions(+), 3 deletions(-)
>
> diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
> index ea989d8..0e1d5b7 100644
> --- a/arch/arm64/mm/init.c
> +++ b/arch/arm64/mm/init.c
> @@ -306,7 +306,8 @@ static void __init free_unused_memmap(void)

How about let free_unused_memmap() support for CONFIG_SPARSEMEM_VMEMMAP?

Thanks,
Xishi Qiu

> struct memblock_region *reg;
>
> for_each_memblock(memory, reg) {
> - start = __phys_to_pfn(reg->base);
> + start = round_down(__phys_to_pfn(reg->base),
> + MAX_ORDER_NR_PAGES);
>
> #ifdef CONFIG_SPARSEMEM
> /*
> @@ -327,8 +328,8 @@ static void __init free_unused_memmap(void)
> * memmap entries are valid from the bank end aligned to
> * MAX_ORDER_NR_PAGES.
> */
> - prev_end = ALIGN(__phys_to_pfn(reg->base + reg->size),
> - MAX_ORDER_NR_PAGES);
> + prev_end = round_up(__phys_to_pfn(reg->base + reg->size),
> + MAX_ORDER_NR_PAGES);
> }
>
> #ifdef CONFIG_SPARSEMEM

2016-04-11 18:11:31

by Laura Abbott

[permalink] [raw]

Subject: Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64

On 04/11/2016 03:57 AM, Chen Feng wrote:
> Hi Will,
>
> On 2016/4/11 18:40, Will Deacon wrote:
>> On Mon, Apr 11, 2016 at 12:31:53PM +0200, Ard Biesheuvel wrote:
>>> On 11 April 2016 at 11:59, Chen Feng <[email protected]> wrote:
>>>> Please see the pg-tables below.
>>>>
>>>>
>>>> With sparse and vmemmap enable.
>>>>
>>>> ---[ vmemmap start ]---
>>>> 0xffffffbdc0200000-0xffffffbdc4800000 70M RW NX SHD AF UXN MEM/NORMAL
>>>> ---[ vmemmap end ]---
>>>>
>>>
>>> OK, I see what you mean now. Sorry for taking so long to catch up.
>>>
>>>> The board is 4GB, and the memap is 70MB
>>>> 1G memory --- 14MB mem_map array.
>>>
>>> No, this is incorrect. 1 GB corresponds with 16 MB worth of struct
>>> pages assuming sizeof(struct page) == 64
>>>
>>> So you are losing 6 MB to rounding here, which I agree is significant.
>>> I wonder if it makes sense to use a lower value for SECTION_SIZE_BITS
>>> on 4k pages kernels, but perhaps we're better off asking the opinion
>>> of the other cc'ees.
>>
>> You need to be really careful making SECTION_SIZE_BITS smaller because
>> it has a direct correlation on the use of page->flags and you can end up
>> running out of bits fairly easily.
>
> Yes, making SECTION_SIZE_BITS smaller can solve the current situation.
>
> But if the phys-addr is 64GB, but only 4GB ddr is the valid address. And the
>
> holes are not always 512MB.
>
> But, can you tell us why *smaller SIZE makes running out of bits fairly easily*?
>

Think about page tables and TLB pressure. A larger page size can cover the
same memory area with fewer page table entries. The same type of logic applies
to memory sections here as well. If the section size is smaller, you need
more bits to represent the number of sections used. page->flags is a long

In include/linux/mm.h

/* Page flags: | [SECTION] | [NODE] | ZONE | [LAST_CPUPID] | ... | FLAGS | */

and

#if SECTIONS_WIDTH+NODES_WIDTH+ZONES_WIDTH > BITS_PER_LONG - NR_PAGEFLAGS
#error SECTIONS_WIDTH+NODES_WIDTH+ZONES_WIDTH > BITS_PER_LONG - NR_PAGEFLAGS
#endif

So it's a trade off of what can be encoded in an unsigned long.

We're hitting the upper bound on zones as well (see 033fbae988fc 'mm:
ZONE_DEVICE for "device memory"')

> And how about the flat-mem model?
>
>>
>> Will
>>
>> .
>>
>

2016-04-12 14:03:58

by Jungseok Lee

[permalink] [raw]

Subject: Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64

On Apr 11, 2016, at 11:49 AM, Chen Feng wrote:

Dear Chen,

> Hi will,
> Thanks for review.
>
> On 2016/4/7 22:21, Will Deacon wrote:
>> On Tue, Apr 05, 2016 at 04:22:51PM +0800, Chen Feng wrote:
>>> We can reduce the memory allocated at mem-map
>>> by flatmem.
>>>
>>> currently, the default memory-model in arm64 is
>>> sparse memory. The mem-map array is not freed in
>>> this scene. If the physical address is too long,
>>> it will reserved too much memory for the mem-map
>>> array.
>>
>> Can you elaborate a bit more on this, please? We use the vmemmap, so any
>> spaces between memory banks only burns up virtual space. What exactly is
>> the problem you're seeing that makes you want to use flatmem (which is
>> probably unsuitable for the majority of arm64 machines).
>>
> The root cause we want to use flat-mem is the mam_map alloced in sparse-mem
> is not freed.
>
> take a look at here:
> arm64/mm/init.c
> void __init mem_init(void)
> {
> #ifndef CONFIG_SPARSEMEM_VMEMMAP
> free_unused_memmap();
> #endif
> }
>
> Memory layout (3GB)
>
> 0 1.5G 2G 3.5G 4G
> | | | | |
> +--------------+------+---------------+--------------+
> | MEM | hole | MEM | IO (regs) |
> +--------------+------+---------------+--------------+
>
>
> Memory layout (4GB)
>
> 0 3.5G 4G 4.5G
> | | | |
> +-------------------------------------+--------------+-------+
> | MEM | IO (regs) | MEM |
> +-------------------------------------+--------------+-------+
>
> Currently, the sparse memory section is 1GB.
>
> 3GB ddr: the 1.5 ~2G and 3.5 ~ 4G are holes.
> 3GB ddr: the 3.5 ~ 4G and 4.5 ~ 5G are holes.
>
> This will alloc 1G/4K * (struct page) memory for mem_map array.
>
> We want to use flat-mem to reduce the alloced mem_map.
>
> I don't know why you tell us the flatmem is unsuitable for the
> majority of arm64 machines. Can tell us the reason of it?
>
> And we are not going to limit the memdel in arm64, we just want to
> make the flat-mem is an optional item in arm64.

I've experienced the same problem and considered the ideas mentioned
in this thread: flatmem and small SECTION_SIZE_BITS. However, I was
reluctant to post any patch since the issue is highly related to memory
map design document, [1], saying 1GB aligned RAM. The majority of arm64
platforms might follow the information although it is not spec. IOW,
a machine I've played was at least unusual *at that time*, so I didn't
consider upstream work.

[1] http://infocenter.arm.com/help/topic/com.arm.doc.den0001c/DEN0001C_principles_of_arm_memory_maps.pdf

Best Regards
Jungseok Lee

2016-04-12 14:44:43

[permalink] [raw]

Subject: Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64

On Mon, Apr 11, 2016 at 11:40:13AM +0100, Will Deacon wrote:
> On Mon, Apr 11, 2016 at 12:31:53PM +0200, Ard Biesheuvel wrote:
> > On 11 April 2016 at 11:59, Chen Feng <[email protected]> wrote:
> > > Please see the pg-tables below.
> > >
> > >
> > > With sparse and vmemmap enable.
> > >
> > > ---[ vmemmap start ]---
> > > 0xffffffbdc0200000-0xffffffbdc4800000 70M RW NX SHD AF UXN MEM/NORMAL
> > > ---[ vmemmap end ]---
> > >
> >
> > OK, I see what you mean now. Sorry for taking so long to catch up.
> >
> > > The board is 4GB, and the memap is 70MB
> > > 1G memory --- 14MB mem_map array.
> >
> > No, this is incorrect. 1 GB corresponds with 16 MB worth of struct
> > pages assuming sizeof(struct page) == 64
> >
> > So you are losing 6 MB to rounding here, which I agree is significant.
> > I wonder if it makes sense to use a lower value for SECTION_SIZE_BITS
> > on 4k pages kernels, but perhaps we're better off asking the opinion
> > of the other cc'ees.
>
> You need to be really careful making SECTION_SIZE_BITS smaller because
> it has a direct correlation on the use of page->flags and you can end up
> running out of bits fairly easily.

With SPARSEMEM_VMEMMAP, SECTION_SIZE_BITS no longer affect the page
flags since we no longer need to encode the section number in
page->flags.

--
Catalin

2016-04-12 14:59:11

[permalink] [raw]

Subject: Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64

On Mon, Apr 11, 2016 at 12:31:53PM +0200, Ard Biesheuvel wrote:
> On 11 April 2016 at 11:59, Chen Feng <[email protected]> wrote:
> > On 2016/4/11 16:00, Ard Biesheuvel wrote:
> >> On 11 April 2016 at 09:55, Chen Feng <[email protected]> wrote:
> >>> On 2016/4/11 15:35, Ard Biesheuvel wrote:
> >>>> On 11 April 2016 at 04:49, Chen Feng <[email protected]> wrote:
> >>>>> 0 1.5G 2G 3.5G 4G
> >>>>> | | | | |
> >>>>> +--------------+------+---------------+--------------+
> >>>>> | MEM | hole | MEM | IO (regs) |
> >>>>> +--------------+------+---------------+--------------+
> >>> The hole in 1.5G ~ 2G is also allocated mem-map array. And also with the 3.5G ~ 4G.
> >>>
> >>
> >> No, it is not. It may be covered by a section, but that does not mean
> >> sparsemem vmemmap will actually allocate backing for it. The
> >> granularity used by sparsemem vmemmap on a 4k pages kernel is 128 MB,
> >> due to the fact that the backing is performed at PMD granularity.
> >>
> >> Please, could you share the contents of the vmemmap section in
> >> /sys/kernel/debug/kernel_page_tables of your system running with
> >> sparsemem vmemmap enabled? You will need to set CONFIG_ARM64_PTDUMP=y
> >
> > Please see the pg-tables below.
> >
> > With sparse and vmemmap enable.
> >
> > ---[ vmemmap start ]---
> > 0xffffffbdc0200000-0xffffffbdc4800000 70M RW NX SHD AF UXN MEM/NORMAL
> > ---[ vmemmap end ]---
[...]
> > The board is 4GB, and the memap is 70MB
> > 1G memory --- 14MB mem_map array.
>
> No, this is incorrect. 1 GB corresponds with 16 MB worth of struct
> pages assuming sizeof(struct page) == 64
>
> So you are losing 6 MB to rounding here, which I agree is significant.
> I wonder if it makes sense to use a lower value for SECTION_SIZE_BITS
> on 4k pages kernels, but perhaps we're better off asking the opinion
> of the other cc'ees.

IIRC, SECTION_SIZE_BITS was chosen to be the maximum sane value we were
thinking of at the time, assuming that 1GB RAM alignment to be fairly
normal. For the !SPARSEMEM_VMEMMAP case, we should probably be fine with
29 but, as Will said, we need to be careful with the page flags. At a
quick look, we have 25 page flags, 2 bits per zone, NUMA nodes and (48 -
section_size_bits) for the section width. We also need to take into
account 4 more bits for 52-bit PA support (ARMv8.2). So, without NUMA
nodes, we are currently at 49 bits used in page->flags.

For the SPARSEMEM_VMEMMAP case, we can decrease the SECTION_SIZE_BITS in
the MAX_ORDER limit.

An alternative would be to free the vmemmap holes later (but still keep
the vmemmap mapping alias). Yet another option would be to change the
sparse_mem_map_populate() logic get the actual section end rather than
always assuming PAGES_PER_SECTION. But I don't think any of these are
worth if we can safely reduce SECTION_SIZE_BITS.

--
Catalin

2016-04-12 15:00:44

[permalink] [raw]

Subject: Re: [PATCH 2/2] arm64: mm: make pfn always valid with flat memory

On Mon, Apr 11, 2016 at 07:08:38PM +0800, Xishi Qiu wrote:
> On 2016/4/5 16:22, Chen Feng wrote:
>
> > Make the pfn always valid when using flat memory.
> > If the reserved memory is not align to memblock-size,
> > there will be holes in zone.
> >
> > This patch makes the memory in buddy always in the
> > array of mem-map.
> >
> > Signed-off-by: Chen Feng <[email protected]>
> > Signed-off-by: Fu Jun <[email protected]>
> > ---
> > arch/arm64/mm/init.c | 7 ++++---
> > 1 file changed, 4 insertions(+), 3 deletions(-)
> >
> > diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
> > index ea989d8..0e1d5b7 100644
> > --- a/arch/arm64/mm/init.c
> > +++ b/arch/arm64/mm/init.c
> > @@ -306,7 +306,8 @@ static void __init free_unused_memmap(void)
>
> How about let free_unused_memmap() support for CONFIG_SPARSEMEM_VMEMMAP?

We would need extra care to check that the memmap was actually allocated
in the first place.

--
Catalin

2016-04-20 03:19:59

[permalink] [raw]

Subject: Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64

Hi Catalin,

Thanks for your reply.
On 2016/4/12 22:59, Catalin Marinas wrote:
> On Mon, Apr 11, 2016 at 12:31:53PM +0200, Ard Biesheuvel wrote:
>> On 11 April 2016 at 11:59, Chen Feng <[email protected]> wrote:
>>> On 2016/4/11 16:00, Ard Biesheuvel wrote:
>>>> On 11 April 2016 at 09:55, Chen Feng <[email protected]> wrote:
>>>>> On 2016/4/11 15:35, Ard Biesheuvel wrote:
>>>>>> On 11 April 2016 at 04:49, Chen Feng <[email protected]> wrote:
>>>>>>> 0 1.5G 2G 3.5G 4G
>>>>>>> | | | | |
>>>>>>> +--------------+------+---------------+--------------+
>>>>>>> | MEM | hole | MEM | IO (regs) |
>>>>>>> +--------------+------+---------------+--------------+
>>>>> The hole in 1.5G ~ 2G is also allocated mem-map array. And also with the 3.5G ~ 4G.
>>>>>
>>>>
>>>> No, it is not. It may be covered by a section, but that does not mean
>>>> sparsemem vmemmap will actually allocate backing for it. The
>>>> granularity used by sparsemem vmemmap on a 4k pages kernel is 128 MB,
>>>> due to the fact that the backing is performed at PMD granularity.
>>>>
>>>> Please, could you share the contents of the vmemmap section in
>>>> /sys/kernel/debug/kernel_page_tables of your system running with
>>>> sparsemem vmemmap enabled? You will need to set CONFIG_ARM64_PTDUMP=y
>>>
>>> Please see the pg-tables below.
>>>
>>> With sparse and vmemmap enable.
>>>
>>> ---[ vmemmap start ]---
>>> 0xffffffbdc0200000-0xffffffbdc4800000 70M RW NX SHD AF UXN MEM/NORMAL
>>> ---[ vmemmap end ]---
> [...]
>>> The board is 4GB, and the memap is 70MB
>>> 1G memory --- 14MB mem_map array.
>>
>> No, this is incorrect. 1 GB corresponds with 16 MB worth of struct
>> pages assuming sizeof(struct page) == 64
>>
>> So you are losing 6 MB to rounding here, which I agree is significant.
>> I wonder if it makes sense to use a lower value for SECTION_SIZE_BITS
>> on 4k pages kernels, but perhaps we're better off asking the opinion
>> of the other cc'ees.
>
> IIRC, SECTION_SIZE_BITS was chosen to be the maximum sane value we were
> thinking of at the time, assuming that 1GB RAM alignment to be fairly
> normal. For the !SPARSEMEM_VMEMMAP case, we should probably be fine with
> 29 but, as Will said, we need to be careful with the page flags. At a
> quick look, we have 25 page flags, 2 bits per zone, NUMA nodes and (48 -
> section_size_bits) for the section width. We also need to take into
> account 4 more bits for 52-bit PA support (ARMv8.2). So, without NUMA
> nodes, we are currently at 49 bits used in page->flags.
>
> For the SPARSEMEM_VMEMMAP case, we can decrease the SECTION_SIZE_BITS in
> the MAX_ORDER limit.
>
> An alternative would be to free the vmemmap holes later (but still keep
> the vmemmap mapping alias). Yet another option would be to change the
> sparse_mem_map_populate() logic get the actual section end rather than
> always assuming PAGES_PER_SECTION. But I don't think any of these are
> worth if we can safely reduce SECTION_SIZE_BITS.
>
Yes,
currently,it's safely to reduce the SECTION_SIZE_BITS to match this issue
very well.

As I mentioned before, if the memory layout is not like this scene. There
will be not suitable to reduce the SECTION_SIZE_BITS.

We have 4G memory, and 64GB phys address.

There will be a lot of holes in the memory layout.
And the *holes size are not always the same*.

So,it's the reason I want to enable flat-mem in ARM64-ARCH. Why not makes
the flat-mem an optional setting for arm64？

2016-04-20 09:32:43