To build this driver as a kernel module, we cannot use
the unexported symbol "max_pfn" to setup enable_4GB.
Use totalram_pages() instead to setup enable_4GB.
Suggested-by: Mike Rapoport <[email protected]>
Signed-off-by: Miles Chen <[email protected]>
Cc: David Hildenbrand <[email protected]>
Cc: Yong Wu <[email protected]>
Cc: Chao Hao <[email protected]>
---
drivers/iommu/mtk_iommu.c | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
diff --git a/drivers/iommu/mtk_iommu.c b/drivers/iommu/mtk_iommu.c
index 5f4d6df59cf6..c2798a6e0e38 100644
--- a/drivers/iommu/mtk_iommu.c
+++ b/drivers/iommu/mtk_iommu.c
@@ -3,7 +3,6 @@
* Copyright (c) 2015-2016 MediaTek Inc.
* Author: Yong Wu <[email protected]>
*/
-#include <linux/memblock.h>
#include <linux/bug.h>
#include <linux/clk.h>
#include <linux/component.h>
@@ -626,8 +625,8 @@ static int mtk_iommu_probe(struct platform_device *pdev)
return -ENOMEM;
data->protect_base = ALIGN(virt_to_phys(protect), MTK_PROTECT_PA_ALIGN);
- /* Whether the current dram is over 4GB */
- data->enable_4GB = !!(max_pfn > (BIT_ULL(32) >> PAGE_SHIFT));
+ /* Whether the current dram is over 4GB, note: DRAM start at 1GB */
+ data->enable_4GB = !!(totalram_pages() > ((SZ_2G + SZ_1G) >> PAGE_SHIFT));
if (!data->plat_data->has_4gb_mode)
data->enable_4GB = false;
--
2.18.0
On 04.06.20 10:01, Miles Chen wrote:
> To build this driver as a kernel module, we cannot use
> the unexported symbol "max_pfn" to setup enable_4GB.
>
> Use totalram_pages() instead to setup enable_4GB.
>
> Suggested-by: Mike Rapoport <[email protected]>
> Signed-off-by: Miles Chen <[email protected]>
> Cc: David Hildenbrand <[email protected]>
> Cc: Yong Wu <[email protected]>
> Cc: Chao Hao <[email protected]>
> ---
> drivers/iommu/mtk_iommu.c | 5 ++---
> 1 file changed, 2 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/iommu/mtk_iommu.c b/drivers/iommu/mtk_iommu.c
> index 5f4d6df59cf6..c2798a6e0e38 100644
> --- a/drivers/iommu/mtk_iommu.c
> +++ b/drivers/iommu/mtk_iommu.c
> @@ -3,7 +3,6 @@
> * Copyright (c) 2015-2016 MediaTek Inc.
> * Author: Yong Wu <[email protected]>
> */
> -#include <linux/memblock.h>
> #include <linux/bug.h>
> #include <linux/clk.h>
> #include <linux/component.h>
> @@ -626,8 +625,8 @@ static int mtk_iommu_probe(struct platform_device *pdev)
> return -ENOMEM;
> data->protect_base = ALIGN(virt_to_phys(protect), MTK_PROTECT_PA_ALIGN);
>
> - /* Whether the current dram is over 4GB */
> - data->enable_4GB = !!(max_pfn > (BIT_ULL(32) >> PAGE_SHIFT));
> + /* Whether the current dram is over 4GB, note: DRAM start at 1GB */
> + data->enable_4GB = !!(totalram_pages() > ((SZ_2G + SZ_1G) >> PAGE_SHIFT));
A similar thing seems to be done by
drivers/media/platform/mtk-vpu/mtk_vpu.c:
vpu->enable_4GB = !!(totalram_pages() > (SZ_2G >> PAGE_SHIFT));
I do wonder if some weird memory hotplug setups might give you false
negatives.
E.g., start a VM with 1GB and hotplug 1GB - it will be hotplugged on
x86-64 above 4GB, turning max_pfn into 5GB. totalram_pages() should
return something < 2GB.
Same can happen when you have a VM and use ballooning to fake-unplug
memory, making totalram_pages() return something < 4GB, but leaving
usable pfns >= 4GB.
but
... I don't know if I understood what "enable_4GB" needs/implies
... I don't know if this is applicable to VMs at all (on real HW such
memory hotplug setups should not exist)
... I don't know how this code would react to memory hotplug, so if the
condition changes after the driver loaded and enable_4GB would
suddenly apply. Again, most probably not relevant on real HW, only
for VMs.
--
Thanks,
David / dhildenb
On Thu, 2020-06-04 at 10:25 +0200, David Hildenbrand wrote:
> On 04.06.20 10:01, Miles Chen wrote:
> > To build this driver as a kernel module, we cannot use
> > the unexported symbol "max_pfn" to setup enable_4GB.
> >
> > Use totalram_pages() instead to setup enable_4GB.
> >
> > Suggested-by: Mike Rapoport <[email protected]>
> > Signed-off-by: Miles Chen <[email protected]>
> > Cc: David Hildenbrand <[email protected]>
> > Cc: Yong Wu <[email protected]>
> > Cc: Chao Hao <[email protected]>
> > ---
> > drivers/iommu/mtk_iommu.c | 5 ++---
> > 1 file changed, 2 insertions(+), 3 deletions(-)
> >
> > diff --git a/drivers/iommu/mtk_iommu.c b/drivers/iommu/mtk_iommu.c
> > index 5f4d6df59cf6..c2798a6e0e38 100644
> > --- a/drivers/iommu/mtk_iommu.c
> > +++ b/drivers/iommu/mtk_iommu.c
> > @@ -3,7 +3,6 @@
> > * Copyright (c) 2015-2016 MediaTek Inc.
> > * Author: Yong Wu <[email protected]>
> > */
> > -#include <linux/memblock.h>
> > #include <linux/bug.h>
> > #include <linux/clk.h>
> > #include <linux/component.h>
> > @@ -626,8 +625,8 @@ static int mtk_iommu_probe(struct platform_device *pdev)
> > return -ENOMEM;
> > data->protect_base = ALIGN(virt_to_phys(protect), MTK_PROTECT_PA_ALIGN);
> >
> > - /* Whether the current dram is over 4GB */
> > - data->enable_4GB = !!(max_pfn > (BIT_ULL(32) >> PAGE_SHIFT));
> > + /* Whether the current dram is over 4GB, note: DRAM start at 1GB */
> > + data->enable_4GB = !!(totalram_pages() > ((SZ_2G + SZ_1G) >> PAGE_SHIFT));
>
> A similar thing seems to be done by
> drivers/media/platform/mtk-vpu/mtk_vpu.c:
> vpu->enable_4GB = !!(totalram_pages() > (SZ_2G >> PAGE_SHIFT));
>
> I do wonder if some weird memory hotplug setups might give you false
> negatives.
>
> E.g., start a VM with 1GB and hotplug 1GB - it will be hotplugged on
> x86-64 above 4GB, turning max_pfn into 5GB. totalram_pages() should
> return something < 2GB.
>
> Same can happen when you have a VM and use ballooning to fake-unplug
> memory, making totalram_pages() return something < 4GB, but leaving
> usable pfns >= 4GB
Yes. Yingjoe also told me that this patch is not correct.
Thanks for pointing this out. totalram_pages() does not work
for some cases:
e.g., DRAM start @0x4000_0000 and DRAM size is 0x1_0000_0000 but we
reserve large amount of memory, which makes totalram_pages() < 3GB but
it is possible to allocate a pfn >= 4GB.
I will discuss this internally.
Miles
> .
>
> but
> ... I don't know if I understood what "enable_4GB" needs/implies
> ... I don't know if this is applicable to VMs
> at all (on real HW such
> memory hotplug setups should not exist)
> ... I don't know how this code would react to memory hotplug, so if the
> condition changes after the driver loaded and enable_4GB would
> suddenly apply. Again, most probably not relevant on real HW, only
> for VMs.
>
On 04.06.20 11:49, Miles Chen wrote:
> On Thu, 2020-06-04 at 10:25 +0200, David Hildenbrand wrote:
>> On 04.06.20 10:01, Miles Chen wrote:
>>> To build this driver as a kernel module, we cannot use
>>> the unexported symbol "max_pfn" to setup enable_4GB.
>>>
>>> Use totalram_pages() instead to setup enable_4GB.
>>>
>>> Suggested-by: Mike Rapoport <[email protected]>
>>> Signed-off-by: Miles Chen <[email protected]>
>>> Cc: David Hildenbrand <[email protected]>
>>> Cc: Yong Wu <[email protected]>
>>> Cc: Chao Hao <[email protected]>
>>> ---
>>> drivers/iommu/mtk_iommu.c | 5 ++---
>>> 1 file changed, 2 insertions(+), 3 deletions(-)
>>>
>>> diff --git a/drivers/iommu/mtk_iommu.c b/drivers/iommu/mtk_iommu.c
>>> index 5f4d6df59cf6..c2798a6e0e38 100644
>>> --- a/drivers/iommu/mtk_iommu.c
>>> +++ b/drivers/iommu/mtk_iommu.c
>>> @@ -3,7 +3,6 @@
>>> * Copyright (c) 2015-2016 MediaTek Inc.
>>> * Author: Yong Wu <[email protected]>
>>> */
>>> -#include <linux/memblock.h>
>>> #include <linux/bug.h>
>>> #include <linux/clk.h>
>>> #include <linux/component.h>
>>> @@ -626,8 +625,8 @@ static int mtk_iommu_probe(struct platform_device *pdev)
>>> return -ENOMEM;
>>> data->protect_base = ALIGN(virt_to_phys(protect), MTK_PROTECT_PA_ALIGN);
>>>
>>> - /* Whether the current dram is over 4GB */
>>> - data->enable_4GB = !!(max_pfn > (BIT_ULL(32) >> PAGE_SHIFT));
>>> + /* Whether the current dram is over 4GB, note: DRAM start at 1GB */
>>> + data->enable_4GB = !!(totalram_pages() > ((SZ_2G + SZ_1G) >> PAGE_SHIFT));
>>
>> A similar thing seems to be done by
>> drivers/media/platform/mtk-vpu/mtk_vpu.c:
>> vpu->enable_4GB = !!(totalram_pages() > (SZ_2G >> PAGE_SHIFT));
>>
>> I do wonder if some weird memory hotplug setups might give you false
>> negatives.
>>
>> E.g., start a VM with 1GB and hotplug 1GB - it will be hotplugged on
>> x86-64 above 4GB, turning max_pfn into 5GB. totalram_pages() should
>> return something < 2GB.
>>
>> Same can happen when you have a VM and use ballooning to fake-unplug
>> memory, making totalram_pages() return something < 4GB, but leaving
>> usable pfns >= 4GB
>
> Yes. Yingjoe also told me that this patch is not correct.
>
> Thanks for pointing this out. totalram_pages() does not work
> for some cases:
>
Just a thought: If memory hotplug is applicable as well, you might
either want to always assume data->enable_4GB, or handle memory hotplug
events from the memory notifier, when new memory gets onlined (not sure
how tricky that is).
--
Thanks,
David / dhildenb
On Thu, Jun 04, 2020 at 01:32:40PM +0200, David Hildenbrand wrote:
> Just a thought: If memory hotplug is applicable as well, you might
> either want to always assume data->enable_4GB, or handle memory hotplug
> events from the memory notifier, when new memory gets onlined (not sure
> how tricky that is).
We probably want a highest_pfn_possible() or similar API instead of
having drivers poking into random VM internals.
On 04.06.20 17:06, Christoph Hellwig wrote:
> On Thu, Jun 04, 2020 at 01:32:40PM +0200, David Hildenbrand wrote:
>> Just a thought: If memory hotplug is applicable as well, you might
>> either want to always assume data->enable_4GB, or handle memory hotplug
>> events from the memory notifier, when new memory gets onlined (not sure
>> how tricky that is).
>
> We probably want a highest_pfn_possible() or similar API instead of
> having drivers poking into random VM internals.
Well, memory notifiers are a reasonable api used accross the kernel to
get notified when new memory is onlined to the buddy that could be used
for allocations.
highest_pfn_possible() would have to default to something linked to
MAX_PHYSMEM_BITS whenever memory hotplug is configured, I am not sure
how helpful that is (IOW, you can just default to enable_4GB=true in
that case instead in most cases).
--
Thanks,
David / dhildenb
On 04.06.20 17:27, David Hildenbrand wrote:
> On 04.06.20 17:06, Christoph Hellwig wrote:
>> On Thu, Jun 04, 2020 at 01:32:40PM +0200, David Hildenbrand wrote:
>>> Just a thought: If memory hotplug is applicable as well, you might
>>> either want to always assume data->enable_4GB, or handle memory hotplug
>>> events from the memory notifier, when new memory gets onlined (not sure
>>> how tricky that is).
>>
>> We probably want a highest_pfn_possible() or similar API instead of
>> having drivers poking into random VM internals.
>
> Well, memory notifiers are a reasonable api used accross the kernel to
> get notified when new memory is onlined to the buddy that could be used
> for allocations.
>
> highest_pfn_possible() would have to default to something linked to
> MAX_PHYSMEM_BITS whenever memory hotplug is configured, I am not sure
> how helpful that is (IOW, you can just default to enable_4GB=true in
> that case instead in most cases).
Correction: At least on x86-64 we have max_possible_pfn, which will
consult the ACPI SRAT table to figure out the maximum possible PFN.
(Without SRAT, max_possible_pfn will point at the end of initial boot
memory and not consider hotplug memory - something that e.g., newer QEMU
versions work around by creating SRAT tables if memory hotplug might be
possible, even if there is no actual NUMA configuration).
pci-swiotlb.c similarly relies on that to figure out if there are any
!DMA addresses to handle.
--
Thanks,
David / dhildenb