by Dan Williams

[permalink] [raw]

Subject: Re: [PATCH v2 6/6] ACPI: HMAT: Attach a device for each soft-reserved range

On Wed, Mar 25, 2020 at 4:10 AM Will Deacon <[email protected]> wrote:
>
> On Sun, Mar 22, 2020 at 09:12:58AM -0700, Dan Williams wrote:
> > The hmem enabling in commit 'cf8741ac57ed ("ACPI: NUMA: HMAT: Register
> > "soft reserved" memory as an "hmem" device")' only registered ranges to
> > the hmem driver for each soft-reservation that also appeared in the
> > HMAT. While this is meant to encourage platform firmware to "do the
> > right thing" and publish an HMAT, the corollary is that platforms that
> > fail to publish an accurate HMAT will strand memory from Linux usage.
> > Additionally, the "efi_fake_mem" kernel command line option enabling
> > will strand memory by default without an HMAT.
> >
> > Arrange for "soft reserved" memory that goes unclaimed by HMAT entries
> > to be published as raw resource ranges for the hmem driver to consume.
> >
> > Include a module parameter to disable either this fallback behavior, or
> > the hmat enabling from creating hmem devices. The module parameter
> > requires the hmem device enabling to have unique name in the module
> > namespace: "device_hmem".
> >
> > Rather than mark this x86-only, include an interim phys_to_target_node()
> > implementation for arm64.
> >
> > Cc: Jonathan Cameron <[email protected]>
> > Cc: Brice Goglin <[email protected]>
> > Cc: Ard Biesheuvel <[email protected]>
> > Cc: "Rafael J. Wysocki" <[email protected]>
> > Cc: Jeff Moyer <[email protected]>
> > Cc: Catalin Marinas <[email protected]>
> > Cc: Will Deacon <[email protected]>
> > Signed-off-by: Dan Williams <[email protected]>
> > ---
> > arch/arm64/mm/numa.c | 13 +++++++++++++
> > drivers/dax/Kconfig | 1 +
> > drivers/dax/hmem/Makefile | 3 ++-
> > drivers/dax/hmem/device.c | 33 +++++++++++++++++++++++++++++++++
> > 4 files changed, 49 insertions(+), 1 deletion(-)
> >
> > diff --git a/arch/arm64/mm/numa.c b/arch/arm64/mm/numa.c
> > index 4decf1659700..00fba21eaec0 100644
> > --- a/arch/arm64/mm/numa.c
> > +++ b/arch/arm64/mm/numa.c
> > @@ -468,3 +468,16 @@ int memory_add_physaddr_to_nid(u64 addr)
> > pr_warn("Unknown node for memory at 0x%llx, assuming node 0\n", addr);
> > return 0;
> > }
> > +
> > +/*
> > + * device-dax instance registrations want a valid target-node in case
> > + * they are ever onlined as memory (see hmem_register_device()).
> > + *
> > + * TODO: consult cached numa info
> > + */
> > +int phys_to_target_node(phys_addr_t addr)
> > +{
> > + pr_warn_once("Unknown target node for memory at 0x%llx, assuming node 0\n",
> > + addr);
> > + return 0;
> > +}
>
> Could you implement a generic version of this by iterating over the nodes
> with for_each_{,online_}node() and checking for intersection with
> node_{start,end}_pfn()?

Interesting. The gap is that node_{start,end}_pfn() requires
node_data[] which to date architectures have only setup for online
nodes. Recall a target node is an offline node that could come online
later. However, reworking offline data into node_data could be the
local solution for arm64, I'd just ask that each of the 6
memory-hotplug capable archs go make that opt-in decision themselves.

2020-03-25 22:33:42

by Dan Williams

[permalink] [raw]

Subject: Re: [PATCH v2 4/6] ACPI: HMAT: Refactor hmat_register_target_device to hmem_register_device

On Tue, Mar 24, 2020 at 2:04 PM Dan Williams <[email protected]> wrote:
>
> On Tue, Mar 24, 2020 at 12:41 PM Joao Martins <[email protected]> wrote:
> >
> >
> > On 3/22/20 4:12 PM, Dan Williams wrote:
> > > In preparation for exposing "Soft Reserved" memory ranges without an
> > > HMAT, move the hmem device registration to its own compilation unit and
> > > make the implementation generic.
> > >
> > > The generic implementation drops usage acpi_map_pxm_to_online_node()
> > > that was translating ACPI proximity domain values and instead relies on
> > > numa_map_to_online_node() to determine the numa node for the device.
> > >
> > > Cc: "Rafael J. Wysocki" <[email protected]>
> > > Link: https://lore.kernel.org/r/158318761484.2216124.2049322072599482736.stgit@dwillia2-desk3.amr.corp.intel.com
> > > Signed-off-by: Dan Williams <[email protected]>
> > > ---
> > > drivers/acpi/numa/hmat.c | 68 ++++-----------------------------------------
> > > drivers/dax/Kconfig | 4 +++
> > > drivers/dax/Makefile | 3 +-
> > > drivers/dax/hmem.c | 56 -------------------------------------
> > > drivers/dax/hmem/Makefile | 5 +++
> > > drivers/dax/hmem/device.c | 64 ++++++++++++++++++++++++++++++++++++++++++
> > > drivers/dax/hmem/hmem.c | 56 +++++++++++++++++++++++++++++++++++++
> > > include/linux/dax.h | 8 +++++
> > > 8 files changed, 144 insertions(+), 120 deletions(-)
> > > delete mode 100644 drivers/dax/hmem.c
> > > create mode 100644 drivers/dax/hmem/Makefile
> > > create mode 100644 drivers/dax/hmem/device.c
> > > create mode 100644 drivers/dax/hmem/hmem.c
> > >
> > > diff --git a/drivers/acpi/numa/hmat.c b/drivers/acpi/numa/hmat.c
> > > index a12e36a12618..134bcb40b2af 100644
> > > --- a/drivers/acpi/numa/hmat.c
> > > +++ b/drivers/acpi/numa/hmat.c
> > > @@ -24,6 +24,7 @@
> > > #include <linux/mutex.h>
> > > #include <linux/node.h>
> > > #include <linux/sysfs.h>
> > > +#include <linux/dax.h>
> > >
> > > static u8 hmat_revision;
> > > static int hmat_disable __initdata;
> > > @@ -640,66 +641,6 @@ static void hmat_register_target_perf(struct memory_target *target)
> > > node_set_perf_attrs(mem_nid, &target->hmem_attrs, 0);
> > > }
> > >
> > > -static void hmat_register_target_device(struct memory_target *target,
> > ^^^^ int ?
> >
> > > - struct resource *r)
> > > -{
> > > - /* define a clean / non-busy resource for the platform device */
> > > - struct resource res = {
> > > - .start = r->start,
> > > - .end = r->end,
> > > - .flags = IORESOURCE_MEM,
> > > - };
> > > - struct platform_device *pdev;
> > > - struct memregion_info info;
> > > - int rc, id;
> > > -
> > > - rc = region_intersects(res.start, resource_size(&res), IORESOURCE_MEM,
> > > - IORES_DESC_SOFT_RESERVED);
> > > - if (rc != REGION_INTERSECTS)
> > > - return;
> > ^ return -ENXIO;
> >
> > > -
> > > - id = memregion_alloc(GFP_KERNEL);
> > > - if (id < 0) {
> > > - pr_err("memregion allocation failure for %pr\n", &res);
> > > - return;
> > ^ return -ENOMEM;
> >
> > > - }
> > > -
> > > - pdev = platform_device_alloc("hmem", id);
> > > - if (!pdev) {
> >
> > rc = -ENOMEM;
> >
> > > - pr_err("hmem device allocation failure for %pr\n", &res);
> > > - goto out_pdev;
> > > - }
> > > -
> > > - pdev->dev.numa_node = acpi_map_pxm_to_online_node(target->memory_pxm);
> > > - info = (struct memregion_info) {
> > > - .target_node = acpi_map_pxm_to_node(target->memory_pxm),
> > > - };
> > > - rc = platform_device_add_data(pdev, &info, sizeof(info));
> > > - if (rc < 0) {
> > > - pr_err("hmem memregion_info allocation failure for %pr\n", &res);
> > > - goto out_pdev;
> > > - }
> > > -
> > > - rc = platform_device_add_resources(pdev, &res, 1);
> > > - if (rc < 0) {
> > > - pr_err("hmem resource allocation failure for %pr\n", &res);
> > > - goto out_resource;
> > > - }
> > > -
> > > - rc = platform_device_add(pdev);
> > > - if (rc < 0) {
> > > - dev_err(&pdev->dev, "device add failed for %pr\n", &res);
> > > - goto out_resource;
> > > - }
> > > -
> > > - return;
> > ^^^^^^ return 0;
> > > -
> > > -out_resource:
> > > - put_device(&pdev->dev);
> > > -out_pdev:
> > > - memregion_free(id);
> >
> > return rc;
> >
> > > -}
> > > -
> > > static void hmat_register_target_devices(struct memory_target *target)
> > > {
> > > struct resource *res;
> > > @@ -711,8 +652,11 @@ static void hmat_register_target_devices(struct memory_target *target)
> > > if (!IS_ENABLED(CONFIG_DEV_DAX_HMEM))
> > > return;
> > >
> > > - for (res = target->memregions.child; res; res = res->sibling)
> > > - hmat_register_target_device(target, res);
> > > + for (res = target->memregions.child; res; res = res->sibling) {
> > > + int target_nid = acpi_map_pxm_to_node(target->memory_pxm);
> > > +
> > > + hmem_register_device(target_nid, res);
> > > + }
> > > }
> > >
> >
> > If it makes sense to propagate error to hmem_register_device (as noted above),
> > then here perhaps a pr_err() if hmem registration fails mainly for reporting
> > purposes?
>
> Yeah, I guess it makes sense to at least log that something went wrong
> if someone wonders where their memory device went. I'll add that
> rework as a follow-on.

Now that I look again hmat_register_target_device() already has
multiple pr_err() indicating that something went wrong. The error code
would not go anywhere useful.