Changes since v4 [1]:
- Rebased on
device-dax-move-instance-creation-parameters-to-struct-dev_dax_data.patch
in -mm [2]. I.e. patches that did not need fixups from v4 are not
included.
- Folded all fixes
- Replaced "device-dax: kill dax_kmem_res" with:
device-dax/kmem: introduce dax_kmem_range()
device-dax/kmem: move resource name tracking to drvdata
device-dax/kmem: replace release_resource() with release_mem_region()
...to address David's request to make those cleanups easier to review.
Note that I dropped changes to how IORESOURCE_BUSY is manipulated since
David and I are still debating the best way forward there.
- Broke out some of dax-bus reworks in "device-dax: introduce 'seed'
devices" to a new "device-dax: introduce 'struct dev_dax' typed-driver
operations"
- Added a conversion of xen_alloc_unallocated_pages() from pgmap.res to
pgmap.range. I found it odd that there is no corresponding
memunmap_pages() triggered by xen_free_unallocated_pages()?
- Not included, a conversion of virtio_fs to use pgmap.range for its new
usage of devm_memremap_pages(). It appears the virtio_fs changes are
merged after -mm? My mental model of -mm was that it applies on top of
linux-next? In any event, Vivek, you will need to coordinate a
conversion to pgmap.range for the virtio_fs dax-support merge. Maybe
that should go through Andrew as well?
- Lowercase all the subject lines per akpm's preference
- Received a 0day robot build-success notification over 122 configs
- Thanks to Joao for looking after this set while I was out.
[1]: http://lore.kernel.org/r/159625229779.3040297.11363509688097221416.stgit@dwillia2-desk3.amr.corp.intel.com
[2]: https://ozlabs.org/~akpm/mmots/broken-out/device-dax-move-instance-creation-parameters-to-struct-dev_dax_data.patch
---
Andrew, this series replaces
device-dax-make-pgmap-optional-for-instance-creation.patch
...through...
dax-hmem-introduce-dax_hmemregion_idle-parameter.patch
...in your stack.
Let me know if there is a different / preferred way to refresh a bulk of
patches in your queue when only a subset need updates.
---
The device-dax facility allows an address range to be directly mapped
through a chardev, or optionally hotplugged to the core kernel page
allocator as System-RAM. It is the mechanism for converting persistent
memory (pmem) to be used as another volatile memory pool i.e. the
current Memory Tiering hot topic on linux-mm.
In the case of pmem the nvdimm-namespace-label mechanism can sub-divide
it, but that labeling mechanism is not available / applicable to
soft-reserved ("EFI specific purpose") memory [3]. This series provides
a sysfs-mechanism for the daxctl utility to enable provisioning of
volatile-soft-reserved memory ranges.
The motivations for this facility are:
1/ Allow performance differentiated memory ranges to be split between
kernel-managed and directly-accessed use cases.
2/ Allow physical memory to be provisioned along performance relevant
address boundaries. For example, divide a memory-side cache [4] along
cache-color boundaries.
3/ Parcel out soft-reserved memory to VMs using device-dax as a security
/ permissions boundary [5]. Specifically I have seen people (ab)using
memmap=nn!ss (mark System-RAM as Persistent Memory) just to get the
device-dax interface on custom address ranges. A follow-on for the VM
use case is to teach device-dax to dynamically allocate 'struct page' at
runtime to reduce the duplication of 'struct page' space in both the
guest and the host kernel for the same physical pages.
[3]: http://lore.kernel.org/r/157309097008.1579826.12818463304589384434.stgit@dwillia2-desk3.amr.corp.intel.com
[4]: http://lore.kernel.org/r/154899811738.3165233.12325692939590944259.stgit@dwillia2-desk3.amr.corp.intel.com
[5]: http://lore.kernel.org/r/[email protected]
---
Dan Williams (14):
device-dax: make pgmap optional for instance creation
device-dax/kmem: introduce dax_kmem_range()
device-dax/kmem: move resource name tracking to drvdata
device-dax/kmem: replace release_resource() with release_mem_region()
device-dax: add an allocation interface for device-dax instances
device-dax: introduce 'struct dev_dax' typed-driver operations
device-dax: introduce 'seed' devices
drivers/base: make device_find_child_by_name() compatible with sysfs inputs
device-dax: add resize support
mm/memremap_pages: convert to 'struct range'
mm/memremap_pages: support multiple ranges per invocation
device-dax: add dis-contiguous resource support
device-dax: introduce 'mapping' devices
device-dax: add an 'align' attribute
Joao Martins (3):
device-dax: make align a per-device property
dax/hmem: introduce dax_hmem.region_idle parameter
device-dax: add a range mapping allocation attribute
arch/powerpc/kvm/book3s_hv_uvmem.c | 14
drivers/base/core.c | 2
drivers/dax/bus.c | 1039 ++++++++++++++++++++++++++++++--
drivers/dax/bus.h | 11
drivers/dax/dax-private.h | 58 ++
drivers/dax/device.c | 112 ++-
drivers/dax/hmem/hmem.c | 17 -
drivers/dax/kmem.c | 178 +++--
drivers/dax/pmem/compat.c | 2
drivers/dax/pmem/core.c | 14
drivers/gpu/drm/nouveau/nouveau_dmem.c | 15
drivers/nvdimm/badrange.c | 26 -
drivers/nvdimm/claim.c | 13
drivers/nvdimm/nd.h | 3
drivers/nvdimm/pfn_devs.c | 13
drivers/nvdimm/pmem.c | 27 -
drivers/nvdimm/region.c | 21 -
drivers/pci/p2pdma.c | 12
drivers/xen/unpopulated-alloc.c | 45 +
include/linux/memremap.h | 11
include/linux/range.h | 6
lib/test_hmm.c | 15
mm/memremap.c | 299 +++++----
tools/testing/nvdimm/dax-dev.c | 22 -
tools/testing/nvdimm/test/iomap.c | 2
25 files changed, 1557 insertions(+), 420 deletions(-)
base-commit: 6764736525f27a411ba2c0c430aaa2df7375f3ac
Hey Dan,
On 9/25/20 8:11 PM, Dan Williams wrote:
> Changes since v4 [1]:
> - Rebased on
> device-dax-move-instance-creation-parameters-to-struct-dev_dax_data.patch
> in -mm [2]. I.e. patches that did not need fixups from v4 are not
> included.
>
> - Folded all fixes
>
Hmm, perhaps you missed the fixups before the above mentioned patch?
From:
https://www.ozlabs.org/~akpm/mmots/series
under "mm/dax", I am listing those fixups here:
x86-numa-add-nohmat-option-fix.patch
acpi-hmat-refactor-hmat_register_target_device-to-hmem_register_device-fix.patch
mm-memory_hotplug-introduce-default-phys_to_target_node-implementation-fix.patch
acpi-hmat-attach-a-device-for-each-soft-reserved-range-fix.patch
(in https://www.ozlabs.org/~akpm/mmots/broken-out/)
[...]
> ---
>
> Andrew, this series replaces
>
> device-dax-make-pgmap-optional-for-instance-creation.patch
>
> ...through...
>
> dax-hmem-introduce-dax_hmemregion_idle-parameter.patch
>
> ...in your stack.
>
> Let me know if there is a different / preferred way to refresh a bulk of
> patches in your queue when only a subset need updates.
>
On Fri, Sep 25, 2020 at 1:52 PM Joao Martins <[email protected]> wrote:
>
> Hey Dan,
>
> On 9/25/20 8:11 PM, Dan Williams wrote:
> > Changes since v4 [1]:
> > - Rebased on
> > device-dax-move-instance-creation-parameters-to-struct-dev_dax_data.patch
> > in -mm [2]. I.e. patches that did not need fixups from v4 are not
> > included.
> >
> > - Folded all fixes
> >
>
> Hmm, perhaps you missed the fixups before the above mentioned patch?
>
> From:
>
> https://www.ozlabs.org/~akpm/mmots/series
>
> under "mm/dax", I am listing those fixups here:
>
> x86-numa-add-nohmat-option-fix.patch
> acpi-hmat-refactor-hmat_register_target_device-to-hmem_register_device-fix.patch
> mm-memory_hotplug-introduce-default-phys_to_target_node-implementation-fix.patch
> acpi-hmat-attach-a-device-for-each-soft-reserved-range-fix.patch
>
> (in https://www.ozlabs.org/~akpm/mmots/broken-out/)
I left those for Andrew to handle. I actually should have started this
set one more down in his stack because that's where my new changes
start.
;In preparation for a facility that enables dax regions to be sub-divided,
introduce infrastructure to track and allocate region capacity.
The new dax_region/available_size attribute is only enabled for volatile
hmem devices, not pmem devices that are defined by nvdimm namespace
boundaries. This is per Jeff's feedback the last time dynamic device-dax
capacity allocation support was discussed.
Link: https://lore.kernel.org/linux-nvdimm/[email protected]
Link: https://lkml.kernel.org/r/159643101035.4062302.6785857915652647857.stgit@dwillia2-desk3.amr.corp.intel.com
Cc: Vishal Verma <[email protected]>
Cc: Brice Goglin <[email protected]>
Cc: Dave Hansen <[email protected]>
Cc: Dave Jiang <[email protected]>
Cc: David Hildenbrand <[email protected]>
Cc: Ira Weiny <[email protected]>
Cc: Jia He <[email protected]>
Cc: Joao Martins <[email protected]>
Cc: Jonathan Cameron <[email protected]>
Signed-off-by: Dan Williams <[email protected]>
---
drivers/dax/bus.c | 120 +++++++++++++++++++++++++++++++++++++++++----
drivers/dax/bus.h | 7 ++-
drivers/dax/dax-private.h | 2 -
drivers/dax/hmem/hmem.c | 7 +--
drivers/dax/pmem/core.c | 8 +--
5 files changed, 121 insertions(+), 23 deletions(-)
diff --git a/drivers/dax/bus.c b/drivers/dax/bus.c
index 96bd64ba95a5..0a48ce378686 100644
--- a/drivers/dax/bus.c
+++ b/drivers/dax/bus.c
@@ -130,6 +130,11 @@ ATTRIBUTE_GROUPS(dax_drv);
static int dax_bus_match(struct device *dev, struct device_driver *drv);
+static bool is_static(struct dax_region *dax_region)
+{
+ return (dax_region->res.flags & IORESOURCE_DAX_STATIC) != 0;
+}
+
static struct bus_type dax_bus_type = {
.name = "dax",
.uevent = dax_bus_uevent,
@@ -185,7 +190,48 @@ static ssize_t align_show(struct device *dev,
}
static DEVICE_ATTR_RO(align);
+#define for_each_dax_region_resource(dax_region, res) \
+ for (res = (dax_region)->res.child; res; res = res->sibling)
+
+static unsigned long long dax_region_avail_size(struct dax_region *dax_region)
+{
+ resource_size_t size = resource_size(&dax_region->res);
+ struct resource *res;
+
+ device_lock_assert(dax_region->dev);
+
+ for_each_dax_region_resource(dax_region, res)
+ size -= resource_size(res);
+ return size;
+}
+
+static ssize_t available_size_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct dax_region *dax_region = dev_get_drvdata(dev);
+ unsigned long long size;
+
+ device_lock(dev);
+ size = dax_region_avail_size(dax_region);
+ device_unlock(dev);
+
+ return sprintf(buf, "%llu\n", size);
+}
+static DEVICE_ATTR_RO(available_size);
+
+static umode_t dax_region_visible(struct kobject *kobj, struct attribute *a,
+ int n)
+{
+ struct device *dev = container_of(kobj, struct device, kobj);
+ struct dax_region *dax_region = dev_get_drvdata(dev);
+
+ if (is_static(dax_region) && a == &dev_attr_available_size.attr)
+ return 0;
+ return a->mode;
+}
+
static struct attribute *dax_region_attributes[] = {
+ &dev_attr_available_size.attr,
&dev_attr_region_size.attr,
&dev_attr_align.attr,
&dev_attr_id.attr,
@@ -195,6 +241,7 @@ static struct attribute *dax_region_attributes[] = {
static const struct attribute_group dax_region_attribute_group = {
.name = "dax_region",
.attrs = dax_region_attributes,
+ .is_visible = dax_region_visible,
};
static const struct attribute_group *dax_region_attribute_groups[] = {
@@ -226,7 +273,8 @@ static void dax_region_unregister(void *region)
}
struct dax_region *alloc_dax_region(struct device *parent, int region_id,
- struct resource *res, int target_node, unsigned int align)
+ struct resource *res, int target_node, unsigned int align,
+ unsigned long flags)
{
struct dax_region *dax_region;
@@ -249,12 +297,17 @@ struct dax_region *alloc_dax_region(struct device *parent, int region_id,
return NULL;
dev_set_drvdata(parent, dax_region);
- memcpy(&dax_region->res, res, sizeof(*res));
kref_init(&dax_region->kref);
dax_region->id = region_id;
dax_region->align = align;
dax_region->dev = parent;
dax_region->target_node = target_node;
+ dax_region->res = (struct resource) {
+ .start = res->start,
+ .end = res->end,
+ .flags = IORESOURCE_MEM | flags,
+ };
+
if (sysfs_create_groups(&parent->kobj, dax_region_attribute_groups)) {
kfree(dax_region);
return NULL;
@@ -267,6 +320,32 @@ struct dax_region *alloc_dax_region(struct device *parent, int region_id,
}
EXPORT_SYMBOL_GPL(alloc_dax_region);
+static int alloc_dev_dax_range(struct dev_dax *dev_dax, resource_size_t size)
+{
+ struct dax_region *dax_region = dev_dax->region;
+ struct resource *res = &dax_region->res;
+ struct device *dev = &dev_dax->dev;
+ struct resource *alloc;
+
+ device_lock_assert(dax_region->dev);
+
+ /* TODO: handle multiple allocations per region */
+ if (res->child)
+ return -ENOMEM;
+
+ alloc = __request_region(res, res->start, size, dev_name(dev), 0);
+
+ if (!alloc)
+ return -ENOMEM;
+
+ dev_dax->range = (struct range) {
+ .start = alloc->start,
+ .end = alloc->end,
+ };
+
+ return 0;
+}
+
static ssize_t size_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
@@ -361,6 +440,15 @@ void kill_dev_dax(struct dev_dax *dev_dax)
}
EXPORT_SYMBOL_GPL(kill_dev_dax);
+static void free_dev_dax_range(struct dev_dax *dev_dax)
+{
+ struct dax_region *dax_region = dev_dax->region;
+ struct range *range = &dev_dax->range;
+
+ device_lock_assert(dax_region->dev);
+ __release_region(&dax_region->res, range->start, range_len(range));
+}
+
static void dev_dax_release(struct device *dev)
{
struct dev_dax *dev_dax = to_dev_dax(dev);
@@ -385,6 +473,7 @@ static void unregister_dev_dax(void *dev)
dev_dbg(dev, "%s\n", __func__);
kill_dev_dax(dev_dax);
+ free_dev_dax_range(dev_dax);
device_del(dev);
put_device(dev);
}
@@ -397,7 +486,7 @@ struct dev_dax *devm_create_dev_dax(struct dev_dax_data *data)
struct dev_dax *dev_dax;
struct inode *inode;
struct device *dev;
- int rc = -ENOMEM;
+ int rc;
if (data->id < 0)
return ERR_PTR(-EINVAL);
@@ -406,11 +495,25 @@ struct dev_dax *devm_create_dev_dax(struct dev_dax_data *data)
if (!dev_dax)
return ERR_PTR(-ENOMEM);
+ dev_dax->region = dax_region;
+ dev = &dev_dax->dev;
+ device_initialize(dev);
+ dev_set_name(dev, "dax%d.%d", dax_region->id, data->id);
+
+ rc = alloc_dev_dax_range(dev_dax, data->size);
+ if (rc)
+ goto err_range;
+
if (data->pgmap) {
+ dev_WARN_ONCE(parent, !is_static(dax_region),
+ "custom dev_pagemap requires a static dax_region\n");
+
dev_dax->pgmap = kmemdup(data->pgmap,
sizeof(struct dev_pagemap), GFP_KERNEL);
- if (!dev_dax->pgmap)
+ if (!dev_dax->pgmap) {
+ rc = -ENOMEM;
goto err_pgmap;
+ }
}
/*
@@ -427,12 +530,7 @@ struct dev_dax *devm_create_dev_dax(struct dev_dax_data *data)
kill_dax(dax_dev);
/* from here on we're committed to teardown via dev_dax_release() */
- dev = &dev_dax->dev;
- device_initialize(dev);
-
dev_dax->dax_dev = dax_dev;
- dev_dax->region = dax_region;
- dev_dax->range = data->range;
dev_dax->target_node = dax_region->target_node;
kref_get(&dax_region->kref);
@@ -444,7 +542,6 @@ struct dev_dax *devm_create_dev_dax(struct dev_dax_data *data)
dev->class = dax_class;
dev->parent = parent;
dev->type = &dev_dax_type;
- dev_set_name(dev, "dax%d.%d", dax_region->id, data->id);
rc = device_add(dev);
if (rc) {
@@ -458,9 +555,12 @@ struct dev_dax *devm_create_dev_dax(struct dev_dax_data *data)
return ERR_PTR(rc);
return dev_dax;
+
err_alloc_dax:
kfree(dev_dax->pgmap);
err_pgmap:
+ free_dev_dax_range(dev_dax);
+err_range:
kfree(dev_dax);
return ERR_PTR(rc);
diff --git a/drivers/dax/bus.h b/drivers/dax/bus.h
index 4aeb36da83a4..44592a8cac0f 100644
--- a/drivers/dax/bus.h
+++ b/drivers/dax/bus.h
@@ -10,8 +10,11 @@ struct resource;
struct dax_device;
struct dax_region;
void dax_region_put(struct dax_region *dax_region);
+
+#define IORESOURCE_DAX_STATIC (1UL << 0)
struct dax_region *alloc_dax_region(struct device *parent, int region_id,
- struct resource *res, int target_node, unsigned int align);
+ struct resource *res, int target_node, unsigned int align,
+ unsigned long flags);
enum dev_dax_subsys {
DEV_DAX_BUS = 0, /* zeroed dev_dax_data picks this by default */
@@ -22,7 +25,7 @@ struct dev_dax_data {
struct dax_region *dax_region;
struct dev_pagemap *pgmap;
enum dev_dax_subsys subsys;
- struct range range;
+ resource_size_t size;
int id;
};
diff --git a/drivers/dax/dax-private.h b/drivers/dax/dax-private.h
index 12a2dbc43b40..99b1273bb232 100644
--- a/drivers/dax/dax-private.h
+++ b/drivers/dax/dax-private.h
@@ -22,7 +22,7 @@ void dax_bus_exit(void);
* @kref: to pin while other agents have a need to do lookups
* @dev: parent device backing this region
* @align: allocation and mapping alignment for child dax devices
- * @res: physical address range of the region
+ * @res: resource tree to track instance allocations
*/
struct dax_region {
int id;
diff --git a/drivers/dax/hmem/hmem.c b/drivers/dax/hmem/hmem.c
index af82d6ba820a..e7b64539e23e 100644
--- a/drivers/dax/hmem/hmem.c
+++ b/drivers/dax/hmem/hmem.c
@@ -20,17 +20,14 @@ static int dax_hmem_probe(struct platform_device *pdev)
mri = dev->platform_data;
dax_region = alloc_dax_region(dev, pdev->id, res, mri->target_node,
- PMD_SIZE);
+ PMD_SIZE, 0);
if (!dax_region)
return -ENOMEM;
data = (struct dev_dax_data) {
.dax_region = dax_region,
.id = 0,
- .range = {
- .start = res->start,
- .end = res->end,
- },
+ .size = resource_size(res),
};
dev_dax = devm_create_dev_dax(&data);
if (IS_ERR(dev_dax))
diff --git a/drivers/dax/pmem/core.c b/drivers/dax/pmem/core.c
index 4fa81d3d2f65..4fe700884338 100644
--- a/drivers/dax/pmem/core.c
+++ b/drivers/dax/pmem/core.c
@@ -54,7 +54,8 @@ struct dev_dax *__dax_pmem_probe(struct device *dev, enum dev_dax_subsys subsys)
memcpy(&res, &pgmap.res, sizeof(res));
res.start += offset;
dax_region = alloc_dax_region(dev, region_id, &res,
- nd_region->target_node, le32_to_cpu(pfn_sb->align));
+ nd_region->target_node, le32_to_cpu(pfn_sb->align),
+ IORESOURCE_DAX_STATIC);
if (!dax_region)
return ERR_PTR(-ENOMEM);
@@ -63,10 +64,7 @@ struct dev_dax *__dax_pmem_probe(struct device *dev, enum dev_dax_subsys subsys)
.id = id,
.pgmap = &pgmap,
.subsys = subsys,
- .range = {
- .start = res.start,
- .end = res.end,
- },
+ .size = resource_size(&res),
};
dev_dax = devm_create_dev_dax(&data);
On 9/25/20 10:01 PM, Dan Williams wrote:
> On Fri, Sep 25, 2020 at 1:52 PM Joao Martins <[email protected]> wrote:
>>
>> Hey Dan,
>>
>> On 9/25/20 8:11 PM, Dan Williams wrote:
>>> Changes since v4 [1]:
>>> - Rebased on
>>> device-dax-move-instance-creation-parameters-to-struct-dev_dax_data.patch
>>> in -mm [2]. I.e. patches that did not need fixups from v4 are not
>>> included.
>>>
>>> - Folded all fixes
>>>
>>
>> Hmm, perhaps you missed the fixups before the above mentioned patch?
>>
>> From:
>>
>> https://www.ozlabs.org/~akpm/mmots/series
>>
>> under "mm/dax", I am listing those fixups here:
>>
>> x86-numa-add-nohmat-option-fix.patch
>> acpi-hmat-refactor-hmat_register_target_device-to-hmem_register_device-fix.patch
>> mm-memory_hotplug-introduce-default-phys_to_target_node-implementation-fix.patch
>> acpi-hmat-attach-a-device-for-each-soft-reserved-range-fix.patch
>>
>> (in https://www.ozlabs.org/~akpm/mmots/broken-out/)
>
> I left those for Andrew to handle. I actually should have started this
> set one more down in his stack because that's where my new changes
> start.
>
Ah, got it!
In preparation for introducing seed devices the dax-bus core needs to be
able to intercept ->probe() and ->remove() operations. Towards that end
arrange for the bus and drivers to switch from raw 'struct device'
driver operations to 'struct dev_dax' typed operations.
Cc: Jason Yan <[email protected]>
Cc: Vishal Verma <[email protected]>
Cc: Brice Goglin <[email protected]>
Cc: Dave Hansen <[email protected]>
Cc: Dave Jiang <[email protected]>
Cc: David Hildenbrand <[email protected]>
Cc: Ira Weiny <[email protected]>
Cc: Jia He <[email protected]>
Cc: Joao Martins <[email protected]>
Cc: Jonathan Cameron <[email protected]>
Reported-by: Hulk Robot <[email protected]>
Signed-off-by: Dan Williams <[email protected]>
---
drivers/dax/bus.c | 18 ++++++++++++++++++
drivers/dax/bus.h | 4 +++-
drivers/dax/device.c | 12 +++++-------
drivers/dax/kmem.c | 18 ++++++++----------
drivers/dax/pmem/compat.c | 2 +-
5 files changed, 35 insertions(+), 19 deletions(-)
diff --git a/drivers/dax/bus.c b/drivers/dax/bus.c
index 0a48ce378686..9549f11ebed0 100644
--- a/drivers/dax/bus.c
+++ b/drivers/dax/bus.c
@@ -135,10 +135,28 @@ static bool is_static(struct dax_region *dax_region)
return (dax_region->res.flags & IORESOURCE_DAX_STATIC) != 0;
}
+static int dax_bus_probe(struct device *dev)
+{
+ struct dax_device_driver *dax_drv = to_dax_drv(dev->driver);
+ struct dev_dax *dev_dax = to_dev_dax(dev);
+
+ return dax_drv->probe(dev_dax);
+}
+
+static int dax_bus_remove(struct device *dev)
+{
+ struct dax_device_driver *dax_drv = to_dax_drv(dev->driver);
+ struct dev_dax *dev_dax = to_dev_dax(dev);
+
+ return dax_drv->remove(dev_dax);
+}
+
static struct bus_type dax_bus_type = {
.name = "dax",
.uevent = dax_bus_uevent,
.match = dax_bus_match,
+ .probe = dax_bus_probe,
+ .remove = dax_bus_remove,
.drv_groups = dax_drv_groups,
};
diff --git a/drivers/dax/bus.h b/drivers/dax/bus.h
index 44592a8cac0f..da27ea70a19a 100644
--- a/drivers/dax/bus.h
+++ b/drivers/dax/bus.h
@@ -38,6 +38,8 @@ struct dax_device_driver {
struct device_driver drv;
struct list_head ids;
int match_always;
+ int (*probe)(struct dev_dax *dev);
+ int (*remove)(struct dev_dax *dev);
};
int __dax_driver_register(struct dax_device_driver *dax_drv,
@@ -48,7 +50,7 @@ void dax_driver_unregister(struct dax_device_driver *dax_drv);
void kill_dev_dax(struct dev_dax *dev_dax);
#if IS_ENABLED(CONFIG_DEV_DAX_PMEM_COMPAT)
-int dev_dax_probe(struct device *dev);
+int dev_dax_probe(struct dev_dax *dev_dax);
#endif
/*
diff --git a/drivers/dax/device.c b/drivers/dax/device.c
index 287cf0a3db23..9833fa83b537 100644
--- a/drivers/dax/device.c
+++ b/drivers/dax/device.c
@@ -392,11 +392,11 @@ static void dev_dax_kill(void *dev_dax)
kill_dev_dax(dev_dax);
}
-int dev_dax_probe(struct device *dev)
+int dev_dax_probe(struct dev_dax *dev_dax)
{
- struct dev_dax *dev_dax = to_dev_dax(dev);
struct dax_device *dax_dev = dev_dax->dax_dev;
struct range *range = &dev_dax->range;
+ struct device *dev = &dev_dax->dev;
struct dev_pagemap *pgmap;
struct inode *inode;
struct cdev *cdev;
@@ -446,17 +446,15 @@ int dev_dax_probe(struct device *dev)
}
EXPORT_SYMBOL_GPL(dev_dax_probe);
-static int dev_dax_remove(struct device *dev)
+static int dev_dax_remove(struct dev_dax *dev_dax)
{
/* all probe actions are unwound by devm */
return 0;
}
static struct dax_device_driver device_dax_driver = {
- .drv = {
- .probe = dev_dax_probe,
- .remove = dev_dax_remove,
- },
+ .probe = dev_dax_probe,
+ .remove = dev_dax_remove,
.match_always = 1,
};
diff --git a/drivers/dax/kmem.c b/drivers/dax/kmem.c
index e56fc688bdc5..c2ac465cc342 100644
--- a/drivers/dax/kmem.c
+++ b/drivers/dax/kmem.c
@@ -29,10 +29,10 @@ static struct range dax_kmem_range(struct dev_dax *dev_dax)
return range;
}
-int dev_dax_kmem_probe(struct device *dev)
+static int dev_dax_kmem_probe(struct dev_dax *dev_dax)
{
- struct dev_dax *dev_dax = to_dev_dax(dev);
struct range range = dax_kmem_range(dev_dax);
+ struct device *dev = &dev_dax->dev;
struct resource *res;
char *res_name;
int numa_node;
@@ -88,12 +88,12 @@ int dev_dax_kmem_probe(struct device *dev)
}
#ifdef CONFIG_MEMORY_HOTREMOVE
-static int dev_dax_kmem_remove(struct device *dev)
+static int dev_dax_kmem_remove(struct dev_dax *dev_dax)
{
- struct dev_dax *dev_dax = to_dev_dax(dev);
+ int rc;
+ struct device *dev = &dev_dax->dev;
struct range range = dax_kmem_range(dev_dax);
const char *res_name = dev_get_drvdata(dev);
- int rc;
/*
* We have one shot for removing memory, if some memory blocks were not
@@ -116,7 +116,7 @@ static int dev_dax_kmem_remove(struct device *dev)
return 0;
}
#else
-static int dev_dax_kmem_remove(struct device *dev)
+static int dev_dax_kmem_remove(struct dev_dax *dev_dax)
{
/*
* Without hotremove purposely leak the request_mem_region() for the
@@ -131,10 +131,8 @@ static int dev_dax_kmem_remove(struct device *dev)
#endif /* CONFIG_MEMORY_HOTREMOVE */
static struct dax_device_driver device_dax_kmem_driver = {
- .drv = {
- .probe = dev_dax_kmem_probe,
- .remove = dev_dax_kmem_remove,
- },
+ .probe = dev_dax_kmem_probe,
+ .remove = dev_dax_kmem_remove,
};
static int __init dax_kmem_init(void)
diff --git a/drivers/dax/pmem/compat.c b/drivers/dax/pmem/compat.c
index d7b15e6f30c5..863c114fd88c 100644
--- a/drivers/dax/pmem/compat.c
+++ b/drivers/dax/pmem/compat.c
@@ -22,7 +22,7 @@ static int dax_pmem_compat_probe(struct device *dev)
return -ENOMEM;
device_lock(&dev_dax->dev);
- rc = dev_dax_probe(&dev_dax->dev);
+ rc = dev_dax_probe(dev_dax);
device_unlock(&dev_dax->dev);
devres_close_group(&dev_dax->dev, dev_dax);
From: Joao Martins <[email protected]>
Introduce @align to struct dev_dax.
When creating a new device, we still initialize to the default dax_region
@align. Child devices belonging to a region may wish to keep a different
alignment property instead of a global region-defined one.
Link: https://lkml.kernel.org/r/159643105377.4062302.4159447829955683131.stgit@dwillia2-desk3.amr.corp.intel.com
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Joao Martins <[email protected]>
Signed-off-by: Dan Williams <[email protected]>
---
drivers/dax/bus.c | 1 +
drivers/dax/dax-private.h | 3 +++
drivers/dax/device.c | 41 +++++++++++++++--------------------------
3 files changed, 19 insertions(+), 26 deletions(-)
diff --git a/drivers/dax/bus.c b/drivers/dax/bus.c
index 005fa3e6d41c..852899084d13 100644
--- a/drivers/dax/bus.c
+++ b/drivers/dax/bus.c
@@ -1218,6 +1218,7 @@ struct dev_dax *devm_create_dev_dax(struct dev_dax_data *data)
dev_dax->dax_dev = dax_dev;
dev_dax->target_node = dax_region->target_node;
+ dev_dax->align = dax_region->align;
ida_init(&dev_dax->ida);
kref_get(&dax_region->kref);
diff --git a/drivers/dax/dax-private.h b/drivers/dax/dax-private.h
index 13780f62b95e..5fd3a26cfcea 100644
--- a/drivers/dax/dax-private.h
+++ b/drivers/dax/dax-private.h
@@ -62,6 +62,7 @@ struct dax_mapping {
struct dev_dax {
struct dax_region *region;
struct dax_device *dax_dev;
+ unsigned int align;
int target_node;
int id;
struct ida ida;
@@ -84,4 +85,6 @@ static inline struct dax_mapping *to_dax_mapping(struct device *dev)
{
return container_of(dev, struct dax_mapping, dev);
}
+
+phys_addr_t dax_pgoff_to_phys(struct dev_dax *dev_dax, pgoff_t pgoff, unsigned long size);
#endif
diff --git a/drivers/dax/device.c b/drivers/dax/device.c
index bf389712a20b..25e0b84a4296 100644
--- a/drivers/dax/device.c
+++ b/drivers/dax/device.c
@@ -17,7 +17,6 @@
static int check_vma(struct dev_dax *dev_dax, struct vm_area_struct *vma,
const char *func)
{
- struct dax_region *dax_region = dev_dax->region;
struct device *dev = &dev_dax->dev;
unsigned long mask;
@@ -32,7 +31,7 @@ static int check_vma(struct dev_dax *dev_dax, struct vm_area_struct *vma,
return -EINVAL;
}
- mask = dax_region->align - 1;
+ mask = dev_dax->align - 1;
if (vma->vm_start & mask || vma->vm_end & mask) {
dev_info_ratelimited(dev,
"%s: %s: fail, unaligned vma (%#lx - %#lx, %#lx)\n",
@@ -78,21 +77,19 @@ static vm_fault_t __dev_dax_pte_fault(struct dev_dax *dev_dax,
struct vm_fault *vmf, pfn_t *pfn)
{
struct device *dev = &dev_dax->dev;
- struct dax_region *dax_region;
phys_addr_t phys;
unsigned int fault_size = PAGE_SIZE;
if (check_vma(dev_dax, vmf->vma, __func__))
return VM_FAULT_SIGBUS;
- dax_region = dev_dax->region;
- if (dax_region->align > PAGE_SIZE) {
+ if (dev_dax->align > PAGE_SIZE) {
dev_dbg(dev, "alignment (%#x) > fault size (%#x)\n",
- dax_region->align, fault_size);
+ dev_dax->align, fault_size);
return VM_FAULT_SIGBUS;
}
- if (fault_size != dax_region->align)
+ if (fault_size != dev_dax->align)
return VM_FAULT_SIGBUS;
phys = dax_pgoff_to_phys(dev_dax, vmf->pgoff, PAGE_SIZE);
@@ -111,7 +108,6 @@ static vm_fault_t __dev_dax_pmd_fault(struct dev_dax *dev_dax,
{
unsigned long pmd_addr = vmf->address & PMD_MASK;
struct device *dev = &dev_dax->dev;
- struct dax_region *dax_region;
phys_addr_t phys;
pgoff_t pgoff;
unsigned int fault_size = PMD_SIZE;
@@ -119,16 +115,15 @@ static vm_fault_t __dev_dax_pmd_fault(struct dev_dax *dev_dax,
if (check_vma(dev_dax, vmf->vma, __func__))
return VM_FAULT_SIGBUS;
- dax_region = dev_dax->region;
- if (dax_region->align > PMD_SIZE) {
+ if (dev_dax->align > PMD_SIZE) {
dev_dbg(dev, "alignment (%#x) > fault size (%#x)\n",
- dax_region->align, fault_size);
+ dev_dax->align, fault_size);
return VM_FAULT_SIGBUS;
}
- if (fault_size < dax_region->align)
+ if (fault_size < dev_dax->align)
return VM_FAULT_SIGBUS;
- else if (fault_size > dax_region->align)
+ else if (fault_size > dev_dax->align)
return VM_FAULT_FALLBACK;
/* if we are outside of the VMA */
@@ -154,7 +149,6 @@ static vm_fault_t __dev_dax_pud_fault(struct dev_dax *dev_dax,
{
unsigned long pud_addr = vmf->address & PUD_MASK;
struct device *dev = &dev_dax->dev;
- struct dax_region *dax_region;
phys_addr_t phys;
pgoff_t pgoff;
unsigned int fault_size = PUD_SIZE;
@@ -163,16 +157,15 @@ static vm_fault_t __dev_dax_pud_fault(struct dev_dax *dev_dax,
if (check_vma(dev_dax, vmf->vma, __func__))
return VM_FAULT_SIGBUS;
- dax_region = dev_dax->region;
- if (dax_region->align > PUD_SIZE) {
+ if (dev_dax->align > PUD_SIZE) {
dev_dbg(dev, "alignment (%#x) > fault size (%#x)\n",
- dax_region->align, fault_size);
+ dev_dax->align, fault_size);
return VM_FAULT_SIGBUS;
}
- if (fault_size < dax_region->align)
+ if (fault_size < dev_dax->align)
return VM_FAULT_SIGBUS;
- else if (fault_size > dax_region->align)
+ else if (fault_size > dev_dax->align)
return VM_FAULT_FALLBACK;
/* if we are outside of the VMA */
@@ -267,9 +260,8 @@ static int dev_dax_split(struct vm_area_struct *vma, unsigned long addr)
{
struct file *filp = vma->vm_file;
struct dev_dax *dev_dax = filp->private_data;
- struct dax_region *dax_region = dev_dax->region;
- if (!IS_ALIGNED(addr, dax_region->align))
+ if (!IS_ALIGNED(addr, dev_dax->align))
return -EINVAL;
return 0;
}
@@ -278,9 +270,8 @@ static unsigned long dev_dax_pagesize(struct vm_area_struct *vma)
{
struct file *filp = vma->vm_file;
struct dev_dax *dev_dax = filp->private_data;
- struct dax_region *dax_region = dev_dax->region;
- return dax_region->align;
+ return dev_dax->align;
}
static const struct vm_operations_struct dax_vm_ops = {
@@ -319,13 +310,11 @@ static unsigned long dax_get_unmapped_area(struct file *filp,
{
unsigned long off, off_end, off_align, len_align, addr_align, align;
struct dev_dax *dev_dax = filp ? filp->private_data : NULL;
- struct dax_region *dax_region;
if (!dev_dax || addr)
goto out;
- dax_region = dev_dax->region;
- align = dax_region->align;
+ align = dev_dax->align;
off = pgoff << PAGE_SHIFT;
off_end = off + len;
off_align = round_up(off, align);
Make the device-dax 'size' attribute writable to allow capacity to be
split between multiple instances in a region. The intended consumers of
this capability are users that want to split a scarce memory resource
between device-dax and System-RAM access, or users that want to have
multiple security domains for a large region.
By default the hmem instance provider allocates an entire region to the
first instance. The process of creating a new instance (assuming a
region-id of 0) is find the region and trigger the 'create' attribute
which yields an empty instance to configure. For example:
cd /sys/bus/dax/devices
echo dax0.0 > dax0.0/driver/unbind
echo $new_size > dax0.0/size
echo 1 > $(readlink -f dax0.0)../dax_region/create
seed=$(cat $(readlink -f dax0.0)../dax_region/seed)
echo $new_size > $seed/size
echo dax0.0 > ../drivers/{device_dax,kmem}/bind
echo dax0.1 > ../drivers/{device_dax,kmem}/bind
Instances can be destroyed by:
echo $device > $(readlink -f $device)../dax_region/delete
Link: https://lkml.kernel.org/r/159643102625.4062302.7431838945566033852.stgit@dwillia2-desk3.amr.corp.intel.com
Cc: Vishal Verma <[email protected]>
Cc: Brice Goglin <[email protected]>
Cc: Dave Hansen <[email protected]>
Cc: Dave Jiang <[email protected]>
Cc: David Hildenbrand <[email protected]>
Cc: Ira Weiny <[email protected]>
Cc: Jia He <[email protected]>
Cc: Joao Martins <[email protected]>
Cc: Jonathan Cameron <[email protected]>
Signed-off-by: Dan Williams <[email protected]>
---
drivers/dax/bus.c | 161 ++++++++++++++++++++++++++++++++++++++++++++++++++---
1 file changed, 152 insertions(+), 9 deletions(-)
diff --git a/drivers/dax/bus.c b/drivers/dax/bus.c
index dce9413a4394..53d07f2f1285 100644
--- a/drivers/dax/bus.c
+++ b/drivers/dax/bus.c
@@ -6,6 +6,7 @@
#include <linux/list.h>
#include <linux/slab.h>
#include <linux/dax.h>
+#include <linux/io.h>
#include "dax-private.h"
#include "bus.h"
@@ -562,7 +563,8 @@ struct dax_region *alloc_dax_region(struct device *parent, int region_id,
}
EXPORT_SYMBOL_GPL(alloc_dax_region);
-static int alloc_dev_dax_range(struct dev_dax *dev_dax, resource_size_t size)
+static int alloc_dev_dax_range(struct dev_dax *dev_dax, u64 start,
+ resource_size_t size)
{
struct dax_region *dax_region = dev_dax->region;
struct resource *res = &dax_region->res;
@@ -580,12 +582,7 @@ static int alloc_dev_dax_range(struct dev_dax *dev_dax, resource_size_t size)
return 0;
}
- /* TODO: handle multiple allocations per region */
- if (res->child)
- return -ENOMEM;
-
- alloc = __request_region(res, res->start, size, dev_name(dev), 0);
-
+ alloc = __request_region(res, start, size, dev_name(dev), 0);
if (!alloc)
return -ENOMEM;
@@ -597,6 +594,29 @@ static int alloc_dev_dax_range(struct dev_dax *dev_dax, resource_size_t size)
return 0;
}
+static int adjust_dev_dax_range(struct dev_dax *dev_dax, struct resource *res, resource_size_t size)
+{
+ struct dax_region *dax_region = dev_dax->region;
+ struct range *range = &dev_dax->range;
+ int rc = 0;
+
+ device_lock_assert(dax_region->dev);
+
+ if (size)
+ rc = adjust_resource(res, range->start, size);
+ else
+ __release_region(&dax_region->res, range->start, range_len(range));
+ if (rc)
+ return rc;
+
+ dev_dax->range = (struct range) {
+ .start = range->start,
+ .end = range->start + size - 1,
+ };
+
+ return 0;
+}
+
static ssize_t size_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
@@ -605,7 +625,127 @@ static ssize_t size_show(struct device *dev,
return sprintf(buf, "%llu\n", size);
}
-static DEVICE_ATTR_RO(size);
+
+static bool alloc_is_aligned(struct dax_region *dax_region,
+ resource_size_t size)
+{
+ /*
+ * The minimum mapping granularity for a device instance is a
+ * single subsection, unless the arch says otherwise.
+ */
+ return IS_ALIGNED(size, max_t(unsigned long, dax_region->align,
+ memremap_compat_align()));
+}
+
+static int dev_dax_shrink(struct dev_dax *dev_dax, resource_size_t size)
+{
+ struct dax_region *dax_region = dev_dax->region;
+ struct range *range = &dev_dax->range;
+ struct resource *res, *adjust = NULL;
+ struct device *dev = &dev_dax->dev;
+
+ for_each_dax_region_resource(dax_region, res)
+ if (strcmp(res->name, dev_name(dev)) == 0
+ && res->start == range->start) {
+ adjust = res;
+ break;
+ }
+
+ if (dev_WARN_ONCE(dev, !adjust, "failed to find matching resource\n"))
+ return -ENXIO;
+ return adjust_dev_dax_range(dev_dax, adjust, size);
+}
+
+static ssize_t dev_dax_resize(struct dax_region *dax_region,
+ struct dev_dax *dev_dax, resource_size_t size)
+{
+ resource_size_t avail = dax_region_avail_size(dax_region), to_alloc;
+ resource_size_t dev_size = range_len(&dev_dax->range);
+ struct resource *region_res = &dax_region->res;
+ struct device *dev = &dev_dax->dev;
+ const char *name = dev_name(dev);
+ struct resource *res, *first;
+
+ if (dev->driver)
+ return -EBUSY;
+ if (size == dev_size)
+ return 0;
+ if (size > dev_size && size - dev_size > avail)
+ return -ENOSPC;
+ if (size < dev_size)
+ return dev_dax_shrink(dev_dax, size);
+
+ to_alloc = size - dev_size;
+ if (dev_WARN_ONCE(dev, !alloc_is_aligned(dax_region, to_alloc),
+ "resize of %pa misaligned\n", &to_alloc))
+ return -ENXIO;
+
+ /*
+ * Expand the device into the unused portion of the region. This
+ * may involve adjusting the end of an existing resource, or
+ * allocating a new resource.
+ */
+ first = region_res->child;
+ if (!first)
+ return alloc_dev_dax_range(dev_dax, dax_region->res.start, to_alloc);
+ for (res = first; to_alloc && res; res = res->sibling) {
+ struct resource *next = res->sibling;
+ resource_size_t free;
+
+ /* space at the beginning of the region */
+ free = 0;
+ if (res == first && res->start > dax_region->res.start)
+ free = res->start - dax_region->res.start;
+ if (free >= to_alloc && dev_size == 0)
+ return alloc_dev_dax_range(dev_dax, dax_region->res.start, to_alloc);
+
+ free = 0;
+ /* space between allocations */
+ if (next && next->start > res->end + 1)
+ free = next->start - res->end + 1;
+
+ /* space at the end of the region */
+ if (free < to_alloc && !next && res->end < region_res->end)
+ free = region_res->end - res->end;
+
+ if (free >= to_alloc && strcmp(name, res->name) == 0)
+ return adjust_dev_dax_range(dev_dax, res, resource_size(res) + to_alloc);
+ else if (free >= to_alloc && dev_size == 0)
+ return alloc_dev_dax_range(dev_dax, res->end + 1, to_alloc);
+ }
+ return -ENOSPC;
+}
+
+static ssize_t size_store(struct device *dev, struct device_attribute *attr,
+ const char *buf, size_t len)
+{
+ ssize_t rc;
+ unsigned long long val;
+ struct dev_dax *dev_dax = to_dev_dax(dev);
+ struct dax_region *dax_region = dev_dax->region;
+
+ rc = kstrtoull(buf, 0, &val);
+ if (rc)
+ return rc;
+
+ if (!alloc_is_aligned(dax_region, val)) {
+ dev_dbg(dev, "%s: size: %lld misaligned\n", __func__, val);
+ return -EINVAL;
+ }
+
+ device_lock(dax_region->dev);
+ if (!dax_region->dev->driver) {
+ device_unlock(dax_region->dev);
+ return -ENXIO;
+ }
+ device_lock(dev);
+ rc = dev_dax_resize(dax_region, dev_dax, val);
+ device_unlock(dev);
+ device_unlock(dax_region->dev);
+
+ return rc == 0 ? len : rc;
+}
+static DEVICE_ATTR_RW(size);
static int dev_dax_target_node(struct dev_dax *dev_dax)
{
@@ -654,11 +794,14 @@ static umode_t dev_dax_visible(struct kobject *kobj, struct attribute *a, int n)
{
struct device *dev = container_of(kobj, struct device, kobj);
struct dev_dax *dev_dax = to_dev_dax(dev);
+ struct dax_region *dax_region = dev_dax->region;
if (a == &dev_attr_target_node.attr && dev_dax_target_node(dev_dax) < 0)
return 0;
if (a == &dev_attr_numa_node.attr && !IS_ENABLED(CONFIG_NUMA))
return 0;
+ if (a == &dev_attr_size.attr && is_static(dax_region))
+ return 0444;
return a->mode;
}
@@ -739,7 +882,7 @@ struct dev_dax *devm_create_dev_dax(struct dev_dax_data *data)
device_initialize(dev);
dev_set_name(dev, "dax%d.%d", dax_region->id, dev_dax->id);
- rc = alloc_dev_dax_range(dev_dax, data->size);
+ rc = alloc_dev_dax_range(dev_dax, dax_region->res.start, data->size);
if (rc)
goto err_range;
Break the requirement that device-dax instances are physically contiguous.
With this constraint removed it allows fragmented available capacity to
be fully allocated.
This capability is useful to mitigate the "noisy neighbor" problem with
memory-side-cache management for virtual machines, or any other scenario
where a platform address boundary also designates a performance boundary.
For example a direct mapped memory side cache might rotate cache colors at
1GB boundaries. With dis-contiguous allocations a device-dax instance
could be configured to contain only 1 cache color.
It also satisfies Joao's use case (see link) for partitioning memory for
exclusive guest access. It allows for a future potential mode where the
host kernel need not allocate 'struct page' capacity up-front.
Link: https://lore.kernel.org/lkml/[email protected]/
Link: https://lkml.kernel.org/r/159643104304.4062302.16561669534797528660.stgit@dwillia2-desk3.amr.corp.intel.com
Reported-by: Joao Martins <[email protected]>
Signed-off-by: Dan Williams <[email protected]>
---
drivers/dax/bus.c | 233 +++++++++++++++++++++++++++++++---------
drivers/dax/dax-private.h | 9 +-
drivers/dax/device.c | 55 ++++++---
drivers/dax/kmem.c | 130 +++++++++++++++-------
tools/testing/nvdimm/dax-dev.c | 20 ++-
5 files changed, 323 insertions(+), 124 deletions(-)
diff --git a/drivers/dax/bus.c b/drivers/dax/bus.c
index 00fa73a8dfb4..06a789aba58a 100644
--- a/drivers/dax/bus.c
+++ b/drivers/dax/bus.c
@@ -136,15 +136,27 @@ static bool is_static(struct dax_region *dax_region)
return (dax_region->res.flags & IORESOURCE_DAX_STATIC) != 0;
}
+static u64 dev_dax_size(struct dev_dax *dev_dax)
+{
+ u64 size = 0;
+ int i;
+
+ device_lock_assert(&dev_dax->dev);
+
+ for (i = 0; i < dev_dax->nr_range; i++)
+ size += range_len(&dev_dax->ranges[i].range);
+
+ return size;
+}
+
static int dax_bus_probe(struct device *dev)
{
struct dax_device_driver *dax_drv = to_dax_drv(dev->driver);
struct dev_dax *dev_dax = to_dev_dax(dev);
struct dax_region *dax_region = dev_dax->region;
- struct range *range = &dev_dax->range;
int rc;
- if (range_len(range) == 0 || dev_dax->id < 0)
+ if (dev_dax_size(dev_dax) == 0 || dev_dax->id < 0)
return -ENXIO;
rc = dax_drv->probe(dev_dax);
@@ -354,15 +366,19 @@ void kill_dev_dax(struct dev_dax *dev_dax)
}
EXPORT_SYMBOL_GPL(kill_dev_dax);
-static void free_dev_dax_range(struct dev_dax *dev_dax)
+static void free_dev_dax_ranges(struct dev_dax *dev_dax)
{
struct dax_region *dax_region = dev_dax->region;
- struct range *range = &dev_dax->range;
+ int i;
device_lock_assert(dax_region->dev);
- if (range_len(range))
+ for (i = 0; i < dev_dax->nr_range; i++) {
+ struct range *range = &dev_dax->ranges[i].range;
+
__release_region(&dax_region->res, range->start,
range_len(range));
+ }
+ dev_dax->nr_range = 0;
}
static void unregister_dev_dax(void *dev)
@@ -372,7 +388,7 @@ static void unregister_dev_dax(void *dev)
dev_dbg(dev, "%s\n", __func__);
kill_dev_dax(dev_dax);
- free_dev_dax_range(dev_dax);
+ free_dev_dax_ranges(dev_dax);
device_del(dev);
put_device(dev);
}
@@ -423,7 +439,7 @@ static ssize_t delete_store(struct device *dev, struct device_attribute *attr,
device_lock(dev);
device_lock(victim);
dev_dax = to_dev_dax(victim);
- if (victim->driver || range_len(&dev_dax->range))
+ if (victim->driver || dev_dax_size(dev_dax))
rc = -EBUSY;
else {
/*
@@ -569,51 +585,86 @@ static int alloc_dev_dax_range(struct dev_dax *dev_dax, u64 start,
struct dax_region *dax_region = dev_dax->region;
struct resource *res = &dax_region->res;
struct device *dev = &dev_dax->dev;
+ struct dev_dax_range *ranges;
+ unsigned long pgoff = 0;
struct resource *alloc;
+ int i;
device_lock_assert(dax_region->dev);
/* handle the seed alloc special case */
if (!size) {
- dev_dax->range = (struct range) {
- .start = res->start,
- .end = res->start - 1,
- };
+ if (dev_WARN_ONCE(dev, dev_dax->nr_range,
+ "0-size allocation must be first\n"))
+ return -EBUSY;
+ /* nr_range == 0 is elsewhere special cased as 0-size device */
return 0;
}
+ ranges = krealloc(dev_dax->ranges, sizeof(*ranges)
+ * (dev_dax->nr_range + 1), GFP_KERNEL);
+ if (!ranges)
+ return -ENOMEM;
+
alloc = __request_region(res, start, size, dev_name(dev), 0);
- if (!alloc)
+ if (!alloc) {
+ /*
+ * If this was an empty set of ranges nothing else
+ * will release @ranges, so do it now.
+ */
+ if (!dev_dax->nr_range) {
+ kfree(ranges);
+ ranges = NULL;
+ }
+ dev_dax->ranges = ranges;
return -ENOMEM;
+ }
- dev_dax->range = (struct range) {
- .start = alloc->start,
- .end = alloc->end,
+ for (i = 0; i < dev_dax->nr_range; i++)
+ pgoff += PHYS_PFN(range_len(&ranges[i].range));
+ dev_dax->ranges = ranges;
+ ranges[dev_dax->nr_range++] = (struct dev_dax_range) {
+ .pgoff = pgoff,
+ .range = {
+ .start = alloc->start,
+ .end = alloc->end,
+ },
};
+ dev_dbg(dev, "alloc range[%d]: %pa:%pa\n", dev_dax->nr_range - 1,
+ &alloc->start, &alloc->end);
+
return 0;
}
static int adjust_dev_dax_range(struct dev_dax *dev_dax, struct resource *res, resource_size_t size)
{
+ int last_range = dev_dax->nr_range - 1;
+ struct dev_dax_range *dax_range = &dev_dax->ranges[last_range];
struct dax_region *dax_region = dev_dax->region;
- struct range *range = &dev_dax->range;
- int rc = 0;
+ bool is_shrink = resource_size(res) > size;
+ struct range *range = &dax_range->range;
+ struct device *dev = &dev_dax->dev;
+ int rc;
device_lock_assert(dax_region->dev);
- if (size)
- rc = adjust_resource(res, range->start, size);
- else
- __release_region(&dax_region->res, range->start, range_len(range));
+ if (dev_WARN_ONCE(dev, !size, "deletion is handled by dev_dax_shrink\n"))
+ return -EINVAL;
+
+ rc = adjust_resource(res, range->start, size);
if (rc)
return rc;
- dev_dax->range = (struct range) {
+ *range = (struct range) {
.start = range->start,
.end = range->start + size - 1,
};
+ dev_dbg(dev, "%s range[%d]: %#llx:%#llx\n", is_shrink ? "shrink" : "extend",
+ last_range, (unsigned long long) range->start,
+ (unsigned long long) range->end);
+
return 0;
}
@@ -621,7 +672,11 @@ static ssize_t size_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
struct dev_dax *dev_dax = to_dev_dax(dev);
- unsigned long long size = range_len(&dev_dax->range);
+ unsigned long long size;
+
+ device_lock(dev);
+ size = dev_dax_size(dev_dax);
+ device_unlock(dev);
return sprintf(buf, "%llu\n", size);
}
@@ -639,32 +694,82 @@ static bool alloc_is_aligned(struct dax_region *dax_region,
static int dev_dax_shrink(struct dev_dax *dev_dax, resource_size_t size)
{
+ resource_size_t to_shrink = dev_dax_size(dev_dax) - size;
struct dax_region *dax_region = dev_dax->region;
- struct range *range = &dev_dax->range;
- struct resource *res, *adjust = NULL;
struct device *dev = &dev_dax->dev;
-
- for_each_dax_region_resource(dax_region, res)
- if (strcmp(res->name, dev_name(dev)) == 0
- && res->start == range->start) {
- adjust = res;
- break;
+ int i;
+
+ for (i = dev_dax->nr_range - 1; i >= 0; i--) {
+ struct range *range = &dev_dax->ranges[i].range;
+ struct resource *adjust = NULL, *res;
+ resource_size_t shrink;
+
+ shrink = min_t(u64, to_shrink, range_len(range));
+ if (shrink >= range_len(range)) {
+ __release_region(&dax_region->res, range->start,
+ range_len(range));
+ dev_dax->nr_range--;
+ dev_dbg(dev, "delete range[%d]: %#llx:%#llx\n", i,
+ (unsigned long long) range->start,
+ (unsigned long long) range->end);
+ to_shrink -= shrink;
+ if (!to_shrink)
+ break;
+ continue;
}
- if (dev_WARN_ONCE(dev, !adjust, "failed to find matching resource\n"))
- return -ENXIO;
- return adjust_dev_dax_range(dev_dax, adjust, size);
+ for_each_dax_region_resource(dax_region, res)
+ if (strcmp(res->name, dev_name(dev)) == 0
+ && res->start == range->start) {
+ adjust = res;
+ break;
+ }
+
+ if (dev_WARN_ONCE(dev, !adjust || i != dev_dax->nr_range - 1,
+ "failed to find matching resource\n"))
+ return -ENXIO;
+ return adjust_dev_dax_range(dev_dax, adjust, range_len(range)
+ - shrink);
+ }
+ return 0;
+}
+
+/*
+ * Only allow adjustments that preserve the relative pgoff of existing
+ * allocations. I.e. the dev_dax->ranges array is ordered by increasing pgoff.
+ */
+static bool adjust_ok(struct dev_dax *dev_dax, struct resource *res)
+{
+ struct dev_dax_range *last;
+ int i;
+
+ if (dev_dax->nr_range == 0)
+ return false;
+ if (strcmp(res->name, dev_name(&dev_dax->dev)) != 0)
+ return false;
+ last = &dev_dax->ranges[dev_dax->nr_range - 1];
+ if (last->range.start != res->start || last->range.end != res->end)
+ return false;
+ for (i = 0; i < dev_dax->nr_range - 1; i++) {
+ struct dev_dax_range *dax_range = &dev_dax->ranges[i];
+
+ if (dax_range->pgoff > last->pgoff)
+ return false;
+ }
+
+ return true;
}
static ssize_t dev_dax_resize(struct dax_region *dax_region,
struct dev_dax *dev_dax, resource_size_t size)
{
resource_size_t avail = dax_region_avail_size(dax_region), to_alloc;
- resource_size_t dev_size = range_len(&dev_dax->range);
+ resource_size_t dev_size = dev_dax_size(dev_dax);
struct resource *region_res = &dax_region->res;
struct device *dev = &dev_dax->dev;
- const char *name = dev_name(dev);
struct resource *res, *first;
+ resource_size_t alloc = 0;
+ int rc;
if (dev->driver)
return -EBUSY;
@@ -685,35 +790,47 @@ static ssize_t dev_dax_resize(struct dax_region *dax_region,
* may involve adjusting the end of an existing resource, or
* allocating a new resource.
*/
+retry:
first = region_res->child;
if (!first)
return alloc_dev_dax_range(dev_dax, dax_region->res.start, to_alloc);
- for (res = first; to_alloc && res; res = res->sibling) {
+
+ rc = -ENOSPC;
+ for (res = first; res; res = res->sibling) {
struct resource *next = res->sibling;
- resource_size_t free;
/* space at the beginning of the region */
- free = 0;
- if (res == first && res->start > dax_region->res.start)
- free = res->start - dax_region->res.start;
- if (free >= to_alloc && dev_size == 0)
- return alloc_dev_dax_range(dev_dax, dax_region->res.start, to_alloc);
+ if (res == first && res->start > dax_region->res.start) {
+ alloc = min(res->start - dax_region->res.start, to_alloc);
+ rc = alloc_dev_dax_range(dev_dax, dax_region->res.start, alloc);
+ break;
+ }
- free = 0;
+ alloc = 0;
/* space between allocations */
if (next && next->start > res->end + 1)
- free = next->start - res->end + 1;
+ alloc = min(next->start - (res->end + 1), to_alloc);
/* space at the end of the region */
- if (free < to_alloc && !next && res->end < region_res->end)
- free = region_res->end - res->end;
+ if (!alloc && !next && res->end < region_res->end)
+ alloc = min(region_res->end - res->end, to_alloc);
- if (free >= to_alloc && strcmp(name, res->name) == 0)
- return adjust_dev_dax_range(dev_dax, res, resource_size(res) + to_alloc);
- else if (free >= to_alloc && dev_size == 0)
- return alloc_dev_dax_range(dev_dax, res->end + 1, to_alloc);
+ if (!alloc)
+ continue;
+
+ if (adjust_ok(dev_dax, res)) {
+ rc = adjust_dev_dax_range(dev_dax, res, resource_size(res) + alloc);
+ break;
+ }
+ rc = alloc_dev_dax_range(dev_dax, res->end + 1, alloc);
+ break;
}
- return -ENOSPC;
+ if (rc)
+ return rc;
+ to_alloc -= alloc;
+ if (to_alloc)
+ goto retry;
+ return 0;
}
static ssize_t size_store(struct device *dev, struct device_attribute *attr,
@@ -767,8 +884,15 @@ static ssize_t resource_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
struct dev_dax *dev_dax = to_dev_dax(dev);
+ struct dax_region *dax_region = dev_dax->region;
+ unsigned long long start;
+
+ if (dev_dax->nr_range < 1)
+ start = dax_region->res.start;
+ else
+ start = dev_dax->ranges[0].range.start;
- return sprintf(buf, "%#llx\n", dev_dax->range.start);
+ return sprintf(buf, "%#llx\n", start);
}
static DEVICE_ATTR(resource, 0400, resource_show, NULL);
@@ -833,6 +957,7 @@ static void dev_dax_release(struct device *dev)
put_dax(dax_dev);
free_dev_dax_id(dev_dax);
dax_region_put(dax_region);
+ kfree(dev_dax->ranges);
kfree(dev_dax->pgmap);
kfree(dev_dax);
}
@@ -941,7 +1066,7 @@ struct dev_dax *devm_create_dev_dax(struct dev_dax_data *data)
err_alloc_dax:
kfree(dev_dax->pgmap);
err_pgmap:
- free_dev_dax_range(dev_dax);
+ free_dev_dax_ranges(dev_dax);
err_range:
free_dev_dax_id(dev_dax);
err_id:
diff --git a/drivers/dax/dax-private.h b/drivers/dax/dax-private.h
index 0cbb2ec81ca7..f863287107fd 100644
--- a/drivers/dax/dax-private.h
+++ b/drivers/dax/dax-private.h
@@ -49,7 +49,8 @@ struct dax_region {
* @id: ida allocated id
* @dev - device core
* @pgmap - pgmap for memmap setup / lifetime (driver owned)
- * @range: resource range for the instance
+ * @nr_range: size of @ranges
+ * @ranges: resource-span + pgoff tuples for the instance
*/
struct dev_dax {
struct dax_region *region;
@@ -58,7 +59,11 @@ struct dev_dax {
int id;
struct device dev;
struct dev_pagemap *pgmap;
- struct range range;
+ int nr_range;
+ struct dev_dax_range {
+ unsigned long pgoff;
+ struct range range;
+ } *ranges;
};
static inline struct dev_dax *to_dev_dax(struct device *dev)
diff --git a/drivers/dax/device.c b/drivers/dax/device.c
index 5f808617672a..bf389712a20b 100644
--- a/drivers/dax/device.c
+++ b/drivers/dax/device.c
@@ -55,15 +55,22 @@ static int check_vma(struct dev_dax *dev_dax, struct vm_area_struct *vma,
__weak phys_addr_t dax_pgoff_to_phys(struct dev_dax *dev_dax, pgoff_t pgoff,
unsigned long size)
{
- struct range *range = &dev_dax->range;
- phys_addr_t phys;
-
- phys = pgoff * PAGE_SIZE + range->start;
- if (phys >= range->start && phys <= range->end) {
+ int i;
+
+ for (i = 0; i < dev_dax->nr_range; i++) {
+ struct dev_dax_range *dax_range = &dev_dax->ranges[i];
+ struct range *range = &dax_range->range;
+ unsigned long long pgoff_end;
+ phys_addr_t phys;
+
+ pgoff_end = dax_range->pgoff + PHYS_PFN(range_len(range)) - 1;
+ if (pgoff < dax_range->pgoff || pgoff > pgoff_end)
+ continue;
+ phys = PFN_PHYS(pgoff - dax_range->pgoff) + range->start;
if (phys + size - 1 <= range->end)
return phys;
+ break;
}
-
return -1;
}
@@ -395,30 +402,40 @@ static void dev_dax_kill(void *dev_dax)
int dev_dax_probe(struct dev_dax *dev_dax)
{
struct dax_device *dax_dev = dev_dax->dax_dev;
- struct range *range = &dev_dax->range;
struct device *dev = &dev_dax->dev;
struct dev_pagemap *pgmap;
struct inode *inode;
struct cdev *cdev;
void *addr;
- int rc;
-
- /* 1:1 map region resource range to device-dax instance range */
- if (!devm_request_mem_region(dev, range->start, range_len(range),
- dev_name(dev))) {
- dev_warn(dev, "could not reserve range: %#llx - %#llx\n",
- range->start, range->end);
- return -EBUSY;
- }
+ int rc, i;
pgmap = dev_dax->pgmap;
+ if (dev_WARN_ONCE(dev, pgmap && dev_dax->nr_range > 1,
+ "static pgmap / multi-range device conflict\n"))
+ return -EINVAL;
+
if (!pgmap) {
- pgmap = devm_kzalloc(dev, sizeof(*pgmap), GFP_KERNEL);
+ pgmap = devm_kzalloc(dev, sizeof(*pgmap) + sizeof(struct range)
+ * (dev_dax->nr_range - 1), GFP_KERNEL);
if (!pgmap)
return -ENOMEM;
- pgmap->range = *range;
- pgmap->nr_range = 1;
+ pgmap->nr_range = dev_dax->nr_range;
+ }
+
+ for (i = 0; i < dev_dax->nr_range; i++) {
+ struct range *range = &dev_dax->ranges[i].range;
+
+ if (!devm_request_mem_region(dev, range->start,
+ range_len(range), dev_name(dev))) {
+ dev_warn(dev, "mapping%d: %#llx-%#llx could not reserve range\n",
+ i, range->start, range->end);
+ return -EBUSY;
+ }
+ /* don't update the range for static pgmap */
+ if (!dev_dax->pgmap)
+ pgmap->ranges[i] = *range;
}
+
pgmap->type = MEMORY_DEVICE_GENERIC;
addr = devm_memremap_pages(dev, pgmap);
if (IS_ERR(addr))
diff --git a/drivers/dax/kmem.c b/drivers/dax/kmem.c
index c2ac465cc342..6c933f2b604e 100644
--- a/drivers/dax/kmem.c
+++ b/drivers/dax/kmem.c
@@ -19,24 +19,28 @@ static const char *kmem_name;
/* Set if any memory will remain added when the driver will be unloaded. */
static bool any_hotremove_failed;
-static struct range dax_kmem_range(struct dev_dax *dev_dax)
+static int dax_kmem_range(struct dev_dax *dev_dax, int i, struct range *r)
{
- struct range range;
+ struct dev_dax_range *dax_range = &dev_dax->ranges[i];
+ struct range *range = &dax_range->range;
/* memory-block align the hotplug range */
- range.start = ALIGN(dev_dax->range.start, memory_block_size_bytes());
- range.end = ALIGN_DOWN(dev_dax->range.end + 1, memory_block_size_bytes()) - 1;
- return range;
+ r->start = ALIGN(range->start, memory_block_size_bytes());
+ r->end = ALIGN_DOWN(range->end + 1, memory_block_size_bytes()) - 1;
+ if (r->start >= r->end) {
+ r->start = range->start;
+ r->end = range->end;
+ return -ENOSPC;
+ }
+ return 0;
}
static int dev_dax_kmem_probe(struct dev_dax *dev_dax)
{
- struct range range = dax_kmem_range(dev_dax);
struct device *dev = &dev_dax->dev;
- struct resource *res;
+ int i, mapped = 0;
char *res_name;
int numa_node;
- int rc;
/*
* Ensure good NUMA information for the persistent memory.
@@ -55,31 +59,58 @@ static int dev_dax_kmem_probe(struct dev_dax *dev_dax)
if (!res_name)
return -ENOMEM;
- /* Region is permanently reserved if hotremove fails. */
- res = request_mem_region(range.start, range_len(&range), res_name);
- if (!res) {
- dev_warn(dev, "could not reserve region [%#llx-%#llx]\n", range.start, range.end);
- kfree(res_name);
- return -EBUSY;
- }
-
- /*
- * Set flags appropriate for System RAM. Leave ..._BUSY clear
- * so that add_memory() can add a child resource. Do not
- * inherit flags from the parent since it may set new flags
- * unknown to us that will break add_memory() below.
- */
- res->flags = IORESOURCE_SYSTEM_RAM;
-
- /*
- * Ensure that future kexec'd kernels will not treat this as RAM
- * automatically.
- */
- rc = add_memory_driver_managed(numa_node, range.start, range_len(&range), kmem_name);
- if (rc) {
- release_mem_region(range.start, range_len(&range));
- kfree(res_name);
- return rc;
+ for (i = 0; i < dev_dax->nr_range; i++) {
+ struct resource *res;
+ struct range range;
+ int rc;
+
+ rc = dax_kmem_range(dev_dax, i, &range);
+ if (rc) {
+ dev_info(dev, "mapping%d: %#llx-%#llx too small after alignment\n",
+ i, range.start, range.end);
+ continue;
+ }
+
+ /* Region is permanently reserved if hotremove fails. */
+ res = request_mem_region(range.start, range_len(&range), res_name);
+ if (!res) {
+ dev_warn(dev, "mapping%d: %#llx-%#llx could not reserve region\n",
+ i, range.start, range.end);
+ /*
+ * Once some memory has been onlined we can't
+ * assume that it can be un-onlined safely.
+ */
+ if (mapped)
+ continue;
+ kfree(res_name);
+ return -EBUSY;
+ }
+
+ /*
+ * Set flags appropriate for System RAM. Leave ..._BUSY clear
+ * so that add_memory() can add a child resource. Do not
+ * inherit flags from the parent since it may set new flags
+ * unknown to us that will break add_memory() below.
+ */
+ res->flags = IORESOURCE_SYSTEM_RAM;
+
+ /*
+ * Ensure that future kexec'd kernels will not treat
+ * this as RAM automatically.
+ */
+ rc = add_memory_driver_managed(numa_node, range.start,
+ range_len(&range), kmem_name);
+
+ if (rc) {
+ dev_warn(dev, "mapping%d: %#llx-%#llx memory add failed\n",
+ i, range.start, range.end);
+ release_mem_region(range.start, range_len(&range));
+ if (mapped)
+ continue;
+ kfree(res_name);
+ return rc;
+ }
+ mapped++;
}
dev_set_drvdata(dev, res_name);
@@ -90,9 +121,8 @@ static int dev_dax_kmem_probe(struct dev_dax *dev_dax)
#ifdef CONFIG_MEMORY_HOTREMOVE
static int dev_dax_kmem_remove(struct dev_dax *dev_dax)
{
- int rc;
+ int i, success = 0;
struct device *dev = &dev_dax->dev;
- struct range range = dax_kmem_range(dev_dax);
const char *res_name = dev_get_drvdata(dev);
/*
@@ -101,17 +131,31 @@ static int dev_dax_kmem_remove(struct dev_dax *dev_dax)
* there is no way to hotremove this memory until reboot because device
* unbind will succeed even if we return failure.
*/
- rc = remove_memory(dev_dax->target_node, range.start, range_len(&range));
- if (rc) {
+ for (i = 0; i < dev_dax->nr_range; i++) {
+ struct range range;
+ int rc;
+
+ rc = dax_kmem_range(dev_dax, i, &range);
+ if (rc)
+ continue;
+
+ rc = remove_memory(dev_dax->target_node, range.start,
+ range_len(&range));
+ if (rc == 0) {
+ release_mem_region(range.start, range_len(&range));
+ success++;
+ continue;
+ }
any_hotremove_failed = true;
- dev_err(dev, "%#llx-%#llx cannot be hotremoved until the next reboot\n",
- range.start, range.end);
- return rc;
+ dev_err(dev,
+ "mapping%d: %#llx-%#llx cannot be hotremoved until the next reboot\n",
+ i, range.start, range.end);
}
- /* Release and free dax resources */
- release_mem_region(range.start, range_len(&range));
- kfree(res_name);
+ if (success >= dev_dax->nr_range) {
+ kfree(res_name);
+ dev_set_drvdata(dev, NULL);
+ }
return 0;
}
diff --git a/tools/testing/nvdimm/dax-dev.c b/tools/testing/nvdimm/dax-dev.c
index 38d8e55c4a0d..fb342a8c98d3 100644
--- a/tools/testing/nvdimm/dax-dev.c
+++ b/tools/testing/nvdimm/dax-dev.c
@@ -9,11 +9,18 @@
phys_addr_t dax_pgoff_to_phys(struct dev_dax *dev_dax, pgoff_t pgoff,
unsigned long size)
{
- struct range *range = &dev_dax->range;
- phys_addr_t addr;
+ int i;
- addr = pgoff * PAGE_SIZE + range->start;
- if (addr >= range->start && addr <= range->end) {
+ for (i = 0; i < dev_dax->nr_range; i++) {
+ struct dev_dax_range *dax_range = &dev_dax->ranges[i];
+ struct range *range = &dax_range->range;
+ unsigned long long pgoff_end;
+ phys_addr_t addr;
+
+ pgoff_end = dax_range->pgoff + PHYS_PFN(range_len(range)) - 1;
+ if (pgoff < dax_range->pgoff || pgoff > pgoff_end)
+ continue;
+ addr = PFN_PHYS(pgoff - dax_range->pgoff) + range->start;
if (addr + size - 1 <= range->end) {
if (get_nfit_res(addr)) {
struct page *page;
@@ -23,9 +30,10 @@ phys_addr_t dax_pgoff_to_phys(struct dev_dax *dev_dax, pgoff_t pgoff,
page = vmalloc_to_page((void *)addr);
return PFN_PHYS(page_to_pfn(page));
- } else
- return addr;
+ }
+ return addr;
}
+ break;
}
return -1;
}