Hi,
I would like to propose the following topic for the MM track. Different
group of people would like to use NVIDMMs as a low cost & slower memory
which is presented to the system as a NUMA node. We do have a NUMA API
but it doesn't really fit to "balance the memory between nodes" needs.
People would like to have hot pages in the regular RAM while cold pages
might be at lower speed NUMA nodes. We do have NUMA balancing for
promotion path but there is notIhing for the other direction. Can we
start considering memory reclaim to move pages to more distant and idle
NUMA nodes rather than reclaim them? There are certainly details that
will get quite complicated but I guess it is time to start discussing
this at least.
--
Michal Hocko
SUSE Labs
On Wed, Jan 30, 2019 at 06:48:47PM +0100, Michal Hocko wrote:
> Hi,
> I would like to propose the following topic for the MM track. Different
> group of people would like to use NVIDMMs as a low cost & slower memory
> which is presented to the system as a NUMA node. We do have a NUMA API
> but it doesn't really fit to "balance the memory between nodes" needs.
> People would like to have hot pages in the regular RAM while cold pages
> might be at lower speed NUMA nodes. We do have NUMA balancing for
> promotion path but there is notIhing for the other direction. Can we
> start considering memory reclaim to move pages to more distant and idle
> NUMA nodes rather than reclaim them? There are certainly details that
> will get quite complicated but I guess it is time to start discussing
> this at least.
Yes, thanks for the proposal. I would be very interested in this
discussion for MM. I think some of the details for determining such a
migration path are related to the heterogeneous memory attributes I'm
currently trying to export.
On Wed, Jan 30, 2019 at 9:48 AM Michal Hocko <[email protected]> wrote:
>
> Hi,
> I would like to propose the following topic for the MM track. Different
> group of people would like to use NVIDMMs as a low cost & slower memory
> which is presented to the system as a NUMA node. We do have a NUMA API
> but it doesn't really fit to "balance the memory between nodes" needs.
> People would like to have hot pages in the regular RAM while cold pages
> might be at lower speed NUMA nodes. We do have NUMA balancing for
> promotion path but there is notIhing for the other direction. Can we
> start considering memory reclaim to move pages to more distant and idle
> NUMA nodes rather than reclaim them? There are certainly details that
> will get quite complicated but I guess it is time to start discussing
> this at least.
I would be interested in this topic too. We (Alibaba) do have some
usecases with using NVDIMM as NUMA node. The node balancing (or
cold/hot data migration) is one of our needs to achieve optimal
performance for some workloads. I also proposed a related topic.
Regards,
Yang
> --
> Michal Hocko
> SUSE Labs
>
Michal Hocko <[email protected]> writes:
> Hi,
> I would like to propose the following topic for the MM track. Different
> group of people would like to use NVIDMMs as a low cost & slower memory
> which is presented to the system as a NUMA node. We do have a NUMA API
> but it doesn't really fit to "balance the memory between nodes" needs.
> People would like to have hot pages in the regular RAM while cold pages
> might be at lower speed NUMA nodes. We do have NUMA balancing for
> promotion path but there is notIhing for the other direction. Can we
> start considering memory reclaim to move pages to more distant and idle
> NUMA nodes rather than reclaim them? There are certainly details that
> will get quite complicated but I guess it is time to start discussing
> this at least.
I would be interested in this topic too. I would like to
understand the API and how it can help exploit the different type of
devices we have on OpenCAPI.
IMHO there are few proposals related to this which we could discuss together
1. HMAT series which want to expose these devices as Numa nodes
2. The patch series from Dave Hansen which just uses Pmem as Numa node.
3. The patch series from Fengguang Wu which does prevent default
allocation from these numa nodes by excluding them from zone list.
4. The patch series from Jerome Glisse which doesn't expose these as
numa nodes.
IMHO (3) is suggesting that we really don't want them as numa nodes. But
since Numa is the only interface we currently have to present them as
memory and control the allocation and migration we are forcing
ourselves to Numa nodes and then excluding them from default allocation.
-aneesh
On Thu, 31 Jan 2019, Aneesh Kumar K.V wrote:
> I would be interested in this topic too. I would like to
> understand the API and how it can help exploit the different type of
> devices we have on OpenCAPI.
So am I. We may want to rethink the whole NUMA API and the way we handle
different types of memory with their divergent performance
characteristics.
We need some way to allow a better selection of memory from the kernel
without creating too much complexity. We have new characteristics to
cover:
1. Persistence (NVRAM) or generally a storage device that allows access to
the medium via a RAM like interface.
2. Coprocessor memory that can be shuffled back and forth to a device
(HMM).
3. On Device memory (important since PCIe limitations are currently a
problem and Intel is stuck on PCIe3 and devices start to bypass the
processor to gain performance)
4. High Density RAM (GDDR f.e.) with different caching behavior
and/or different cacheline sizes.
5. Modifying access characteristics by reserving slice of a cache (f.e.
L3) for a specific memory region.
6. SRAM support (high speed memory on the processor itself or by using
the processor cache to persist a cacheline)
And then the old NUMA stuff where only the latency to memory varies. But
that was a particular solution targeted at scaling SMP system through
interconnects. This was a mostly symmetric approach. The use of
accellerators etc etc and the above characteristics lead to more complex
assymmetric memory approaches that may be difficult to manage and use from
kernel space.
On 02/06/2019 02:03 PM, Christopher Lameter wrote:
> On Thu, 31 Jan 2019, Aneesh Kumar K.V wrote:
>
>> I would be interested in this topic too. I would like to
>> understand the API and how it can help exploit the different type of
>> devices we have on OpenCAPI.
Same here, we/RedHat have quite a bit of experience running on several
large system
(32TB/128nodes/1024CPUs). Some of these systems have NVRAM and can operated
in memory mode as well as storage mode.
Larry
> So am I. We may want to rethink the whole NUMA API and the way we handle
> different types of memory with their divergent performance
> characteristics.
>
> We need some way to allow a better selection of memory from the kernel
> without creating too much complexity. We have new characteristics to
> cover:
>
> 1. Persistence (NVRAM) or generally a storage device that allows access to
> the medium via a RAM like interface.
>
> 2. Coprocessor memory that can be shuffled back and forth to a device
> (HMM).
>
> 3. On Device memory (important since PCIe limitations are currently a
> problem and Intel is stuck on PCIe3 and devices start to bypass the
> processor to gain performance)
>
> 4. High Density RAM (GDDR f.e.) with different caching behavior
> and/or different cacheline sizes.
>
> 5. Modifying access characteristics by reserving slice of a cache (f.e.
> L3) for a specific memory region.
>
> 6. SRAM support (high speed memory on the processor itself or by using
> the processor cache to persist a cacheline)
>
> And then the old NUMA stuff where only the latency to memory varies. But
> that was a particular solution targeted at scaling SMP system through
> interconnects. This was a mostly symmetric approach. The use of
> accellerators etc etc and the above characteristics lead to more complex
> assymmetric memory approaches that may be difficult to manage and use from
> kernel space.
>
On Thu, Jan 31, 2019 at 12:19:47PM +0530, Aneesh Kumar K.V wrote:
>Michal Hocko <[email protected]> writes:
>
>> Hi,
>> I would like to propose the following topic for the MM track. Different
>> group of people would like to use NVIDMMs as a low cost & slower memory
>> which is presented to the system as a NUMA node. We do have a NUMA API
>> but it doesn't really fit to "balance the memory between nodes" needs.
>> People would like to have hot pages in the regular RAM while cold pages
>> might be at lower speed NUMA nodes. We do have NUMA balancing for
>> promotion path but there is notIhing for the other direction. Can we
>> start considering memory reclaim to move pages to more distant and idle
>> NUMA nodes rather than reclaim them? There are certainly details that
>> will get quite complicated but I guess it is time to start discussing
>> this at least.
>
>I would be interested in this topic too. I would like to understand
So do me. I'd be glad to take in the discussions if can attend the slot.
>the API and how it can help exploit the different type of devices we
>have on OpenCAPI.
>
>IMHO there are few proposals related to this which we could discuss together
>
>1. HMAT series which want to expose these devices as Numa nodes
>2. The patch series from Dave Hansen which just uses Pmem as Numa node.
>3. The patch series from Fengguang Wu which does prevent default
>allocation from these numa nodes by excluding them from zone list.
>4. The patch series from Jerome Glisse which doesn't expose these as
>numa nodes.
>
>IMHO (3) is suggesting that we really don't want them as numa nodes. But
>since Numa is the only interface we currently have to present them as
>memory and control the allocation and migration we are forcing
>ourselves to Numa nodes and then excluding them from default allocation.
Regarding (3), we actually made a default policy choice for
"separating fallback zonelists for PMEM/DRAM nodes" for the
typical use scenarios.
In long term, it's better to not build such assumption into kernel.
There may well be workloads that are cost sensitive rather than
performance sensitive. Suppose people buy a machine with tiny DRAM
and large PMEM. In which case the suitable policy may be to
1) prefer (but not bind) slab etc. kernel pages in DRAM
2) allocate LRU etc. pages from either DRAM or PMEM node
In summary, kernel may offer flexibility for different policies for
use by different users. PMEM has different characteristics comparing
to DRAM, users may or may not be treated differently than DRAM through
policies.
Thanks,
Fengguang
On Sat, Feb 23, 2019 at 09:27:48PM +0800, Fengguang Wu wrote:
>On Thu, Jan 31, 2019 at 12:19:47PM +0530, Aneesh Kumar K.V wrote:
>>Michal Hocko <[email protected]> writes:
>>
>>> Hi,
>>> I would like to propose the following topic for the MM track. Different
>>> group of people would like to use NVIDMMs as a low cost & slower memory
>>> which is presented to the system as a NUMA node. We do have a NUMA API
>>> but it doesn't really fit to "balance the memory between nodes" needs.
>>> People would like to have hot pages in the regular RAM while cold pages
>>> might be at lower speed NUMA nodes. We do have NUMA balancing for
>>> promotion path but there is notIhing for the other direction. Can we
>>> start considering memory reclaim to move pages to more distant and idle
>>> NUMA nodes rather than reclaim them? There are certainly details that
>>> will get quite complicated but I guess it is time to start discussing
>>> this at least.
>>
>>I would be interested in this topic too. I would like to understand
>
>So do me. I'd be glad to take in the discussions if can attend the slot.
>
>>the API and how it can help exploit the different type of devices we
>>have on OpenCAPI.
>>
>>IMHO there are few proposals related to this which we could discuss together
>>
>>1. HMAT series which want to expose these devices as Numa nodes
>>2. The patch series from Dave Hansen which just uses Pmem as Numa node.
>>3. The patch series from Fengguang Wu which does prevent default
>>allocation from these numa nodes by excluding them from zone list.
>>4. The patch series from Jerome Glisse which doesn't expose these as
>>numa nodes.
>>
>>IMHO (3) is suggesting that we really don't want them as numa nodes. But
>>since Numa is the only interface we currently have to present them as
>>memory and control the allocation and migration we are forcing
>>ourselves to Numa nodes and then excluding them from default allocation.
>
>Regarding (3), we actually made a default policy choice for
>"separating fallback zonelists for PMEM/DRAM nodes" for the
>typical use scenarios.
>
>In long term, it's better to not build such assumption into kernel.
>There may well be workloads that are cost sensitive rather than
>performance sensitive. Suppose people buy a machine with tiny DRAM
>and large PMEM. In which case the suitable policy may be to
>
>1) prefer (but not bind) slab etc. kernel pages in DRAM
>2) allocate LRU etc. pages from either DRAM or PMEM node
The point is not separating fallback zonelists for DRAM and PMEM in
this case.
>In summary, kernel may offer flexibility for different policies for
>use by different users. PMEM has different characteristics comparing
>to DRAM, users may or may not be treated differently than DRAM through
>policies.
>
>Thanks,
>Fengguang