Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Wed, 15 Nov 2017 21:44:26 -0500
From:   Jerome Glisse <jglisse@redhat.com>
To:     chet l <loke.chetan@gmail.com>
Cc:     Bob Liu <lliubbo@gmail.com>, Bob Liu <liubo95@huawei.com>,
        Dan Williams <dan.j.williams@intel.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        Linux MM <linux-mm@kvack.org>,
        John Hubbard <jhubbard@nvidia.com>,
        David Nellans <dnellans@nvidia.com>,
        Balbir Singh <bsingharora@gmail.com>,
        Michal Hocko <mhocko@kernel.org>,
        Andrew Morton <akpm@linux-foundation.org>
Subject: Re: [PATCH 0/6] Cache coherent device memory (CDM) with HMM v5
Message-ID: <20171116024425.GC2934@redhat.com>
References: <20170905193644.GD19397@redhat.com>
 <CAA_GA1ckfyokvqy3aKi-NoSXxSzwiVsrykC6xNxpa3WUz0bqNQ@mail.gmail.com>
 <20170911233649.GA4892@redhat.com>
 <CAA_GA1ff4mGKfxxRpjYCRjXOvbUuksM0K2gmH1VrhL4qtGWFbw@mail.gmail.com>
 <20170926161635.GA3216@redhat.com>
 <0d7273c3-181c-6d68-3c5f-fa518e782374@huawei.com>
 <20170930224927.GC6775@redhat.com>
 <CAA_GA1dhrs7n-ewZmW4bNtouK8rKnF1_TWv0z+2vrUgJjWpnMQ@mail.gmail.com>
 <20171012153721.GA2986@redhat.com>
 <CAAsGZS7JeH-cxrmZAVraLm5RjSVHJLXMdwZQ7Cxm91KGdVQocg@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <CAAsGZS7JeH-cxrmZAVraLm5RjSVHJLXMdwZQ7Cxm91KGdVQocg@mail.gmail.com>
User-Agent: Mutt/1.9.1 (2017-09-22)
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Wed, Nov 15, 2017 at 06:10:08PM -0800, chet l wrote:
> >> You may think it as a CCIX device or CAPI device.
> >> The requirement is eliminate any extra copy.
> >> A typical usecase/requirement is malloc() and madvise() allocate from
> >> device memory, then CPU write data to device memory directly and
> >> trigger device to read the data/do calculation.
> >
> > I suggest you rely on the device driver userspace API to do a migration after malloc
> > then. Something like:
> >   ptr = malloc(size);
> >   my_device_migrate(ptr, size);
> >
> > Which would call an ioctl of the device driver which itself would migrate memory or
> > allocate device memory for the range if pointer return by malloc is not yet back by
> > any pages.
> >
> 
> So for CCIX, I don't think there is going to be an inline device
> driver that would allocate any memory for you. The expansion memory
> will become part of the system memory as part of the boot process. So,
> if the host DDR is 256GB and the CCIX expansion memory is 4GB, the
> total system mem will be 260GB.
> 
> Assume that the 'mm' is taught to mark/anoint the ZONE_DEVICE(or
> ZONE_XXX) range from 256 to 260 GB. Then, for kmalloc it(mm) won't use
> the ZONE_DEV range. But for a malloc, it will/can use that range.

HMM zone device memory would work with that, you just need to teach the
platform to identify this memory zone and not hotplug it. Again you
should rely on specific device driver API to allocate this memory.

> > There has been several discussions already about madvise/mbind/set_mempolicy/
> > move_pages and at this time i don't think we want to add or change any of them to
> > understand device memory. My personal opinion is that we first need to have enough
> 
> We will visit these APIs when we are more closer to building exotic
> CCIX devices. And the plan is to present/express the CCIX proximity
> attributes just like a NUMA node-proximity attribute today. That way
> there would be minimal disruptions to the existing OS ecosystem.

NUMA have been rejected previously see CDM/CAPI threads. So i don't see
it being accepted for CCIX either. My belief is that we want to hide this
inside device driver and only once we see multiple devices all doing the
same kind of thing we should move toward building something generic that
catter to CCIX devices.

J�r�me

From 1584194472841752725@xxx Thu Nov 16 04:13:00 +0000 2017
X-GM-THRID: 1572843623662560165
X-Gmail-Labels: Inbox,Category Forums,HistoricalUnread