Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates
 216.228.112.34 as permitted sender) receiver=protection.outlook.com;
 client-ip=216.228.112.34; helo=mail.nvidia.com;
From:   Alistair Popple <apopple@nvidia.com>
To:     Peter Xu <peterx@redhat.com>
CC:     Balbir Singh <bsingharora@gmail.com>,
        John Hubbard <jhubbard@nvidia.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        <linux-mm@kvack.org>, <nouveau@lists.freedesktop.org>,
        <bskeggs@redhat.com>, <rcampbell@nvidia.com>,
        <linux-doc@vger.kernel.org>, <linux-kernel@vger.kernel.org>,
        <dri-devel@lists.freedesktop.org>, <hch@infradead.org>,
        <jglisse@redhat.com>, <willy@infradead.org>, <jgg@nvidia.com>,
        <hughd@google.com>, Christoph Hellwig <hch@lst.de>
Subject: Re: [PATCH v9 07/10] mm: Device exclusive memory access
Date:   Thu, 3 Jun 2021 21:39:32 +1000
Message-ID: <3853054.AI2YdRgKcH@nvdebian>
In-Reply-To: <YLeXqp/U0DgylI/u@t490s>
References: <20210524132725.12697-1-apopple@nvidia.com> <YLdGXSw0zdiovn4i@balbir-desktop> <YLeXqp/U0DgylI/u@t490s>
MIME-Version: 1.0
Content-Transfer-Encoding: 7Bit
Content-Type: text/plain; charset="us-ascii"
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Jun 2021 11:39:38.6137
 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: da1a8c05-f034-472a-f7fa-08d926844524
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.112.34];Helo=[mail.nvidia.com]
X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT051.eop-nam11.prod.protection.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Anonymous
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR1201MB0154
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Thursday, 3 June 2021 12:37:30 AM AEST Peter Xu wrote:
> External email: Use caution opening links or attachments
> 
> On Wed, Jun 02, 2021 at 06:50:37PM +1000, Balbir Singh wrote:
> > On Wed, May 26, 2021 at 12:17:18AM -0700, John Hubbard wrote:
> > > On 5/25/21 4:51 AM, Balbir Singh wrote:
> > > ...
> > > 
> > > > > How beneficial is this code to nouveau users?  I see that it permits
> > > > > a
> > > > > part of OpenCL to be implemented, but how useful/important is this
> > > > > in
> > > > > the real world?
> > > > 
> > > > That is a very good question! I've not reviewed the code, but a sample
> > > > program with the described use case would make things easy to parse.
> > > > I suspect that is not easy to build at the moment?
> > > 
> > > The cover letter says this:
> > > 
> > > This has been tested with upstream Mesa 21.1.0 and a simple OpenCL
> > > program
> > > which checks that GPU atomic accesses to system memory are atomic.
> > > Without
> > > this series the test fails as there is no way of write-protecting the
> > > page
> > > mapping which results in the device clobbering CPU writes. For reference
> > > the test is available at https://ozlabs.org/~apopple/opencl_svm_atomics/
> > > 
> > > Further testing has been performed by adding support for testing
> > > exclusive
> > > access to the hmm-tests kselftests.
> > > 
> > > ...so that seems to cover the "sample program" request, at least.
> > 
> > Thanks, I'll take a look
> > 
> > > > I wonder how we co-ordinate all the work the mm is doing, page
> > > > migration,
> > > > reclaim with device exclusive access? Do we have any numbers for the
> > > > worst
> > > > case page fault latency when something is marked away for exclusive
> > > > access?
> > > 
> > > CPU page fault latency is approximately "terrible", if a page is
> > > resident on the GPU. We have to spin up a DMA engine on the GPU and
> > > have it copy the page over the PCIe bus, after all.
> > > 
> > > > I presume for now this is anonymous memory only? SWP_DEVICE_EXCLUSIVE
> > > > would
> > > 
> > > Yes, for now.
> > > 
> > > > only impact the address space of programs using the GPU. Should the
> > > > exclusively marked range live in the unreclaimable list and recycled
> > > > back to active/in-active to account for the fact that
> > > > 
> > > > 1. It is not reclaimable and reclaim will only hurt via page faults?
> > > > 2. It ages the page correctly or at-least allows for that possibility
> > > > when the> > > 
> > > >     page is used by the GPU.
> > > 
> > > I'm not sure that that is *necessarily* something we can conclude. It
> > > depends upon access patterns of each program. For example, a
> > > "reduction" parallel program sends over lots of data to the GPU, and
> > > only a tiny bit of (reduced!) data comes back to the CPU. In that case,
> > > freeing the physical page on the CPU is actually the best decision for
> > > the OS to make (if the OS is sufficiently prescient).> 
> > With a shared device or a device exclusive range, it would be good to get
> > the device usage pattern and update the mm with that knowledge, so that
> > the LRU can be better maintained. With your comment you seem to suggest
> > that a page used by the GPU might be a good candidate for reclaim based
> > on the CPU's understanding of the age of the page should not account for
> > use by the device
> > (are GPU workloads - access once and discard?)
> 
> Hmm, besides the aging info, this reminded me: do we need to isolate the
> page from lru too when marking device exclusive access?
> 
> Afaict the current patch didn't do that so I think it's reclaimable.  If we
> still have the rmap then we'll get a mmu notify CLEAR when unmapping that
> special pte, so device driver should be able to drop the ownership.  However
> we dropped the rmap when marking exclusive.  Now I don't know whether and
> how it'll work if page reclaim runs with the page being exclusively owned
> if without isolating the page..

Reclaim won't run on the page due to the extra references from the special 
swap entries.

> --
> Peter Xu