Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates
 216.228.112.34 as permitted sender) receiver=protection.outlook.com;
 client-ip=216.228.112.34; helo=mail.nvidia.com;
Subject: Re: [PATCH v9 07/10] mm: Device exclusive memory access
To:     Balbir Singh <bsingharora@gmail.com>,
        Andrew Morton <akpm@linux-foundation.org>
CC:     Alistair Popple <apopple@nvidia.com>, <linux-mm@kvack.org>,
        <nouveau@lists.freedesktop.org>, <bskeggs@redhat.com>,
        <rcampbell@nvidia.com>, <linux-doc@vger.kernel.org>,
        <linux-kernel@vger.kernel.org>, <dri-devel@lists.freedesktop.org>,
        <hch@infradead.org>, <jglisse@redhat.com>, <willy@infradead.org>,
        <jgg@nvidia.com>, <peterx@redhat.com>, <hughd@google.com>,
        Christoph Hellwig <hch@lst.de>
References: <20210524132725.12697-1-apopple@nvidia.com>
 <20210524132725.12697-8-apopple@nvidia.com>
 <20210524151157.2dc5d2bb510ff86dc449bf0c@linux-foundation.org>
 <YKzk0ILRsyazMs2W@balbir-desktop>
From:   John Hubbard <jhubbard@nvidia.com>
Message-ID: <8844f8c1-d78c-e0f9-c046-592bd75d4c07@nvidia.com>
Date:   Wed, 26 May 2021 00:17:18 -0700
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.10.2
MIME-Version: 1.0
In-Reply-To: <YKzk0ILRsyazMs2W@balbir-desktop>
Content-Type: text/plain; charset="utf-8"; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 May 2021 07:17:19.2575
 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: 4d5531b0-d882-4dc7-4f54-08d920164c72
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.112.34];Helo=[mail.nvidia.com]
X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT035.eop-nam11.prod.protection.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Anonymous
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB5006
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 5/25/21 4:51 AM, Balbir Singh wrote:
...
>> How beneficial is this code to nouveau users?  I see that it permits a
>> part of OpenCL to be implemented, but how useful/important is this in
>> the real world?
> 
> That is a very good question! I've not reviewed the code, but a sample
> program with the described use case would make things easy to parse.
> I suspect that is not easy to build at the moment?
> 

The cover letter says this:

This has been tested with upstream Mesa 21.1.0 and a simple OpenCL program
which checks that GPU atomic accesses to system memory are atomic. Without
this series the test fails as there is no way of write-protecting the page
mapping which results in the device clobbering CPU writes. For reference
the test is available at https://ozlabs.org/~apopple/opencl_svm_atomics/

Further testing has been performed by adding support for testing exclusive
access to the hmm-tests kselftests.

...so that seems to cover the "sample program" request, at least.

> I wonder how we co-ordinate all the work the mm is doing, page migration,
> reclaim with device exclusive access? Do we have any numbers for the worst
> case page fault latency when something is marked away for exclusive access?

CPU page fault latency is approximately "terrible", if a page is resident on
the GPU. We have to spin up a DMA engine on the GPU and have it copy the page
over the PCIe bus, after all.

> I presume for now this is anonymous memory only? SWP_DEVICE_EXCLUSIVE would

Yes, for now.

> only impact the address space of programs using the GPU. Should the exclusively
> marked range live in the unreclaimable list and recycled back to active/in-active
> to account for the fact that
> 
> 1. It is not reclaimable and reclaim will only hurt via page faults?
> 2. It ages the page correctly or at-least allows for that possibility when the
>     page is used by the GPU.

I'm not sure that that is *necessarily* something we can conclude. It depends upon
access patterns of each program. For example, a "reduction" parallel program sends
over lots of data to the GPU, and only a tiny bit of (reduced!) data comes back
to the CPU. In that case, freeing the physical page on the CPU is actually the
best decision for the OS to make (if the OS is sufficiently prescient).

thanks,
-- 
John Hubbard
NVIDIA