Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates
 216.228.112.34 as permitted sender) receiver=protection.outlook.com;
 client-ip=216.228.112.34; helo=mail.nvidia.com;
Subject: Re: [PATCH v9 07/10] mm: Device exclusive memory access
To:     Balbir Singh <bsingharora@gmail.com>
CC:     Andrew Morton <akpm@linux-foundation.org>,
        Alistair Popple <apopple@nvidia.com>, <linux-mm@kvack.org>,
        <nouveau@lists.freedesktop.org>, <bskeggs@redhat.com>,
        <rcampbell@nvidia.com>, <linux-doc@vger.kernel.org>,
        <linux-kernel@vger.kernel.org>, <dri-devel@lists.freedesktop.org>,
        <hch@infradead.org>, <jglisse@redhat.com>, <willy@infradead.org>,
        <jgg@nvidia.com>, <peterx@redhat.com>, <hughd@google.com>,
        Christoph Hellwig <hch@lst.de>
References: <20210524132725.12697-1-apopple@nvidia.com>
 <20210524132725.12697-8-apopple@nvidia.com>
 <20210524151157.2dc5d2bb510ff86dc449bf0c@linux-foundation.org>
 <YKzk0ILRsyazMs2W@balbir-desktop>
 <8844f8c1-d78c-e0f9-c046-592bd75d4c07@nvidia.com>
 <YLdGXSw0zdiovn4i@balbir-desktop>
From:   John Hubbard <jhubbard@nvidia.com>
Message-ID: <d1cf8ee3-3bad-f31b-770a-26d03d457efe@nvidia.com>
Date:   Thu, 3 Jun 2021 01:37:16 -0700
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.10.2
MIME-Version: 1.0
In-Reply-To: <YLdGXSw0zdiovn4i@balbir-desktop>
Content-Type: text/plain; charset="utf-8"; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Jun 2021 08:37:16.9716
 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: cca83f86-ae35-4faa-f833-08d9266acb74
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.112.34];Helo=[mail.nvidia.com]
X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT054.eop-nam11.prod.protection.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Anonymous
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR12MB1622
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 6/2/21 1:50 AM, Balbir Singh wrote:
...
>>> only impact the address space of programs using the GPU. Should the exclusively
>>> marked range live in the unreclaimable list and recycled back to active/in-active
>>> to account for the fact that
>>>
>>> 1. It is not reclaimable and reclaim will only hurt via page faults?
>>> 2. It ages the page correctly or at-least allows for that possibility when the
>>>      page is used by the GPU.
>>
>> I'm not sure that that is *necessarily* something we can conclude. It depends upon
>> access patterns of each program. For example, a "reduction" parallel program sends
>> over lots of data to the GPU, and only a tiny bit of (reduced!) data comes back
>> to the CPU. In that case, freeing the physical page on the CPU is actually the
>> best decision for the OS to make (if the OS is sufficiently prescient).
>>
> 
> With a shared device or a device exclusive range, it would be good to get the device
> usage pattern and update the mm with that knowledge, so that the LRU can be better

Integrating a GPU (or "device") processor and it's mm behavior with the Linux kernel is
always an interesting concept. Certainly worth exploring, although it's probably
not a small project by any means.

> maintained. With your comment you seem to suggest that a page used by the GPU might
> be a good candidate for reclaim based on the CPU's understanding of the age of
> the page should not account for use by the device
> (are GPU workloads - access once and discard?)
> 

Well, that's a little too narrow of an interpretation. The GPU is a fairly general
purpose processor, and so it has all kinds of workloads. I'm trying to discourage
any hopes that one can know, in advance, precisely how the GPU's pages need to be
managed. It's similar to the the CPU, in that regard. My example was just one, out
of a vast pool of possible behaviors.

thanks,
-- 
John Hubbard
NVIDIA