Received-SPF: pass (google.com: domain of linux-ext4-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Subject: Re: [PATCH v1 03/14] mm: add iomem vma selection for memory migration
To:     Christoph Hellwig <hch@lst.de>
Cc:     "Sierra Guiza, Alejandro (Alex)" <alex.sierra@amd.com>,
        akpm@linux-foundation.org, linux-mm@kvack.org,
        rcampbell@nvidia.com, linux-ext4@vger.kernel.org,
        linux-xfs@vger.kernel.org, amd-gfx@lists.freedesktop.org,
        dri-devel@lists.freedesktop.org, jgg@nvidia.com,
        jglisse@redhat.com, Dan Williams <dan.j.williams@intel.com>
References: <20210825034828.12927-1-alex.sierra@amd.com>
 <20210825034828.12927-4-alex.sierra@amd.com> <20210825074602.GA29620@lst.de>
 <c4241eb3-07d2-c85b-0f48-cce4b8369381@amd.com>
 <a9eb2c4a-d8cc-9553-57b7-fd1622679aaa@amd.com> <20210830082800.GA6836@lst.de>
 <e40b3b79-f548-b87b-7a85-f654f25ed8dd@amd.com>
 <20210901082925.GA21961@lst.de>
 <11d64457-9d61-f82d-6c98-d68762dce85d@amd.com>
 <20210902081826.GA16283@lst.de>
From:   Felix Kuehling <felix.kuehling@amd.com>
Message-ID: <8307c3b6-52c9-fbab-6f02-c00ed3e8a35e@amd.com>
Date:   Thu, 9 Sep 2021 00:02:10 -0400
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.13.0
In-Reply-To: <20210902081826.GA16283@lst.de>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
Content-Language: en-US
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?MHk3NTl2bUp1ZFo5NHQ2U3EvU0JXazVNVjQwbjNEQVhMNklRWW1LM0tMdEwz?=
 =?utf-8?B?bWdHNE5KMlhMbk1HWEVBdWh6Nk54ZDlzaS83VWRKSmZadytGR3J4eEtsUnM2?=
 =?utf-8?B?QmZnek9HZWJ5NFFUdUJHdVhhQ21McFFWRHExRzFTODBqMVVTajYwWFUxYmZr?=
 =?utf-8?B?UlNlaGgwSHlGTDZlV0NubXArbGJmYXNqTEk4bmhDZnZ1RUIxVDlJOFBFQnd2?=
 =?utf-8?B?b2lCWXhpdzJqQW9EbGVoTTQ5SUNZRjc2b2RjVWJNNzZiWnY2M1ZrV3Z1NGNl?=
 =?utf-8?B?UXdRdFpjRTd2VGJpSi9ocGtEVU5jU3AzRXZva0lkZWR0WDdRVXphSjR1WEQy?=
 =?utf-8?B?UEtQc2YrZ1B6Q1MyTys5RlBsMVFYalpJcEl5OWJNTU1WL2RETFZkSWxRV3BM?=
 =?utf-8?B?bmc5WGx3UEd1WkhwdDhXMWxWdkdnN1hvb0xCa2lxN3lEMEVESUNmVTJieW5W?=
 =?utf-8?B?eU44TVkxNUY0SUFSUGV0SXhLUk9XcmNpczZHUVlQK2dQNU9rN3JydDlqZmtE?=
 =?utf-8?B?bW5TVFp6QnRZeitzL0I4WElrV3owQ3FMUUYvQTF0dEtRb1ZaQnlLYmNienVN?=
 =?utf-8?B?aU16a2NDUEpwdEl5SWNvZ3F2WkdlZU16SmxtUWdycjJGNXBnZTY5SFBKYVNw?=
 =?utf-8?B?Ny9LNE5wekhKNnRZcnNrN0dQNGN2Wm1yRTEyUExOa29GQ0xDWExobzVkeE0v?=
 =?utf-8?B?cjRRTVZKY3NhUDhPS2RpbzdqUXRyNHMyQUZpUXNNSjZscXVnLzhtQ1RWd240?=
 =?utf-8?B?VGJ0bUlqdkU4NkRrRGNHWHYway9FR0RWalBHbGNDSEMwNHZiSEF1bnUxRmZw?=
 =?utf-8?B?UTZtd1ArLzRMUkxCQThRYnJ2eU9xcXVlR1RjcVZ2STBOQUNlNmhkcmlhSmZr?=
 =?utf-8?B?VzltZTZSY3lvYy9FQkhwNVh2S3hySTBhTFVIUnE3U0luMTAzSnNuSHAwWGJG?=
 =?utf-8?B?TmlYS1Y0V0pMejUxSFhYYUdLSFVicG5tbUl1QjZwblJkVTFGM01qckIzQlh1?=
 =?utf-8?B?TzE0MGJHNTJnajUvQmpVU3VMOGx2STBYd0lneXZ1dUUzOGY2K24yVnVhWHN4?=
 =?utf-8?B?SlQ5clQ3amJ4WjFWVXF5Sk9Za1Y2U1kzVUFRZENvNlc5S3JKOUNzRlU0TEE2?=
 =?utf-8?B?ckJXRHNrMTdqVGZCOXl5T3dndWx6L2prWnYvQk05T0UxRGZocFZLZ1VYOENv?=
 =?utf-8?B?VnI4UEFLK0FmT0g1TFI4bER2eDBXUzFZMlovRDBJYk05WlAxSmtmMHhEY1J5?=
 =?utf-8?B?eS9jMnBXS3pWVy9vWjY3MVVMcXR6dEFkY0JiTXNNNHZzdTgzQ0pGU0NJYThX?=
 =?utf-8?B?dXo4aWF3bU9MTzJmRjhRSHdFenBGS3lXZUtIOXpUd0o4RXp2VXNueWtFanR3?=
 =?utf-8?B?ZkhlcU1FOGw0elhqcDRjbUlibUpKUW8rekpqT2FjVTMrY0FBSWdkWkVwWDFi?=
 =?utf-8?B?a1pQbGFnZnloNENlcjJoWWg3cWtsajA0cE84VndObFRJQ2pBVnZEYXRhWkZo?=
 =?utf-8?B?UXg4OElaSFl2R1ZLMFlhVkhadk5tbG56MFZYWFNzekovVnJqMHhnOUlteUJu?=
 =?utf-8?B?KzFhdUdpNm8vUnRoRml3bER3cWNBRk1MZTUzZEtjN1hKT05Wb1lKYjNiTGpU?=
 =?utf-8?B?eVJjWFdRaUZpZm9DcHlBZ2FMaEw1VCtJakNwdy9lUERmd01kbmlGUVNYVXpQ?=
 =?utf-8?B?VjZWM0NXemJJd0o1eFYxMnlyeXhFTnJDbERXZStzZG1PY2JPNjVzNmsrMkRZ?=
 =?utf-8?Q?gyrXmVwZAdT9ZEC3qt750WGc8RgSZBZsB6clj4M?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: dfd77367-aa11-45e5-ef40-08d973469a3e
X-MS-Exchange-CrossTenant-AuthSource: DM4PR12MB5133.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Sep 2021 04:02:12.5084
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 7EWvkY1SRC99zunzkk+9JwG2SHeDmCUY2WWVQV9YJLJ0d87x+F2Z93ApOfz03yEa5uAFliTZPhk2E8W368OSaA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB5325
Precedence: bulk
List-ID: <linux-ext4.vger.kernel.org>
X-Mailing-List: linux-ext4@vger.kernel.org

Am 2021-09-02 um 4:18 a.m. schrieb Christoph Hellwig:
> On Wed, Sep 01, 2021 at 11:40:43AM -0400, Felix Kuehling wrote:
>>>>> It looks like I'm totally misunderstanding what you are adding here
>>>>> then.  Why do we need any special treatment at all for memory that
>>>>> has normal struct pages and is part of the direct kernel map?
>>>> The pages are like normal memory for purposes of mapping them in CPU
>>>> page tables and for coherent access from the CPU.
>>> That's the user page tables.  What about the kernel direct map?
>>> If there is a normal kernel struct page backing there really should
>>> be no need for the pgmap.
>> I'm not sure. The physical address ranges are in the UEFI system address
>> map as special-purpose memory. Does Linux create the struct pages and
>> kernel direct map for that without a pgmap call? I didn't see that last
>> time I went digging through that code.
> So doing some googling finds a patch from Dan that claims to hand EFI
> special purpose memory to the device dax driver.  But when I try to
> follow the version that got merged it looks it is treated simply as an
> MMIO region to be claimed by drivers, which would not get a struct page.
>
> Dan, did I misunderstand how E820_TYPE_SOFT_RESERVED works?
>
>>>> From an application
>>>> perspective, we want file-backed and anonymous mappings to be able to
>>>> use DEVICE_PUBLIC pages with coherent CPU access. The goal is to
>>>> optimize performance for GPU heavy workloads while minimizing the need
>>>> to migrate data back-and-forth between system memory and device memory.
>>> I don't really understand that part.  file backed pages are always
>>> allocated by the file system using the pagecache helpers, that is
>>> using the page allocator.  Anonymouns memory also always comes from
>>> the page allocator.
>> I'm coming at this from my experience with DEVICE_PRIVATE. Both
>> anonymous and file-backed pages should be migrateable to DEVICE_PRIVATE
>> memory by the migrate_vma_* helpers for more efficient access by our
>> GPU. (*) It's part of the basic premise of HMM as I understand it. I
>> would expect the same thing to work for DEVICE_PUBLIC memory.
> Ok, so you want to migrate to and from them.  Not use DEVICE_PUBLIC
> for the actual page cache pages.  That maks a lot more sense.
>
>> I see DEVICE_PUBLIC as an improved version of DEVICE_PRIVATE that allows
>> the CPU to map the device memory coherently to minimize the need for
>> migrations when CPU and GPU access the same memory concurrently or
>> alternatingly. But we're not going as far as putting that memory
>> entirely under the management of the Linux memory manager and VM
>> subsystem. Our (and HPE's) system architects decided that this memory is
>> not suitable to be used like regular NUMA system memory by the Linux
>> memory manager.
> So yes.  It is a Memory Mapped I/O region, which unlike the PCIe BARs
> that people typically deal with is fully cache coherent.  I think this
> does make more sense as a description.
>
> But to go back to what start this discussion:  If these are memory
> mapped I/O pfn_valid should generally not return true for them.

As I understand it, pfn_valid should be true for any pfn that's part of
the kernel's physical memory map, i.e. is returned by page_to_pfn or
works with pfn_to_page. Both the hmm_range_fault and the migrate_vma_*
APIs use pfns to refer to regular system memory and ZONE_DEVICE pages
(even DEVICE_PRIVATE). Therefore I believe pfn_valid should be true for
ZONE_DEVICE pages as well.

Regards,
  Felix


>
> And as you already pointed out in reply to Alex we need to tighten the
> selection criteria one way or another.