Received-SPF: softfail (google.com: domain of transitioning linux-ext4-owner@vger.kernel.org does not designate 23.128.96.19 as permitted sender) client-ip=23.128.96.19;
Message-ID: <bf16195e-2570-3687-2b53-3f597ebfcfec@amd.com>
Date:   Thu, 17 Feb 2022 16:12:20 -0500
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.5.0
Subject: Re: [PATCH v6 01/10] mm: add zone device coherent type memory support
Content-Language: en-US
To:     Jason Gunthorpe <jgg@nvidia.com>,
        David Hildenbrand <david@redhat.com>
Cc:     Alistair Popple <apopple@nvidia.com>,
        Christoph Hellwig <hch@lst.de>,
        Alex Sierra <alex.sierra@amd.com>, akpm@linux-foundation.org,
        linux-mm@kvack.org, rcampbell@nvidia.com,
        linux-ext4@vger.kernel.org, linux-xfs@vger.kernel.org,
        amd-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org,
        jglisse@redhat.com, willy@infradead.org
References: <beb38138-2266-1ff8-cc82-8fe914bed862@redhat.com>
 <877d9vd10u.fsf@nvdebian.thelocal> <20220216020357.GD4160@nvidia.com>
 <6156515.kVgMqSaHHm@nvdebian>
 <98d8bbc5-ffc2-8966-fdc1-a844874e7ae8@redhat.com>
 <20220216122600.GG4160@nvidia.com>
From:   Felix Kuehling <felix.kuehling@amd.com>
In-Reply-To: <20220216122600.GG4160@nvidia.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?bGgvK0hBTWcwa1Zac3I4NkVvTjY2S1lKUllZbEg3SUFnRG5iLzlHaFk1ZWUy?=
 =?utf-8?B?QnNlM1BFeWhVcDhkMTNMekxQaGV3bzdHNFBQZDNJYXN4SEVXOXIxUEZIV3NV?=
 =?utf-8?B?YnZxRzJrT3NtSDZxLzl5dkd5UDRDUXV6U3Ryc0xUS1RBTGNDVXh1eE1nMUd1?=
 =?utf-8?B?YmgzcGNKVm8wWUVwVEZ1bWFMTWMybExMNlg2Z3NmRG1jVGhEbWhjYWNYM2pR?=
 =?utf-8?B?S05yTWhteFJxekhwb0RNOHlwaHZqa2ZwWVVYUE5MUDYzdDVVQ1F2UkhNendU?=
 =?utf-8?B?dHRRVlBoalRXK0tCdm1QdTQ0UWlCMEc4UHFmWVhYT21PVVBmMktaVUUvOGxj?=
 =?utf-8?B?ZEk2VFU1cGpIZnZwdUFMZ296bGd2dlRuRGszbzVOQ0lOVzd1dlhUVjd4ejJI?=
 =?utf-8?B?Q2ExVUh1QmtGUHJRZkhkbTdvbVBrakRUMGFTa0JOWEFWS0dscENsMWhndGk2?=
 =?utf-8?B?SWVvQUp0b2w3SDlHUkpqaVJKNmN4cFhUZUx1TGFmWDE1VDBTdm95TS9pbHlM?=
 =?utf-8?B?ZVhYd2g1WDVaSlJlTC9vS1MzSlBZZVIyVThLUjJkWHF0Z2tSVzVMd0FRODlC?=
 =?utf-8?B?d0ZscklVMEVFbWwzVUY0ZHBiZDd5dE1rVkhKR2E1RDVtcUp5QllNSjNEOURJ?=
 =?utf-8?B?bFRQWTFaa3Z3bFJwYlVKdUFOc1FidGNJZUhpRUFlMjZBMzhiNXdDRkdYUVAy?=
 =?utf-8?B?RVlYb1VnUkdQSlZHWFgxbjFkUlc1R3dXMHoxaU5GU21TU3ZlTFdiNVZhZFV3?=
 =?utf-8?B?S21SWEVUbzBrVVlJUHIwRnhEMWJmS0x5TStvTTZVK1ozZFh2Y0d3L2RvR3dX?=
 =?utf-8?B?dnN1a2F1bXBUeDBpZU1pWkdrK3V1YStST1pBdm1lNUNyRVc2bHRhZGMzampJ?=
 =?utf-8?B?dFNxWEdvdXlWNHhhaElkR3JCQUFoNEd6TGowMktJN2Z1WTJhZm9aK0JSaGtu?=
 =?utf-8?B?YzNXaWJpMk1Gb3dESlFzK28zajRJbGFoNkROQXI1Q2RmR3psa2lRK2t2dFNw?=
 =?utf-8?B?ai9KR1RreGdtSXlEdDd0UmgyR3dia0YrWjR2dk5FcDJ4Tk9mbUFqYVhVTmw2?=
 =?utf-8?B?Z2NxMThONURwV2VYK1h1cUs4Rm1qdTdpK2dQakFWSFhFT1g3R000dkJXSlNw?=
 =?utf-8?B?UXUvaEdFWGZoLzc2ZnYxSVA1ZXlGTWRqd3g5cG56THUxK05yYjh2bUE3eVZP?=
 =?utf-8?B?VVc0NjQxd3d0alk2S21RN2RQcjlXTzQ2OU52bVN3NVF0NnFYOGZuRk5uVWZ6?=
 =?utf-8?B?TmFOdlVKUjJPOXVKZDBEL3NnRmpObzduSkZ2NzRIcjI5cVVNV1JIVjIybHpW?=
 =?utf-8?B?MXMzelhqbkxjWVU4VzVEL3hzQ3JJNkhnM2NvZm5hUGFic0pqZTV5cnJUMWpu?=
 =?utf-8?B?MllwcUtPMnp6V1pzZHZNTjBmRTc5bWJXUHZ0VjNWQ0ZSUzNhaFNpSFpQdFZ3?=
 =?utf-8?B?a3N0THEzdDB4QWcra2xQZ0pVSnlnUTBPdWJrTjBaNU5lNkxIbFAvZFlSOEZj?=
 =?utf-8?B?VXBrSE9LeExjb1lWNzJ5TTZwOGs1V1YycHB4MG8xK1o2WmEyOGdHTWcwWFNP?=
 =?utf-8?B?L2tPcEhmdGRlMDErdk9hS2srNzBya2x4YW1id2RxRzBHMS9rNlhlOEt3QVFk?=
 =?utf-8?B?WWtMcFBDVG04VHFOdXoybDRQVWdVRkRTMmRPTE9JMWtoQ0JGYVlYRHluZDZQ?=
 =?utf-8?B?M0xvSWY5cEdwemxHOXBJRi9CamlCMlpGQU1GdHhNcVFJN1RhU1ROUnJVdlZk?=
 =?utf-8?B?UUVBbStoWjRydStIMVJ2LzRUM0FpMnR5cDBuWDFoOXlWTUJ4MTFOc0dYbkQw?=
 =?utf-8?B?OElvSk9ad1JPdnlJWG16RTRIK1oxYmhnbEsxMWpUdjBMQTFMUTF3MGNuWXZh?=
 =?utf-8?B?dkhUZjg1Z2lBdFBjMU1MLzUyMUNPS0JLYzNWZVE0eDc0bmU5ZEUvaDYra2ZB?=
 =?utf-8?B?K0o5bnc3S2tTWlpxVGlGMkVRUmtPUTBpZyt3MlByQXc1dzlkUE5xS3ZYeEZh?=
 =?utf-8?B?TXFRQWhmYXlNTzIxZWpYR3JlWk9OcjRLMUZOWWFOb3VjZEZrM2Z6U3kxMzJn?=
 =?utf-8?B?YlpQWS9Yenh1RjVMTGI4b2JaTWM3dS9qR0QzdE9SbWxTY3BTZkIrcU9ZNXY0?=
 =?utf-8?B?R3JRQUtiMFF5MHFUVThWUGpGM2FuRmJlZDhacDNEQmg2VHlEdXU3QnFMdHJM?=
 =?utf-8?Q?cDM7DF/yrdi82qo0TrLi3FQ=3D?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 83202489-23e7-4b52-3dce-08d9f25a306d
X-MS-Exchange-CrossTenant-AuthSource: BN9PR12MB5115.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Feb 2022 21:12:22.4498
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: XS+SgGaKTdizF99QaQObmvMvQmPDzfSwaDoANF8w4s6LTwgpubJCDcEcQpgYO55qcK9PiPs8DrS2K1S3Ko/SDA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR12MB1500
X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,NICE_REPLY_A,RDNS_NONE,SPF_HELO_NONE,
	T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-ext4.vger.kernel.org>
X-Mailing-List: linux-ext4@vger.kernel.org

Am 2022-02-16 um 07:26 schrieb Jason Gunthorpe:
> The other place that needs careful audit is all the callers using
> vm_normal_page() - they must all be able to accept a ZONE_DEVICE page
> if we don't set pte_devmap.

How much code are we talking about here? A quick search finds 26 
call-sites in 12 files in current master:

    fs/proc/task_mmu.c
    mm/hmm.c
    mm/gup.c
    mm/huge_memory.c (vm_normal_page_pmd)
    mm/khugepaged.c
    mm/madvise.c
    mm/mempolicy.c
    mm/memory.c
    mm/mlock.c
    mm/migrate.c
    mm/mprotect.c
    mm/memcontrol.c

I'm thinking of a more theoretical approach: Instead of auditing all 
users, I'd ask, what are the invariants that a vm_normal_page should 
have. Then check, whether our DEVICE_COHERENT pages satisfy them. But 
maybe the concept of a vm_normal_page isn't defined clearly enough for that.

That said, I think we (Alex and myself) made an implicit assumption from 
the start, that a DEVICE_COHERENT page should behave a lot like a normal 
page in terms of VMA mappings, even if we didn't know what that means in 
detail.

I can now at least name some differences between DEVICE_COHERENT and 
normal pages: how the memory is allocated, how data is migrated into 
DEVICE_COHERENT pages and that it can't be on any LRU list (because the 
lru list_head in struct page is aliased by pgmap and zone_device_data). 
Maybe I'll find more differences if I keep digging.

Regards,
   Felix


>
> Jason