Received-SPF: pass (google.com: domain of linux-ext4-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Wed, 16 Feb 2022 08:26:00 -0400
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     David Hildenbrand <david@redhat.com>
Cc:     Alistair Popple <apopple@nvidia.com>,
        Felix Kuehling <felix.kuehling@amd.com>,
        Christoph Hellwig <hch@lst.de>,
        Alex Sierra <alex.sierra@amd.com>, akpm@linux-foundation.org,
        linux-mm@kvack.org, rcampbell@nvidia.com,
        linux-ext4@vger.kernel.org, linux-xfs@vger.kernel.org,
        amd-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org,
        jglisse@redhat.com, willy@infradead.org
Subject: Re: [PATCH v6 01/10] mm: add zone device coherent type memory support
Message-ID: <20220216122600.GG4160@nvidia.com>
References: <beb38138-2266-1ff8-cc82-8fe914bed862@redhat.com>
 <877d9vd10u.fsf@nvdebian.thelocal>
 <20220216020357.GD4160@nvidia.com>
 <6156515.kVgMqSaHHm@nvdebian>
 <98d8bbc5-ffc2-8966-fdc1-a844874e7ae8@redhat.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <98d8bbc5-ffc2-8966-fdc1-a844874e7ae8@redhat.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?9wAkmSlpUxgyqXI1fJbtzBoO4SEsd1Vjy3h3sLlspZ/wwORJ7cg5UIHE7ufe?=
 =?us-ascii?Q?69n1GKqTtqS6C2azHKbwzZSUlOx8zSIs6XegglhAFEkADhM5BB1a2iw2QJJ4?=
 =?us-ascii?Q?gTCsUepgcKH30m4++lXx96/4OpkSesxZR36UhvuQd28yST6/fPBtwveVGoXw?=
 =?us-ascii?Q?DFsR1dC2JzPqh52hHywoiL/iCa1Zdke/poJo6TkGH5+2rLQgGyiPq8tGvyMv?=
 =?us-ascii?Q?i9C/qJxArsUNarb/5VO1KO6KXyqRKY3YhJoOYqUccZauVJQJjJyZxWsVfR1p?=
 =?us-ascii?Q?NXCKKm8+xeEgjmVa/ISZBQYwfKWACcHXHy/BDRomHBQkJ/jkXuQwhR0HS4fS?=
 =?us-ascii?Q?qxrbPFuODrxwkTOf8HpcD0wdg9WtL4pD5SQ1276ob+gikzS38dTrnXVLcpuh?=
 =?us-ascii?Q?hQxLWISgL5ToNxN1reP/33xbQ2TzPO6U5R8yZMBHEDs4phnPa9oMJvBmnzOu?=
 =?us-ascii?Q?IIYhEyTBYRUquuajQ+DiBm7AHCGa3Y7V7nb+ReaLTIyqKqyE6YvD1hP2Zqt5?=
 =?us-ascii?Q?gUFZIXLZ3GIgvUSAvXcCc9u8X6V8NJX8+dHlHwouoVjt/h7rs0x+jgqXHphk?=
 =?us-ascii?Q?DwyhnGowm+v0J6MHJgtAKwd6Xzycn9jk8cX99zWSTG2H7di3+CPwIaEJOZut?=
 =?us-ascii?Q?/FFCsJ6bD192taQaD6ZNjGkNUuKl0rguKU6jNKd43Kp4fDloHA4gG4xiNo3E?=
 =?us-ascii?Q?U9/VkBMj+W6msF5eKWQ/YVfLZCTDjA5cNkZbcUH/hOAqPfVIzHAoIYTIIrms?=
 =?us-ascii?Q?GRYcDplL72XUuxZukIJq90dtUhHpkaTQQEI9j6ZVjS2lJ1fDZNjQSHsWOMO3?=
 =?us-ascii?Q?LnqcOTYlV17YeCQgwmz++jycynnS9D/Zrr5H69rqmzrkiMYk79PTqYC83Zb0?=
 =?us-ascii?Q?pIjnv1lEJdsq1HJ/zisEjFO/mnFtsTrYREfcs0SgqIDUyGu+hQ5b5vNbHmJm?=
 =?us-ascii?Q?WUAGvf/suMLl2BqqenV8apT/JIhLqlRdMHFsmbKHmP86kZC+RBOcUizhnkSS?=
 =?us-ascii?Q?/LvvKZV6UbT7ZMp7NmzAQYcq9OcytGffPeBZhE0pBdnqc+/is5P/PU/2QsDj?=
 =?us-ascii?Q?eeOIgczg0UQQ4AkkBvYBdlYLQhGdX0AlaXAxQGRlE7oofAUq4gn5M1GopeZk?=
 =?us-ascii?Q?0s8g2v91GOAEkgJ6o4RNVof1yoN5W6M1+5TSZTIyENrhXMbkaZYo8CmA1b7N?=
 =?us-ascii?Q?6W8Rnr+1fith9PUYFdoIFea96LpIv4igrNcLv+/W/OCKOvQti6A2u1lovrSx?=
 =?us-ascii?Q?GPLjmgh/rjq+7tSBQ7gMjmqOvZPrTYCs6aLCx+AWacysY/TbyF+ju+vkMUs3?=
 =?us-ascii?Q?tnCVv73T/EzKb05VV/3Fdhzlj8uODBpMjEJz1sgFMAf4F+5kn92rXC+G2dms?=
 =?us-ascii?Q?1jw1UXxG8tazPYF7KZTgNPi9nQxyTCpdAOYRwfD4Cracm+PCa9CH2Ne983wC?=
 =?us-ascii?Q?eaPEkzQntE8ynYV6vDoIUZRX3NY/iI51LpQjcU/MVYlMm0B/pLe9HRXZoxGS?=
 =?us-ascii?Q?fc18Hs79nCOziqlrO913gJcGmMYt+C0zR43bkFQeMF+69ScqX/7o5MiqnfLV?=
 =?us-ascii?Q?evQ2a37VJz/oe3q+vBs=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: a28a825f-d610-4c42-3d29-08d9f1477e52
X-MS-Exchange-CrossTenant-AuthSource: MN2PR12MB4192.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Feb 2022 12:26:01.5488
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: bSwZv7JQNTYdaBkloha6pKk3Kyh6I3ATKUg+/LEEE8KhQHTGzfqoXesuMo7CyT2t
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR12MB4999
X-Spam-Status: No, score=-1.2 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-ext4.vger.kernel.org>
X-Mailing-List: linux-ext4@vger.kernel.org

On Wed, Feb 16, 2022 at 09:31:03AM +0100, David Hildenbrand wrote:
> On 16.02.22 03:36, Alistair Popple wrote:
> > On Wednesday, 16 February 2022 1:03:57 PM AEDT Jason Gunthorpe wrote:
> >> On Wed, Feb 16, 2022 at 12:23:44PM +1100, Alistair Popple wrote:
> >>
> >>> Device private and device coherent pages are not marked with pte_devmap and they
> >>> are backed by a struct page. The only way of inserting them is via migrate_vma.
> >>> The refcount is decremented in zap_pte_range() on munmap() with special handling
> >>> for device private pages. Looking at it again though I wonder if there is any
> >>> special treatment required in zap_pte_range() for device coherent pages given
> >>> they count as present pages.
> >>
> >> This is what I guessed, but we shouldn't be able to just drop
> >> pte_devmap on these pages without any other work?? Granted it does
> >> very little already..
> > 
> > Yes, I agree we need to check this more closely. For device private pages
> > not having pte_devmap is fine, because they are non-present swap entries so
> > they always get special handling in the swap entry paths but the same isn't
> > true for coherent device pages.
> 
> I'm curious, how does the refcount of a PageAnon() DEVICE_COHERENT page
> look like when mapped? I'd assume it's also (currently) still offset by
> one, meaning, if it's mapped into a single page table it's always at
> least 2.

Christoph fixed this offset by one and updated the DEVICE_COHERENT
patchset, I hope we will see that version merged.

> >> I thought at least gup_fast needed to be touched or did this get
> >> handled by scanning the page list after the fact?
> > 
> > Right, for gup I think the only special handling required is to prevent
> > pinning. I had assumed that check_and_migrate_movable_pages() would still get
> > called for gup_fast but unless I've missed something I don't think it does.
> > That means gup_fast could still pin movable and coherent pages. Technically
> > that is ok for coherent pages, but it's undesirable.
> 
> We really should have the same pinning rules for GUP vs. GUP-fast.
> is_pinnable_page() should be the right place for such checks (similarly
> as indicated in my reply to the migration series).

Yes, I think this is a bug too.

The other place that needs careful audit is all the callers using
vm_normal_page() - they must all be able to accept a ZONE_DEVICE page
if we don't set pte_devmap.

Jason