Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <400baba7-1cd6-09d4-4de9-2a73f08afc79@deltatee.com>
Date:   Thu, 2 Jun 2022 10:16:10 -0600
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.9.0
Content-Language: en-CA
To:     Jason Gunthorpe <jgg@ziepe.ca>
Cc:     linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org,
        linux-block@vger.kernel.org, linux-pci@vger.kernel.org,
        linux-mm@kvack.org, iommu@lists.linux-foundation.org,
        Stephen Bates <sbates@raithlin.com>,
        Christoph Hellwig <hch@lst.de>,
        Dan Williams <dan.j.williams@intel.com>,
        =?UTF-8?Q?Christian_K=c3=b6nig?= <christian.koenig@amd.com>,
        John Hubbard <jhubbard@nvidia.com>,
        Don Dutile <ddutile@redhat.com>,
        Matthew Wilcox <willy@infradead.org>,
        Daniel Vetter <daniel.vetter@ffwll.ch>,
        Jakowski Andrzej <andrzej.jakowski@intel.com>,
        Minturn Dave B <dave.b.minturn@intel.com>,
        Jason Ekstrand <jason@jlekstrand.net>,
        Dave Hansen <dave.hansen@linux.intel.com>,
        Xiong Jianxin <jianxin.xiong@intel.com>,
        Bjorn Helgaas <helgaas@kernel.org>,
        Ira Weiny <ira.weiny@intel.com>,
        Robin Murphy <robin.murphy@arm.com>,
        Martin Oliveira <martin.oliveira@eideticom.com>,
        Chaitanya Kulkarni <ckulkarnilinux@gmail.com>,
        Ralph Campbell <rcampbell@nvidia.com>,
        Bjorn Helgaas <bhelgaas@google.com>
References: <20220407154717.7695-1-logang@deltatee.com>
 <20220407154717.7695-21-logang@deltatee.com>
 <20220527125501.GD2960187@ziepe.ca>
 <a2590e27-41e8-59dc-3576-b5b8d716a198@deltatee.com>
 <20220527190307.GG2960187@ziepe.ca>
 <d336cfe8-2451-04c3-a2ce-0e8e47afd1e3@deltatee.com>
 <20220602000038.GK2960187@ziepe.ca>
From:   Logan Gunthorpe <logang@deltatee.com>
In-Reply-To: <20220602000038.GK2960187@ziepe.ca>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Subject: Re: [PATCH v6 20/21] PCI/P2PDMA: Introduce pci_mmap_p2pmem()
Precedence: bulk


On 2022-06-01 18:00, Jason Gunthorpe wrote:
> On Fri, May 27, 2022 at 04:41:08PM -0600, Logan Gunthorpe wrote:
>>>
>>> IIRC this is the last part:
>>>
>>> https://lore.kernel.org/linux-mm/20220524190632.3304-1-alex.sierra@amd.com/
>>>
>>> And the earlier bit with Christoph's pieces looks like it might get
>>> merged to v5.19..
>>>
>>> The general idea is once pte_devmap is not set then all the
>>> refcounting works the way it should. This is what all new ZONE_DEVICE
>>> users should do..
>>
>> Ok, I don't actually follow how those patches relate to this.
>>
>> Based on your description I guess I don't need to set PFN_DEV and
> 
> Yes
> 
>> perhaps not use vmf_insert_mixed()? And then just use vm_normal_page()?
> 
> I'm not sure ATM the best function to use, but yes, a function that
> doesn't set PFN_DEV is needed here.
>  
>> But the refcounting of the pages seemed like it was already sane to me,
>> unless you mean that the code no longer has to synchronize_rcu() before
>> returning the pages... 
> 
> Right. It also doesn't need to call unmap range or keep track of the
> inode, or do any of that stuff unless it really needs mmap revokation
> semantics (which I doubt this use case does)
> 
> unmap range was only necessary because the refcounting is wrong -
> since the pte's don't hold a ref on the page in PFN_DEV mode it is
> necessary to wipe all the PTE explicitly before going ahead to
> decrement the refcount on this path.
> 
> Just stuff the pages into the mmap, and your driver unprobe will
> automatically block until all the mmaps are closed - no different than
> having an open file descriptor or something.

Oh is that what we want? With the current method the mmaps are unmapped
on unbind so that it doesn't block indefinitely. It seems more typical
for resources to be dropped quickly on unbind and processes that are
using them will get an error on next use.

Logan