Received-SPF: pass (google.com: domain of linux-ext4-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates
 216.228.112.32 as permitted sender) receiver=protection.outlook.com;
 client-ip=216.228.112.32; helo=mail.nvidia.com;
Subject: Re: [PATCH v3 2/8] mm: remove extra ZONE_DEVICE struct page refcount
To:     Felix Kuehling <felix.kuehling@amd.com>,
        Alex Sierra <alex.sierra@amd.com>, <akpm@linux-foundation.org>,
        <linux-mm@kvack.org>, <linux-ext4@vger.kernel.org>,
        <linux-xfs@vger.kernel.org>, "Yang, Philip" <Philip.Yang@amd.com>
CC:     <amd-gfx@lists.freedesktop.org>, <dri-devel@lists.freedesktop.org>,
        <hch@lst.de>, <jgg@nvidia.com>, <jglisse@redhat.com>
References: <20210617151705.15367-1-alex.sierra@amd.com>
 <20210617151705.15367-3-alex.sierra@amd.com>
 <7163dbb6-67b5-6eef-5772-500fd2107e5c@nvidia.com>
 <cecd2164-ebfd-382b-af73-992cdc4304b7@amd.com>
From:   Ralph Campbell <rcampbell@nvidia.com>
Message-ID: <4f6dd918-d79b-1aa7-3a4c-caa67ddc29bc@nvidia.com>
Date:   Tue, 29 Jun 2021 17:23:24 -0700
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.10.1
MIME-Version: 1.0
In-Reply-To: <cecd2164-ebfd-382b-af73-992cdc4304b7@amd.com>
Content-Type: text/plain; charset="utf-8"; format=flowed
Content-Transfer-Encoding: 8bit
Content-Language: en-US
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Jun 2021 00:23:25.6836
 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: 1d9fe82a-6e48-4743-715d-08d93b5d46f9
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.112.32];Helo=[mail.nvidia.com]
X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT019.eop-nam11.prod.protection.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Anonymous
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR12MB5406
Precedence: bulk
List-ID: <linux-ext4.vger.kernel.org>
X-Mailing-List: linux-ext4@vger.kernel.org

On 6/28/21 9:46 AM, Felix Kuehling wrote:
> Am 2021-06-17 um 3:16 p.m. schrieb Ralph Campbell:
>> On 6/17/21 8:16 AM, Alex Sierra wrote:
>>> From: Ralph Campbell <rcampbell@nvidia.com>
>>>
>>> ZONE_DEVICE struct pages have an extra reference count that
>>> complicates the
>>> code for put_page() and several places in the kernel that need to
>>> check the
>>> reference count to see that a page is not being used (gup, compaction,
>>> migration, etc.). Clean up the code so the reference count doesn't
>>> need to
>>> be treated specially for ZONE_DEVICE.
>>>
>>> v2:
>>> AS: merged this patch in linux 5.11 version
>>>
>>> Signed-off-by: Ralph Campbell <rcampbell@nvidia.com>
>>> Signed-off-by: Alex Sierra <alex.sierra@amd.com>
>>> ---
>>>    arch/powerpc/kvm/book3s_hv_uvmem.c     |  2 +-
>>>    drivers/gpu/drm/nouveau/nouveau_dmem.c |  2 +-
>>>    fs/dax.c                               |  4 +-
>>>    include/linux/dax.h                    |  2 +-
>>>    include/linux/memremap.h               |  7 +--
>>>    include/linux/mm.h                     | 44 -----------------
>>>    lib/test_hmm.c                         |  2 +-
>>>    mm/internal.h                          |  8 +++
>>>    mm/memremap.c                          | 68 +++++++-------------------
>>>    mm/migrate.c                           |  5 --
>>>    mm/page_alloc.c                        |  3 ++
>>>    mm/swap.c                              | 45 ++---------------
>>>    12 files changed, 45 insertions(+), 147 deletions(-)
>>>
>> I think it is great that you are picking this up and trying to revive it.
>>
>> However, I have a number of concerns about how it affects existing
>> ZONE_DEVICE
>> MEMORY_DEVICE_GENERIC and MEMORY_DEVICE_FS_DAX users and I don't see this
>> addressing them. For example, dev_dax_probe() allocates
>> MEMORY_DEVICE_GENERIC
>> struct pages and then:
>>    dev_dax_fault()
>>      dev_dax_huge_fault()
>>        __dev_dax_pte_fault()
>>          vmf_insert_mixed()
>> which just inserts the PFN into the CPU page tables without increasing
>> the page
>> refcount so it is zero (whereas it was one before). But using
>> get_page() will
>> trigger VM_BUG_ON_PAGE() if it is enabled. There isn't any current
>> notion of
>> free verses allocated for these struct pages. I suppose init_page_count()
>> could be called on all the struct pages in dev_dax_probe() to fix that
>> though.
> Hi Ralph,
>
> For DEVICE_GENERIC pages free_zone_device_page doesn't do anything. So
> I'm not sure what the reference counting is good for in this case.
>
> Alex is going to add free_zone_device_page support for DEVICE_GENERIC
> pages (patch 8 of this series). However, even then, it only does
> anything if there is an actual call to put_page. Where would that call
> come from in the dev_dax driver case?

Correct, the drivers/dax/device.c driver allocates MEMORY_DEVICE_GENERIC
struct pages and doesn't seem to allocate/free the page nor increment/decrement
the reference count but it does call vmf_insert_mixed() if the /dev/file
is mmap()'ed into a user process' address space. If devm_memremap_pages()
returns the array of ZONE_DEVICE struct pages initialized with a reference
count of zero, then the CPU page tables will have a PTE/PFN that points to
a struct page with a zero reference count. This goes against the normal
expectation in the rest of the mm code that assumes a page mapped by a CPU
has a non-zero reference count.
So yes, nothing "bad" happens because put_page() isn't called but the
reference count will differ from other drivers that call vmf_insert_mixed()
or vm_insert_page() where the page was allocated with alloc_pages() or
similar.

>> I'm even less clear about how to fix MEMORY_DEVICE_FS_DAX. File
>> systems have clear
>> allocate and free states for backing storage but there are the
>> complications with
>> the page cache references, etc. to consider. The >1 to 1 reference
>> count seems to
>> be used to tell when a page is idle (no I/O, reclaim scanners) rather
>> than free
>> (not allocated to any file) but I'm not 100% sure about that since I
>> don't really
>> understand all the issues around why a file system needs to have a DAX
>> mount option
>> besides knowing that the storage block size has to be a multiple of
>> the page size.
> The only thing that happens in free_zone_device_page for FS_DAX pages is
> wake_up_var(&page->_refcount). I guess, whoever is waiting for this
> wake-up will need to be prepared to see a refcount 0 instead of 1 now. I
> see these callers that would need to be updated:
>
> ./fs/ext4/inode.c:        error = ___wait_var_event(&page->_refcount,
> ./fs/ext4/inode.c-                atomic_read(&page->_refcount) == 1,
> ./fs/ext4/inode.c-                TASK_INTERRUPTIBLE, 0, 0,
> ./fs/ext4/inode.c-                ext4_wait_dax_page(ei));
> --
> ./fs/fuse/dax.c:    return ___wait_var_event(&page->_refcount,
> ./fs/fuse/dax.c-            atomic_read(&page->_refcount) == 1,
> TASK_INTERRUPTIBLE,
> ./fs/fuse/dax.c-            0, 0, fuse_wait_dax_page(inode));
> --
> ./fs/xfs/xfs_file.c:    return ___wait_var_event(&page->_refcount,
> ./fs/xfs/xfs_file.c-            atomic_read(&page->_refcount) == 1,
> TASK_INTERRUPTIBLE,
> ./fs/xfs/xfs_file.c-            0, 0, xfs_wait_dax_page(inode));
>
> Regarding your page-cache comment, doesn't DAX mean that those file
> pages are not in the page cache?
>
> Regards,
>    Felix
>
I don't really understand the FS_DAX code. I can see the __wait_var_event()
is being used when truncating or punching holes in files but I'm not
quite sure if it is using the >1 to 1 reference count to know when a
page has no "extra" references or if it means the page is actually
"free" and no longer assigned to a file.
I really think some FS_DAX expert needs to weigh in on these reference count
changes.