Received-SPF: pass (google.com: domain of linux-ext4-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Thu, 22 Sep 2022 14:55:57 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Dan Williams <dan.j.williams@intel.com>
Cc:     akpm@linux-foundation.org, Matthew Wilcox <willy@infradead.org>,
        Jan Kara <jack@suse.cz>, "Darrick J. Wong" <djwong@kernel.org>,
        Christoph Hellwig <hch@lst.de>,
        John Hubbard <jhubbard@nvidia.com>,
        linux-fsdevel@vger.kernel.org, nvdimm@lists.linux.dev,
        linux-xfs@vger.kernel.org, linux-mm@kvack.org,
        linux-ext4@vger.kernel.org
Subject: Re: [PATCH v2 10/18] fsdax: Manage pgmap references at entry
 insertion and deletion
Message-ID: <YyyhrTxFJZlMGYY6@nvidia.com>
References: <166329930818.2786261.6086109734008025807.stgit@dwillia2-xfh.jf.intel.com>
 <166329936739.2786261.14035402420254589047.stgit@dwillia2-xfh.jf.intel.com>
 <YysZrdF/BSQhjWZs@nvidia.com>
 <632b2b4edd803_66d1a2941a@dwillia2-xfh.jf.intel.com.notmuch>
 <632b8470d34a6_34962946d@dwillia2-xfh.jf.intel.com.notmuch>
 <YyuLLsindwo0prz4@nvidia.com>
 <632ba8eaa5aea_349629422@dwillia2-xfh.jf.intel.com.notmuch>
 <YyurdXnW7SyEndHV@nvidia.com>
 <632bc5c4363e9_349629486@dwillia2-xfh.jf.intel.com.notmuch>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <632bc5c4363e9_349629486@dwillia2-xfh.jf.intel.com.notmuch>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?FLHSvSyGxDWSppy9l5AfVS/FeJZNn3RjsLG4wqV08IlUwM5FQ3PacwiVJah8?=
 =?us-ascii?Q?U3V6/jyTKO+Mpvp/0xN76uYyGz9KoqjDlrCHYC84aJH4YFpobsTXNUTprIAe?=
 =?us-ascii?Q?H7e01Wl15cfXB7N7RJNk/Cq1sR5FU6U4oSNDzjkx1/ZlugPOFoEarBFWhaWT?=
 =?us-ascii?Q?J2UjovucuUJy9MAg5YsYrmviJrv+7T2/2EYsNlwt8GmtxlP7df9WwzMhs/G4?=
 =?us-ascii?Q?5rp5E4Ac1geUQnoQ03YJ4vDlhchvvtAImW5lmV6MC2dN+eo8xIQye6lGZv6+?=
 =?us-ascii?Q?dCCu8zzOnccMXkpo6yeN0uATJu0YxAsryMX3wIFI3E65w4F4aFOescM7TZhM?=
 =?us-ascii?Q?Io1rHwERU5M9X/Y7orZKfFzd72D1yfKMh18mZei89dfFC/sGx4D7XM2b2kRJ?=
 =?us-ascii?Q?q9P+LhOi0HARTP8HoPbNVKsQwNKhJqXz4PvfSCGSeAkcMSdpI1H0sKH2XbAg?=
 =?us-ascii?Q?3ZQWW0FD3qtoTI0hA8NeXyDmklN2IalPG03eQ5VpXgJxlZJ2BoYbdhgXadIW?=
 =?us-ascii?Q?ryX3vTW+gOwujzuA/RG64yzIvbgyK4EdR+8OjG+Y9Msowe3s86CwQKSGtcDC?=
 =?us-ascii?Q?rXG5bCZEDhT+U972G6HXR9gXRHErjKSoSEVrnbBgwTNZbFkRe+3GWojlGg4a?=
 =?us-ascii?Q?lBsE+rsVDV3x1foVtZsJgu2duh+MiG+ILsuHCX/uNvGi1gWUg2ob1lBi6qHN?=
 =?us-ascii?Q?DZLc/mWEpkH38L6172OwinweXa7b90CnD2KeewnONZHEhWYAY6HBtspwxzSn?=
 =?us-ascii?Q?ssRpkarZbFujzZSoFN2CS5qLX/gctWMZSUfAu/z/RBK9VhP6F82UdcpXuXYz?=
 =?us-ascii?Q?BxMtUWBrS8u9HpihoZPIKpo4baPYefdiBSLUis/PR9H1rmNT2YxxpWCHVQ9u?=
 =?us-ascii?Q?OV8P+pJVt+4pwVPy9yY+MEqBm7yPD6TPXGFHv5oA2dEj5gZPYeZRJDCNWayv?=
 =?us-ascii?Q?okjOqiSyUQMVegEMkbZ5786DaXdQ5Fdx7gW/j8sZ/yn1DZN0GGpR1GjIDo2n?=
 =?us-ascii?Q?1TvnM33jXpNn9sZz04nCQD9LMl+tWkCsVfejF4IBgtUb5D2fBfgqjQ5IBnFH?=
 =?us-ascii?Q?+CYTTpPTw09Ohm7/FERV8STD3FZVy/0mAEsvHRZ792JeJ8VClK9RAsdz9lht?=
 =?us-ascii?Q?cSUofaBpwxSrIEcCTWx+1PVOhr5rWIhuic3YYHn9DsKDmVraUyiJY4IpJcqM?=
 =?us-ascii?Q?cDNx269BOOpYGCuJS/K7md/oNMZ+GtXZrVkVk3oI+Hi7+1pmmz4MggnEK4Yx?=
 =?us-ascii?Q?+ODJ76tHrv9Uf3sIahDzJOl0jeq8Np4rnSgHsmoYLHbRG9PNTMrowmSdjeDC?=
 =?us-ascii?Q?bP74HXNjeSGzOWGB/VTNmRAydbYZ7j85XEX5QvJoxOhtULXJFILa+GRNkHSa?=
 =?us-ascii?Q?I/GztcfVrwYFRAubiMSUXSLCkgT/GM1KkKUy35qN6tAZNK2KVBBP2pggsCoD?=
 =?us-ascii?Q?yA83DnI3Uj1bo9IpSaTVjfm5doYfAizDK9oEfJLxsxw20AwPQDE42cWF6UZP?=
 =?us-ascii?Q?9j+e8AVr1UoDeGC4rUovEBhe/+Sy7rKNN6c8mjUaB0SLQAtUwqPVLphihKxk?=
 =?us-ascii?Q?A0ofNjafndtNJ1Z8fIu1LgY6r5C8wpdon23XHAkH?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: bf053442-991b-4b96-0bec-08da9cc3b47d
X-MS-Exchange-CrossTenant-AuthSource: MN2PR12MB4192.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Sep 2022 17:55:58.9339
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: BCLf8z0i7ihkSVpuTU9r2fqVSlXKnybTXXNCSOfoox9uNcClL0m50cM+yvV/tgBk
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5850
X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE
        autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-ext4.vger.kernel.org>
X-Mailing-List: linux-ext4@vger.kernel.org

On Wed, Sep 21, 2022 at 07:17:40PM -0700, Dan Williams wrote:
> Jason Gunthorpe wrote:
> > On Wed, Sep 21, 2022 at 05:14:34PM -0700, Dan Williams wrote:
> > 
> > > > Indeed, you could reasonably put such a liveness test at the moment
> > > > every driver takes a 0 refcount struct page and turns it into a 1
> > > > refcount struct page.
> > > 
> > > I could do it with a flag, but the reason to have pgmap->ref managed at
> > > the page->_refcount 0 -> 1 and 1 -> 0 transitions is so at the end of
> > > time memunmap_pages() can look at the one counter rather than scanning
> > > and rescanning all the pages to see when they go to final idle.
> > 
> > That makes some sense too, but the logical way to do that is to put some
> > counter along the page_free() path, and establish a 'make a page not
> > free' path that does the other side.
> > 
> > ie it should not be in DAX code, it should be all in common pgmap
> > code. The pgmap should never be freed while any page->refcount != 0
> > and that should be an intrinsic property of pgmap, not relying on
> > external parties.
> 
> I just do not know where to put such intrinsics since there is nothing
> today that requires going through the pgmap object to discover the pfn
> and 'allocate' the page.

I think that is just a new API that wrappers the set refcount = 1,
percpu refcount and maybe building appropriate compound pages too.

Eg maybe something like:

  struct folio *pgmap_alloc_folios(pgmap, start, length)

And you get back maximally sized allocated folios with refcount = 1
that span the requested range.

> In other words make dax_direct_access() the 'allocation' event that pins
> the pgmap? I might be speaking a foreign language if you're not familiar
> with the relationship of 'struct dax_device' to 'struct dev_pagemap'
> instances. This is not the first time I have considered making them one
> in the same.

I don't know enough about dax, so yes very foreign :)

I'm thinking broadly about how to make pgmap usable to all the other
drivers in a safe and robust way that makes some kind of logical sense.

Jason