Received-SPF: pass (google.com: domain of linux-ext4-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Wed, 21 Sep 2022 08:18:38 -0700
From:   Dan Williams <dan.j.williams@intel.com>
To:     Jason Gunthorpe <jgg@nvidia.com>,
        Dan Williams <dan.j.williams@intel.com>
CC:     <akpm@linux-foundation.org>, Matthew Wilcox <willy@infradead.org>,
        "Jan Kara" <jack@suse.cz>, "Darrick J. Wong" <djwong@kernel.org>,
        "Christoph Hellwig" <hch@lst.de>,
        John Hubbard <jhubbard@nvidia.com>,
        <linux-fsdevel@vger.kernel.org>, <nvdimm@lists.linux.dev>,
        <linux-xfs@vger.kernel.org>, <linux-mm@kvack.org>,
        <linux-ext4@vger.kernel.org>
Subject: Re: [PATCH v2 10/18] fsdax: Manage pgmap references at entry
 insertion and deletion
Message-ID: <632b2b4edd803_66d1a2941a@dwillia2-xfh.jf.intel.com.notmuch>
References: <166329930818.2786261.6086109734008025807.stgit@dwillia2-xfh.jf.intel.com>
 <166329936739.2786261.14035402420254589047.stgit@dwillia2-xfh.jf.intel.com>
 <YysZrdF/BSQhjWZs@nvidia.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <YysZrdF/BSQhjWZs@nvidia.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?+xjFArwJyi19KjNXS1n2b+NRYObKNRxykIbTERcXujVIGxRyRLnA0Urz5DiL?=
 =?us-ascii?Q?9EwuL0bWSntvlhY9nKAOuL7buR5cSh6qupilCq3PtBCEZNOjrRu1Q5SL5Q+X?=
 =?us-ascii?Q?FNbV6+yGuV5PdwJ72kL3MTj//JzhBdlqv8rM/nrbck2CMd4ayNm7ENzZHCvk?=
 =?us-ascii?Q?dh2dxXQQpDUsFqHQ1B4AbY/PmRmUT1HovF9llrwZhU7zoyj5jWKMcBFrnGNU?=
 =?us-ascii?Q?aoArxPR8mE24k1PciJixhshdV5HeYJB0JGXBPnULZY5nWQ+rR50kc2IHCfPH?=
 =?us-ascii?Q?C/1yzzpbhjAF/y3cGZD5DrqtE66NQNWLPJ8qnT242Vw7BdHCip2WP/a417k1?=
 =?us-ascii?Q?JlVI56zo+CG/oh7Pc4Ipjl9R85Ln07kz/vvRaS+LFB6Z91DXmMMXlvGXRvAK?=
 =?us-ascii?Q?GiXGjSjWlWzJ1OqLfHoCI+2lLfZ+e4/7AaaTfgnmj42V/2ckZZMR5VWIrnaU?=
 =?us-ascii?Q?a/B70Jua7j+zlfMyPFIhpjPtlEVROrgPhSQQ3H7GVtDDj7CL3YSxjX8PRQCP?=
 =?us-ascii?Q?WqbKiHUi30fkGgVEuTjBPiV2mlSLtwqX7i4SDsVWycSS+xeBRNYZ9vbfcX8Y?=
 =?us-ascii?Q?nAY570aFI0BxrmcqHPaEDCFq/snux7SJ3x8nntq1Sru6+VvwYi58cgesMvhm?=
 =?us-ascii?Q?jCGeGA0ssOWJhdz7UDAI9gcQTpy22GXuh5zod6l3UiBAaMsqM2OwRORIHm7l?=
 =?us-ascii?Q?lpf1yY54dVphUGUX8uWgukGrHFZxa/mna7kb8BxGXBWTvEHsYvvVC+svcbKf?=
 =?us-ascii?Q?WJd/2dbE7xSkL7YtEwRcE6HaifTTuI/MemHGIBDjksIoYxmtTomY7F1YIIgx?=
 =?us-ascii?Q?r1xhLMq0Izrxx6O2CkMfuShsSRUCrX7eJfge4U4vJDl55nF5WOQiUcdQTCFf?=
 =?us-ascii?Q?PjB5ZnZRC9Zoee8Ith3xAAlzpxuiHBooYD646VZyu7kacVdRszuxHNZSjx/z?=
 =?us-ascii?Q?44efWzhV6ybxHhyl5LaaVeXfD19IiNU3f/3EWL2DZipvo7oOsWd0i6qYN3GP?=
 =?us-ascii?Q?LJBgQnCDXSySM8TM+c7530C22tgCr3xPk9qVoojTqNzd7MfK3uquGHhqgkBw?=
 =?us-ascii?Q?HcuB41pZgPlS6KTMFRSDcLI2+woqj0huljVsMT4D8B1nPzO7xsZ9JCE7NVWq?=
 =?us-ascii?Q?VS5pp8YoQB+l7F2j2ZNFh2EbvcEV2Bhj+5iJGy11Gn3726y2uc4NaI3vWW7b?=
 =?us-ascii?Q?PmU+bqB0X4xXUPbn6Xgdp5Hl3kVYEPbehAWH09ca/WwRnIAVZRqQ2prUZXU8?=
 =?us-ascii?Q?gG0huRZrLwv3TtzLsC0zmT6PMBMsUE4QF3kbmjQxg38sdm5n+RmazUpCvdba?=
 =?us-ascii?Q?9H4AmxsFORp468rMZaHdvqo0+vJ5OunjALX3XtPsnPDJI5B9RkPRMaytVS4s?=
 =?us-ascii?Q?2l/NquB004vwvQtnlO/gw5E5szvcc5W9rru34hhBsxLZB3xSnPmgVqCK7LdO?=
 =?us-ascii?Q?xSPWyqbzqXTlhsQJ3whFw1rlFIVlSSUtZoU/1GpUyxivDzVYPhzgo4TiytGX?=
 =?us-ascii?Q?tml2wJJnM9j7V/QOecDfBHeGayACOu7CPnimib4L3je95/GAO5n8qW2QRKP1?=
 =?us-ascii?Q?hN2mr8jR6YszsSk1JAZOjGhcR9OhZaDW3GtmWhRGG76QM/OpBIA9aQz5jix6?=
 =?us-ascii?Q?gQ=3D=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 2dccb2ca-07d9-4a6b-19b4-08da9be490fa
X-MS-Exchange-CrossTenant-AuthSource: MWHPR1101MB2126.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Sep 2022 15:18:41.5067
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 1hUdP0M8tIB9QHVN2DCghDNtlGaQFa0qEkmA0w2pcDwf5BEhRHUCUVCPn8unAOM6ujGdfHLnsqWgmSNxc6aNUiX63z6gNVntKnOvir9Xj5k=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR11MB7350
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-ext4.vger.kernel.org>
X-Mailing-List: linux-ext4@vger.kernel.org

Jason Gunthorpe wrote:
> On Thu, Sep 15, 2022 at 08:36:07PM -0700, Dan Williams wrote:
> > The percpu_ref in 'struct dev_pagemap' is used to coordinate active
> > mappings of device-memory with the device-removal / unbind path. It
> > enables the semantic that initiating device-removal (or
> > device-driver-unbind) blocks new mapping and DMA attempts, and waits for
> > mapping revocation or inflight DMA to complete.
> 
> This seems strange to me
> 
> The pagemap should be ref'd as long as the filesystem is mounted over
> the dax. The ref should be incrd when the filesystem is mounted and
> decrd when it is unmounted.
> 
> When the filesystem unmounts it should zap all the mappings (actually
> I don't think you can even unmount a filesystem while mappings are
> open) and wait for all page references to go to zero, then put the
> final pagemap back.
> 
> The rule is nothing can touch page->pgmap while page->refcount == 0,
> and if page->refcount != 0 then page->pgmap must be valid, without any
> refcounting on the page map itself.
> 
> So, why do we need pgmap refcounting all over the place? It seems like
> it only existed before because of the abuse of the page->refcount?

Recall that this percpu_ref is mirroring the same function as
blk_queue_enter() whereby every new request is checking to make sure the
device is still alive, or whether it has started exiting.

So pgmap 'live' reference taking in fs/dax.c allows the core to start
failing fault requests once device teardown has started. It is a 'block
new, and drain old' semantic.