Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Thu, 14 Jul 2022 11:21:44 -0700
From:   Dan Williams <dan.j.williams@intel.com>
To:     "ruansy.fnst@fujitsu.com" <ruansy.fnst@fujitsu.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linux-xfs@vger.kernel.org" <linux-xfs@vger.kernel.org>,
        "nvdimm@lists.linux.dev" <nvdimm@lists.linux.dev>,
        "linux-mm@kvack.org" <linux-mm@kvack.org>,
        "linux-fsdevel@vger.kernel.org" <linux-fsdevel@vger.kernel.org>
CC:     "djwong@kernel.org" <djwong@kernel.org>,
        "dan.j.williams@intel.com" <dan.j.williams@intel.com>,
        "david@fromorbit.com" <david@fromorbit.com>,
        "hch@infradead.org" <hch@infradead.org>,
        "jane.chu@oracle.com" <jane.chu@oracle.com>
Subject: RE: [RFC PATCH v6] mm, pmem, xfs: Introduce MF_MEM_REMOVE for unbind
Message-ID: <62d05eb8e663c_1643dc294fa@dwillia2-xfh.jf.intel.com.notmuch>
References: <20220410171623.3788004-1-ruansy.fnst@fujitsu.com>
 <20220714103421.1988696-1-ruansy.fnst@fujitsu.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20220714103421.1988696-1-ruansy.fnst@fujitsu.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?v7Z77dRSvLV9xguhdNVMaIsm25FHaliYApNlM7xE58VUFoO2f4eIOZrhIvUI?=
 =?us-ascii?Q?MWo8Q/rZcSOu8RmJJ1o3ul1X16ebHOjXxXChnOA82szoNLB12aoiahjwifnW?=
 =?us-ascii?Q?RDgMXSlp5zAOKHd26lXnZAj9luMqPmPDsGKzPgLUaHYNFA9useaFMS2ad0PZ?=
 =?us-ascii?Q?mWmB6YLFUSZIqkDNTF+6cnGF3bWmv0vTLWEuq30s5q3Q8S6hjKZH/Y/q4d2o?=
 =?us-ascii?Q?nNTkn0qSWQ9duCUGrRcDQca3FQjbpf27eBicJk/C6cnquwDo14BPDzDXOKDl?=
 =?us-ascii?Q?U8/UWI7MRS2SJ+ggZeuNHt2N61Z3DpkW7xo2ae5AAqdhtm+7GvHPa42M2vWm?=
 =?us-ascii?Q?dM3VtKbUkiyNUvJABsRCG1iir3F1mf4dVZYJFko25/TRsYI3n4I6qf/P16eQ?=
 =?us-ascii?Q?mXJUarD4wXBS+qzfJKkr8B967j0Gb1pz8mIJWXkkbHYfepPBsOI88wFkhnu6?=
 =?us-ascii?Q?ON8lANOSnsQ7+TGSAXkfTTL9NWPvQYu4c2bjLQmsxZR1IL5qoud+yPRjxWGq?=
 =?us-ascii?Q?kskz75IwpG8b7SkzQGoppB9E0J/jKG05Mk+3Lw07rVVOwJMebHou3Iqftroz?=
 =?us-ascii?Q?3NChVVZso5VpsFLYStVseSqvos4mVmKaWQ0u7UtH52UJmWB8rkYzfxBMr0hM?=
 =?us-ascii?Q?2pFDU/5R5WaGubvojdBvoJvH02AyIZkqAypfbk5Y/M2Pyaw5d1yDkaCxywEG?=
 =?us-ascii?Q?0lIHs9EYDwzWx9MmV6CaKaEbA0UUV/Ta5FB6Cqnv+jIl96kxDGWDM0VTND6V?=
 =?us-ascii?Q?/fZ2cA6ClbthfSM1FcAKbrQTmMGQ74HmyY9fMixUL4VXsRpSqTPMjl0jWP5J?=
 =?us-ascii?Q?l1qOJCMX9aPaVOHdz007TDR09LMXP/OlYqTBfRdHViOr54uedx7ypW/G/9Xr?=
 =?us-ascii?Q?mqURn8X7IICGvIrz0mY7GA8csYQuR24QtGa/AtS7lYVlNFbsR/S23MJYA/cL?=
 =?us-ascii?Q?y3g/N7Q3wpLq7LNoupXLJbKrWG2Dbg20kdPJRyi1gXbEjUtPpjCn9AEdwEKZ?=
 =?us-ascii?Q?dn7cK1Da/bKZ4UcMPQ7u/enxFj9cQeboxDcEHvgvFcKyczE8CQj0yodRxaIR?=
 =?us-ascii?Q?tpTKWMT1gXYI2GDsdgGal7Ugwd8gotZnpudYOiPzYmG781d+yTO7EGKO3nNo?=
 =?us-ascii?Q?q2SP72030yhQsu29xRVPbiwmDwCHLziCoYfmujGKN7Kl34umBKc/Z5T3fA3z?=
 =?us-ascii?Q?PC8Exy7H39jV3HfOTa+1548QhGYauaOgoTcTBpLnWH5yEBc2fU3cpfh6SA/K?=
 =?us-ascii?Q?RAqL76JJcbLvDbUAgSbIdFvYzokl2MfNw9SVmP9gMN0ZfNvm9D9Eqrk+VTmz?=
 =?us-ascii?Q?gFV1jFyqVbPeyDhWo8DVxVARwWui7UOO911uS0eQR1apGbyzMBSXm1WPWgXZ?=
 =?us-ascii?Q?lljg7VVuC1zymMkOxDR6RNAUKIJv5WSWEMc5lZwcE2moCwOxLUDuh4B7nueR?=
 =?us-ascii?Q?sS2A1sdkqQCXeFTbG/rURkaaaKVvNZe8XWDCUJ8aEHteieWvTDuSJdzB/N94?=
 =?us-ascii?Q?9IcT3mudvU/T7M7TU+Dczh7xrqvgoNI0dN1ScmHbFRNuzJbty76NCvzQQ2WX?=
 =?us-ascii?Q?9YjPuTacslr8057Om2rHl2W3dAykzkIsP+0sTK96cF8GNOzzCJQ++vzpeabX?=
 =?us-ascii?Q?bQ=3D=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: d592c3fe-6727-4e02-ec94-08da65c5b67c
X-MS-Exchange-CrossTenant-AuthSource: MWHPR1101MB2126.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Jul 2022 18:21:47.2355
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: oIlp0v8zRAqhheKRsqvgNlefPQIMH8s6FCxCVdD06hOznqnMUU8RCVuf8bfthxYRjtahQhiXKgfG7+AczmB4rgcw2lBW7cZf0dfr5zZitjo=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR11MB6169
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-5.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

ruansy.fnst@fujitsu.com wrote:
> This patch is inspired by Dan's "mm, dax, pmem: Introduce
> dev_pagemap_failure()"[1].  With the help of dax_holder and
> ->notify_failure() mechanism, the pmem driver is able to ask filesystem
> (or mapped device) on it to unmap all files in use and notify processes
> who are using those files.
> 
> Call trace:
> trigger unbind
>  -> unbind_store()
>   -> ... (skip)
>    -> devres_release_all()   # was pmem driver ->remove() in v1
>     -> kill_dax()
>      -> dax_holder_notify_failure(dax_dev, 0, U64_MAX, MF_MEM_PRE_REMOVE)
>       -> xfs_dax_notify_failure()
> 
> Introduce MF_MEM_PRE_REMOVE to let filesystem know this is a remove
> event.  So do not shutdown filesystem directly if something not
> supported, or if failure range includes metadata area.  Make sure all
> files and processes are handled correctly.
> 
> ==
> Changes since v5:
>   1. Renamed MF_MEM_REMOVE to MF_MEM_PRE_REMOVE
>   2. hold s_umount before sync_filesystem()
>   3. move sync_filesystem() after SB_BORN check
>   4. Rebased on next-20220714
> 
> Changes since v4:
>   1. sync_filesystem() at the beginning when MF_MEM_REMOVE
>   2. Rebased on next-20220706
> 
> [1]: https://lore.kernel.org/linux-mm/161604050314.1463742.14151665140035795571.stgit@dwillia2-desk3.amr.corp.intel.com/
> 
> Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>
> ---
>  drivers/dax/super.c         |  3 ++-
>  fs/xfs/xfs_notify_failure.c | 15 +++++++++++++++
>  include/linux/mm.h          |  1 +
>  3 files changed, 18 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/dax/super.c b/drivers/dax/super.c
> index 9b5e2a5eb0ae..cf9a64563fbe 100644
> --- a/drivers/dax/super.c
> +++ b/drivers/dax/super.c
> @@ -323,7 +323,8 @@ void kill_dax(struct dax_device *dax_dev)
>  		return;
>  
>  	if (dax_dev->holder_data != NULL)
> -		dax_holder_notify_failure(dax_dev, 0, U64_MAX, 0);
> +		dax_holder_notify_failure(dax_dev, 0, U64_MAX,
> +				MF_MEM_PRE_REMOVE);
>  
>  	clear_bit(DAXDEV_ALIVE, &dax_dev->flags);
>  	synchronize_srcu(&dax_srcu);
> diff --git a/fs/xfs/xfs_notify_failure.c b/fs/xfs/xfs_notify_failure.c
> index 69d9c83ea4b2..6da6747435eb 100644
> --- a/fs/xfs/xfs_notify_failure.c
> +++ b/fs/xfs/xfs_notify_failure.c
> @@ -76,6 +76,9 @@ xfs_dax_failure_fn(
>  
>  	if (XFS_RMAP_NON_INODE_OWNER(rec->rm_owner) ||
>  	    (rec->rm_flags & (XFS_RMAP_ATTR_FORK | XFS_RMAP_BMBT_BLOCK))) {
> +		/* Do not shutdown so early when device is to be removed */
> +		if (notify->mf_flags & MF_MEM_PRE_REMOVE)
> +			return 0;
>  		xfs_force_shutdown(mp, SHUTDOWN_CORRUPT_ONDISK);
>  		return -EFSCORRUPTED;
>  	}
> @@ -174,12 +177,22 @@ xfs_dax_notify_failure(
>  	struct xfs_mount	*mp = dax_holder(dax_dev);
>  	u64			ddev_start;
>  	u64			ddev_end;
> +	int			error;
>  
>  	if (!(mp->m_sb.sb_flags & SB_BORN)) {
>  		xfs_warn(mp, "filesystem is not ready for notify_failure()!");
>  		return -EIO;
>  	}
>  
> +	if (mf_flags & MF_MEM_PRE_REMOVE) {
> +		xfs_info(mp, "device is about to be removed!");
> +		down_write(&mp->m_super->s_umount);
> +		error = sync_filesystem(mp->m_super);
> +		up_write(&mp->m_super->s_umount);

Are all mappings invalidated after this point?

The goal of the removal notification is to invalidate all DAX mappings
that are no pointing to pfns that do not exist anymore, so just syncing
does not seem like enough, and the shutdown is skipped above. What am I
missing?

Notice that kill_dev_dax() does unmap_mapping_range() after invalidating
the dax device and that ensures that all existing mappings are gone and
cannot be re-established. As far as I can see a process with an existing
dax mapping will still be able to use it after this runs, no?