Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Tue, 30 Mar 2021 08:16:20 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Dan Williams <dan.j.williams@intel.com>
Cc:     linux-cxl@vger.kernel.org, linux-kernel@vger.kernel.org,
        vishal.l.verma@intel.com, ira.weiny@intel.com,
        alison.schofield@intel.com
Subject: Re: [PATCH v2 2/4] cxl/mem: Fix synchronization mechanism for device
 removal vs ioctl operations
Message-ID: <20210330111620.GK2356281@nvidia.com>
References: <161707245893.2072157.6743322596719518693.stgit@dwillia2-desk3.amr.corp.intel.com>
 <161707246948.2072157.2116502455691653472.stgit@dwillia2-desk3.amr.corp.intel.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <161707246948.2072157.2116502455691653472.stgit@dwillia2-desk3.amr.corp.intel.com>
MIME-Version: 1.0
X-MS-Exchange-MessageSentRepresentingType: 1
Received: from mlx.ziepe.ca (206.223.160.26) by YT1PR01CA0039.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:2e::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3977.31 via Frontend Transport; Tue, 30 Mar 2021 11:16:22 +0000
Received: from jgg by mlx with local (Exim 4.94)        (envelope-from <jgg@nvidia.com>)        id 1lRCM8-005qN4-Mq; Tue, 30 Mar 2021 08:16:20 -0300
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: 5ee534fb-0363-40c1-ae65-08d8f36d400d
X-MS-TrafficTypeDiagnostic: DM5PR1201MB0108:
X-Microsoft-Antispam-PRVS: <DM5PR1201MB0108B5FDE9D6D8C1B026B3AAC27D9@DM5PR1201MB0108.namprd12.prod.outlook.com>
X-MS-Oob-TLC-OOBClassifiers: OLM:4502;
X-MS-Exchange-SenderADCheck: 1
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info: UF++gLaPLae1F1QFPqLp871oyk2KBNDEN/yMYFx5xtBX4DPJSJ7aYG4xUoFrd1T9CaOFbjtLNhT3NGTCTN+YKsg/kxTGbV5+pQ73p09k5XNy+xw4ZKU69Tnqj1oAEKx08YxuzDRvzM234WNB7BOKzqBUUPFz430vrPz3+aF9WqU7edU37xPmDtxePy1jRg2gjr9icJZ9OI26SZEpAQO/xfQVakmqQea6ZsnobdnN2k0oeIMfMM4t7NJQH9dkNXt+4K1wKoxdpGWl7onjcuufmLh4RVtMku5R1DuyiykFisKkhiWo3lRibW2bQxsKa83k3XyZydU3xUf9Y/jo3Ar1Mngfg0SCQYcvIKcXwJma3/9RG3znnoZNIgWOPMGziMgsV7y3q/BOty877jVkZNJ4iv9WwA+VGq+HYssGLN9gRauMTtVR0I5prZ+E+dPuWs7Dv7SIARw0WdFYtvw/+zgoBCKXQ2QDYh0W5m6xLMuAkBt2Xqj63vdhLLP2E4xYzNtjIn0LD3OJrT2sYfnGNLJSgKy5mOJyWW7qjKy1TrfRuKEl4/ed/B5aR7Qn9CKj1QpYWXMUdWko7p9WlqbI8CILEmLOQgz+9ZM7nxSbUsnIIUUimUqThwzXBgMY4OIDbnhRnxWCuIzV0MKCbt037pGG6hFrZoR5981TW5XUaCg2si4=
X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3834.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(346002)(39860400002)(376002)(136003)(396003)(366004)(26005)(186003)(9786002)(5660300002)(38100700001)(86362001)(9746002)(478600001)(1076003)(4744005)(2906002)(8676002)(316002)(426003)(6916009)(4326008)(83380400001)(8936002)(66556008)(66946007)(2616005)(66476007)(36756003)(33656002);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData: =?us-ascii?Q?0IGeUMDXTyzmMB4T7yds9uEmjp415+ooHAcnKD2CvQLkqOl8ybpljDyb+tEZ?=
 =?us-ascii?Q?rBkC6ISBjj/jwzErRUVtOiCHS5CRjx5BopsPzCTqOJXKHJL9v7hqR27Egylx?=
 =?us-ascii?Q?eW1L5LtCBbhVMAfboMR15oyNfKQP1fDn95bp+McdwvjJ6NK/z4scyqajFRPI?=
 =?us-ascii?Q?nS+ZF4Nazld85rt9/9tv4vGJKLh0HQHGS6HAyTNYLTL1UBfCM8di6aDpj9Vt?=
 =?us-ascii?Q?QnuV6RGCpnIcISCQf4TcTUsvt247r0SayuLAG8hPLuxcJFopXULyOVo1MSB1?=
 =?us-ascii?Q?l6Ow13lbrz6EhZ3GCsEF72MSAkfbdcLAMowJol2qNoujafUyCVifMB+cLq2O?=
 =?us-ascii?Q?367MDkpFtmZKDnOB1vWORw652ezll3Fvc+lkL74aN4suGf9+vHzL0MQwanmF?=
 =?us-ascii?Q?z+lEvnHVhZcrS/8UGjR7TKmErhM8GFPiX0/HTgEKy1BuC0p8xacQtllqxZE9?=
 =?us-ascii?Q?iP2dB7oBZsa4eEIVawctxR3riCw1l+3jLGeN9JYhazHx6LDwWF6QBPniaR3O?=
 =?us-ascii?Q?QQ+hv7vop0u9LMrFh63vdZGLlP6ejwOT5ely7u74npEVWHYRayy69wO/pc4z?=
 =?us-ascii?Q?G1TUoMPDR3wR4OQOJrgyO+bj/6RzrP5cb4LbxyepZ+GCEGpmVk9Q7f7cywU7?=
 =?us-ascii?Q?Zk1nUgWPBhGGJdG7ofC+mnSxPIb2ltv4BykKfUjexTtN/0idmNRlYuJBxDn/?=
 =?us-ascii?Q?WvXL6M6v8BUwlTYQ8yji0pJga2zENrNThrPeZ/Kz5TsIdcZp1jI8iSHgIy5S?=
 =?us-ascii?Q?vo//p9rOD62TIy7h2FNlW8zFXFiJkuH/1KBNXODOE5j6o9zVdYhx1ro6pI1b?=
 =?us-ascii?Q?g2DK8lfwyCbJ/tIDOpVJXtZJcKJnvuUdGBjrl3s9BPgCHJOosSjon1WNmw3C?=
 =?us-ascii?Q?cIKfw94c/T0cZfptsZLZYV1oaN7VhWjRWRvPUT3mrUgZKfs2eyvt6xopgli4?=
 =?us-ascii?Q?T8Hr7Hx8Cr/C/sWg5/zj7ycuB34dtabqBirObzkTP2dPTpBMFdjfV0c08c82?=
 =?us-ascii?Q?tn9Hnn6m5sLnq3+urNYI+mQTCj86yLtG4GqAgjEGTGKYS5aqeBuMff0KoWQp?=
 =?us-ascii?Q?pd+TnA2s/gzGeW2Pofuo5xivmHlqZs2jZ9+eHM3rNAVplLcpz08FbFl+iuiY?=
 =?us-ascii?Q?bGPJUYNmKJlg5vtTlV98Zdh7pc7cXcjC/ClbNqJ23eyC3P9yzl5uLqCVzSUF?=
 =?us-ascii?Q?mC+qGfVJkrr+40Jili+sry3Px/HqTuIhdzfKn8Ckeeo1UJa+VFP6jSuC3dHr?=
 =?us-ascii?Q?jtPlouSX9L5hzn6qSdHwbBIJKcMtptoofkl6436tL5laXtTvcGhNCjctNxAI?=
 =?us-ascii?Q?dan7h9EQ5amrkePkNxnGt59d?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 5ee534fb-0363-40c1-ae65-08d8f36d400d
X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3834.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Mar 2021 11:16:22.6897
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 6OoOUVa0DZ+Onhxd8i/AEpD1+ngow7NN38oCBRglz4lIUzuIdiMJxbEODNx+Vasw
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR1201MB0108
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Mon, Mar 29, 2021 at 07:47:49PM -0700, Dan Williams wrote:

> @@ -1155,21 +1175,12 @@ static void cxlmdev_unregister(void *_cxlmd)
>  	struct cxl_memdev *cxlmd = _cxlmd;
>  	struct device *dev = &cxlmd->dev;
>  
> -	percpu_ref_kill(&cxlmd->ops_active);
>  	cdev_device_del(&cxlmd->cdev, dev);
> -	wait_for_completion(&cxlmd->ops_dead);
> +	synchronize_srcu(&cxl_memdev_srcu);

This needs some kind of rcu protected pointer for SRCU to to
work.. The write side has to null the pointer and the read side has to
copy the pointer to the stack and check for NULL. 

Otherwise the read side can't detect when the write side is shutting
down.

Basically you must use rcu_derference(), rcu_assign_pointer(), etc
when working with RCU.

Something like 'ops' is usually a reasonable choice

This really can't just be a simple rwsem?

Jason