Received-SPF: pass (google.com: domain of linux-kernel+bounces-44894-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
Message-ID: <49259a2e-4840-4009-aaa6-0c51239c5c81@intel.com>
Date: Tue, 30 Jan 2024 10:01:03 -0600
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH] dmaengine: idxd: Change wmb() to smp_wmb() when copying
 completion record to user space
To: Fenghua Yu <fenghua.yu@intel.com>, Vinod Koul <vkoul@kernel.org>, "Dave
 Jiang" <dave.jiang@intel.com>
CC: <dmaengine@vger.kernel.org>, linux-kernel <linux-kernel@vger.kernel.org>,
	Nikhil Rao <nikhil.rao@intel.com>, Tony Zhu <tony.zhu@intel.com>
References: <20240130025806.2027284-1-fenghua.yu@intel.com>
Content-Language: en-US
From: Lijun Pan <lijun.pan@intel.com>
In-Reply-To: <20240130025806.2027284-1-fenghua.yu@intel.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?QmFySWdRUnY2VU1kZmhXS3o4QkFKS0xDUEtpT0QwLzViY1NXWElEQ3lHWlkv?=
 =?utf-8?B?U3VvU0lYam9uTllZYlNZNEpVMGdjaDBQOFRoM3ZtVjREblMxRXlzdC81STgv?=
 =?utf-8?B?ZU9lZU4xU3A3S0dOZWNIc0FhTWVuQVNDd243M2ZQK08xOGRldFFLbXVielF2?=
 =?utf-8?B?WG1UYlo3Nmp1Smc0Uk1ERVBIMUhYSGpnZS8weVQxZm5wVG9vY0htUndHSG0z?=
 =?utf-8?B?VlB2dHFlQW5qbHZuOXRCYVRLVUkxckN2d3Y2dmw0U2EyMEhEQ3dORVZYNU42?=
 =?utf-8?B?MHY1RmtnWXEwaG4rNkhBNjVyUElyQ05sUEo5N3dmMDhVVWczQXU1bEYrYk9U?=
 =?utf-8?B?WGZBNGVQZ2FJSXpOOXpVNGlsK2tuTTJ6U3VyUlNDVUt0OCtTdDI1N01IdHZ0?=
 =?utf-8?B?YlVKK0FIaGs3L0pPN1hMSjN5bnUxd2R2Q2haTCtjajYyS0RLMzBZdEJVWEhj?=
 =?utf-8?B?VmFlNk0yTUVGSno1U3hYZEpiOVd1Y201bVkyalF3cHNQeUducGZYYjJNZkpU?=
 =?utf-8?B?eUZFVStaakU5YWplOElnWFp4M016anJ2cG9rSUw5ZDdSd1VpN2QrSmpMQVJZ?=
 =?utf-8?B?R3UvenZCbXpQd0lJZlV5TS93T3BPYmdwekVOdlF5bkxBMHV0Y21kOFB2eVZP?=
 =?utf-8?B?RDNYTlUxTW5YUWhhWWE3Z3FsaHVJVG9Sb3g4WVlsbmZZQy9KNVRKRm9CSGNV?=
 =?utf-8?B?Rzhzc0JpZGNsNjVxSGpxd1NiL0R0cCt5VkxIUDFvNENCeGVCejZrOWx3VUp4?=
 =?utf-8?B?eHp6OGkrRjVGcm1iQ2NQRndQYmNSSmJTN21nYi9DUGpBSFRwTVNqV05tQ0po?=
 =?utf-8?B?WGRtTktYMmNNYUFjVGdWVW5yL1hMeDdpcWlxSHlFSTZjMlVjU0laVU9FV2t3?=
 =?utf-8?B?Z1BLOTdrMjd6ajA4MDZsUmFuempJNC82QnErTXR0dDZuWEJzVzhUZTB1blRt?=
 =?utf-8?B?YTFHa09PUURhMDFTUTBOZkVLSUoreU94VFRPb1o3VGtQZWpTYzNWdmZoTXJr?=
 =?utf-8?B?YzR6QnRxYTVGZE4vdlIyckZGL05FUG5YUXN3S2llQXVJaDl0VmVOTmZaeUk2?=
 =?utf-8?B?aWFKaStwMmNsNGZsZ2wrS1UvK2IrcDZOMHRQdGoxZ2xybUJuTW1CcExvWVN3?=
 =?utf-8?B?UmEvdThPZVJwbG9hdWxQNnpUVmU4bmtIWXN3ZlFOQXlJUDJMeURrMklBSHJI?=
 =?utf-8?B?MS8xYzNDaDIvVzJpZWtodG5uL1lrN2tZRWpQSVFOZ1U4azMySm1kREtnNXRV?=
 =?utf-8?B?UlNmdmpkcVNwTzBMbjI1ZnpOODVPUXM0UDBiczV4RzZZL0NiSlRuYWg1VzlE?=
 =?utf-8?B?OVJCclJpUDRaL2hNZk5nVVo1djYyZmg1NkV1amRUOU1vNHRvZG5jaXNGSWcz?=
 =?utf-8?B?ZWQwRWlmNlF2ZkYrRFpBQkpSOWlDKzM0dktEbTBNN3JzN1FQS2RjYWxwWjVY?=
 =?utf-8?B?U0NIUVY1WVVhck9BeEU5MWZGYU5US3ZaNmlZeHZYWWIvb0tVaWRwUmNXVk9P?=
 =?utf-8?B?QlUwQnoxVDBsTVp1a2dwa0lvZ1MrZkZ3Um9VQmNLUUVGTjM5WUxkeHh0L0x1?=
 =?utf-8?B?NS84K2tDNi9IVXlPMTR3bFNZdHBCSzErVno2eGtJbmt1T3MvODA2UFBsSmV5?=
 =?utf-8?B?TnV0MFR5Sm1QZEYxU2RXVEwrbTVmWW9rZERGVSt2dHNqWnczVXRKU3NtTTJa?=
 =?utf-8?B?bHlpN1FEWjk1OXhSNTl3a2FtZ1RvNm44dS9jL0l3RnZXRGRYODJaMlh6d2w5?=
 =?utf-8?B?ZnI3WlRjbER2ME9LYURXV3V0WlJHc0tBSnFqOFFTNElGNC8xMnZJR2V4QU85?=
 =?utf-8?B?NDJtdFFENzBPeWZtc0grcjZNUTVMOTBXd3JaQXhGSmZyYUp3S1dHVVV4VlRJ?=
 =?utf-8?B?K2NqZHU1M1c1MElrV09OQkpheFVyMmdDZkthcUFwUnREN1N2V3ZzbEU3allU?=
 =?utf-8?B?ODlLMGhmak11N1Vyd3JjLytPcjBuRVVSQVB1RjFFUWlDNlM2bHpROXhiUGFM?=
 =?utf-8?B?aXVGeWZHN0YxVHpHdmgxaGtKNUM3Q2R3VUtObnhJQU5lRUZCYWJ0UEVIdU55?=
 =?utf-8?B?SFBCN3lJUmNwTE04OXNaUUxIYU9NQ2hra0JXeUY0bDhubW4xb1p2UW8rZ2pO?=
 =?utf-8?Q?W6IzUYnsipMJlWOmi8KCv6tfu?=
X-MS-Exchange-CrossTenant-Network-Message-Id: da91fac3-d8a4-4687-8ecb-08dc21acab35
X-MS-Exchange-CrossTenant-AuthSource: PH8PR11MB7141.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Jan 2024 16:01:07.1675
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: uH0DezhrCuKigQeXatC5vOR8pKQ4AGSx3wvN3kxjCBnoDvV4vfresdpar5cbrHDjEl22MSHtvQnfWH7/BYaF/Q==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR11MB7123
X-OriginatorOrg: intel.com


On 1/29/2024 8:58 PM, Fenghua Yu wrote:
> wmb() is used to ensure status in the completion record is written
> after the rest of the completion record, making it visible to the user.
> However, on SMP systems, this may not guarantee visibility across
> different CPUs.
> 
> Considering this scenario that event log handler is running on CPU1 while
> user app is polling completion record (cr) status on CPU2:
> 
> 	CPU1				CPU2
> event log handler			user app
> 
> 					1. cr = 0 (status = 0)
> 2. copy X to user cr except "status"
> 3. wmb()
> 4. copy Y to user cr "status"
> 					5. poll status value Y
> 				 	6. read rest cr which is still 0.
> 					   cr handling fails
> 					7. cr value X visible now
> 
> Although wmb() ensure value Y is written and visible after X is written
> on CPU1, the order is not guaranteed on CPU2. So user app may see status
> value Y while cr value X is still not visible yet on CPU2. This will
> cause reading 0 from the rest of cr and cr handling fails.
> 
> Changing wmb() to smp_wmb() ensures Y is written after X on both CPU1
> and CPU2. This guarantees that user app can consume cr in right order.
> 
> Fixes: b022f59725f0 ("dmaengine: idxd: add idxd_copy_cr() to copy user completion record during page fault handling")
> Suggested-by: Nikhil Rao <nikhil.rao@intel.com>
> Tested-by: Tony Zhu <tony.zhu@intel.com>
> Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>

Acked-by: Lijun Pan <lijun.pan@intel.com>

some minor comments below.

> ---
>   drivers/dma/idxd/cdev.c | 5 +++--
>   1 file changed, 3 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/dma/idxd/cdev.c b/drivers/dma/idxd/cdev.c
> index 77f8885cf407..9b7388a23cbe 100644
> --- a/drivers/dma/idxd/cdev.c
> +++ b/drivers/dma/idxd/cdev.c
> @@ -681,9 +681,10 @@ int idxd_copy_cr(struct idxd_wq *wq, ioasid_t pasid, unsigned long addr,
>   		 * Ensure that the completion record's status field is written
>   		 * after the rest of the completion record has been written.
>   		 * This ensures that the user receives the correct completion
> -		 * record information once polling for a non-zero status.
> +		 * record information on any CPU once polling for a non-zero

Maybe add "smp system" to the sentence to be more explicit since people 
usually only read above comments instead of the commit message later on.
say,
"record information on any CPUs of a SMP system once polling for a 
non-zero"

> +		 * status.
>   		 */
> -		wmb();
> +		smp_wmb();
>   		status = *(u8 *)cr;
>   		if (put_user(status, (u8 __user *)addr))
>   			left += status_size;