Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <3921846a-5560-44fd-c4ce-329acad454c0@amd.com>
Date:   Tue, 8 Aug 2023 16:19:10 +0530
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Thunderbird/102.14.0
Subject: Re: [PATCH v5 3/3] vfio-cdx: add bus mastering device feature support
Content-Language: en-US
To:     Alex Williamson <alex.williamson@redhat.com>
Cc:     gregkh@linuxfoundation.org, linux-kernel@vger.kernel.org,
        git@amd.com, pieter.jansen-van-vuuren@amd.com,
        nikhil.agarwal@amd.com, michal.simek@amd.com,
        abhijit.gangurde@amd.com, Shubham Rohila <shubham.rohila@amd.com>
References: <20230803143253.7817-1-nipun.gupta@amd.com>
 <20230803143253.7817-3-nipun.gupta@amd.com>
 <20230803161736.52dd2d33.alex.williamson@redhat.com>
From:   "Gupta, Nipun" <nipun.gupta@amd.com>
In-Reply-To: <20230803161736.52dd2d33.alex.williamson@redhat.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?RlFNMG55elhnRWo1c25hTDFXek80emFJcHFjSDFEdXZEWk1INWxVSVFQekNm?=
 =?utf-8?B?OXFOWkJqK3RjdU9sdTVJZVhRNG1WbDRoRU94QU5ydGFrbG9KZTNURnphUWhU?=
 =?utf-8?B?WlkxWkRqdkhHNFVYWWg2SUhleTZLQnY2NFZrQXJOUTdjZkZPLzJVSnUwYUFE?=
 =?utf-8?B?a2JzQlh0N284YnRmU09KRUw0K3NMUzhGNDQ2ZUVGWnFwOGRva3JvNEtBQTJP?=
 =?utf-8?B?bHJQOFVMOFhWSFkzdEMrV0Vic3dIMExVVkFtU3JqMjdXeEduQSszTlhEei9n?=
 =?utf-8?B?ZTR3bGRJWXloRzl6ZFhoTW5YNVdGZ0JSMDdFZ0lWZ1RKbUdpRzhscXZJd253?=
 =?utf-8?B?NWtCZ0hlbS8rd1M3YlAvS0QwQVBHWlRkSFpKQWJueDZIOUx4aTlZSit5dmNS?=
 =?utf-8?B?RDNqUFFlSEl6MUhpdFBwd0dVUVdDSk1BckVhMkZSWGpDWWhyODlFTXJNSVQz?=
 =?utf-8?B?RzJZZC9CemZNbHkrN0s0Tk9mS0VXbHg4OEhMVWJMNzQva2FDNldZYm9MQUIw?=
 =?utf-8?B?R3JsL05nYXJ4QmZsNFFEMkRLK0FHSnJ1ZVZ1RUV2TmNaZ3I0bUNRK1lEeTll?=
 =?utf-8?B?Nkx4N09SVHJEaCtmTHVCNDFjckh2Wkdvd21tNUhYODByUzE5bW9ZYVFSYXlI?=
 =?utf-8?B?MTFWUFdVenpQa3ZIMm50RzN1UkQvN1VCeU9EZi9XYmYwME1FRlFiYlZvVm9q?=
 =?utf-8?B?QnB2a04zMFlkbWNpdjd0NkM2VlI4dXBuME4xbWl0L0NGME8vcXFrUTZUTVJr?=
 =?utf-8?B?cXRYdlVKRllPTWxqeGJxREcvd2RqUk52b0Fuby9uUS80dHV6b1JjNE8vVDQx?=
 =?utf-8?B?R3dYNm5yZ0lKWDV1US9VUFQxQWhTOFUrOW1YRWNYNGxxb1Rqam1nVkZmV2Np?=
 =?utf-8?B?cU5sdkRUSmdOQTZIcVBQSUJuU2JabzZKTWRCTzB6dDFjSW4zdTBxY1dUKzNm?=
 =?utf-8?B?OFlOVllJYnZFRSt3NUtRUmdndCtkY0ZXcFQ4djhWN0t0WE9COWllemF3TzdS?=
 =?utf-8?B?L2QvazhNUnVlT2NCMFhWbm1TVE40b3RZRGZYV04zd3RGNitWVDR6eGNtb3lY?=
 =?utf-8?B?cUMvcWF1M3AxbzlvbjRuaHFLZnBFZDQ1azkyL2lyOVhFTmhSbmRQZEJqWGhS?=
 =?utf-8?B?dHFYeHdmMFhPRVdBckRzS1l3U2x2NnUvZzRqekpZeTFuazUxcEliK1Y3aXRh?=
 =?utf-8?B?ckpyWlB1UUp2TmtmSjgyNmVweVFydXVuYTRiZiswaGZON2NVajBaYldLUnV4?=
 =?utf-8?B?cnlTMTZFWHlaRURHTzE4ZUxXVnQrNWVyeEhlYVZOaDZWU1l6ZG56YU9Lak8z?=
 =?utf-8?B?SFlRc0pudWszcEpTUldUL0ZRQ0htSTNZLzRtRG9qVHVKVkVsS25MYW95V1pt?=
 =?utf-8?B?Si9SU1J5Z21XcnhpbTg3ZVNjU3BEdXprUnJwYzl1MlNLT0V0eVBabVlPYUJu?=
 =?utf-8?B?UzVlWmwvdmZrb2tkWDBoYTQwd1IrMUZHQjQ3Njg1amRZZlk4eC80N0RtZ0RR?=
 =?utf-8?B?ZkptR0tNM0FCdVNCcngvb1dkK29adUJ4VUIwbzdyY1M2K2hWUzBsSFhpSFpY?=
 =?utf-8?B?MGRDM0JBZDlYSkVnVndWaEVNVHU2TDhMWExadWc2SUd2OTJxTzB1a0RkVENi?=
 =?utf-8?B?TVByWHZCNTQzU1hpL0UzM25PcEZDR2tyb0lldWFKWHdkVlV6TEl2WGVOWm83?=
 =?utf-8?B?M1VpV3RoNUJuNnZDNnFxMXlOMk9zTHJaTnVHVGxvYWF6anlFT0h4VjJ4OVB6?=
 =?utf-8?B?aFRkeWhVRVY3TkJXY3hLQXBqTzZWb2VJS2RDck5jVmNkUHdzMDVMREhvRzhu?=
 =?utf-8?B?bk42MUtZRmZBZ2tYbkMxczVmRlZZU1RvQXRudFYxZnlRODE1cW9OQjc5ZGll?=
 =?utf-8?B?Z3cyVDZLZ00wL0t3RTY4L2VuZmhVaVVFUXRkaWpMeHlWbEE3WXpRRmpxbzB6?=
 =?utf-8?B?QmFKdm5iSnduMkVKay9hbjk2R1p5aFBoMmM5dFoxdTF5ZE1QL3RuSWtkNWpL?=
 =?utf-8?B?emZNWDVPSUpJMFYxY05NbjlieEdYaW9MUDFpQXVGNElIZWxvWG11cmxEVVZ0?=
 =?utf-8?B?SUxtS1pBQTVad1d6NEpkZERHWldxNW1acXY4bFh5dzUzbGxNZ1JhVDBtNUJ5?=
 =?utf-8?Q?Uv0EWr8igf1eG3BKTkitH0zHQ?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 5e8b9516-8aa1-43a4-29d1-08db97fd2196
X-MS-Exchange-CrossTenant-AuthSource: CH3PR12MB8308.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Aug 2023 10:49:25.2986
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: U8FrCDhwOEX/inymzdzFFL9XgGTOidBDgKQXr+ojRNHvTw+4WE339jUtoTVqmp5q
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB9169
X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        NICE_REPLY_A,RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,
        SPF_NONE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 8/4/2023 3:47 AM, Alex Williamson wrote:
> On Thu, 3 Aug 2023 20:02:53 +0530
> Nipun Gupta <nipun.gupta@amd.com> wrote:
> 
>> Support Bus master enable and disable on VFIO-CDX devices using
>> VFIO_DEVICE_FEATURE_BUS_MASTER flag over VFIO_DEVICE_FEATURE IOCTL.
>>
>> Co-developed-by: Shubham Rohila <shubham.rohila@amd.com>
>> Signed-off-by: Shubham Rohila <shubham.rohila@amd.com>
>> Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
>> ---
>>
>> Changes v4->v5:
>> - Use device feature IOCTL instead of adding a new VFIO IOCTL
>>    for bus master feature.
>>
>> Changes in v4:
>> - This patch is newly added which uses cdx_set_master() and
>>    cdx_clear_master() APIs.
>>
>>   drivers/vfio/cdx/main.c | 40 ++++++++++++++++++++++++++++++++++++++++
>>   1 file changed, 40 insertions(+)
>>
>> diff --git a/drivers/vfio/cdx/main.c b/drivers/vfio/cdx/main.c
>> index c376a69d2db2..6420cf6eb2f9 100644
>> --- a/drivers/vfio/cdx/main.c
>> +++ b/drivers/vfio/cdx/main.c
>> @@ -52,6 +52,45 @@ static void vfio_cdx_close_device(struct vfio_device *core_vdev)
>>   	cdx_dev_reset(core_vdev->dev);
>>   }
>>   
>> +static int vfio_cdx_bm_ctrl(struct vfio_device *core_vdev, u32 flags,
>> +			    void __user *arg, size_t argsz)
>> +{
>> +	size_t minsz =
>> +		offsetofend(struct vfio_device_feature_bus_master, op);
>> +	struct cdx_device *cdx_dev = to_cdx_device(core_vdev->dev);
>> +	struct vfio_device_feature_bus_master ops;
>> +	int ret;
>> +
>> +	ret = vfio_check_feature(flags, argsz, VFIO_DEVICE_FEATURE_SET,
>> +				 sizeof(ops));
>> +	if (ret != 1)
>> +		return ret;
>> +
>> +	if (copy_from_user(&ops, arg, minsz))
>> +		return -EFAULT;
>> +
>> +	switch (ops.op) {
>> +	case VFIO_DEVICE_FEATURE_CLEAR_MASTER:
>> +		cdx_clear_master(cdx_dev);
>> +		return 0;
>> +	case VFIO_DEVICE_FEATURE_SET_MASTER:
>> +		return cdx_set_master(cdx_dev);
> 
> It's curious that the implementation of set and clear in CDX call
> through to functions with non-void returns, but we simply ignore the
> return in cdx_clear_master().  Does something prevent clear from
> failing?

Would update cdx_clear_master() with non-void return.

> 
> I also note internally that true is used for enabling and false for
> disabling, which is effectively opposite of the proposed uAPI in the
> previous patch.

Yes, will fix in the uAPI.

> 
> If the idea here is that the user should assume bus master is disabled
> when opening the device, what happens if the user closes the device
> with bus master enabled?  What would cleanup that state for the next
> user?

cdx_dev_reset() clears the bus mastering and user would enable bus 
mastering once ready for DMA. Probably we also need to add reset in 
device open. So user should assume that bus master is disabled when 
opening the device, and close would also clear the bus mastering (by 
calling reset).

> 
> Is there a use case for the GET operation in userspace?

There is no use-case for get operation in CDX as of now.

Thanks,
Nipun

> Thanks,
> 
> Alex
> 
>> +	default:
>> +		return -EINVAL;
>> +	}
>> +}
>> +
>> +static int vfio_cdx_ioctl_feature(struct vfio_device *device, u32 flags,
>> +				  void __user *arg, size_t argsz)
>> +{
>> +	switch (flags & VFIO_DEVICE_FEATURE_MASK) {
>> +	case VFIO_DEVICE_FEATURE_BUS_MASTER:
>> +		return vfio_cdx_bm_ctrl(device, flags, arg, argsz);
>> +	default:
>> +		return -ENOTTY;
>> +	}
>> +}
>> +
>>   static int vfio_cdx_ioctl_get_info(struct vfio_cdx_device *vdev,
>>   				   struct vfio_device_info __user *arg)
>>   {
>> @@ -169,6 +208,7 @@ static const struct vfio_device_ops vfio_cdx_ops = {
>>   	.open_device	= vfio_cdx_open_device,
>>   	.close_device	= vfio_cdx_close_device,
>>   	.ioctl		= vfio_cdx_ioctl,
>> +	.device_feature = vfio_cdx_ioctl_feature,
>>   	.mmap		= vfio_cdx_mmap,
>>   	.bind_iommufd	= vfio_iommufd_physical_bind,
>>   	.unbind_iommufd	= vfio_iommufd_physical_unbind,
>