Received-SPF: pass (google.com: domain of linux-kernel+bounces-185024-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161;
Message-ID: <b55cccfc-67b0-4f21-9070-127af6fd07a7@oracle.com>
Date: Tue, 21 May 2024 09:38:52 -0400
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH V3 2/3] vduse: suspend
To: Jason Wang <jasowang@redhat.com>
Cc: virtualization@lists.linux-foundation.org, linux-kernel@vger.kernel.org,
        "Michael S. Tsirkin" <mst@redhat.com>,
        Si-Wei Liu <si-wei.liu@oracle.com>,
        Eugenio Perez Martin <eperezma@redhat.com>,
        Xie Yongji <xieyongji@bytedance.com>,
        Stefano Garzarella <sgarzare@redhat.com>
References: <1716218462-84587-1-git-send-email-steven.sistare@oracle.com>
 <1716218462-84587-3-git-send-email-steven.sistare@oracle.com>
 <CACGkMEtfjHzdBBwgUWgRHDc5opekvgTuXp-_z4ku6L5cqOeAJA@mail.gmail.com>
Content-Language: en-US
From: Steven Sistare <steven.sistare@oracle.com>
Organization: Oracle Corporation
In-Reply-To: <CACGkMEtfjHzdBBwgUWgRHDc5opekvgTuXp-_z4ku6L5cqOeAJA@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?utf-8?B?RDByUUhTM00yN0pkcjUwK0liNG1nT1RteWdFcVdFMW1EOTJBckkyYVFIQVY0?=
 =?utf-8?B?TGpnUk5FY210MFp6VXRtUjFYV09adGdlUjZwTXFENjZ4RkpOMEpDc0hQbFYx?=
 =?utf-8?B?SDVvSWhEaFpyWUNzUWNWL0pLK0RBeVdLMUFoTlFTQkliSHNIMjV3YlR2NnJV?=
 =?utf-8?B?R05naFF1LzVXWGdpWUYwV3lOS2J1bnVrc0RkWXRuTjhOVTNtVlZGb2UvOFFn?=
 =?utf-8?B?S3FTb21JUS9oY0d1M2FIL0t5YTJEU0hhNllwdTdRaTRWcm9mWldsYmpKeEhl?=
 =?utf-8?B?U2ZSMFJzSTY1Mm5Mei92MHJ2VndKbE5DeUpuaGNuTHJpZGRLSzloc2lLaW9T?=
 =?utf-8?B?d1pmZTFRbGJGeFhIbVdNNTh2VkFhQzNSUnBOTUZYNXRoZGlGemdJaHYwRHFE?=
 =?utf-8?B?aHdwc2hCUFVXRW1vbnRIWDNjc3BWWmo4WHF4SHJkREVXRFY3RjJqMjBjVmVE?=
 =?utf-8?B?Y0dqdDhuVlhOU3BiNE9CaFpMTjFwNGxWdmo3SHJTOGcyc0xGbmpIV0FuL3ZL?=
 =?utf-8?B?b2VLQWZUa1Z3SFF6aEFoMlJDM2NvVitPRU5xajkxWVdKRDJJRDdWMzVSYnNW?=
 =?utf-8?B?MC8ybE9Kbm5Yc1RmMzlST2JtODZ3SnViZHV6RTZPcHErZmJVdlJZbW1ETkdS?=
 =?utf-8?B?bEl1SXdqSVFLVnhHa0xuQURuanhoOHFuaUVzQXVUeUpKYVg4M0NXSkluTFox?=
 =?utf-8?B?ZmJiMFNyU3ZDZHdSalBzSFMxdzByalMvbnQ3SWIxMjRVdWg1Z1E5bFZMM2Mx?=
 =?utf-8?B?ZmRBWDV3SUZ6eGcyMFc1Z2FVYjQwOGxpaXJEY0J5UjZvTFZtUGxZZlRudTJX?=
 =?utf-8?B?WDc4ZmVPM2Y2WDZqWE5xbkxFdlkvMjUwVHl2a1JQcGRBYTZqVmR2bVFneUZN?=
 =?utf-8?B?SnBCNjZvb0l1UGMzNTFjeUhPMkl0bVdXWXNBMFJZb2s4OTBIeEE0cmNLeUVz?=
 =?utf-8?B?Z0xZOW1WclRjUzlZb1NhRllScVFPMm9FK2lTTyt6cmhZWnQ4WmN2bGgwc3VJ?=
 =?utf-8?B?ZTVoWWRGYk85N3Z6UGFnSWpOZnVkWDkzYk9nblBrOGd1b2lraFZGWnlCT0ZF?=
 =?utf-8?B?bHZ6bVlFR0V4K3BJVUY1blpiVm5LVWUzQ0RpeHZZMjJXUFpwSWpxNjUwOXBh?=
 =?utf-8?B?QmpmR2pyY3BrTERqUHoyK1dqWlB2NjN4dUdxdUtUb2NsZlcyRnVvM2tuZ01O?=
 =?utf-8?B?RXpnKzJ6RG8wQ3ZPc2MvYjZTZGFsY2NjeDZnTmdDTWc1MXJoVXRId3VzNEdZ?=
 =?utf-8?B?RDJ3NmtqNnVqVGFpQXg4Yk5icnZ5YUR3UWNVM1pkWVdGeEZnc2kyUy9ZWkR6?=
 =?utf-8?B?TVAycXg2SCtDRlFCcXNpekZscDlldGtWT1F5dlJROHVtUHlHbnFBbGtHZk1W?=
 =?utf-8?B?R1g1VGZNSUJKLzdyNWRlUFVSZFhiUjNnbzUvRTYwRWxlNDBuMjFvYjRGTkUw?=
 =?utf-8?B?ZUJLUEwxVHg0Ukg0ck93WGl6czEvU0xOOFgvVGZYcFZTSXF4b1lScG5GMDRP?=
 =?utf-8?B?dHZNWkY3b3RITXdCK3p2VUlidDVmSzk2N3czckVoc2lMYkdOemJ2OHRiV2I0?=
 =?utf-8?B?dGlGenFoLzREblBqOXppbWYvVCtjYTdjYUk3cS9jOERqaXdLUkhybWVhNkpP?=
 =?utf-8?B?eXVQUWgrWTFBYnlUMUxqYTBFZUlkTEQvbzQ3R3FSMGNUckZTdTByMHZzVllZ?=
 =?utf-8?B?QUFPMm9XZkdJVmg1ZHJscHROUVczN3UyaHh5OHIxb2Q2b3pUUE8vTHZRd1BJ?=
 =?utf-8?B?TmN5NGdlMkp4MGVsOVlXUEVlN0NjajUzS2IyUk9IUmlndFE1cTFNQUtxdDNi?=
 =?utf-8?B?YVI1Q0t0U0NtRWV6ZU1nN1NUaFhjVUg5cGhQN0dTTjVQQWFmNWtaanBVMk45?=
 =?utf-8?B?WjYwbXQxdzhaWXorRWpZTk9JQjZkU0kweDB6Rzg5dWdkZnl0dXlQRGZlaGdW?=
 =?utf-8?B?Z05WQ2hhbGtWK0NXdU1XUnRBMHFEVm5ydlBwM3l2SDhxMzRuZjhhSzdFOHVY?=
 =?utf-8?B?cDZXK01TRzVGallyLzhWem1ybVpUdGp2THlMMW9RMEFMbmFNVXlaTHlaM3R6?=
 =?utf-8?B?a3hZM3VZMFk2ZGtQSkplYzJ2enRsRjI4WHAvcDRoR0NkaU5zWjFScXIxYjE2?=
 =?utf-8?B?UWUyUGlZT3NtNDBrVks0cVpnaW5qcEExaFVpMzhaOE5kMTBnQ1drWWJvclB4?=
 =?utf-8?B?cXc9PQ==?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	HsOoCXVGQEU/hHpI4GrwylM2TF9LhxAA4RjJi7vCN87rbj8DfZ92BatXXbrRjnMH4VxbD4baoxj9oS5yg5KuDEAVptI3rciSx6CXOQAKVnAjTzpZvtCrx9I1752W/R7m70sNyzlYtijPYA2oLvgzTcftdKvD0/K7VMqhllhyKjumXAR/ZEtPgIiQa77H8Xx6D/1IW/10pnZX6br1LeMC/SOIqZknxAr+fgCFwJTdnirCBwjAuPrWU4xMTa1On1ZGxae7N65BFM/WjuFOyXjJWKeyxpoPbcWR8TVTvuo8NUG0Kofzg8q4fwSBkSygVcTCDFYoDe83AEej0YyAwYzRcGRt6zxxF9DCJ1cgCa3RdzkT8wwzPTnrlLKKSSmhiWrTkqtrAhb70ZEBTnO/jEmqlfYH2jetryMvGsbiD6dRt1ALt/0oX6sa9IOtaOZeZNFbL8egWF2eg7huYGwh3epYzNRm4Alm7kC2St4cg6nXPNJcyrUrxgAFsKHhrTqAmxcxsei/TeRT72iPR+xlH2kvdVw8Y32iNm21F2nSU97DjLUi5vmIcJKk50lptifdbOV+eOrbWamcDzHvPvpP2D/vJ3eyIJv9miyhH+T1wQCllYI=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 183273f9-9dc2-47b0-0814-08dc799b5eb5
X-MS-Exchange-CrossTenant-AuthSource: SA2PR10MB4684.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 May 2024 13:38:59.7432
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: pcu+93DNrvkhCW3QhdAdGlpCRQ4PoVVn79wajrJiiREM0Ehgk5jTBp7IpRdBVH2to3SlM1zCwvjC12ouuOb1b8crl09M/76MsPyJo0zF5uU=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR10MB4710
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.650,FMLib:17.12.28.16
 definitions=2024-05-21_08,2024-05-21_01,2024-05-17_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 adultscore=0
 mlxlogscore=999 bulkscore=0 spamscore=0 malwarescore=0 phishscore=0
 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2405010000 definitions=main-2405210103
X-Proofpoint-GUID: G6VjSAdO_KMnmIpy_JLs9QKwMl_5roZB
X-Proofpoint-ORIG-GUID: G6VjSAdO_KMnmIpy_JLs9QKwMl_5roZB

On 5/20/2024 10:30 PM, Jason Wang wrote:
> On Mon, May 20, 2024 at 11:21 PM Steve Sistare
> <steven.sistare@oracle.com> wrote:
>>
>> Support the suspend operation.  There is little to do, except flush to
>> guarantee no workers are running when suspend returns.
>>
>> Signed-off-by: Steve Sistare <steven.sistare@oracle.com>
>> ---
>>   drivers/vdpa/vdpa_user/vduse_dev.c | 24 ++++++++++++++++++++++++
>>   1 file changed, 24 insertions(+)
>>
>> diff --git a/drivers/vdpa/vdpa_user/vduse_dev.c b/drivers/vdpa/vdpa_user/vduse_dev.c
>> index 73c89701fc9d..7dc46f771f12 100644
>> --- a/drivers/vdpa/vdpa_user/vduse_dev.c
>> +++ b/drivers/vdpa/vdpa_user/vduse_dev.c
>> @@ -472,6 +472,18 @@ static void vduse_dev_reset(struct vduse_dev *dev)
>>          up_write(&dev->rwsem);
>>   }
>>
>> +static void vduse_flush_work(struct vduse_dev *dev)
>> +{
>> +       flush_work(&dev->inject);
>> +
>> +       for (int i = 0; i < dev->vq_num; i++) {
>> +               struct vduse_virtqueue *vq = dev->vqs[i];
>> +
>> +               flush_work(&vq->inject);
>> +               flush_work(&vq->kick);
>> +       }
>> +}
>> +
>>   static int vduse_vdpa_set_vq_address(struct vdpa_device *vdpa, u16 idx,
>>                                  u64 desc_area, u64 driver_area,
>>                                  u64 device_area)
>> @@ -724,6 +736,17 @@ static int vduse_vdpa_reset(struct vdpa_device *vdpa)
>>          return ret;
>>   }
>>
>> +static int vduse_vdpa_suspend(struct vdpa_device *vdpa)
>> +{
>> +       struct vduse_dev *dev = vdpa_to_vduse(vdpa);
>> +
>> +       down_write(&dev->rwsem);
>> +       vduse_flush_work(dev);
>> +       up_write(&dev->rwsem);
> 
> Can this forbid the new work to be scheduled?

Are you suggesting I return an error below if the dev is suspended?
I can do that.

However, I now suspect this implementation of vduse_vdpa_suspend is not
complete in other ways, so I withdraw this patch pending future work.
Thanks for looking at it.

- Steve

> static int vduse_dev_queue_irq_work(struct vduse_dev *dev,
>                                      struct work_struct *irq_work,
>                                      int irq_effective_cpu)
> {
>          int ret = -EINVAL;
> 
>          down_read(&dev->rwsem);
>          if (!(dev->status & VIRTIO_CONFIG_S_DRIVER_OK))
>                  goto unlock;
> 
>          ret = 0;
>          if (irq_effective_cpu == IRQ_UNBOUND)
>                  queue_work(vduse_irq_wq, irq_work);
>          else
>                  queue_work_on(irq_effective_cpu,
>                        vduse_irq_bound_wq, irq_work);
> unlock:
>          up_read(&dev->rwsem);
> 
>          return ret;
> }
> 
> Thanks
> 
>> +
>> +       return 0;
>> +}
>> +
>>   static u32 vduse_vdpa_get_generation(struct vdpa_device *vdpa)
>>   {
>>          struct vduse_dev *dev = vdpa_to_vduse(vdpa);
>> @@ -806,6 +829,7 @@ static const struct vdpa_config_ops vduse_vdpa_config_ops = {
>>          .set_vq_affinity        = vduse_vdpa_set_vq_affinity,
>>          .get_vq_affinity        = vduse_vdpa_get_vq_affinity,
>>          .reset                  = vduse_vdpa_reset,
>> +       .suspend                = vduse_vdpa_suspend,
>>          .set_map                = vduse_vdpa_set_map,
>>          .free                   = vduse_vdpa_free,
>>   };
>> --
>> 2.39.3
>>
>