Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) client-ip=23.128.96.36;
Message-ID: <27312106-07b9-4719-970c-b8e1aed7c4eb@oracle.com>
Date:   Tue, 12 Dec 2023 15:44:00 -0800
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH vhost v2 4/8] vdpa/mlx5: Mark vq addrs for modification in
 hw vq
Content-Language: en-US
To:     Eugenio Perez Martin <eperezma@redhat.com>,
        Dragos Tatulea <dtatulea@nvidia.com>
Cc:     "Michael S . Tsirkin" <mst@redhat.com>,
        Jason Wang <jasowang@redhat.com>,
        Saeed Mahameed <saeedm@nvidia.com>,
        Leon Romanovsky <leon@kernel.org>,
        virtualization@lists.linux-foundation.org,
        Gal Pressman <gal@nvidia.com>, kvm@vger.kernel.org,
        linux-kernel@vger.kernel.org, Parav Pandit <parav@nvidia.com>,
        Xuan Zhuo <xuanzhuo@linux.alibaba.com>
References: <20231205104609.876194-1-dtatulea@nvidia.com>
 <20231205104609.876194-5-dtatulea@nvidia.com>
 <CAJaqyWeEY9LNTE8QEnJgLhgS7HiXr5gJEwwPBrC3RRBsAE4_7Q@mail.gmail.com>
From:   Si-Wei Liu <si-wei.liu@oracle.com>
Organization: Oracle Corporation
In-Reply-To: <CAJaqyWeEY9LNTE8QEnJgLhgS7HiXr5gJEwwPBrC3RRBsAE4_7Q@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?bEIreVdySGFWR0pjVUtqLzQzT1Y2cnV1S2JPeWlwdStKaXo4ZGJUSXhsUDgx?=
 =?utf-8?B?cEhIMEF0N0pZNDQwem1LNUg5TE1DeTdwbkU4a2M0TzR0elg5UzB2ZFlyYWNV?=
 =?utf-8?B?R3dZbUlRSXpvWXVIbHhaMVZnTjk0RnBlMWhzUWtiTDQxUzRidnhib2hET1Yv?=
 =?utf-8?B?bXZkaFpONXlZSitMaS9pb2QwUmdPaTkwOXllRURPT2JTTmN1Q3VVK0dWd0sv?=
 =?utf-8?B?d2pCZk5Eakh5RlR2Z1MyaFZNdzJSUXY0OHBlZXNKMWxCMFlYNFRVSGdaOTYw?=
 =?utf-8?B?cUVSMVlacTIzMnVQRUZxbm5ldXJjMW91dnF3dk9DSTMwMFZSWUZlUXJmNnRV?=
 =?utf-8?B?RStGTlpWL2ZzeXdReWZTMlo3aEh1UkdvK3BxeFBlUTJJQkdUSkplaFNzellL?=
 =?utf-8?B?dVJieHJNVHBQSERHcWFQYzV4T2JHRldPbHMyNkY3dG5NdEp5SEs5ZFVGQVkv?=
 =?utf-8?B?eWxsbW4vS2l2Zi9jSU50K2VxUGNoQXJTenI1TGhGOXovenJDc3RPekJ4Z0tV?=
 =?utf-8?B?RjZlNTZuTVppRTU5RW8rSzRmakRXTVFLY1lwQlBMSENzcXFKZTYvcUdoZmpF?=
 =?utf-8?B?VmVNSHdSZEIzaXF5V25zMjR6aFRvQXJjUWhmNUdpOUlPQmVVVWF2c3hKc242?=
 =?utf-8?B?QWZlWWFKb2JpV2k1a0ZlQXlDQVB2NXRJQWs3TlJiUjNYYjQ4WTlwYkhNZldw?=
 =?utf-8?B?WGV3bkh0MDRtdCtWSXFWQnczTTVoOXRlaDNUTGNsMUZlZnBKM3o3bjdWSWRQ?=
 =?utf-8?B?cjRDblgyaG1iQW9qQ0FzTDJRNFRxUUtLSkUrMkdVb3pGS1FRbW1ybGJyRlYv?=
 =?utf-8?B?bUllV2N2dWs2eDBFTDBhUEF2Q1FPOUZ4U1A2QXo5T3FuQlBlRkFSWVB4Wll4?=
 =?utf-8?B?L0dPMkxGRzgySEROenFGTFpFWm80eDVPQ2xUeUlXMEZFbXBXTFlkTlJNemw3?=
 =?utf-8?B?YUR3T05kRHF1eUpGU2VGcFlHek4zeWlISEQ3R2lBdUtkTmlFdHZtWlBhNEtR?=
 =?utf-8?B?MDhzcExxb3lvUHdwR0U0NTJJNTB2b09qcEsweldVMUUrZnJ1R1VaZnhJalMw?=
 =?utf-8?B?T0ppTDlRWUYwREE2d3E3SnpDdkVNUWREWjBWN0ZjWGUvcXR5cXJHSzA0dkdu?=
 =?utf-8?B?bDhPNWcrZTdVUjVaQm1jVk1GejQ0UUZ4TEFYTlpCVkVpS2xpVjZoMm0zcFlO?=
 =?utf-8?B?WjRwVzNrZjNEelpEMmJCUEZPZENPN3dMZlRveHlWL3RBdUhkWDFTZHBkWmlH?=
 =?utf-8?B?SlRBMG1aQlp0Y2hVUE1OcTFkSXo3bktwZzYrQUJnNmJmVjVWM3oreERTMkw0?=
 =?utf-8?B?LzVjU1d0Z2VBZzBXdEs5dkpCRWRxY0tmVElTR0E2OVFwWnJSbXFwNFJFeHM0?=
 =?utf-8?B?OUFnUjF1MmN3VXdpOHZjV2NFMS9QK1pEdnFVcThxTmVBbkhxMmFxeTdhMlJm?=
 =?utf-8?B?bTF3QWd3RHNPc2liWFNBYnQvNld1SE1XQnVMbDRpbXZwM2ZNNkUvZ0IwZ3p3?=
 =?utf-8?B?NXdMMXc3Vjk3K1RXSkYyRnBnNnZrdmRZN3FNUnVOZ3JJMExsQk05OGJMNjhQ?=
 =?utf-8?B?Q2ZpelI2U29PZkZLdnVZTmJ0dnM2WHBQWEt3ekZrcjJORWYxRFptamNjM3Nz?=
 =?utf-8?B?Z21FWHlsRVVObFAxT2plb2pReEkvcXhhTDRMSGpsOEQ0c2Zkais4Y0dPN3pE?=
 =?utf-8?B?TzIvdWx3RmxNYnBEaC9PNGppRE10M0hjajBKcDY4Zll3b1VjZ3UrbHFCL0Rl?=
 =?utf-8?B?TEJ0a0xNaVdxc2R3d1hlc0JLRkdsSEtRTVlGVWNSZnRFUGxBMGFrZVdLVGUw?=
 =?utf-8?B?UTlJQ2ZBM2c0NTZZNjF3YXhJQjRwOCt2NWtaa3lxbW5oditYT21pSTJtOFpw?=
 =?utf-8?B?RU14TDdzOWtVZEgzR0x2bUtpK1N1UnEreHVsNStYRFIxWnk0OVVaOGwwUjBn?=
 =?utf-8?B?dDJuMnVmbU1FNEFpRFFjYTNtbnFFVjR1RDdERTZGQytiYmdYRUNxUjZ5dUpC?=
 =?utf-8?B?SUphTDBlR04rdFlFRk5kQVBBYzFPc1dQNzd1QWtvMDFEVUtCZWlocnB4bGNQ?=
 =?utf-8?B?VmxwWXdEMHNGYjN1bU5ud1hkSm52cERPdVZtNHNQaE1LRjFQNUxiTlo0TjFE?=
 =?utf-8?Q?8/xTIqaCIDB81uoDiNRFkVQxA?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: =?utf-8?B?RXh1eUpDaDlXMHNybENWNjFpbkgvK05tTmVXTGZYdnA4N0pqSERrdi9kVHJl?=
 =?utf-8?B?Nm9ONk03Q0pwZWVpZ3NmUExwWVcrNVo0dmkwS0lkbVJPWHowYWZKQVZGRVor?=
 =?utf-8?B?M2Y2Q0ZKeEVYck1BNlc3NHVYc0loeDhYSmpLYjFaNGJKMFRFRlk2RUZqTEZy?=
 =?utf-8?B?WFFrYkorMXFLRGFaQ2hpTVZmWWYxWUk1Z3ZzSXhEc3NlUXNkQldhUmFtOHk5?=
 =?utf-8?B?UEQvZzdHa05ON04xbEpISTFMRUhPams5MG56RENHMGJJdm1scVphZHcwa0I2?=
 =?utf-8?B?L2dWcGI2ZUFJdUUrMXZzNlM0bXArZEpURDlJOVNHYXMwMi9uZlhqenBnL0RR?=
 =?utf-8?B?RDJJQ1hFVWN3c0VkM0pvVjJ4ZmdISjYzQ1oyb3lvTlFzckpTdkhVYVJTY3pI?=
 =?utf-8?B?T05yd3U2YnBhZmtKT2g1eVZTaGxWS1NRVDZxelh6ei9kVkxDd1JKYUhzMjBi?=
 =?utf-8?B?cTVPZFBEMnB4aVRvaWRhS3dyMmtCbmwzakRkMWhyU0JEMmpxTWs5ak5MSHBK?=
 =?utf-8?B?UjNxdkNaUXlKNjJuVk9RRjl1cy96dDk3RjdoOE52WmVnY1RNbXNJS1lzMGhL?=
 =?utf-8?B?Yis2N1VIOC9yWDhyWnF1aDNwOFd2ZExmTEdRZXlrTmhHb0Z3VkdyOW9UaHI4?=
 =?utf-8?B?Z1ZLa2NZanNGOFFqTkRleEk3SVZVVFF1MjMwYkhEeFIxYUxuOEVLUXFIL3RB?=
 =?utf-8?B?eGd4ZnkxSjJibXVBTWZRS095cFNVKzczUndIREpROW41Q2h4VXVkZVhTQWN2?=
 =?utf-8?B?MHZwZzF3UWxjb0tqeG5Xa08zWC9UWEZMeE5Zc0V3ZUJsaXNhU0NuMWVtUE9o?=
 =?utf-8?B?N3hGWlNmRlRkOHhkWm9icERidzV3YkNsdlhKYkcwbCtWaXEwSElTVXRQNC9x?=
 =?utf-8?B?WktHU09KYUZnSWZ4Y0dTbWVNc1prbzk1c2lDczFZb3h6U3pQSnR1Y3dXNnVq?=
 =?utf-8?B?TFhhOEQwVDFEWE1PcjRvZ05JRzFISVBkL0hqVDMzQUxkWXVxRnRYZytOMmYr?=
 =?utf-8?B?UDl4MW5xZW5ucWJHUHBuUEVwQ3RzOFVaazJya2VJbi8veXR4U3QxUnk5cnU2?=
 =?utf-8?B?ZnZWT1FjZmVvcXFucWszd0FadUxIZkdMK1lxUWJucVBiZTdwbWVYT1lLQ1NI?=
 =?utf-8?B?UWdQWklIeHpaS3NKRHFKVzBhTzV1Z0I4UTN6aDVKSHU0UmVyaDRzU253SGRJ?=
 =?utf-8?B?eEk1eHppWXpNdG9nSzZnNzVYZC9xdU5pa2tpZHZwcjhLY0haSC82NWszdWk1?=
 =?utf-8?B?SlROVFk1SWR3azRVYUtteUxSM25aVWNGSmx0d1YzY0cyUnBiaEJNVGRnWDZj?=
 =?utf-8?Q?OerhoTpciFxI8=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: ee9c561d-b00c-41a9-ab8b-08dbfb6c3898
X-MS-Exchange-CrossTenant-AuthSource: MW4PR10MB6535.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Dec 2023 23:44:02.8676
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: EjbXeLABUiiBrGRWnmwS6OUnA576sXspPkwZPCkbj0EgDpaYR57b+UywHtthJ5IVBZSQizkI5DOR7cLC3+h19A==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB5614
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2023-12-12_12,2023-12-12_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 malwarescore=0
 mlxlogscore=999 adultscore=0 phishscore=0 suspectscore=0 mlxscore=0
 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2311290000 definitions=main-2312120185
X-Proofpoint-GUID: ady3aXqUu02jlwIoIFgaMjr74eTfwt_r
X-Proofpoint-ORIG-GUID: ady3aXqUu02jlwIoIFgaMjr74eTfwt_r
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID,
	DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,
	SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Tue, 12 Dec 2023 15:44:36 -0800 (PST)


On 12/12/2023 11:21 AM, Eugenio Perez Martin wrote:
> On Tue, Dec 5, 2023 at 11:46 AM Dragos Tatulea <dtatulea@nvidia.com> wrote:
>> Addresses get set by .set_vq_address. hw vq addresses will be updated on
>> next modify_virtqueue.
>>
>> Signed-off-by: Dragos Tatulea <dtatulea@nvidia.com>
>> Reviewed-by: Gal Pressman <gal@nvidia.com>
>> Acked-by: Eugenio Pérez <eperezma@redhat.com>
> I'm kind of ok with this patch and the next one about state, but I
> didn't ack them in the previous series.
>
> My main concern is that it is not valid to change the vq address after
> DRIVER_OK in VirtIO, which vDPA follows. Only memory maps are ok to
> change at this moment. I'm not sure about vq state in vDPA, but vhost
> forbids changing it with an active backend.
>
> Suspend is not defined in VirtIO at this moment though, so maybe it is
> ok to decide that all of these parameters may change during suspend.
> Maybe the best thing is to protect this with a vDPA feature flag.
I think protect with vDPA feature flag could work, while on the other 
hand vDPA means vendor specific optimization is possible around suspend 
and resume (in case it helps performance), which doesn't have to be 
backed by virtio spec. Same applies to vhost user backend features, 
variations there were not backed by spec either. Of course, we should 
try best to make the default behavior backward compatible with 
virtio-based backend, but that circles back to no suspend definition in 
the current virtio spec, for which I hope we don't cease development on 
vDPA indefinitely. After all, the virtio based vdap backend can well 
define its own feature flag to describe (minor difference in) the 
suspend behavior based on the later spec once it is formed in future.

Regards,
-Siwei


>
> Jason, what do you think?
>
> Thanks!
>
>> ---
>>   drivers/vdpa/mlx5/net/mlx5_vnet.c  | 9 +++++++++
>>   include/linux/mlx5/mlx5_ifc_vdpa.h | 1 +
>>   2 files changed, 10 insertions(+)
>>
>> diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c
>> index f8f088cced50..80e066de0866 100644
>> --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c
>> +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c
>> @@ -1209,6 +1209,7 @@ static int modify_virtqueue(struct mlx5_vdpa_net *ndev,
>>          bool state_change = false;
>>          void *obj_context;
>>          void *cmd_hdr;
>> +       void *vq_ctx;
>>          void *in;
>>          int err;
>>
>> @@ -1230,6 +1231,7 @@ static int modify_virtqueue(struct mlx5_vdpa_net *ndev,
>>          MLX5_SET(general_obj_in_cmd_hdr, cmd_hdr, uid, ndev->mvdev.res.uid);
>>
>>          obj_context = MLX5_ADDR_OF(modify_virtio_net_q_in, in, obj_context);
>> +       vq_ctx = MLX5_ADDR_OF(virtio_net_q_object, obj_context, virtio_q_context);
>>
>>          if (mvq->modified_fields & MLX5_VIRTQ_MODIFY_MASK_STATE) {
>>                  if (!is_valid_state_change(mvq->fw_state, state, is_resumable(ndev))) {
>> @@ -1241,6 +1243,12 @@ static int modify_virtqueue(struct mlx5_vdpa_net *ndev,
>>                  state_change = true;
>>          }
>>
>> +       if (mvq->modified_fields & MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_ADDRS) {
>> +               MLX5_SET64(virtio_q, vq_ctx, desc_addr, mvq->desc_addr);
>> +               MLX5_SET64(virtio_q, vq_ctx, used_addr, mvq->device_addr);
>> +               MLX5_SET64(virtio_q, vq_ctx, available_addr, mvq->driver_addr);
>> +       }
>> +
>>          MLX5_SET64(virtio_net_q_object, obj_context, modify_field_select, mvq->modified_fields);
>>          err = mlx5_cmd_exec(ndev->mvdev.mdev, in, inlen, out, sizeof(out));
>>          if (err)
>> @@ -2202,6 +2210,7 @@ static int mlx5_vdpa_set_vq_address(struct vdpa_device *vdev, u16 idx, u64 desc_
>>          mvq->desc_addr = desc_area;
>>          mvq->device_addr = device_area;
>>          mvq->driver_addr = driver_area;
>> +       mvq->modified_fields |= MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_ADDRS;
>>          return 0;
>>   }
>>
>> diff --git a/include/linux/mlx5/mlx5_ifc_vdpa.h b/include/linux/mlx5/mlx5_ifc_vdpa.h
>> index b86d51a855f6..9594ac405740 100644
>> --- a/include/linux/mlx5/mlx5_ifc_vdpa.h
>> +++ b/include/linux/mlx5/mlx5_ifc_vdpa.h
>> @@ -145,6 +145,7 @@ enum {
>>          MLX5_VIRTQ_MODIFY_MASK_STATE                    = (u64)1 << 0,
>>          MLX5_VIRTQ_MODIFY_MASK_DIRTY_BITMAP_PARAMS      = (u64)1 << 3,
>>          MLX5_VIRTQ_MODIFY_MASK_DIRTY_BITMAP_DUMP_ENABLE = (u64)1 << 4,
>> +       MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_ADDRS           = (u64)1 << 6,
>>          MLX5_VIRTQ_MODIFY_MASK_DESC_GROUP_MKEY          = (u64)1 << 14,
>>   };
>>
>> --
>> 2.42.0
>>