Received-SPF: pass (google.com: domain of linux-kernel+bounces-59556-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
Message-ID: <80eaded6-b6b6-4273-a8e2-186a6a56659c@oracle.com>
Date: Fri, 9 Feb 2024 10:49:33 -0500
User-Agent: Mozilla Thunderbird
Subject: Re: [RFC V1 05/13] vhost-vdpa: VHOST_IOTLB_REMAP
Content-Language: en-US
To: Eugenio Perez Martin <eperezma@redhat.com>
Cc: virtualization@lists.linux-foundation.org, linux-kernel@vger.kernel.org,
        "Michael S. Tsirkin" <mst@redhat.com>,
        Jason Wang <jasowang@redhat.com>, Si-Wei Liu <si-wei.liu@oracle.com>,
        Xuan Zhuo <xuanzhuo@linux.alibaba.com>,
        Dragos Tatulea <dtatulea@nvidia.com>, Eli Cohen <elic@nvidia.com>,
        Xie Yongji <xieyongji@bytedance.com>
References: <1704919215-91319-1-git-send-email-steven.sistare@oracle.com>
 <1704919215-91319-6-git-send-email-steven.sistare@oracle.com>
 <CAJaqyWc+nSMPv7248x2QWOgAPOjwZ2BJ52LrOhKUu51t+3QViw@mail.gmail.com>
From: Steven Sistare <steven.sistare@oracle.com>
Organization: Oracle Corporation
In-Reply-To: <CAJaqyWc+nSMPv7248x2QWOgAPOjwZ2BJ52LrOhKUu51t+3QViw@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?utf-8?B?b013VkR5dkwxUW40SmhmTEFIVW5LK3hIN1pmK1dTQXAwYVUydHJXaFVrMHR1?=
 =?utf-8?B?eHhQUGJnTlFkMDdvY3BFMlQ2WjBPeWVlaGV2VnBzancwTzRuTjZUZVBFa1hM?=
 =?utf-8?B?ekl5MjJ0V0VBTnhTbDlXbU0reVhneXkzMFBTUHVnMkR0WnVOQm5OUXdLMjdC?=
 =?utf-8?B?YTJ5bEVGU2JacWVubEk2UkJ4eDd1azFtb1VOTk1GSmlqbVJFSURMNTRrNVlr?=
 =?utf-8?B?NWtjVkxuT2l1MWlYa085K0p6QlUxcUtnN2ZQL2N1THJ0QThudXdvM0NjRUN4?=
 =?utf-8?B?Q2JYM010aDRya2FQaTJldU03ZGhkNTlXVXlKRDFvMVErUzNCdURnNDAxRG90?=
 =?utf-8?B?YlBsZTVBLzB6TXE3dG5JTkxHaEltdnd0cDIyYWQ4RndnalBLdEpTREdHOXRt?=
 =?utf-8?B?UXRRcXdEczNYYWVieWd3a0hGTUpnTVNVREV0TTllUVBDYWlvUU02alpkdVQz?=
 =?utf-8?B?WEVPWFQxQWYwVjdudGRzOE5mN25OS1lZem40ZHJOUXNQNVFvM0hiOHpWay9T?=
 =?utf-8?B?bzYreWwwTWM2dnJ5OGZOVXAva2t5OXBiMmt4bDVSenhMVWxXMTJJWFJuSGVG?=
 =?utf-8?B?Ykp6RXlPNnRDbDBMSHl0Z2xpY2c2UllUVTI2TkxaOCt0czdZWmNiSGFMa1c4?=
 =?utf-8?B?OVQvZ1hwcXBRTXlpRzhzcmJvcEo3YjF4RldhallzNWl3OGVSTGdyYVdCTDZ6?=
 =?utf-8?B?UHpleW1BQ3JJQXphdGhWRHhuaE9vaVk5VHRjNWN5Rm9TZW41T3hUYTdwQnpY?=
 =?utf-8?B?UVR6TmtDNnJ4SG1YYlkzWnJPbWxvdDBCKzdSUFZMN2l0L2ZhT0JJOEtrVHJ1?=
 =?utf-8?B?TXVaNU9iMms5cHdyd1dmT3pwME9HazBQYy9ockxLcnYzVGI3NklieGppSjRQ?=
 =?utf-8?B?RkdmNCs5TW5mNS9UMmRnZStQMDVNQzU1TlYvQ3lXSUw0Q2pHR3ZYMWw2cUpF?=
 =?utf-8?B?RXdIYzByTU9UTWNFNzkzNjZDbzM1WkowRjk4RzM4cjdDRjJ2bGRyd2ZJZ3Rj?=
 =?utf-8?B?aEFiSVlGUm9hMmMwV1JMcnUxOGpIWGppUTM4ZUYrVjhTd1NaMGRtbWswV0RD?=
 =?utf-8?B?RnMwYWJlaXE2Zm1QMkRhM3JReGRmLzRpUm1hM1BycnU4Z2JIbWVNRE4yMTFC?=
 =?utf-8?B?a0RPTWJnRXBZK3RzaUVKYW0xbVgva1lZbjUvZnNzRk1DYVp5elRxZnNwVG1K?=
 =?utf-8?B?YzNCQTZtZzVaZ3lkVlk2THpXUTQ4UzQzYmJxaGhtWFlHMzVpVEorZVRZS0xv?=
 =?utf-8?B?V1RzWTlENEJHNWVQZmlESmxGRjdXOU95UFFlemZiTGk2UW9uaU9CUDhZWVlx?=
 =?utf-8?B?Q09tUllVYnNsc3EvRFRnVHNVbzZicDFWY3RWbEJYRGQ5Rm9nMW8rKzJZQ0lI?=
 =?utf-8?B?WGpXS1VtYUhIL1JMZXJhWjladHJxRFdaMU54TEhiT2ZrMGE0MWVFRk1ha2pr?=
 =?utf-8?B?cGFwZ3U4Sk1vaVh3YXJTcXV4QUd1MzBkU1FWa0N4K1lJaFFacnNESU9VOFRR?=
 =?utf-8?B?WGY5ajdaTGVoS2R5amxGaTRnYTdzVXJIb3d2c1h1aDVjZ3ZvbUV1K1IrQ2NL?=
 =?utf-8?B?NTRmeXVON0QzdDQ2OW16M2FBd3dxeFZvbzY4R0ZlbndpRVc2eDE5T0N3NXQ5?=
 =?utf-8?B?SjJvQnBvS2gyNTRvRTdhZXl3eGlsWUk0M2NyVmlRYXZHaHJiRW1QR0d0Y25P?=
 =?utf-8?B?YTBwYnpNRnVjcTRTcU1Da0RYZ1l6WG5NbkQ3cXFld3JLelhURDZHR25rVDJB?=
 =?utf-8?B?aEZFcWFrVGJKdEZPOFBOTDNyb1g3OEJTODZFek5zc2UrZjRYY3o5dEprVnM5?=
 =?utf-8?B?T1c1aU1DR2Y4Uk9Dc0FNODlOdkhSVytEWTloNjFBRFpPMEZ1QzNHaTVNN25z?=
 =?utf-8?B?VUdneG5xYXJqY3FJVzRMZHBjdHlWRzdkcEdzYjhkbmR1c1VrckJtY2VFVzhT?=
 =?utf-8?B?WkYxODAwRDJyelhUMzBGd0ZaVEp5Z09ER1RwU2FwbWdEZkMwQnBoQW9aZ0dC?=
 =?utf-8?B?ckk2SHhwbFFEc0dCTExZbFpaU3ZXY3lOdlZLdERUWExNd3hpYmUyTmIrSzBJ?=
 =?utf-8?B?OUZyZFFjYVVudno2MWlCUUlZWDg2cUpBTGVVRUM1d3VOSGVyYjlzeHBveFo0?=
 =?utf-8?B?VUZHLzg0dEdJYUZpTlFlbkt6cXVyL3pjZER4TlEwNlVCV1o3dVNPb2wwallC?=
 =?utf-8?B?Q3c9PQ==?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	26ZeGl8BnORsOnXgR2QU8Ong5/UaiecWRPDwaZld1VfRYwBDkkYuWDW42UoFx8JQCweXPWx3OjPyqxN8VgESUTu2NlaTZ34Y/TUONvdy+IEtOyBI13peDUZvP4TGuGnlo92mCXbXLLoLwR+h1ltFXKGZHs94NigsLESHKhD/3mkGsb7cz1SRlLFoJacc+KHy8aqENvjsxWukMhQldbuIumJ7eIQy3ft3cTtpDNP/7qhBh6whgYurYp1enPdQ55/FljZa36mNSIiA/S/2023HOJM4rgRsMM7m9nHkTf5is4L2ZUKnjCB6mFMryGbB4J2uIJ8KrDXjPqJEIJI0QcVy3+P9GKaf757QOL5OP4Tp0Fxj3ckxlDtPaKtqbjroanEYO8GZdDmBaNVjQMWHD1WQC7pmDx5r4AC4AbAdHrmgqTRuzPeoktOdMxNxMDeu6NkqYFGBpNPEzJAPpOwj7gYWbuV/+s6kGKCICNq2NPsBKshuFNTDbYVGJKHt9UirAkmoxgeADrGb3BzmNxLdlV3WWBpkr9WSWt3nol1cNuF8/gS8OHD7sIbXzPTWX8jhDfduQI5XQ4RmDiJb174oZVXH+WwNq2ZQg/MIIAAkP45Ryuw=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 2ddfe19c-578c-435a-1c99-08dc2986b954
X-MS-Exchange-CrossTenant-AuthSource: SA2PR10MB4684.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Feb 2024 15:49:39.3479
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: twqLSwNF5e/Kf6eYcFDXbmguTije/E7Ynq3zgdWXzYCRs0W2Em7AsKfwiaqs17qe6W5F8GfEalKsp+vITRBaXJEG7RMh9HZZSdzLPlFatGg=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR10MB5263
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2024-02-09_13,2024-02-08_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 spamscore=0
 mlxlogscore=999 bulkscore=0 mlxscore=0 phishscore=0 adultscore=0
 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2311290000 definitions=main-2402090117
X-Proofpoint-ORIG-GUID: 3XTkTWpNp-DtttPT_wE1she8bhXzhxp6
X-Proofpoint-GUID: 3XTkTWpNp-DtttPT_wE1she8bhXzhxp6

On 1/16/2024 1:14 PM, Eugenio Perez Martin wrote:
> On Wed, Jan 10, 2024 at 9:40 PM Steve Sistare <steven.sistare@oracle.com> wrote:
>>
>> When device ownership is passed to a new process via VHOST_NEW_OWNER,
>> some devices need to know the new userland addresses of the dma mappings.
>> Define the new iotlb message type VHOST_IOTLB_REMAP to update the uaddr
>> of a mapping.  The new uaddr must address the same memory object as
>> originally mapped.
> 
> I think this new ioctl is very interesting, and can be used to
> optimize some parts of live migration with shadow virtqueue if it
> allows to actually replace the uaddr. Would you be interested in that
> capability?

Sure.  Please share your thoughts on how it would be used.  I don't follow,
because with live migration, you are creating a new vdpa instance with new
shadow queues on the destination, versus relocating old shadow queues (which 
we could do during live update which preserves the same vdpa instance).

(Sorry for the late response, I stashed this email and forgot to respond.)

>> The user must suspend the device before the old address is invalidated,
>> and cannot resume it until after VHOST_IOTLB_REMAP is called, but this
>> requirement is not enforced by the API.
>>
>> Signed-off-by: Steve Sistare <steven.sistare@oracle.com>
>> ---
>>  drivers/vhost/vdpa.c             | 34 ++++++++++++++++++++++++++++++++
>>  include/uapi/linux/vhost_types.h | 11 ++++++++++-
>>  2 files changed, 44 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/vhost/vdpa.c b/drivers/vhost/vdpa.c
>> index faed6471934a..ec5ca20bd47d 100644
>> --- a/drivers/vhost/vdpa.c
>> +++ b/drivers/vhost/vdpa.c
>> @@ -1219,6 +1219,37 @@ static int vhost_vdpa_pa_map(struct vhost_vdpa *v,
>>
>>  }
>>
>> +static int vhost_vdpa_process_iotlb_remap(struct vhost_vdpa *v,
>> +                                         struct vhost_iotlb *iotlb,
>> +                                         struct vhost_iotlb_msg *msg)
>> +{
>> +       struct vdpa_device *vdpa = v->vdpa;
>> +       const struct vdpa_config_ops *ops = vdpa->config;
>> +       u32 asid = iotlb_to_asid(iotlb);
>> +       u64 start = msg->iova;
>> +       u64 last = start + msg->size - 1;
>> +       struct vhost_iotlb_map *map;
>> +       int r = 0;
>> +
>> +       if (msg->perm || !msg->size)
>> +               return -EINVAL;
>> +
>> +       map = vhost_iotlb_itree_first(iotlb, start, last);
>> +       if (!map)
>> +               return -ENOENT;
>> +
>> +       if (map->start != start || map->last != last)
>> +               return -EINVAL;
>> +
>> +       /* batch will finish with remap.  non-batch must do it now. */
>> +       if (!v->in_batch)
>> +               r = ops->set_map(vdpa, asid, iotlb);
> 
> I'm missing how these cases are handled:
>
> * The device does not expose set_map but dma_map / dma_unmap (you can
> check this case with the simulator)

I chose not to support dma_map, because set_map looks to be more commonly supported,
and batch-mode plus set_map is the most efficient way to support remapping.
I could define a dma_remap entry point if folks think that is important.
Regardless, I will check that ops->set_map != NULL before calling it.

> * The device uses platform iommu (!set_map && !dma_unmap vdpa_ops).

I believe this just works, because iommu_map() never saves userland address, so it 
does not need to be informed when uaddr changes.  That is certainly true for the 
code path:

  vhost_vdpa_pa_map()
    vhost_vdpa_map()
      if !dma_map && !set_map
        iommu_map()

However, this code path confuses me:

  vhost_vdpa_process_iotlb_update()
    if (vdpa->use_va)
      vhost_vdpa_va_map(... uaddr)
        vhost_vdpa_map(... uaddr)
          iommu_map(... uaddr)

AFAICT uaddr is never translated to physical.
Maybe use_va is always false if !dma_map && !set_map  ?

- Steve

>> +       if (!r)
>> +               map->addr = msg->uaddr;
>> +
>> +       return r;
>> +}
>> +
>>  static int vhost_vdpa_process_iotlb_update(struct vhost_vdpa *v,
>>                                            struct vhost_iotlb *iotlb,
>>                                            struct vhost_iotlb_msg *msg)
>> @@ -1298,6 +1329,9 @@ static int vhost_vdpa_process_iotlb_msg(struct vhost_dev *dev, u32 asid,
>>                         ops->set_map(vdpa, asid, iotlb);
>>                 v->in_batch = false;
>>                 break;
>> +       case VHOST_IOTLB_REMAP:
>> +               r = vhost_vdpa_process_iotlb_remap(v, iotlb, msg);
>> +               break;
>>         default:
>>                 r = -EINVAL;
>>                 break;
>> diff --git a/include/uapi/linux/vhost_types.h b/include/uapi/linux/vhost_types.h
>> index 9177843951e9..35908315ff55 100644
>> --- a/include/uapi/linux/vhost_types.h
>> +++ b/include/uapi/linux/vhost_types.h
>> @@ -79,7 +79,7 @@ struct vhost_iotlb_msg {
>>  /*
>>   * VHOST_IOTLB_BATCH_BEGIN and VHOST_IOTLB_BATCH_END allow modifying
>>   * multiple mappings in one go: beginning with
>> - * VHOST_IOTLB_BATCH_BEGIN, followed by any number of
>> + * VHOST_IOTLB_BATCH_BEGIN, followed by any number of VHOST_IOTLB_REMAP or
>>   * VHOST_IOTLB_UPDATE messages, and ending with VHOST_IOTLB_BATCH_END.
>>   * When one of these two values is used as the message type, the rest
>>   * of the fields in the message are ignored. There's no guarantee that
>> @@ -87,6 +87,15 @@ struct vhost_iotlb_msg {
>>   */
>>  #define VHOST_IOTLB_BATCH_BEGIN    5
>>  #define VHOST_IOTLB_BATCH_END      6
>> +
>> +/*
>> + * VHOST_IOTLB_REMAP registers a new uaddr for the existing mapping at iova.
>> + * The new uaddr must address the same memory object as originally mapped.
>> + * Failure to do so will result in user memory corruption and/or device
>> + * misbehavior.  iova and size must match the arguments used to create the
>> + * an existing mapping.  Protection is not changed, and perm must be 0.
>> + */
>> +#define VHOST_IOTLB_REMAP          7
>>         __u8 type;
>>  };
>>
>> --
>> 2.39.3
>>
>