Received-SPF: pass (google.com: domain of linux-kernel+bounces-29424-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Message-ID: <aea1681c-e04d-4678-b161-6dbd2b13b82e@oracle.com>
Date: Wed, 17 Jan 2024 15:31:36 -0500
User-Agent: Mozilla Thunderbird
Subject: Re: [RFC V1 00/13] vdpa live update
Content-Language: en-US
To: Jason Wang <jasowang@redhat.com>
Cc: virtualization@lists.linux-foundation.org, linux-kernel@vger.kernel.org,
        "Michael S. Tsirkin" <mst@redhat.com>,
        Si-Wei Liu <si-wei.liu@oracle.com>,
        Eugenio Perez Martin <eperezma@redhat.com>,
        Xuan Zhuo <xuanzhuo@linux.alibaba.com>,
        Dragos Tatulea
 <dtatulea@nvidia.com>, Eli Cohen <elic@nvidia.com>,
        Xie Yongji <xieyongji@bytedance.com>
References: <1704919215-91319-1-git-send-email-steven.sistare@oracle.com>
 <CACGkMEuFTRBYU+z3ZSWzMMv2650PQ=kduGxcGXaf0T5222Bh4g@mail.gmail.com>
From: Steven Sistare <steven.sistare@oracle.com>
Organization: Oracle Corporation
In-Reply-To: <CACGkMEuFTRBYU+z3ZSWzMMv2650PQ=kduGxcGXaf0T5222Bh4g@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?utf-8?B?MnJKT09XQmhBYkF0ZkpraXA0YTVqbW9NU3BOMEdJaXMyS0xiU2VTUWdheHRL?=
 =?utf-8?B?eVRuUkZpam9ZQXFPSUhjalNXQlZqRmZSY2pVL2xzZTNFbzBadlN6eUtUZEsw?=
 =?utf-8?B?SXp0TXBkYmFEUjY1UnU4aFNENVJYL2piUUpzWThqVnJvRjFzMXFVeVo0Sndi?=
 =?utf-8?B?N0I3dnlWeUZCSWJ6MldWRy9aS0UzRDMyNlU3dGgwK2prc0MvclladU1jZXF3?=
 =?utf-8?B?Tk81Wm5xT2pVMG0wbVdvUVlVWjNxMXVnLythMjFseUJXM0xySGF3aDBqdHgy?=
 =?utf-8?B?S3cyR2FIWVE0TDB4T0VGbGdsS1dSb29BTGxtbnBvOXZzNEVlVnNEVFNZY2lD?=
 =?utf-8?B?d3hmaEF2b1ZKRy9vdFpjbXJlRzZidW9qUFRFV3dibFdmcVJoa1p0NklKV05v?=
 =?utf-8?B?YnYrUWhyZUlpTHNucWplSDNxbVVwV3dnUUxtSThEbHRERElUUTZMNlB4QU1n?=
 =?utf-8?B?bUpySmVKRlY5cTdJQ3p0b2tud2wxdzd5OWZ5V3ZpQmovTUhSMitWSUxwMTlT?=
 =?utf-8?B?RHhXa0RVdTlTcGNCOHZlc2dTaUtvZWxuOVh1cGppT3JraGZiRnVBcU1WcDcv?=
 =?utf-8?B?T2hEeGMzd1VMUkNqclYvVzRhRDU0alA0M3llMDdnbWVMaElQZEJGbFVvbkxY?=
 =?utf-8?B?aDlqZ0VDNlJrY2YzbTFEQVVaUUFLQVA4ajQvOWlkRXp0WWJKSXdNamNGSXlB?=
 =?utf-8?B?S3QwZjFVYTM4cVp3WnRWdU01Qk9lN280REUxV1lvMUZNZWtHZUpQYlpPU0NK?=
 =?utf-8?B?c09tTW8wSEpmY3prVVVLcHF5d0t1UXNyakhzbjdOb01PbEdvOTl5MU9zSkQw?=
 =?utf-8?B?V3hsVzZTdzJRNElvUWUyQkk0M1ZhL1JVWW5EVzFnK0dFcDFKNDFIRGM4L1NJ?=
 =?utf-8?B?amJuSHM2R1VmRzFyR1hGYTdxNWtRUmtJR0l5bllZSFFtais3ZGwzdlNqUGhX?=
 =?utf-8?B?cTVMVlNWOXNXVnZEc0RtdFQ0bXpDTlg2L2NuT3RTOEQ3ZVZMck9nZGZJQkQ4?=
 =?utf-8?B?MDNwV1FJMkdONit0WEJ2SlFyYmNCSjlWMWF5YlFQMjhWUS9BenUzZHJVVEQ2?=
 =?utf-8?B?WTFTdDFxVFBuak5aY2duSkxWbXp6WFM5Mk9ISXUxaFM0dHdRZFJ2OHdCU2pl?=
 =?utf-8?B?dXBUSnV2ZzZ6VENpWDdVWnE1aWRqTVZCcG1QZ0o3RldoZGxTaXZManNNSE5y?=
 =?utf-8?B?d095OGo0a3hRUG9WYXZwWjNnd2VEOUp5VnYvVWw5K3ZXWThOZ25SSTFIamJy?=
 =?utf-8?B?MzAwaEdzSExoU1BzQTZrRU96ODRCeFB3WWhqeXY5dnMzTVBHRjVoS1NkYTZE?=
 =?utf-8?B?bStzdjk3R1V4Q25pT1dVdUhtaDhBWG8reS80WFJ4RW5KeFhXaG04NklXRmtC?=
 =?utf-8?B?cW84Qnhzb0pybkJDUncxYW16YTlzNVNIVU1PTCt0R0ZRV2ExbkJ6TW9iZmxz?=
 =?utf-8?B?dmZlazJPVmpyTjVKcDc0ZStycXlSaDF2UkE5T3VUNFkrV0dRVVY5czZ0VzBI?=
 =?utf-8?B?Z2FIMzRneHFSMG9hWDUxUjZuTFJvUUVPUHM1WjNZSTlkV2hEVVBIQS9acEZo?=
 =?utf-8?B?Znc0SXR5NU1NZTZhQmZ3ZWRCYTZQeVVVcE5kZ0hSVzRnb2dYTGJ4YVJwUExY?=
 =?utf-8?B?UlM0RlpESUJNYTh6VXVSTWFGdGgzMmpPeFRYOGRqUDZuR1VhbWJ5Zmg2SnNx?=
 =?utf-8?B?VkNqUUNNVDYvblpNc01DWUcxa3YzTzVJNWhVRU5ndXNjajZQUjJuYjdlVThN?=
 =?utf-8?B?WHA1VFlydmRJMGZaQVpVcTdNS3VkZkZhcldUcWNqQUxBRU9mL090K1RaKy8r?=
 =?utf-8?B?cUxqdkI2eG44Z21JUmlxbVpvZVVPeGphaGgySDMzUENIdGpENkFIeVZiY1J5?=
 =?utf-8?B?cFpoNmFpSDFoenRscWprV0ZiUUZIYTA5WGQ2R0s5NTlpSlNpaTkxMFErMndn?=
 =?utf-8?B?ckNBc09qWmhVQVZFQzJEU3p5U1BVK3M0S2IrYzVZcXdoQTFONkEzajIvUjZx?=
 =?utf-8?B?NHEzOEo1SVpESWlYN0lraStMY3NlSnduTzJBZ0cwZFQwQnRKcnNMb1ZXVE5i?=
 =?utf-8?B?UGpjU0t1ZkJOVHF2ZTlCTGpoWEp2MXBuYk9KMkhaSi9XM3ZpZjNJSzVpMEdl?=
 =?utf-8?B?M0YvNGZvbDE3MXVnZ1lET3UrQUVGbEE0Vm1ydmdqYy9TaWQ0NkM1NDdOdWZO?=
 =?utf-8?B?bWc9PQ==?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	UkMDgqvZK6C6zAlTCbvBH2ultJrV6VEG4JILAOyCF0JLwpa0cVAqUNfRpvrMyLzmX5ZUWl61uoE6Yj1mgu+vVQaWJNk2OCEQSgtEc0FGAyOVH1abDgLM6N5HTklV3uI2pzY+sDPHpAkS/GijiDOpSaF0xBhMA9GzjQfFAd1tkM8NGnHYR8QyzPaNCgIsKfvBI+5q6vzrFId0HxkBstlGn+VHZ8XDWZ5oAYfsLnxLHzEOKijkLOlbblyytGOwcMkhzDHQa6OufJbE38lnBp105rL8bQ5a4SHoq8IZ1N3vlQGga4HHr5eQrllvXIiYE7gxgDzYjdIq1zB9iHNM1XHlz4Br9p960xfosaJI5FImpGDCsGt0SVW0kZcVby7Vd7TcBEEodpSSlfMVegz28Jnl9fifXnlgvIEwHZsWadEISmh4H0gmYya+FnO9EQWxFHl8gQSj/H0ZNCplnGzRcgYLLA3Rnir1dgoQ5FINAb8eJgNFnTlxF40YyVYkFIZJsldrUB++ppASADIrocn+QTalv/fNZx2iEtylE6UNLTM9kLpyG3MKT/NBqqZbUX21W8eMl3eqAHN9NbT1R6RpRZlGBWO5RPtekij8F2sA0aOz47Y=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: dc9e63cd-8e1a-41ec-67dc-08dc179b4e9f
X-MS-Exchange-CrossTenant-AuthSource: SA2PR10MB4684.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Jan 2024 20:31:38.7266
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: fCQQhn9GmZ+FBCXtR8o5/NscMx4FED+ourJP1463QVOdxLj7FhzOt2L4mDwesCjLYl1cnOAMPgCswRMe3Ws27hlBjXzPypSn7jRTDc2ySZs=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR10MB5942
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2024-01-17_12,2024-01-17_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 adultscore=0 phishscore=0
 suspectscore=0 malwarescore=0 spamscore=0 mlxlogscore=999 mlxscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000
 definitions=main-2401170150
X-Proofpoint-ORIG-GUID: CvoQpOU2Y3gHV1diVh6ewQDnPubACy_7
X-Proofpoint-GUID: CvoQpOU2Y3gHV1diVh6ewQDnPubACy_7

On 1/10/2024 9:55 PM, Jason Wang wrote:
> On Thu, Jan 11, 2024 at 4:40 AM Steve Sistare <steven.sistare@oracle.com> wrote:
>>
>> Live update is a technique wherein an application saves its state, exec's
>> to an updated version of itself, and restores its state.  Clients of the
>> application experience a brief suspension of service, on the order of
>> 100's of milliseconds, but are otherwise unaffected.
>>
>> Define and implement interfaces that allow vdpa devices to be preserved
>> across fork or exec, to support live update for applications such as qemu.
>> The device must be suspended during the update, but its dma mappings are
>> preserved, so the suspension is brief.
>>
>> The VHOST_NEW_OWNER ioctl transfers device ownership and pinned memory
>> accounting from one process to another.
>>
>> The VHOST_BACKEND_F_NEW_OWNER backend capability indicates that
>> VHOST_NEW_OWNER is supported.
>>
>> The VHOST_IOTLB_REMAP message type updates a dma mapping with its userland
>> address in the new process.
>>
>> The VHOST_BACKEND_F_IOTLB_REMAP backend capability indicates that
>> VHOST_IOTLB_REMAP is supported and required.  Some devices do not
>> require it, because the userland address of each dma mapping is discarded
>> after being translated to a physical address.
>>
>> Here is a pseudo-code sequence for performing live update, based on
>> suspend + reset because resume is not yet available.  The vdpa device
>> descriptor, fd, remains open across the exec.
>>
>>   ioctl(fd, VHOST_VDPA_SUSPEND)
>>   ioctl(fd, VHOST_VDPA_SET_STATUS, 0)
>>   exec
> 
> Is there a userspace implementation as a reference?

I have working patches for qemu that use these ioctl's, but they depend on other 
qemu cpr patches that are a work in progress, and not posted yet.  I'm working on
that.

>>   ioctl(fd, VHOST_NEW_OWNER)
>>
>>   issue ioctls to re-create vrings
>>
>>   if VHOST_BACKEND_F_IOTLB_REMAP
>>       foreach dma mapping
>>           write(fd, {VHOST_IOTLB_REMAP, new_addr})
> 
> I think I need to understand the advantages of this approach. For
> example, why it is better than
> 
> ioctl(VHOST_RESET_OWNER)
> exec
> 
> ioctl(VHOST_SET_OWNER)
> 
> for each dma mapping
>      ioctl(VHOST_IOTLB_UPDATE)

That is slower.  VHOST_RESET_OWNER unbinds physical pages, and VHOST_IOTLB_UPDATE
rebinds them.  It costs multiple seconds for large memories, and is incurred during the
virtual machine's pause time during live update.  For comparison, the total pause time
for live update with vfio interfaces is ~100 millis.

However, the interaction with userland is so similar that the same code paths can be used.
In my qemu prototype, after cpr exec's new qemu:
  - vhost_vdpa_set_owner() calls VHOST_NEW_OWNER instead of VHOST_SET_OWNER
  - vhost_vdpa_dma_map() sets type VHOST_IOTLB_REMAP instead of VHOST_IOTLB_UPDATE

- Steve