Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) client-ip=23.128.96.35;
Message-ID: <7b86e89a-9eac-4711-840d-c677cc892cba@oracle.com>
Date:   Tue, 17 Oct 2023 22:21:34 -0700
User-Agent: Mozilla Thunderbird
Subject: Re: [RFC PATCH] vdpa_sim: implement .reset_map support
Content-Language: en-US
To:     Stefano Garzarella <sgarzare@redhat.com>
Cc:     jasowang@redhat.com, mst@redhat.com, eperezma@redhat.com,
        virtualization@lists.linux-foundation.org,
        linux-kernel@vger.kernel.org
References: <1697185420-27213-1-git-send-email-si-wei.liu@oracle.com>
 <4o4yuuezigilikolv2paxb7icrsm3gdnj5aeoe47uotzju7xve@s5vqaafrppdv>
 <2222edd1-4eb1-4b06-87ef-df2d4f591931@oracle.com>
 <kuiqvqcspmx2fo6sylivhzopxnsxlqxahicgbp3lrltewz2puj@bka6ceakl6x3>
From:   Si-Wei Liu <si-wei.liu@oracle.com>
Organization: Oracle Corporation
In-Reply-To: <kuiqvqcspmx2fo6sylivhzopxnsxlqxahicgbp3lrltewz2puj@bka6ceakl6x3>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?K2xmN0VUcmdFbmZwWHFTY0kwQWZhRXJrcG5KWU5yaFVpNmYwRDZrUFZOQWIw?=
 =?utf-8?B?Yk5nYk13QXlWUU1uRkUyNVRRWmVWeXlBTzJhaUxvUGdhTnpFcEZhSnh5cWJR?=
 =?utf-8?B?QmJsM25nb2ZXUFh0dllCNjBLQW1JQkxjM3Arek1ZRUZua2lYTGwxVDR2R3E2?=
 =?utf-8?B?S3psUjJXN3hBakk4bXZ3eEluZWtuVklqcHFFdjRjT0RoMFNXcGl4UjhiSTh3?=
 =?utf-8?B?alJTd2NlbHFKSE5PN3E0VTVqSkVhR2FPTEhsdTF0aHd3cHRFTWpTZDZ6U3Zl?=
 =?utf-8?B?ZmNaRDVOY2ZrN09tUmQ1Q1pLZStXZUJGem1FbVBUUWxwbjMwcVJUYStKY0Rx?=
 =?utf-8?B?M0o2WGZvaEEwMDVkRWZJYzhQUjNoeUl0ZjhGWG1YVVB5emNxeUYyWkNRREZi?=
 =?utf-8?B?by9jeCtjaW1iMEFqWm9EUWdYbmJ1dStIeWZyREtNMk5ySWQwN0NiZ0RDVGNT?=
 =?utf-8?B?akVobHlkQndrN2pPVTRYanNFOHoyeXZnRWhjOHRqTDY3Mm5keWw2TWxnditS?=
 =?utf-8?B?OVNXTktldnZORHQ3Yi81elN5eUl0em1OaUp3cXY2UnVxbmNoMVJ3emRtdkE4?=
 =?utf-8?B?WGRKR3gxWFkwWDEwM0V1cTJxYXovUm1rY1VoQ3poWkhkalFaVGZMSjd3dTFV?=
 =?utf-8?B?VE43VGNodWU0elVWUUdaRWRucmZPdXBvcmxjaVFja3hLZHhGNFZNeUlhTVZn?=
 =?utf-8?B?UXVRWVN6ZjE4ZHA5UDExdHF0aUhjbUFhd1kxdG9sNndDbTFWUXE4cGpEYSt1?=
 =?utf-8?B?MktiUVgvbElnRHZ6dTVLRkR4WW5jN20reGNKTFM0N1NuWkNtdkNaSmNLWTBS?=
 =?utf-8?B?S2tPQmViNWhPSVg1bTVRNjhuY1FIKzhwUytieDU5TEJ6R3BGR0NPQnN3VWk0?=
 =?utf-8?B?MWphYTRDaVc0MlhrQ1F0QUgxV1pxM2hnNmVaNkY2d0VwUmRTL2lQVEpmeXFp?=
 =?utf-8?B?UUtaZWdiWVhQVzJ6dWkxS1d3QkRvTSt5RkN1ZWdrQjl3bWZRT3FlQ2VHZ09N?=
 =?utf-8?B?YVA3RnNINWpWbWd1REhEdGFVRURUNFNjVUZpOVVRbi9IR21QK2pOdFpMRmE0?=
 =?utf-8?B?NHpFdEZSVGlIRVdLeHpqOXNJM1U2aDZWK3NwTUhWRTJWVDVBNmdxZlJ1eHZm?=
 =?utf-8?B?RTFQOUg2NHNKZUx5TCtMSEZsSUw4NXVyWjNmWGZLTWI4ZE9BalFDRUlRZGNa?=
 =?utf-8?B?c0FFUmttSEhaMFY2MkJoTEFqQjdFVitCZzJpZ2VNbXBjQUlvVFg3eUlRcXk0?=
 =?utf-8?B?dFg1eUN0VmZRWG81T1JEK3p4MTZQaVBJOXBQbnNaUmdlQmJhTXBNc3BuVGl5?=
 =?utf-8?B?Y1ZrQnV4SEQ0d0ZwZzFTMEFZK2dBN0toWS9RcWN0WlNodFdnL0RCQVdoc083?=
 =?utf-8?B?UnhNYUZQd1Z1eXlUa0NvejZDdmlvaTdvOU1XNHkzYnRtbzZNZVdUcHFIODZL?=
 =?utf-8?B?RWVxVW0rNHpqSzhwcHJROEtWUnJxZVpISGpIUGtwUForY24ra09QelhUbFFo?=
 =?utf-8?B?VzBnRVI1eEREdVFxY1FoZjh2SHFBbXdtbWxGY1poUVlRMklKL21zb2pLQ3Iy?=
 =?utf-8?B?K1pGUmxtNjdhT21nZTk2RnIzSkFHN25nNlFhK0p5LzJaTjRBRk95dk5GM2Ux?=
 =?utf-8?B?ZG1PTFpRTTU0eGRHYWJpWEJTMnAwTWNBUjh6QkM2SDJUZ3E3OE5BTktySU85?=
 =?utf-8?B?aFZzTTdWWTdVV0R4V1RkMWZyUkRLZWNXMVFLcndMV3I1RUgvcG1xSjJSZG5C?=
 =?utf-8?B?OXBBdUZwWXUxR0UyRkQ3d1o3TVVHVk82M3hVaUZJL2F2OVFFTjdNc2ZMWjFw?=
 =?utf-8?B?aEZDT1YrL0hXUkpQM3FoS3BkN0oydWgvODJSMmRFZms0cmFKQXhZNzk2dVNO?=
 =?utf-8?B?ZXVPQkkzVHU3QUE2SWVBd1NpQU1HQTh0S3BMZ011K1htcFJaSkd0UWpZU0h3?=
 =?utf-8?B?Sm4xamVTcXFacGxXcU9wTUdnN1V1MFhoV2E2VHpCTEtQK2FuSnNBMnJ3RS9k?=
 =?utf-8?B?d0hBUkdWRWJoa3BndFp6SDVNY3VJVi91a2Fha0hLanJrYVhKa3VWSTBaeXZO?=
 =?utf-8?B?SzJSSkVHVEU3bWRZS2dYTDZVMitEdTRKUG1iaFc0d2M0L2pyYXpaQmczVXlB?=
 =?utf-8?Q?0aQSeZGyd1Qnzz00WaphDXbLC?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 7DlQBHPChiw1YR4bReo1v1KGMh7bLrZ766NeM8jTFIhoNCwx4O599nVNuWTCWu9JTrccE9FewYx9xjgOhj8mr8JiX6WkKkAhaz8pYXMnqTofSjuvmaNkmh/qtaC12Epz0g09nBadORzBWJh476/DW5LusrkwxIBJgSsldSleNONrC/ZEuXgcB6i/36JcvgTUaCoJkCI1xvGz7BXGZCEcfvEnxjwzWAJfxmliTPLyARkpV1NHFTYKvIcmSURYqVAa4RvVMj1I5QmAubhsD0sBpakdLdH4uNI/h8nCyCwIVx1ZDy8hKvWDkJQrg+41N00zGrSFg168gsERzLpR4VHFH1RtDDY1aaqD8+30a23iPfpMVvXFYgB+mMpkHJIqLfrAFfo9nEZqZSQFRE/lWP/yIng8B5qSARBxrA7E1GQUXj8wf5u9e1nvLfIIo4rxOuvFpwNP9iR05NJYBJoPf9TJcxa0n8pIBBRv5Ymc5iNvSPzC54NCcMJTxfsQH1nr3Mic/BjNuPldKa3hjVS1MiePovvpQDXqILUGkZO2TzNXH3wrlJVgnpG6n7OgNyY/olMRba9VkpJ95AZ68+EcjWGYBYnxKOKZi2kS52JC/gLof5VGOQ5pS5BGP4N7VcBasUMJgQX2uJB37zSzKpIw2g7hib30/Aomq6Xq/XwBL9mSpzVydUgE/G57iNX49qO36M9K4vC98rTcaitR1wo14dibiRe2fLO8yljddgHGnEtSgVFCfdUAe+wCcIkAuZKZI+HSP70m+1jLG5aY2abD7W6Jl4YC7Qv6kTCtXfzkU0kqygb9MV1DEkZS16emewa7mAJEQi7r9CxowBWdhlfvA5gaLwn2swKu7053KHI2H7bOuXI=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 9056179c-fb92-4f82-c887-08dbcf9a1b43
X-MS-Exchange-CrossTenant-AuthSource: MW4PR10MB6535.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Oct 2023 05:21:39.4167
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: SoPh0KUApzHUWNU3/tPAw0LiF4vhDa6xTriMTVUAd3dlBCk+3vbdgC3jm3LlA/EdzRKey7q3c0pQ/cPJ6ivEdQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR10MB7161
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.980,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2023-10-18_03,2023-10-17_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 spamscore=0 malwarescore=0
 adultscore=0 suspectscore=0 mlxscore=0 mlxlogscore=999 bulkscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2309180000
 definitions=main-2310180045
X-Proofpoint-ORIG-GUID: y3YemqkcoXtSqr7WaYQZQA82JJvHO7z9
X-Proofpoint-GUID: y3YemqkcoXtSqr7WaYQZQA82JJvHO7z9
X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIM_SIGNED,DKIM_VALID,
	DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,
	SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no
	version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Tue, 17 Oct 2023 22:22:00 -0700 (PDT)

Hi Stefano,

On 10/17/2023 6:44 AM, Stefano Garzarella wrote:
> On Fri, Oct 13, 2023 at 10:29:26AM -0700, Si-Wei Liu wrote:
>> Hi Stefano,
>>
>> On 10/13/2023 2:22 AM, Stefano Garzarella wrote:
>>> Hi Si-Wei,
>>>
>>> On Fri, Oct 13, 2023 at 01:23:40AM -0700, Si-Wei Liu wrote:
>>>> RFC only. Not tested on vdpa-sim-blk with user virtual address.
>>>
>>> I can test it, but what I should stress?
>> Great, thank you! As you see, my patch moved vhost_iotlb_reset out of 
>> vdpasim_reset for the sake of decoupling mapping from vdpa device 
>> reset. For hardware devices this decoupling makes sense as platform 
>> IOMMU already did it. But I'm not sure if there's something in the 
>> software device (esp. with vdpa-blk and the userspace library stack) 
>> that may have to rely on the current .reset behavior that clears the 
>> vhost_iotlb. So perhaps you can try to exercise every possible case 
>> involving blk device reset, and see if anything (related to mapping) 
>> breaks?
>
> I just tried these steps without using a VM and the host kernel hangs
> after adding the device:
>
> [root@f38-vm-build ~]# modprobe virtio-vdpa
> [root@f38-vm-build ~]# modprobe vdpa-sim-blk
> [root@f38-vm-build ~]# vdpa dev add mgmtdev vdpasim_blk name blk0
> [   35.284575][  T563] virtio_blk virtio6: 1/0/0 default/read/poll queues
> [   35.286372][  T563] virtio_blk virtio6: [vdb] 262144 512-byte 
> logical blocks (134 MB/128 MiB)
> [   35.295271][  T564] vringh:
>
> Reverting this patch (so building "vdpa/mlx5: implement .reset_map 
> driver op") worked here.
I'm sorry, the previous RFC patch was incomplete - please see the v2 I 
just posted. Tested both use_va and !use_va on vdpa-sim-blk, and raw 
disk copy to the vdpa block simulator using dd seems fine. Just let me 
know how it goes on your side this time.

Thanks,
-Siwei

>
>>
>>>
>>>> Works fine with vdpa-sim-net which uses physical address to map.
>>>
>>> Can you share your tests? so I'll try to do the same with blk.
>> Basically everything involving virtio device reset in the guest, 
>> e.g.  reboot the VM, remove/unbind then reprobe/bind the virtio-net 
>> module/driver, then see if device I/O (which needs mapping properly) 
>> is still flowing as expected. And then everything else that could 
>> trigger QEMU's vhost_dev_start/stop paths ending up as passive 
>> vhos-vdpa backend reset, for e.g. link status change, 
>> suspend/hibernate, SVQ switch and live migration. I am not sure if 
>> vdpa-blk supports live migration through SVQ or not, if not you don't 
>> need to worry about.
>>
>>>
>>>>
>>>> This patch is based on top of [1].
>>>>
>>>> [1] 
>>>> https://lore.kernel.org/virtualization/1696928580-7520-1-git-send-email-si-wei.liu@oracle.com/
>>>
>>> The series does not apply well on master or vhost tree.
>>> Where should I apply it?
>> Sent the link through another email offline.
>
> Received thanks!
>
> Stefano
>