Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) client-ip=2620:137:e000::3:6;
Message-ID: <389942ac-962f-4198-b87a-14bd6dd3e188@oracle.com>
Date:   Thu, 26 Oct 2023 16:13:51 -0400
User-Agent: Mozilla Thunderbird
Subject: Re: [RFC] vdpa/mlx5: preserve CVQ vringh index
Content-Language: en-US
To:     virtualization@lists.linux-foundation.org,
        linux-kernel@vger.kernel.org
Cc:     "Michael S. Tsirkin" <mst@redhat.com>,
        Jason Wang <jasowang@redhat.com>, Eli Cohen <elic@nvidia.com>,
        Si-Wei Liu <si-wei.liu@oracle.com>,
        Dragos Tatulea <dtatulea@nvidia.com>,
        Xuan Zhuo <xuanzhuo@linux.alibaba.com>
References: <1698351075-415989-1-git-send-email-steven.sistare@oracle.com>
From:   Steven Sistare <steven.sistare@oracle.com>
Organization: Oracle Corporation
In-Reply-To: <1698351075-415989-1-git-send-email-steven.sistare@oracle.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?dFViZS9DYkJLYnI3UTdzcVVsY2ZyY05DUmk1UmdVT3FobGdRRHpiMWZxNE5Q?=
 =?utf-8?B?dy9ZbmlyYW9MbHpkRk92MkVCMjJmQ0VWR3dqWTZZSjFqSXVIYUhERm94UGRS?=
 =?utf-8?B?UmxveWdkMytDY2FDT0Y1c2l4b3JlbFhxUWo5T0FRS1JTZ0dqK2xuanI1MkEy?=
 =?utf-8?B?TzVDMnk2eEVSL2RnaUY1TkdmNTlkSGU2Z0xNVEIwbk1TMXkxTmxIUGVzaGJy?=
 =?utf-8?B?UXh1cXp5U202V0t6cWpnSlZIWVNsQlFWdHpiOElvRTlGVjByVCt2YWYwSUxO?=
 =?utf-8?B?R3ZBNFRYMHBkUVZEMlpDaWlBbHZyNkQxSlFFaitYeVFVRDA3czNSZU9MMHdP?=
 =?utf-8?B?V0E2dWhJbFYzajJVSkVLUklDK2VMNUlsc2pCdkF1NHBtTHpFelNrQm1rNm9D?=
 =?utf-8?B?c3NLMXZxNzk3d21yWGttZkQ5KzI2bWhYNTVSNlc5WTZ1djVWS25GUGRDQm96?=
 =?utf-8?B?bnJXd0lKTHhBVW5mK3ZaVlU5dkVGcituOHJhc1E3cGgzZ0ZCa280YTZBTzFj?=
 =?utf-8?B?YzRVVC9QUmRZWWlJMTdzMnAyWFI3MWFBblY5WFFGSFR3YjdTM3lBUmlKejFJ?=
 =?utf-8?B?Nk00dG1HOWtMZDFULy9IREU1NnlkZW93WlkyTUdnbXJjNW15RllaNm9SYldT?=
 =?utf-8?B?N09ydFgrOGdTcDdQdTdWQTlJQnJBRlFkQ0xoU0lMaCtQTkgyNUlMWVNjM0Qy?=
 =?utf-8?B?bWdQbEc5dFZ2UTd3NFdvVGhhOWNzZm45ZXRaWEgvL2ZUZnd0dTZRa3FMcyt0?=
 =?utf-8?B?ak5meFZvQjdmUEw1NW50MlRoQzNhbGRUckZJTGRmNThPd1dEc2loRVBoZXg3?=
 =?utf-8?B?bWd0dThFdTd1cS9JM00xdUt0dllPanpiVzlNZkgwVnRsbS94Vk40OUx5Z3hw?=
 =?utf-8?B?TEJha0xueXRTNFF2OVdqZXBDanh5UG9QM0hyMk9GZjdjMUFzOE9VU3ZtTFZ5?=
 =?utf-8?B?SlVxVlh3di91enI1QXBQcExoQUowYlFWUnhyNzUxb1JFYjV2a2pkc0ZieWhD?=
 =?utf-8?B?Tm1pWndoRjR0c2w0S3lCQkwyb21nbzdnbE9XV2NsTTdtenNIbnFXTkxtbElU?=
 =?utf-8?B?dWtTTU82elowUkh3MjhraG45VTBUVDIyc25HeERkdjZ2WHZuRjNBVWJHVENv?=
 =?utf-8?B?WWR6ZWgyMkExTDJEZ2tyMG0xc0h0akd6Z1U5am53MzI5ZGF2MTEvYVk1Z2Zl?=
 =?utf-8?B?WVVlQmJtRXRIdkVvb2VXcklWVmtPcUl3bjFVNWlxVHZBcEJ0UG5hckdmYXhx?=
 =?utf-8?B?SmJML3A0UHFDTDhhZmFJbkd2eER3RmlwSzdwS3ppUGpGMEVUYTI0M1VVZS9k?=
 =?utf-8?B?YkJVSzAzMXROdS83ZklsaktPeEEvNzJJQzZ3OFlqdTZrWVBkQzQ0ZjkwaXA4?=
 =?utf-8?B?azF2MEhURUVMQWttQ2Vad3l4Ymw0TElqaHV1VUc3eVczbTMreld2d3MwUlE1?=
 =?utf-8?B?ekQ2RzBUWVpiSmR3aVpwOUR5V2hSdEdmZjJ1V29wamtqTDArb012S1ptckRF?=
 =?utf-8?B?bGRHMCtKejJrQW1CblFaTmpsb3NuT05pYmRjbWJ2VEdMMnFranZ0K2U0aUY1?=
 =?utf-8?B?Q3NOQlRtNzdqN1ozdFdkWXUwOXVnakFBUkZBSUhHZlZRUmlBYmlGd3I4NUoz?=
 =?utf-8?B?SXdFVk0rQ3RqbW1pZlYyVExqQ29WK3VoUS9vOUJmWGFHTVB2RkRkREMwLzdq?=
 =?utf-8?B?dDdONmhWYkwrMllkMHh1Q3VOSGhyNlc2bFFiSEQxQ1dFdk04eFpuZ1NLZVEz?=
 =?utf-8?B?OW9JcVRUem0yRlVVay9UN0dxVURaRkJqMjhVOHNJanJRRCtwRFh0ZkUzV3V4?=
 =?utf-8?B?YXQrZmNUbUQ3TUxBWGpwaWNVZ0ZqM05Rd3ZqRm54cEVGT2xnUnQ0MFk4ZHp4?=
 =?utf-8?B?UThuZWZaakRndlFuazhXRitiTWVmSi9PU0MxZFpJaGwxTDJJUGZuZkNiYVY0?=
 =?utf-8?B?NEZwenJHaXZpRnNoRnpOZVRTNjMvMkJkN1VEYzhxaXdGWHN2c2ZGNjdEL292?=
 =?utf-8?B?dWlLQWFvTnptQUY3MDRicWVSVnFJdHVyK1ViYmlFclNCNHJzMmN2Z0NCSXpF?=
 =?utf-8?B?bW9XbVE4elgxaGlDcTVSSmhRL2hYdFRXRTVNL0IvRXNYbUJyRWlyT1lSZXhV?=
 =?utf-8?B?dFdhYTRDTWNobWxUck1VNEk2K0cwUllCY214L0dMeXFhMHdNT1h3RXhaZUhL?=
 =?utf-8?B?SHc9PQ==?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: =?utf-8?B?ejVlS3dRV1hxbkdFTW5WZE4zdWh6c05kSXYvUkhWNk5IYWJSQVJRa1RpOWR1?=
 =?utf-8?B?M3QzVUJBVGUwbEZXdmNGN29DdEVZL3kzbG0vRjNEeWJWeURBb041M09BOWtL?=
 =?utf-8?B?dFdzOXhaNjg4dXhSVzhWZ1lROGNheng3aDFYNmlmYXd3VEZLTWxYTGpYZ09M?=
 =?utf-8?B?WStCV3o5NDVLVWdJaXdQNFU4NXVPaXZ6Qkl6V3l6Mk1OajlKSW0yRVp2Sm15?=
 =?utf-8?B?MHRQelFaRlk3SkV4YklMWEhydnkvaVpoR2hmV0VXLytmbjE3cGVaN24vQ05n?=
 =?utf-8?B?T2I1SlhnOVYvTE4wYk5tQXdRNUIwQm5rcldxVzRZdk11NlZtMWRjUHdWNTk0?=
 =?utf-8?B?UW8ra1RCSHZJcVd2VnlKNVZJS1MwYzl4NUNRcStFOGwwMVBYQXZJemV6eXU2?=
 =?utf-8?B?ellPY3JTNS9MTjU2dVp4c0hwRHJVYVIxZVhZclhoWXY1UUVzTHVuS2svVGVp?=
 =?utf-8?B?ekRxdW5PSDhTeHNWNnV6aXZWdmZ6aTk0TkU3K1NhNm9xU2UxQUxrS3hIaTUw?=
 =?utf-8?B?bnZ5RGp3YkFJVDhlUVIvaCtQUThPdm5qV0htMVBwZFV6TE1JMGZSN05ZQyt0?=
 =?utf-8?B?d2plZmFiYjJhVVZ4Z2JjRjd0TmI2cUkxaFBBbkd2clFZRVBtWHQ2cFIzZFY0?=
 =?utf-8?B?Z2hobTJBRGN2aGx2SFRzTWpVOFNoTTRlcVA0YjlQU3hpOGhRU1hFZDlLbEt1?=
 =?utf-8?B?aHRIWG9nRUhYVWVyTWxiYTBjVHJFWHBHYzZxRTBnSjFtRGxoK2hLbUx3Q0JC?=
 =?utf-8?B?MDVEOWxiMFlxazA0UzNoTU9MeHpZWkRKTkxha3FicWtuS3dyM05aOFdRVkVW?=
 =?utf-8?B?c2g2dyszL3lQdVd5UUtDVEJRVFB2bHVYUk55Z0c4YVlEVjFwZGVBK2YxWVpI?=
 =?utf-8?B?cmlNQ1hPZmwrcEJLUnRLNHVNNmw4MC9ZWUpxU0srWWlJWjg2UlduK3ByeGxo?=
 =?utf-8?B?SnorenJudGg3MmlhU3RxaG1VR0ZSYjNrRENNTHVlQ1dFS0EwVFlUM0twRzk4?=
 =?utf-8?B?U2s0bGxkZEZiZ3ZpL2FRY3MzNFdnVTNTUmxqUGV1eUZrcW9PZzk3WW50WU9L?=
 =?utf-8?B?RTk3aFIrQUxuVHlSQTZ3eFg0dU84WUkrNmt0TjY5SzVJT1NFY2JRQTBURXZB?=
 =?utf-8?B?UGFGeGtvVTBhYllDSkNlUE5qMWpFZVZLK3Fna0VzaFhkRWVIajJqUHpsYkNm?=
 =?utf-8?B?eGMvRGlHU28zUFRkMldHYkNQNWoreFp4Q2I5SGwxdXBhVm1CR2pYRml2R3ZL?=
 =?utf-8?Q?6iFJU5sz3CcWumc?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 5e6f3289-020d-4c4e-5b0f-08dbd660147e
X-MS-Exchange-CrossTenant-AuthSource: SA2PR10MB4684.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Oct 2023 20:13:55.3280
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 2it8zo3KOXQdiruZeipUsR7Envi617enVFiaDHT9lkfCV3lSamoPQBkxBHOrxzx6Fl+wb0VHnWvIeXXIeXeUHUWQ5v/HymBBs4Ok/arx8z8=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR10MB6016
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.987,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2023-10-26_19,2023-10-26_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 suspectscore=0 spamscore=0
 malwarescore=0 mlxscore=0 adultscore=0 mlxlogscore=999 bulkscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2310240000
 definitions=main-2310260175
X-Proofpoint-GUID: SPkEtEctOZL1EBZJvvH9RKP5qEjKyS9Z
X-Proofpoint-ORIG-GUID: SPkEtEctOZL1EBZJvvH9RKP5qEjKyS9Z
X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIM_SIGNED,DKIM_VALID,
	DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,
	SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no
	version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Thu, 26 Oct 2023 13:14:23 -0700 (PDT)

On 10/26/2023 4:11 PM, Steve Sistare wrote:
> mlx5_vdpa does not preserve userland's view of vring base for the control
> queue in the following sequence:
> 
> ioctl VHOST_SET_VRING_BASE
> ioctl VHOST_VDPA_SET_STATUS VIRTIO_CONFIG_S_DRIVER_OK
>   mlx5_vdpa_set_status()
>     setup_cvq_vring()
>       vringh_init_iotlb()
>         vringh_init_kern()
>           vrh->last_avail_idx = 0;
> ioctl VHOST_GET_VRING_BASE
> 
> To fix, restore the value of cvq->vring.last_avail_idx after calling
> vringh_init_iotlb.
> 
> Signed-off-by: Steve Sistare <steven.sistare@oracle.com>

This is a resend, I forgot to cc myself the first time.

I don't know if we expect vring_base to be preserved after reset, because the
uapi comments say nothing about it.  mlx5 *does* preserve base across reset
for the the data vq's, but perhaps that is an accident of the implementation.

I posted this patch to perhaps avoid future problems. The bug(?) bit me while
developing with an older version of qemu, and I can work around it in qemu
code.  Further, the latest version of qemu always enables svq for the cvq
and is not affected by this behavior AFAICT.

- Steve