Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) client-ip=2620:137:e000::3:8;
Message-ID: <6ccc182f-5b1f-43ee-aa61-db0bc9a3d1ab@oracle.com>
Date:   Thu, 26 Oct 2023 13:33:00 -0700
User-Agent: Mozilla Thunderbird
Subject: Re: [RFC] vdpa/mlx5: preserve CVQ vringh index
Content-Language: en-US
To:     Steven Sistare <steven.sistare@oracle.com>,
        virtualization@lists.linux-foundation.org,
        linux-kernel@vger.kernel.org
Cc:     "Michael S. Tsirkin" <mst@redhat.com>,
        Jason Wang <jasowang@redhat.com>, Eli Cohen <elic@nvidia.com>,
        Dragos Tatulea <dtatulea@nvidia.com>,
        Xuan Zhuo <xuanzhuo@linux.alibaba.com>,
        Eugenio Perez Martin <eperezma@redhat.com>
References: <1698351075-415989-1-git-send-email-steven.sistare@oracle.com>
 <389942ac-962f-4198-b87a-14bd6dd3e188@oracle.com>
From:   Si-Wei Liu <si-wei.liu@oracle.com>
Organization: Oracle Corporation
In-Reply-To: <389942ac-962f-4198-b87a-14bd6dd3e188@oracle.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?SGlCbTlJdzJMbUFHMkZtaGJkR3JHZU5QajdKckpabEtheFlsL09ob0lnOWRw?=
 =?utf-8?B?TnJ3UXdFWkdzdEoyRzBTTHA4YjZLWmF0azJnRzNIRERYTXVOTGE5RkYvY1lB?=
 =?utf-8?B?b2ZjdU83T1JIYTNuUjdUZzlsemk3TUpFUWM1WE1GTmwzR3dwb0QyQlY2dWl1?=
 =?utf-8?B?L1RQK0dmeXlSeTJVVkxGWVZEWDM4RiszZXptUVl0QUNZNUhIKzA5SGZqTGRR?=
 =?utf-8?B?NWRGN04yQ1dqQUlHd2Nsd1VLRFppSVFrdkpkTEtWRGNYQ1gzaldySm00WnN2?=
 =?utf-8?B?d3pGRnpYckl3SFJHOE90cTRjQS9TT3F2TmFlTWZwWUNBbWZENktrN1A5T00z?=
 =?utf-8?B?WVJPTnM1UkxFRHFjU1lqbGxaNHg4bTRmRDlxWTdSSldVWjVZNC8xalFsakxJ?=
 =?utf-8?B?dVZjSVNxZXV1aVVrcUlFSGNWMWlBZDlBcWZxZG5tbXAyWHZJczJRdEx6cGtv?=
 =?utf-8?B?TEl1YlZvd1RBWVRNRkNjR2Y0TjBLdXlrZnUwVGM2MmxjQzVDKzdRVWxmVnBF?=
 =?utf-8?B?TUNIaUY2bGRiMkR0alcyVnJHR2hvRzVjUGx0YVd2ZmROcmVFcnJ5bG1ZYkhS?=
 =?utf-8?B?TVNlUXY5Mm5EMVgrNGl6LzJ2dVVZWWlTanY3YTZpaUlyK0N6NHpERjJnVHNT?=
 =?utf-8?B?a2tteVZ5RTFpaXdrM2g0VldsV0tVOUFBaENEU21SeEhMRTFMSWtsNllmVm9j?=
 =?utf-8?B?UlB4c0hhWlVEQ3ppRFVlNXZtckM5bDdFSDNzbi96SGU4bnhuWmdwUzFxMEdS?=
 =?utf-8?B?R2w1QmVZRUFUWXlBdjVDWHBheERlc0RuRE5LR1hnNWRUd0lLMjdmMFEwbUlh?=
 =?utf-8?B?Nzd6cTc5amZpUkFPVjhXWFdQSTB4L3NHak55YmZobGFEU1ZjdTBEZzk3SWRT?=
 =?utf-8?B?NlBHYks4MFE3b2poUWlyL0dlNkhBZUdpdVBNWkNqbDBtbUNTdklKUTJaVzhn?=
 =?utf-8?B?cmxPZXF0YmRmbWlBNU4wRjZ5alNKRlVsSXJLRmswZjd5cWtFckVTLyt5V1FL?=
 =?utf-8?B?SjY4VG9BYWxHTFcxNFhxRnQwTWNyR2tOYytTaXUwMXA1aEQ0Z0RvQTJYSjlz?=
 =?utf-8?B?ellacU91aWlPaWE3WjJZS1dsSDY0TExJZnkvNngrK3VmUFVIdUF6SDdDNjhY?=
 =?utf-8?B?c1cvT1VIM2FIbkcrQnVrUVptUll4TE5FMjdneDZQVVR4eXFjOGZzLzBFVTl0?=
 =?utf-8?B?Y1JmaDFtZ3cwK29jckhpdE0wSktZb1lVZG9uak96TFkySHdQZjNyNjg3Nmlv?=
 =?utf-8?B?cjdMSzJJalhHUnRZV0M4MHhVWWtGZ2xUUmdTazRPeldLWWV2OStQcHRHUFlW?=
 =?utf-8?B?MVI5SVdQbmdhUUpudXhGb3R5dEhZTk9CNC9LV0gwMGRkZnB5S0NYQy9wcGtR?=
 =?utf-8?B?K0E3akVXQ1JYVzR6UGNseFlCd0Y3YXFiNGpGRVJZMUk2bWVSTEppQWMxWTJI?=
 =?utf-8?B?eGExR29TRHJad0paVlFKVjJsNEtkblA3UnVIck51TE1PMkQ1S0xqMmJYa1F3?=
 =?utf-8?B?T0dXVnJtNXhYLzEvWklvejlyQmwxWmpSZUJjVzJLcmVZQVQwblI2L2JyVXcx?=
 =?utf-8?B?UzZjVHlGZG9VSHkxUnZKcEdNL0gzSEIzSi9hbmlGdlQ0alpJVjBrTzFWTElp?=
 =?utf-8?B?cVFaMjYvUWlTU0IxeEpKYVRzb2N0MnpVak92TDdPZTFyeDErRUlrU2ZJbWhu?=
 =?utf-8?B?Y3dVQzgxdmlBSlZKazlRVG9KU0lEU1QyVGxLUVlBZi9FRGVrQUFQNDd6Q0Zy?=
 =?utf-8?B?Q1RaczdTcjAvZnBhYXRsTktNcDhJSEZKd2xGWUxjNHJNc213a1lmdWR1MEtk?=
 =?utf-8?B?eDZoMkxnam1XLzJIL0tVblJ3azIrQnRLQlpBOXlqeExFc0dhMHVvdktjWEpL?=
 =?utf-8?B?c1FBc2tROVl5R1RPSE5iZmk4Q0t2dGt1ekppdEJYRStyWTVFUnNpVTczZytC?=
 =?utf-8?B?dDdLRkxwcEs3RmljWkcwL3czL3d0VFdYNHFUcWxaZWliaFViYXF2Yi9yb1dj?=
 =?utf-8?B?TGNULzJtazYwTVMvK1FMVjhuZERLb3AxeEorYUphUWJxakpEVkpQUTMxQ0VS?=
 =?utf-8?B?OXNUM2JZeEFBQzV5cUR1WloxMUwzNVZnenY1V2JxVEpIMkU4NFV6T28rN3ZC?=
 =?utf-8?Q?zI4+dqf0Hw8VQIreV7IB7G7xM?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: =?utf-8?B?OHNDWTM4QzJlN0ZONVlTUnh2TFRSMTFTeTFjdGk3ZXdIQ0xNUGtSQnJrdDNv?=
 =?utf-8?B?SGFiK2dmYXN5QXlnZ2NNYUM0dVNQZTBNdm1HL0pzZ1ZHNXBaWDlkMUo5U08x?=
 =?utf-8?B?WEhPY2JHbnA5aEFjY2drdXR3N1J2SmxjTXVCNHUvTEFMTVY0MVdiTWxiRU9p?=
 =?utf-8?B?RzRacmVPQU5uUVd4UFlxeHF6ODA5Wnl0MThFbyt3TDJmOHFvbWY3LzFUMWE3?=
 =?utf-8?B?MldrUG9XUXk3VlZFNmxoYlJiRmlPcFdZaDBwblA0NlkrMUE4WUcyUmljWXNP?=
 =?utf-8?B?eklFL1gwT3NIVm93VVpIRjM0KyswRnhqUEo3cUxBS09LMVl0SlRGQ1pFUlNH?=
 =?utf-8?B?dXdnU3crZGFuNU1ZZVl1MklsazRrZC8xTUUzRTMraGhldzg5dmpRNCtSNW80?=
 =?utf-8?B?Zi9NbU1tdlBKWVlzeS8xeitrTU1FMXdRKzZBdnFIYThjemRDVTE0bm5Ed2pU?=
 =?utf-8?B?ekJUUEs1SVl6L3U3cFVmNUtqd3pUcXFReHhUSGFGMkR0VDBFczZBNDJsNm04?=
 =?utf-8?B?Qk5hVVFDZjFJcDBCb0FYS2ZCZThhVmdva3RydEgreHdhTFl4QSt5bEtDb1VR?=
 =?utf-8?B?NUNDRCtqRUtPRjBrR0RGNUYxWTZlQlZwUkhjdyt4NGdtZlFNZllhRnU1RWZm?=
 =?utf-8?B?WDkwc2ZPZHE5eGJlSzlUUkV4QW9yZmFZbFh0SmFFK01vVHdtL3FxVGt0eDhH?=
 =?utf-8?B?TlpCVGpoWWhMelNkcG9aUys1Tm9HZXJDZDZaSFl5L2Q0cXU4RkpCc0NUbElk?=
 =?utf-8?B?cHQ0a1pPUnh6Rk9zSENRSlNlNlJBL3NKTXo1aG9tSm9TRlJ4UnNjN28vT2xC?=
 =?utf-8?B?elczWnpreXdCWnpUTVR3THZQMGZsbm1Udi9ydEl1aG5nM0E2K0M1RVZSTGxI?=
 =?utf-8?B?aUlYMjhsN3YrYUhOQnkyM1hra0xBT1RiUnVCcytOaWRUL2ZGbEI4S3BxRmdJ?=
 =?utf-8?B?YUpoc2QvVFJqRVpabk5kaXFpOVA5bkFrVWNjVVVQVEJiYkVET3c3Yk1nZCt3?=
 =?utf-8?B?em1PTFcxQjRqTW9VMkFXSDR3OXBjRWlqZVhOQWFLbzBXd1FnbnlyWVhEc0I0?=
 =?utf-8?B?RFZFbWZLUW5KYVUrOGVSOXdaZTB2c3UrMU9yMDU1dzZyUmNrYWNPZWszV0g5?=
 =?utf-8?B?V3pMVFlCTTNRMXQ4ZGFyR3dhVmFLL2ZZVzJyZkNKU1EwM1JWTWU0UHRQR2Uv?=
 =?utf-8?B?S09Ib0ZkZzFsdnJQZ1lTWmo3ZlgwTVZCdVFMVFc3TUM3L3I0RXVNUFFWWWc5?=
 =?utf-8?Q?NF8IkkuzOJCATGp?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: fe533b16-c7dc-4c29-1a20-08dbd662c3dc
X-MS-Exchange-CrossTenant-AuthSource: MW4PR10MB6535.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Oct 2023 20:33:08.5191
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: IWvyLfdpEl/z4Zap0gxprI9wEc3FkjHoaf/oW5GTBDPeah0to6LYa9Yn+aH3/Ypb9DdS+f5qh01mrmwM/8JMhg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR10MB5951
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.987,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2023-10-26_19,2023-10-26_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 mlxlogscore=999 mlxscore=0
 spamscore=0 suspectscore=0 adultscore=0 bulkscore=0 malwarescore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2310240000
 definitions=main-2310260178
X-Proofpoint-ORIG-GUID: HeITk5H6VYNdePh_hzL1F4Rv9b37sndd
X-Proofpoint-GUID: HeITk5H6VYNdePh_hzL1F4Rv9b37sndd
X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIM_SIGNED,DKIM_VALID,
	DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,
	SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no
	version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Thu, 26 Oct 2023 13:33:50 -0700 (PDT)


Steve, I think this is a loose end that I myself am not sure if worth 
fixing, copy Eugenio for his awareness. Reason is that when CVQ is in 
place it always has to cope with device state saving and restoration 
using shadowed virtqueue for a lot of cases not just migration, and 
that's the reason why SVQ is always enabled for CVQ in the latest QEMU. 
But I agree this is a nice to have, possibly there could be value to 
support vDPA VM instances without solely depending on SVQ for e.g. for 
use case like memory encrypted VM. Thanks for posting the fix and lets 
see what other people think about it.

-Siwei

On 10/26/2023 1:13 PM, Steven Sistare wrote:
> On 10/26/2023 4:11 PM, Steve Sistare wrote:
>> mlx5_vdpa does not preserve userland's view of vring base for the control
>> queue in the following sequence:
>>
>> ioctl VHOST_SET_VRING_BASE
>> ioctl VHOST_VDPA_SET_STATUS VIRTIO_CONFIG_S_DRIVER_OK
>>    mlx5_vdpa_set_status()
>>      setup_cvq_vring()
>>        vringh_init_iotlb()
>>          vringh_init_kern()
>>            vrh->last_avail_idx = 0;
>> ioctl VHOST_GET_VRING_BASE
>>
>> To fix, restore the value of cvq->vring.last_avail_idx after calling
>> vringh_init_iotlb.
>>
>> Signed-off-by: Steve Sistare <steven.sistare@oracle.com>
> This is a resend, I forgot to cc myself the first time.
>
> I don't know if we expect vring_base to be preserved after reset, because the
> uapi comments say nothing about it.  mlx5 *does* preserve base across reset
> for the the data vq's, but perhaps that is an accident of the implementation.
>
> I posted this patch to perhaps avoid future problems. The bug(?) bit me while
> developing with an older version of qemu, and I can work around it in qemu
> code.  Further, the latest version of qemu always enables svq for the cvq
> and is not affected by this behavior AFAICT.
>
> - Steve