Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <8338fe24-04ab-130a-1324-ab8f8e14816d@oracle.com>
Date:   Mon, 14 Feb 2022 13:34:15 +0000
Subject: Re: [RFC v2 0/4] vfio/hisilicon: add acc live migration driver
Content-Language: en-US
To:     Jason Gunthorpe <jgg@nvidia.com>
Cc:     Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linux-crypto@vger.kernel.org" <linux-crypto@vger.kernel.org>,
        "alex.williamson@redhat.com" <alex.williamson@redhat.com>,
        "mgurtovoy@nvidia.com" <mgurtovoy@nvidia.com>,
        Linuxarm <linuxarm@huawei.com>,
        liulongfang <liulongfang@huawei.com>,
        "Zengtao (B)" <prime.zeng@hisilicon.com>,
        yuzenghui <yuzenghui@huawei.com>,
        Jonathan Cameron <jonathan.cameron@huawei.com>,
        "Wangzhou (B)" <wangzhou1@hisilicon.com>
References: <20220202153945.GT1786498@nvidia.com>
 <c8a0731c589e49068a78afcc73d66bfa@huawei.com>
 <20220202170329.GV1786498@nvidia.com>
 <c6c3007e-3a2e-a376-67a0-b28801bf93aa@oracle.com>
 <20220203151856.GD1786498@nvidia.com>
 <e67654ce-5646-8fe7-1230-00bd7ee66ff3@oracle.com>
 <20220204230750.GR1786498@nvidia.com>
 <0cd1d33a-39b1-9104-a4f8-ff2338699a41@oracle.com>
 <20220211174933.GQ4160@nvidia.com>
 <34ac016b-000f-d6d6-acf1-f39ca1ca9c54@oracle.com>
 <20220212000117.GS4160@nvidia.com>
From:   Joao Martins <joao.m.martins@oracle.com>
In-Reply-To: <20220212000117.GS4160@nvidia.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?UGplN0I5aml1REorTGlQekhBUENYRS8rNmtKQVltWFBrTTcxUVJWek9TdXh5?=
 =?utf-8?B?eS83SlpNYVlTdWpsSFZFeloxREltZ3Fob3RtYS9TQXlTUldiNzU2RFNqUDkz?=
 =?utf-8?B?QzRCaEV0TkZIOFB1Q1B3N2o0Z3ppN2E1dVFUNDBjeXpLeUtlSHg0MWJ2a21y?=
 =?utf-8?B?aDVxYiswSko1djFxRUFmOVhuLzdQajZESjFGUkttMFgvSHVwUXM0U0kyNkNm?=
 =?utf-8?B?ZlBkN2dnMWVORmJWZFZsSVZDRGw2QkxDcVdYbkJlOHdmT3Z3UU02NHl4d3BX?=
 =?utf-8?B?ZE9TZ1ZlZTgwQitteHVLaEFQU3A0V1prcHZoUGdZc0NjOTdCM2lYTDRzTjZH?=
 =?utf-8?B?d0hEZzBWbDhRSlQzYjFPQUsrOGllN2ZUSFczUnpTWHpNdXJPb0JZVzJaNHRN?=
 =?utf-8?B?ZkxBalp5OFJYQjZjZEhFY0tBM09CVk1vM0JFQzEycGdRK29mZ1hFOUFtU3k3?=
 =?utf-8?B?ZnM5bDRIWno1cWhDcm1wN2swU2pqeFJmSjFZOTIvNXlXMG1sTHlaZVVLbjhC?=
 =?utf-8?B?Qmoyb0IxVjZQZk1WK0tzNWJZWVYvU3RtRTl0UlRVVVNIUDFUYzd4ck4yNGpG?=
 =?utf-8?B?dFpvRnpXek9Qa1g5em1IZkdNNHFkVE5iOUxaeEsxZmN5OUk4ZnFzSm9Ibkxt?=
 =?utf-8?B?RHd3M0pKVTZVS2pXWHh6MlpKR2Jid3NuaHVFOFpKU3JvYXY2ZkFxd3NFMkVT?=
 =?utf-8?B?VTFubmtLamREWjNqYW1ObXQ1YmZkbzRTNUJ3cTUrOEJjTjloMnlxWUxMZXVW?=
 =?utf-8?B?M1ZNRCsvTXhWSExraXp6Q3F2T0FFZGRCWXlvNTUvbGNWQkxjNUljeVBSTWpN?=
 =?utf-8?B?NmU0UHFaejAyWWZtREZKV1dIU2EwTU5sWXk3cWF4NEZ3RjZnNnozWTFyanZx?=
 =?utf-8?B?SXNMTzFvWit5UWdTb0F1TnV2TjNRUWRDcTI3eGdQUlBGZzRMMDcyR2FXdnlX?=
 =?utf-8?B?VmhsL2ZWTDQ4ZFdMbTRJTUlvSHVqMHU2OHJJS3lINW5xVWRYV2tEMnkwSDF1?=
 =?utf-8?B?N1RCcEw0SmVhTjNleVBndzdCTlNLa091UkxhU2h2N0N1OVdmaktQS1l1QlJ1?=
 =?utf-8?B?aklDVUNTa0FMeEs4NlhWYVJCR1cwbFBJbmNNR1dNWjEwKyszdDcxcUU4emUw?=
 =?utf-8?B?L21GdXdIY1oranZ3eTJaOTJjWTJvZ2VMQWFZcmVZd1pBbnFWTkhFckNJTHV6?=
 =?utf-8?B?dXJMbmNSb2JaazRDRW54U2hvcXBwemxDbkovb2R1KzdpQ1VURnBrSFc5NnVJ?=
 =?utf-8?B?N3BQUDhONktOMm9pYzN4bWk0bllOVFQ0dlc0djVoQ1JDMmJsdkluUzRQbkdJ?=
 =?utf-8?B?VVVvN0lqM0htTDR1UjFvbS9nZnVwdUh1blU4aVArZWs0cktxMGl1WnlKb3RG?=
 =?utf-8?B?aWcrL1oyNVFkcU16V2loRHdHQ0labk1CV1BBbU45ZmJoWGJoSnlidlh2eFJR?=
 =?utf-8?B?bVZ4RXhaOGJwcGV5amlRKzJzUlNDYUVlekRob0h4SThPUXdxUHllTXFSWWxq?=
 =?utf-8?B?YWdYa09nb2Q0LytPUUFzbWxocWNTeUlvbXFHQVl4RExUWHU2V1NwbUxXTW5M?=
 =?utf-8?B?QVI1SnlMaTNzYnNJOE00SmRCMy9ST2grSDhHREVZWWdKOVlQT3B5ZG1PSU9O?=
 =?utf-8?B?b0xZZlc0a2czQ1lHUzF1aGh5VFJ6eERwMTZ0UGhrTWlWdDBWRFdIanFFaG05?=
 =?utf-8?B?VVNocWJFcHpQYkxZNGNQOUhWNzhwcm9ZRUIyV2s0cXloOUVQcHhxR3dzRldL?=
 =?utf-8?B?QUJMMzhoUDcyMEhtQTFidG5nb1U3bm10VjVZMXd3dzNrbThsSVRXM0tReWhC?=
 =?utf-8?B?bUdPWHRVdTk2dGRjcFZydFlIbmFQU3dzSnViTFJoUFdXaDA3MExoN1NQUmpy?=
 =?utf-8?B?ZUNWbjMrS2ZuTnFyaCt0dVVraTFVdmdOOUdaQ3RFRnVMdjJHcUc0RFpUdjR0?=
 =?utf-8?B?WllNdS80elRaSnpDUWRNeHFySm1QaitUSlU0NG1LdDlJRnBFRnlPUmVjWXY5?=
 =?utf-8?B?RGlaUHlpYWo5dGJ2bHFkREU1T2J0dVNDcXlBeit2WmdOdmltQ0U5QWhGWGRz?=
 =?utf-8?B?SkhGYTUwVkdFelByWXBsTUVtbGVUcU5KeGY0WmFJUTJiSUF6QXFBTnp3L2xB?=
 =?utf-8?B?bnlNMnFCNlRIQ2ZPUmNkMlRWQ092WUtyZ1pIOGI3TGRocEJRSjAwQ0ozV3JI?=
 =?utf-8?B?OVdhKy9qMXVLT2EyRkJiVjZjYnhjckR6VE84Y3pmNnA5aUtQZE9sVGw2bE9q?=
 =?utf-8?B?OUtwclYreTdUNVZzR0NTTy9rS2h3PT0=?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 6cfaca5d-e744-447e-83ed-08d9efbeb6b0
X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB4835.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Feb 2022 13:34:23.9513
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: NNoPNzpMrGQaJjYkmDwG6KNlSFLYeGqGACB+qCZux7f2YERpMMmr4waymo2GOzrc+08tBvFr0GBb7hwPjhNOJdgN7Ae9EVaKCB1To76CklM=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLAPR10MB4898
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10257 signatures=673431
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 phishscore=0 bulkscore=0
 malwarescore=0 adultscore=0 spamscore=0 mlxlogscore=881 mlxscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2201110000
 definitions=main-2202140082
X-Proofpoint-GUID: cISn_QsgFajWMF-SiT8vh_TrKqNUJAHy
X-Proofpoint-ORIG-GUID: cISn_QsgFajWMF-SiT8vh_TrKqNUJAHy
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H5,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 2/12/22 00:01, Jason Gunthorpe wrote:
> On Fri, Feb 11, 2022 at 09:43:56PM +0000, Joao Martins wrote:
>> 1. Decodes the read and write intent from the memory access.
>> 2. If P=0 in the page descriptor, fail the access.
>> 3. Compare the A & D bits in the descriptor with the read and write intent in the request.
>> 4. If the A or D bits need to be updated in the descriptor:
> 
> Ah, so the dirty update is actually atomic on the first write before
> any DMA happens - and I suppose all of this happens when the entry is
> first loaded into the IOTLB.
> 
Intel's equivalent feature suggests me that this works the same way there.

ARM update of ioptes looks to be slightly different[*] but this FEAT_BBM
in SMMUv3.2 makes it work in similar way to Intel/AMD. But I could be
misunderstanding something there.

[*] apparently we need to write an invalid entry first, invalidate the {IO}TLB
and then write the new valid entry. Not sure I understood correctly that this
is the 'break-before-make' thingie.

> So the flush is to allow the IOTLB to see the cleared D bit..
> 
Right.

>>> split/collapse seems kind of orthogonal to me it doesn't really
>>> connect to dirty tracking other than being mostly useful during dirty
>>> tracking.
>>>
>>> And I wonder how hard split is when trying to atomically preserve any
>>> dirty bit..
>>>
>> Would would it be hard? The D bit is supposed to be replicated when you
>> split to smaller page size.
> 
> I guess it depends on how the 'acquire' is done, as the CPU has to
> atomically replace a large entry with a pointer to a small entry,
> flush the IOTLB then 'acquire' the dirty bit. If the dirty bit is set
> in the old entry then it has to sprinkle it into the new entries with
> atomics.
> 
ISTR some mention to what we are chatting here in the IOMMU SDM:

When a non-default page size is used , software must OR the Dirty bits in
all of the replicated host PTEs used to map the page. The IOMMU does not
guarantee the Dirty bits are set in all of the replicated PTEs. Any portion
of the page may have been written even if the Dirty bit is set in only one
of the replicated PTEs.

>> I wonder if we could start progressing the dirty tracking as a first initial series and
>> then have the split + collapse handling as a second part? That would be quite
>> nice to get me going! :D
> 
> I think so, and I think we should. It is such a big problem space, it
> needs to get broken up.

OK, cool! I'll stick with the same (slimmed down) IOMMU+VFIO interface as proposed in the
past except with the x86 support only[*]. And we poke holes there I guess.

[*] I might include Intel too, albeit emulated only.