Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <0bda7899-6790-4b24-f622-5f9c8951c239@intel.com>
Date:   Mon, 12 Dec 2022 09:20:09 +0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Firefox/102.0 Thunderbird/102.5.1
Subject: Re: [RFC PATCH 5/5] nvme-vfio: Add a document for the NVMe device
To:     Max Gurtovoy <mgurtovoy@nvidia.com>,
        Christoph Hellwig <hch@lst.de>, Jason Gunthorpe <jgg@ziepe.ca>
CC:     <kbusch@kernel.org>, <axboe@fb.com>, <kch@nvidia.com>,
        <sagi@grimberg.me>, <alex.williamson@redhat.com>,
        <cohuck@redhat.com>, <yishaih@nvidia.com>,
        <shameerali.kolothum.thodi@huawei.com>, <kevin.tian@intel.com>,
        <mjrosato@linux.ibm.com>, <linux-kernel@vger.kernel.org>,
        <linux-nvme@lists.infradead.org>, <kvm@vger.kernel.org>,
        <eddie.dong@intel.com>, <yadong.li@intel.com>,
        <yi.l.liu@intel.com>, <Konrad.wilk@oracle.com>,
        <stephen@eideticom.com>, <hang.yuan@intel.com>,
        Oren Duer <oren@nvidia.com>
References: <20221206055816.292304-1-lei.rao@intel.com>
 <20221206055816.292304-6-lei.rao@intel.com> <20221206062604.GB6595@lst.de>
 <Y48+AaG5rSCviIhl@ziepe.ca> <20221206130901.GB24358@lst.de>
 <Y49JNvdmRPNWw26q@ziepe.ca> <20221206140002.GB27689@lst.de>
 <Y49PqoAhZOeraLVa@ziepe.ca> <20221206143126.GB30297@lst.de>
 <Y49WNo7XWZ2aFfds@ziepe.ca> <20221206150131.GA32365@lst.de>
 <9bc8e614-a687-e419-f9fd-e3177cfb41dd@nvidia.com>
 <cf88c2ec-bdd6-1df3-6c77-64a17dc3eb86@intel.com>
 <d4aeda5c-d7bb-4427-5157-fb7530dfd1fb@nvidia.com>
Content-Language: en-US
From:   "Rao, Lei" <lei.rao@intel.com>
In-Reply-To: <d4aeda5c-d7bb-4427-5157-fb7530dfd1fb@nvidia.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?bjY0aHY1RkhqMHdaZmpHS0xFQlJVOGZWTVFIMVZvSFpzQlVUZUtjWnRLczY3?=
 =?utf-8?B?WjF5N1dVQjJHL2RhSkE1M3hNTlhDMG5CUmFaemgyUnNwQXdlQlozclFzOE04?=
 =?utf-8?B?cE9TWGZOTzU4cGhJKzN2TGQ4cEdBay9hQ3FnM0RvaW5sOGFYcXNJOVdLZ2dI?=
 =?utf-8?B?bnJ4K1BHaTFCUmpRd1l0dWZjZEx5Y3FTYUdlekM1QjRRd2F4WXNaNjlwQkxP?=
 =?utf-8?B?S082Y1pVUUdzYWVZcDJsaFhxckp6dk1Fc1l4RWRydmN1Sm4vL1RKNEFlR1Jt?=
 =?utf-8?B?aHc3NVYvV3J5c1hrMzRydHZBS0Q1MDBCSXlOL0lHL3A0RU5lTVVma3RWV3hX?=
 =?utf-8?B?dUpQcFRYZmNUTlBZTVBCUW9DeExrMyt6QldxZ0ZFUXVCLzAvWXFqOGFZTHFo?=
 =?utf-8?B?Q2hVT2ZwT1lhSXMzNExkQmc3aTFaYXEwNUJrR0VrSlc3UjhOSElCb2lZR2Ny?=
 =?utf-8?B?RUxoOXhNWE13SDZ3WjNMLzF2SDFYZ3V1dXhqRFJMcUY0cDVYcWd6MDU5Ymth?=
 =?utf-8?B?VTBIMHVnWnVhZG56T0ZvUTFvYk1WaGVXbGdtbTQxbGFDRUtIYUJXQ1BHWnhP?=
 =?utf-8?B?b1lGRms5cHVZcFg5VG1ocjVMekFmRXNLZEhrOXY1Z28za3B3bnJvdkpnY28x?=
 =?utf-8?B?RlRMMUpjUGxuSno0LzR2bVVVVTEwOEtvNEVqbnMvUXdKZEw5bk1qVURzTCtR?=
 =?utf-8?B?Y2U4c0Q3SXBGNG9sSGZpMnpMUElyZXAwSFRoWHFZdUJqYmFiZzhOVndRUm1o?=
 =?utf-8?B?Y29OTXBiMXVPdVc4Y1ZxUWVMamZVNlVyL0NveVorVWxGRHB2a2h6c3o2SUs1?=
 =?utf-8?B?bWtKL2xWZlh3TFVuWDErK1FmbmI2UTljRzBkOGlvOUxZWVg4OEJ1RHladVdN?=
 =?utf-8?B?UllKTXlLellVTTdqL3E4bVFKa1EweFljays0N1JNVks0MU9tV2dlek15ZG9u?=
 =?utf-8?B?YTNrRGJHU05CNlprVHFBL1dIaFVzdXljOEpkbEV0VmhiU085RVJnQlEyK01O?=
 =?utf-8?B?eGVGNmY5YURubTdJMlJWU2RTaUdCL1FkVXo5UFVKWC9LYUNwTDRFVmwyamtz?=
 =?utf-8?B?TkhLbTVlWHRydFBWRGdEODBaYUtXQUdGV0RCV1RmSnlndVBDbk43WTNsQ0Ra?=
 =?utf-8?B?Unp2QmZ3clNJYU0zL3dkVVE1YjhpbVMwZWhsaDBkNGxBa2NwM0xRUFpPbElk?=
 =?utf-8?B?K0VNOVF3bDVWZDhyaTRUVHBXZVBhQjRjeXJlckZYVFhIVXlEeW02aFdGeTMw?=
 =?utf-8?B?UUh4dWNpMXg0K2ZDU0kzMHRLTjc1YVJQVVpNMWFTSk05V2kwNzYwQk8rc28x?=
 =?utf-8?B?SlhIZ1Iya0hyMk9rWXZsMEpmc0dmemxzOFh0Q0RRUFZxMFVuNm1hbEN6b1Z3?=
 =?utf-8?B?a20zNnZoelZCaXNaOEsrSmIxVFg1ZXo2NlBlaUc4WU5kZm1lMVRMZnN5L0xJ?=
 =?utf-8?B?MksyREg1UmxPVjBUblZLZ2ptU0JkMnNSSlhCWUNtMkJKUURvZTRGVmFrZ1h3?=
 =?utf-8?B?TVVTd2x0dW1ZeVpORFY2Qng5YUZydFZ4WHU0TmNFeDlyYVhiakxvUUtVbXA2?=
 =?utf-8?B?TnlINXFyYW90VytsQzMwSFBOY2RJRDdyM2ZnVCtHYjZPaUFHekQ5OHlvNytX?=
 =?utf-8?B?OEh1cXlzQkhuNTE5cFc5WHcxUlJZTE9IdkJNU25xTm9wdjY0RERnTDNrUDYy?=
 =?utf-8?B?OGcyU3piMzFhUmNtQlY1QkEzTW5GbDNmcjdOYjBlbjZyNzJhR1daNVRLMmdq?=
 =?utf-8?B?dFhuWklFdWMveU1SQXhBQmNWMXJEd3J0S0VsekR2Tk55US93U3owTjR4cXVq?=
 =?utf-8?B?cnFIWjVURVBKemMvLytDZnFqQ3pURDlTS1FjelFrR24yN3hDaDE4eC9FTWdI?=
 =?utf-8?B?NEdqU1RtbFNzZTRNYWRaOHhaYjhVbjZLTnBRazNlS1hMNGh0TldCS3hJb2JI?=
 =?utf-8?B?Q3labmpPNGNlREVHNUw5WU5DVXhnZC82UW1JKzBYZGZSdVNhTlM0OFB3Qm9m?=
 =?utf-8?B?ZDZhNmlHeUo4Z3dTR052WHRkenhCVGNlTG5zK2lOdFpBRys0MWFBOC9yZmtQ?=
 =?utf-8?B?bTdxekFIUjFvZ3V6SVNPVXoySWtvZ3lZK21mQ0grZ2dYenAvaDlWQkdNR2pz?=
 =?utf-8?Q?0eFWgWU2/OxaLul+1Fp25HpeW?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 5df634dc-96ef-45bb-98a5-08dadbdf0afa
X-MS-Exchange-CrossTenant-AuthSource: DM8PR11MB5640.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Dec 2022 01:20:25.0032
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: +eBWHhHXU3d2ekzy6If5CsDmBZmWKo7aS/uz/H1mVRTDvGzjzFjnGXdoUrX1hjNSdXcj9Z9+4yZpGmK1cg9p4g==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR11MB6968
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_HI,SPF_HELO_NONE,SPF_NONE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 12/11/2022 10:51 PM, Max Gurtovoy wrote:
> 
> On 12/11/2022 3:21 PM, Rao, Lei wrote:
>>
>>
>> On 12/11/2022 8:05 PM, Max Gurtovoy wrote:
>>>
>>> On 12/6/2022 5:01 PM, Christoph Hellwig wrote:
>>>> On Tue, Dec 06, 2022 at 10:48:22AM -0400, Jason Gunthorpe wrote:
>>>>> Sadly in Linux we don't have a SRIOV VF lifecycle model that is any
>>>>> use.
>>>> Beward:  The secondary function might as well be a physical function
>>>> as well.  In fact one of the major customers for "smart" multifunction
>>>> nvme devices prefers multi-PF devices over SR-IOV VFs. (and all the
>>>> symmetric dual ported devices are multi-PF as well).
>>>>
>>>> So this isn't really about a VF live cycle, but how to manage life
>>>> migration, especially on the receive / restore side.  And restoring
>>>> the entire controller state is extremely invasive and can't be done
>>>> on a controller that is in any classic form live.  In fact a lot
>>>> of the state is subsystem-wide, so without some kind of virtualization
>>>> of the subsystem it is impossible to actually restore the state.
>>>
>>> ohh, great !
>>>
>>> I read this subsystem virtualization proposal of yours after I sent my proposal for subsystem virtualization in patch 1/5 thread.
>>> I guess this means that this is the right way to go.
>>> Lets continue brainstorming this idea. I think this can be the way to migrate NVMe controllers in a standard way.
>>>
>>>>
>>>> To cycle back to the hardware that is posted here, I'm really confused
>>>> how it actually has any chance to work and no one has even tried
>>>> to explain how it is supposed to work.
>>>
>>> I guess in vendor specific implementation you can assume some things that we are discussing now for making it as a standard.
>>
>> Yes, as I wrote in the cover letter, this is a reference implementation to
>> start a discussion and help drive standardization efforts, but this series
>> works well for Intel IPU NVMe. As Jason said, there are two use cases:
>> shared medium and local medium. I think the live migration of the local medium
>> is complicated due to the large amount of user data that needs to be migrated.
>> I don't have a good idea to deal with this situation. But for Intel IPU NVMe,
>> each VF can connect to remote storage via the NVMF protocol to achieve storage
>> offloading. This is the shared medium. In this case, we don't need to migrate
>> the user data, which will significantly simplify the work of live migration.
> 
> I don't think that medium migration should be part of the SPEC. We can specify it's out of scope.
> 
> All the idea of live migration is to have a short downtime and I don't think we can guarantee short downtime if we need to copy few terabytes throw the networking.
> If the media copy is taking few seconds, there is no need to do live migration of few milisecs downtime. Just do regular migration of a
> 
>>
>> The series tries to solve the problem of live migration of shared medium.
>> But it still lacks dirty page tracking and P2P support, we are also developing
>> these features.
>>
>> About the nvme device state, As described in my document, the VF states include
>> VF CSR registers, Every IO Queue Pair state, and the AdminQ state. During the
>> implementation, I found that the device state data is small per VF. So, I decided
>> to use the admin queue of the Primary controller to send the live migration
>> commands to save and restore the VF states like MLX5.
> 
> I think and hope we all agree that the AdminQ of the controlling NVMe function will be used to migrate the controlled NVMe function.

Fully agree.

> 
> which document are you refereeing to ?

The fifth patch includes the definition of these commands and how the firmware handles
these live migration commands. It's the documentation that I referenced.

>>
>>>
>>>