Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
From:   Felipe Franciosi <felipe@nutanix.com>
To:     Maxim Levitsky <mlevitsk@redhat.com>
CC:     Keith Busch <kbusch@kernel.org>,
        Stefan Hajnoczi <stefanha@redhat.com>,
        Fam Zheng <fam@euphon.net>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        Wolfram Sang <wsa@the-dreams.de>,
        "linux-nvme@lists.infradead.org" <linux-nvme@lists.infradead.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        Keith Busch <keith.busch@intel.com>,
        Kirti Wankhede <kwankhede@nvidia.com>,
        Mauro Carvalho Chehab <mchehab+samsung@kernel.org>,
        "Paul E . McKenney" <paulmck@linux.ibm.com>,
        Christoph Hellwig <hch@lst.de>,
        Sagi Grimberg <sagi@grimberg.me>,
        "Harris, James R" <james.r.harris@intel.com>,
        Liang Cunming <cunming.liang@intel.com>,
        Jens Axboe <axboe@fb.com>,
        Alex Williamson <alex.williamson@redhat.com>,
        Thanos Makatos <thanos.makatos@nutanix.com>,
        John Ferlan <jferlan@redhat.com>,
        Liu Changpeng <changpeng.liu@intel.com>,
        Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
        Nicolas Ferre <nicolas.ferre@microchip.com>,
        Paolo Bonzini <pbonzini@redhat.com>,
        Amnon Ilan <ailan@redhat.com>,
        "David S . Miller" <davem@davemloft.net>
Subject: Re: 
Thread-Index: AQHU4ISGDFUGO+4rpUu49Zh8vFOt4Q==
Date:   Fri, 22 Mar 2019 07:54:50 +0000
Message-ID: <0E8918CB-F679-4A5C-92AD-239E9CEC260C@nutanix.com>
References: <20190319144116.400-1-mlevitsk@redhat.com>
 <488768D7-1396-4DD1-A648-C86E5CF7DB2F@nutanix.com>
 <42f444d22363bc747f4ad75e9f0c27b40a810631.camel@redhat.com>
 <20190321161239.GH31434@stefanha-x1.localdomain>
 <20190321162140.GA29342@localhost.localdomain>
 <E217011E-C273-442D-AF99-84D3CD63FE77@nutanix.com>
 <8698ad583b1cfe86afc3d5440be630fc3e8e0680.camel@redhat.com>
In-Reply-To: <8698ad583b1cfe86afc3d5440be630fc3e8e0680.camel@redhat.com>
Accept-Language: en-GB, en-US
Content-Language: en-US
received-spf: None (protection.outlook.com: nutanix.com does not designate
 permitted sender hosts)
Content-Type: text/plain; charset="us-ascii"
Content-ID: <06727B797CBF14419894608A12BEBB3F@namprd02.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: 5c6106c6-8b74-44ee-205c-08d6ae9ba94c
X-MS-Exchange-CrossTenant-originalarrivaltime: 22 Mar 2019 07:54:50.1424
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: bb047546-786f-4de1-bd75-24e5b6f79043
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR02MB2480
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-03-22_05:,,
 signatures=0
X-Proofpoint-Spam-Reason: safe
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


> On Mar 21, 2019, at 5:04 PM, Maxim Levitsky <mlevitsk@redhat.com> wrote:
>=20
> On Thu, 2019-03-21 at 16:41 +0000, Felipe Franciosi wrote:
>>> On Mar 21, 2019, at 4:21 PM, Keith Busch <kbusch@kernel.org> wrote:
>>>=20
>>> On Thu, Mar 21, 2019 at 04:12:39PM +0000, Stefan Hajnoczi wrote:
>>>> mdev-nvme seems like a duplication of SPDK.  The performance is not
>>>> better and the features are more limited, so why focus on this approac=
h?
>>>>=20
>>>> One argument might be that the kernel NVMe subsystem wants to offer th=
is
>>>> functionality and loading the kernel module is more convenient than
>>>> managing SPDK to some users.
>>>>=20
>>>> Thoughts?
>>>=20
>>> Doesn't SPDK bind a controller to a single process? mdev binds to
>>> namespaces (or their partitions), so you could have many mdev's assigne=
d
>>> to many VMs accessing a single controller.
>>=20
>> Yes, it binds to a single process which can drive the datapath of multip=
le
>> virtual controllers for multiple VMs (similar to what you described for =
mdev).
>> You can therefore efficiently poll multiple VM submission queues (and mu=
ltiple
>> device completion queues) from a single physical CPU.
>>=20
>> The same could be done in the kernel, but the code gets complicated as y=
ou add
>> more functionality to it. As this is a direct interface with an untruste=
d
>> front-end (the guest), it's also arguably safer to do in userspace.
>>=20
>> Worth noting: you can eventually have a single physical core polling all=
 sorts
>> of virtual devices (eg. virtual storage or network controllers) very
>> efficiently. And this is quite configurable, too. In the interest of fai=
rness,
>> performance or efficiency, you can choose to dynamically add or remove q=
ueues
>> to the poll thread or spawn more threads and redistribute the work.
>>=20
>> F.
>=20
> Note though that SPDK doesn't support sharing the device between host and=
 the
> guests, it takes over the nvme device, thus it makes the kernel nvme driv=
er
> unbind from it.

That is absolutely true. However, I find it not to be a problem in practice=
.

Hypervisor products, specially those caring about performance, efficiency a=
nd fairness, will dedicate NVMe devices for a particular purpose (eg. vDisk=
 storage, cache, metadata) and will not share these devices for other use c=
ases. That's because these products want to deterministically control the p=
erformance aspects of the device, which you just cannot do if you are shari=
ng the device with a subsystem you do not control.

For scenarios where the device must be shared and such fine grained control=
 is not required, it looks like using the kernel driver with io_uring offer=
s very good performance with flexibility.

Cheers,
Felipe=