Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
From:   Felipe Franciosi <felipe@nutanix.com>
To:     Keith Busch <kbusch@kernel.org>
CC:     Maxim Levitsky <mlevitsk@redhat.com>,
        Stefan Hajnoczi <stefanha@redhat.com>,
        Fam Zheng <fam@euphon.net>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        Wolfram Sang <wsa@the-dreams.de>,
        "linux-nvme@lists.infradead.org" <linux-nvme@lists.infradead.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        Keith Busch <keith.busch@intel.com>,
        Kirti Wankhede <kwankhede@nvidia.com>,
        Mauro Carvalho Chehab <mchehab+samsung@kernel.org>,
        "Paul E . McKenney" <paulmck@linux.ibm.com>,
        Christoph Hellwig <hch@lst.de>,
        Sagi Grimberg <sagi@grimberg.me>,
        "Harris, James R" <james.r.harris@intel.com>,
        Liang Cunming <cunming.liang@intel.com>,
        Jens Axboe <axboe@fb.com>,
        Alex Williamson <alex.williamson@redhat.com>,
        Thanos Makatos <thanos.makatos@nutanix.com>,
        John Ferlan <jferlan@redhat.com>,
        Liu Changpeng <changpeng.liu@intel.com>,
        Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
        Nicolas Ferre <nicolas.ferre@microchip.com>,
        Paolo Bonzini <pbonzini@redhat.com>,
        Amnon Ilan <ailan@redhat.com>,
        "David S . Miller" <davem@davemloft.net>
Subject: Re: 
Thread-Index: AQHU4ISH1RctgniOY0qVd01TvSu+tqYXx06AgAS7C4A=
Date:   Mon, 25 Mar 2019 15:44:56 +0000
Message-ID: <0C3E7CCF-DD56-4129-A6F6-4A181AA2D102@nutanix.com>
References: <20190319144116.400-1-mlevitsk@redhat.com>
 <488768D7-1396-4DD1-A648-C86E5CF7DB2F@nutanix.com>
 <42f444d22363bc747f4ad75e9f0c27b40a810631.camel@redhat.com>
 <20190321161239.GH31434@stefanha-x1.localdomain>
 <20190321162140.GA29342@localhost.localdomain>
 <E217011E-C273-442D-AF99-84D3CD63FE77@nutanix.com>
 <8698ad583b1cfe86afc3d5440be630fc3e8e0680.camel@redhat.com>
 <0E8918CB-F679-4A5C-92AD-239E9CEC260C@nutanix.com>
 <20190322153025.GA31194@localhost.localdomain>
In-Reply-To: <20190322153025.GA31194@localhost.localdomain>
Accept-Language: en-GB, en-US
Content-Language: en-US
received-spf: None (protection.outlook.com: nutanix.com does not designate
 permitted sender hosts)
Content-Type: text/plain; charset="us-ascii"
Content-ID: <C029EFDA5CC8DA4BB5BD01D3E2BA1ED9@namprd02.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: c21e09ee-a518-47b5-bab8-08d6b138d4fd
X-MS-Exchange-CrossTenant-originalarrivaltime: 25 Mar 2019 15:44:56.6594
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: bb047546-786f-4de1-bd75-24e5b6f79043
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR02MB2352
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-03-25_09:,,
 signatures=0
X-Proofpoint-Spam-Reason: safe
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Keith,

> On Mar 22, 2019, at 3:30 PM, Keith Busch <kbusch@kernel.org> wrote:
>=20
> On Fri, Mar 22, 2019 at 07:54:50AM +0000, Felipe Franciosi wrote:
>>>=20
>>> Note though that SPDK doesn't support sharing the device between host a=
nd the
>>> guests, it takes over the nvme device, thus it makes the kernel nvme dr=
iver
>>> unbind from it.
>>=20
>> That is absolutely true. However, I find it not to be a problem in pract=
ice.
>>=20
>> Hypervisor products, specially those caring about performance, efficienc=
y and fairness, will dedicate NVMe devices for a particular purpose (eg. vD=
isk storage, cache, metadata) and will not share these devices for other us=
e cases. That's because these products want to deterministically control th=
e performance aspects of the device, which you just cannot do if you are sh=
aring the device with a subsystem you do not control.
>=20
> I don't know, it sounds like you've traded kernel syscalls for IPC,
> and I don't think one performs better than the other.

Sorry, I'm not sure I understand. My point is that if you are packaging a d=
istro to be a hypervisor and you want to use a storage device for VM data, =
you _most likely_ won't be using that device for anything else. To that end=
, driving the device directly from your application definitely gives you mo=
re deterministic control.

>=20
>> For scenarios where the device must be shared and such fine grained cont=
rol is not required, it looks like using the kernel driver with io_uring of=
fers very good performance with flexibility.
>=20
> NVMe's IO Determinism features provide fine grained control for shared
> devices. It's still uncommon to find hardware supporting that, though.

Sure, but then your hypervisor needs to certify devices that support that. =
This will limit your HCL. Moreover, unless the feature is solid, well-estab=
lished and works reliably on all devices you support, it's arguably prefera=
ble to have an architecture which gives you that control in software.

Cheers,
Felipe=