Received: by 2002:a05:6a10:a0d1:0:0:0:0 with SMTP id j17csp3235650pxa; Tue, 18 Aug 2020 09:52:45 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwFGckcIyYAUYOA6hOXlFjc6aXTDGwZnYPqJPRkVPFP+EXHi8BSFZvAcTThBgcFSzsm9slG X-Received: by 2002:aa7:d948:: with SMTP id l8mr21009771eds.371.1597769565358; Tue, 18 Aug 2020 09:52:45 -0700 (PDT) Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id b2si14055819edy.370.2020.08.18.09.52.20; Tue, 18 Aug 2020 09:52:45 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@nvidia.com header.s=n1 header.b=SEzo8G+m; arc=fail (signature failed); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=nvidia.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728338AbgHRQtQ (ORCPT + 99 others); Tue, 18 Aug 2020 12:49:16 -0400 Received: from hqnvemgate24.nvidia.com ([216.228.121.143]:16803 "EHLO hqnvemgate24.nvidia.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726721AbgHRQtL (ORCPT ); Tue, 18 Aug 2020 12:49:11 -0400 Received: from hqpgpgate101.nvidia.com (Not Verified[216.228.121.13]) by hqnvemgate24.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Tue, 18 Aug 2020 09:47:18 -0700 Received: from hqmail.nvidia.com ([172.20.161.6]) by hqpgpgate101.nvidia.com (PGP Universal service); Tue, 18 Aug 2020 09:49:10 -0700 X-PGP-Universal: processed; by hqpgpgate101.nvidia.com on Tue, 18 Aug 2020 09:49:10 -0700 Received: from HQMAIL111.nvidia.com (172.20.187.18) by HQMAIL111.nvidia.com (172.20.187.18) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Tue, 18 Aug 2020 16:49:06 +0000 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (104.47.57.174) by HQMAIL111.nvidia.com (172.20.187.18) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Tue, 18 Aug 2020 16:49:06 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=RsH8eoLcuddzy4lvDlWFz2pOrbayANGKdQHExZ4YG8u/FdiRg0LAfsZPsTm+/d9XLuy7zseVNy1gLPIBcJF3XUE/DvdG422UDjJp3l6oXTGDn9RnOJNBUv6Gq5j89N3YL26KGKs0g7xLH+MqyR0kh4PB9PC4mjlBIVzd+WhAAvi9k7FD6iB45Gane1cLe8EzfjuOZYxjNUWMVP4lk2TVZxH+SPFjs0SVX8yHswT+k3uIROmBRODV0VTk6Mm+sVV7YvKWMxy7Q9nt7I3/UZbCwrCw0FTKVcqJJr/KyDs3U/dLbuZDowewXcehNHmPy9W2mTnICMQm/g/+ZkCeBA4bLw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rkSGjCScfNbjJPQ/qhiAkmrUHqRMHr1gX+zunOCM46A=; b=NVmZS+KebhqFRp3mrV19rgqw+Sq3E7OxdbKmMMEqbYG9lxGQnnduaZk5RkY+5aVOzofVryNUbAN2VQ9PHauSYht9G6PU8lOFnxJw2UdYPy8ot+hQQPZxs/CV/V1Hm/dnLCEJXEbhtls4CiLCNnAlaLHQyUwJGeSry80OCasDPmTFTfgyh2dVqf8VOyT4KXart+dvndKQIz7ioZIX4SmlJ3pIvcT1se7mpxA7/e8r8iokbhIDxgkkXQjByyxm4SRGWHNjeqmSCF1Eb2W701CgaKDpcJdx/ue8fNMgk+EoYu0R3omTyH1424yaB4fV0CJrFSm1i03KfrlQbYfcCyh9pA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none Authentication-Results: redhat.com; dkim=none (message not signed) header.d=none;redhat.com; dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB3834.namprd12.prod.outlook.com (2603:10b6:5:14a::12) by DM5PR12MB2487.namprd12.prod.outlook.com (2603:10b6:4:af::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3283.16; Tue, 18 Aug 2020 16:49:05 +0000 Received: from DM6PR12MB3834.namprd12.prod.outlook.com ([fe80::2d79:7f96:6406:6c76]) by DM6PR12MB3834.namprd12.prod.outlook.com ([fe80::2d79:7f96:6406:6c76%3]) with mapi id 15.20.3283.028; Tue, 18 Aug 2020 16:49:05 +0000 Date: Tue, 18 Aug 2020 13:49:03 -0300 From: Jason Gunthorpe To: Paolo Bonzini CC: "Tian, Kevin" , Alex Williamson , "Jiang, Dave" , "vkoul@kernel.org" , "Dey, Megha" , "maz@kernel.org" , "bhelgaas@google.com" , "rafael@kernel.org" , "gregkh@linuxfoundation.org" , "tglx@linutronix.de" , "hpa@zytor.com" , "Pan, Jacob jun" , "Raj, Ashok" , "Liu, Yi L" , "Lu, Baolu" , "Kumar, Sanjay K" , "Luck, Tony" , "Lin, Jing" , "Williams, Dan J" , "kwankhede@nvidia.com" , "eric.auger@redhat.com" , "parav@mellanox.com" , "Hansen, Dave" , "netanelg@mellanox.com" , "shahafs@mellanox.com" , "yan.y.zhao@linux.intel.com" , "Ortiz, Samuel" , "Hossain, Mona" , "dmaengine@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "x86@kernel.org" , "linux-pci@vger.kernel.org" , "kvm@vger.kernel.org" Subject: Re: [PATCH RFC v2 00/18] Add VFIO mediated device support and DEV-MSI support for the idxd driver Message-ID: <20200818164903.GA1152540@nvidia.com> References: <20200724001930.GS2021248@mellanox.com> <20200805192258.5ee7a05b@x1.home> <20200807121955.GS16789@nvidia.com> <20200814133522.GE1152540@nvidia.com> <20200818004343.GG1152540@nvidia.com> <20200818115003.GM1152540@nvidia.com> <0711a4ce-1e64-a0cb-3e6d-f6653284e2e3@redhat.com> Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: <0711a4ce-1e64-a0cb-3e6d-f6653284e2e3@redhat.com> X-ClientProxiedBy: BL0PR0102CA0058.prod.exchangelabs.com (2603:10b6:208:25::35) To DM6PR12MB3834.namprd12.prod.outlook.com (2603:10b6:5:14a::12) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from mlx.ziepe.ca (156.34.48.30) by BL0PR0102CA0058.prod.exchangelabs.com (2603:10b6:208:25::35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3283.16 via Frontend Transport; Tue, 18 Aug 2020 16:49:04 +0000 Received: from jgg by mlx with local (Exim 4.94) (envelope-from ) id 1k84nH-008VkF-JM; Tue, 18 Aug 2020 13:49:03 -0300 X-Originating-IP: [156.34.48.30] X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 995c9018-6a64-45f9-8b26-08d843969e19 X-MS-TrafficTypeDiagnostic: DM5PR12MB2487: X-LD-Processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8273; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: h0BpFpS1Ul2/3Y4F+i/IQHXmhIZAKPWnsMgl08KkHRhmg0GQQtso1x6ibowR1aq/9c03DM4lX/Nlj0yZaD8FZc+nOwgrztrTrypqCFb+3NM7ORrjOzUAfogPsX3OOvfBTsakSub/YmqyB6PgTox9Ik2vzktPQPztsE69XxAnlKMAvhf0DpwPFPcNbAnZQBcEoWJxire6VJAEz0GJXvllWXi7WlTWTrZHTdxnG5EGuqvlT9nDNDy0zUaS9qYGWjeMzThq8ys/inIIi3YQgt450X/uOZw3zXxOfTMmhx855S/CgThzFxwpRQjCS0zM4kAOlOfSQKSXWxU+4wbHu/wLQg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3834.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(366004)(396003)(376002)(39860400002)(346002)(136003)(36756003)(66476007)(66556008)(7406005)(7416002)(8676002)(9746002)(1076003)(66946007)(2906002)(9786002)(426003)(86362001)(26005)(6916009)(4744005)(8936002)(5660300002)(316002)(186003)(53546011)(478600001)(54906003)(83380400001)(2616005)(33656002)(4326008);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: xudimcu9dVNHMTwohSFRZw726v6mJXnmIjTLbx0vRP5D81Ea7LBcsEHvCSmBQwgFx9hgr05o37ZHTKaiizcQjZU0UUub/0kBmZkR/ifogqFZhpl6UJ2V+spYsuoFOKUECOwLE11IecXLDGBHTXfTMej7FluZKuu6aWP58pJ5H9hOU/ma+3682Cr77+QuRlXjbrT34hMmxT1QhArIWlTmcpdUMBTP6JSNqF+u4ixzhVJ1ZHP/CK12cwoU6SyC/+qU2rJ/4P+I6zUkHY17ajp4M+pjpOoRLE9gUrbLCYLcsS4azzTuS3KJAHPnu3ZycySXmbQ16z+DsKmPJIZZCrard2Mz6GgnhbUiyysn9ficlvaVbY3H8Agr1uUoo/eXsnUWC0/t8U5+yaowBuL3ELSvKDBudwkkNG2g5jvpTs6tG8VPxTjQ3tLxrODrHAL0hZDG57eICcL/wRdOLZkTdMmZyXxSmPiZaKU8nPQog7lN4VEINojA4Gzrtdf0u+OkWU0X0GP0CxMHTozuv+VapHNDo+RBAaN03Uw+BF1ldlV9jaJgs81Xl4HnHpRt+inbiS41GKmHcf4TTYEPRHjHEQSbGXCb+I/nCXKwa+f8OuxQaztLWR6sldvCoQYfio91u2bxPbd9B1W0zYSArGG1Vm90KQ== X-MS-Exchange-CrossTenant-Network-Message-Id: 995c9018-6a64-45f9-8b26-08d843969e19 X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3834.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Aug 2020 16:49:05.1343 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: PRK/RgnnSALI86jemSEnJH8y6bTh7jappx45rx4EeIJbj7GjT2jYoyO1bUEE5Q9b X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR12MB2487 X-OriginatorOrg: Nvidia.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1597769238; bh=rkSGjCScfNbjJPQ/qhiAkmrUHqRMHr1gX+zunOCM46A=; h=X-PGP-Universal:ARC-Seal:ARC-Message-Signature: ARC-Authentication-Results:Authentication-Results:Date:From:To:CC: Subject:Message-ID:References:Content-Type:Content-Disposition: In-Reply-To:X-ClientProxiedBy:MIME-Version: X-MS-Exchange-MessageSentRepresentingType:X-Originating-IP: X-MS-PublicTrafficType:X-MS-Office365-Filtering-Correlation-Id: X-MS-TrafficTypeDiagnostic:X-LD-Processed: X-MS-Exchange-Transport-Forked:X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers:X-MS-Exchange-SenderADCheck: X-Microsoft-Antispam:X-Microsoft-Antispam-Message-Info: X-Forefront-Antispam-Report:X-MS-Exchange-AntiSpam-MessageData: X-MS-Exchange-CrossTenant-Network-Message-Id: X-MS-Exchange-CrossTenant-AuthSource: X-MS-Exchange-CrossTenant-AuthAs: X-MS-Exchange-CrossTenant-OriginalArrivalTime: X-MS-Exchange-CrossTenant-FromEntityHeader: X-MS-Exchange-CrossTenant-Id:X-MS-Exchange-CrossTenant-MailboxType: X-MS-Exchange-CrossTenant-UserPrincipalName: X-MS-Exchange-Transport-CrossTenantHeadersStamped:X-OriginatorOrg; b=SEzo8G+mXp+HcfugAbMSGXk4WmZnX4isWWOgCRDdSPmoH/YtpmfbgEXhQv1TT/d48 2rY6A9gQy+U2Dzmx6uetSdEHcOSTO1nyHlkCMD9bVGJTTWuc3j6GYIE76xJbHzIrxg TkrVj6At00H5eOiESLt1XvBUBCpcH7618Se/Tew71j35+jkydho/2SjH8ppcmsGPlk JRSY+pKCKQbOIKcFfky34/Ret5viBoKP+Y69254dg6moQLtbw0a66+Dscx4OVxEZhM 2SnHF2FzKYi+irElYjwQKhEM5fESv2klCE3XGGDJWql9OgnuOHy2u/VlV5RHxCJxel CxLRE/ddHQ9xw== Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Aug 18, 2020 at 06:27:21PM +0200, Paolo Bonzini wrote: > On 18/08/20 13:50, Jason Gunthorpe wrote: > > For instance, what about suspend/resume of containers using idxd? > > Wouldn't you want to have the same basic approach of controlling the > > wq from userspace that virtualization uses? > > The difference is that VFIO more or less standardizes the approach you > use for live migration. With another interface you'd have to come up > with something for every driver, and add support in CRIU for every > driver as well. VFIO is very unsuitable for use as some general userspace. It only 1:1 with a single process and just can't absorb what the existing idxd userspace is doing. So VFIO is already not a solution for normal userspace idxd where CRIU becomes interesting. Not sure what you are trying to say? My point was the opposite, if you want to enable CRIU for idxd then you probably need all the same stuff as for qemu/VFIO except in the normal idxd user API. Jason