Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
Date:   Wed, 1 Jun 2022 14:30:54 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Alex Williamson <alex.williamson@redhat.com>
Cc:     Abhishek Sahu <abhsahu@nvidia.com>,
        Cornelia Huck <cohuck@redhat.com>,
        Yishai Hadas <yishaih@nvidia.com>,
        Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>,
        Kevin Tian <kevin.tian@intel.com>,
        "Rafael J . Wysocki" <rafael@kernel.org>,
        Max Gurtovoy <mgurtovoy@nvidia.com>,
        Bjorn Helgaas <bhelgaas@google.com>,
        linux-kernel@vger.kernel.org, kvm@vger.kernel.org,
        linux-pm@vger.kernel.org, linux-pci@vger.kernel.org
Subject: Re: [PATCH v3 8/8] vfio/pci: Add the support for PCI D3cold state
Message-ID: <20220601173054.GS1343366@nvidia.com>
References: <9e44e9cc-a500-ab0d-4785-5ae26874b3eb@nvidia.com>
 <20220509154844.79e4915b.alex.williamson@redhat.com>
 <68463d9b-98ee-b9ec-1a3e-1375e50a2ad2@nvidia.com>
 <42518bd5-da8b-554f-2612-80278b527bf5@nvidia.com>
 <20220530122546.GZ1343366@nvidia.com>
 <c73d537b-a653-bf79-68cd-ddc8f0f62a25@nvidia.com>
 <20220531194304.GN1343366@nvidia.com>
 <20220531165209.1c18854f.alex.williamson@redhat.com>
 <00b6e380-ecf4-1eaf-f950-2c418bdb6cac@nvidia.com>
 <20220601102151.75445f6a.alex.williamson@redhat.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20220601102151.75445f6a.alex.williamson@redhat.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?KsuwPF6k0Gk2w8l6XA3gVmKKtEoPZfivgpdC9xCTOz3duO6+jrA1jaskVdlu?=
 =?us-ascii?Q?GpewiUrd29Ut2OD+xSR9KcUBDHbaE2+mbYJormyQ/zN1uOGetYrnFEca0zpo?=
 =?us-ascii?Q?b4Ldx6Ct6QVHrFLGhKiL+7W9XHKCsh0ajEYhxcpoPNqs8ZxYo+Bd4tpBSHfX?=
 =?us-ascii?Q?zWDz3dZeEUzxIinC39/J4GOOF9AI5Seznc7gA22pCN9WoHXE3K60/lmovXgr?=
 =?us-ascii?Q?xEMLRwESZRGZtllEJyRiSAvd5gK7PKal/NBVBYF9VaiNgS6YcJ33UNMuLrxt?=
 =?us-ascii?Q?9UpjTRm8AwipdoQFQ+Io/OTOouETHdjgnuJBnBJ/PakkoiHh9DnzHw5Th5n5?=
 =?us-ascii?Q?V5OhCK9PFFS0nzhwbSEP7wdIvdBzoU9nGRR+DDXd2XMFUWZMGaOG/9sHeTV3?=
 =?us-ascii?Q?jlU5pQ69n3ISLcPNlmcIE0FWn2oQvr9pqrcAwK4TDoOiTQR0+8Np+HGeDzb5?=
 =?us-ascii?Q?K6M8WhagJibo/EmSXK6IDnL6jPlJOKmFjrg86rmQ38qqnWWeuOdjCb8QRfA8?=
 =?us-ascii?Q?SDU7maQZHnKGjofs0gHnuUSlxLNH9msB7S6NGs8qUKexv+9kVPaUf1y08Xyi?=
 =?us-ascii?Q?pIuKc0QEmFrU0kpjspY6rv9HFlHyLNGKuXA70+FSimRTr/ON33pDDhxlkMsw?=
 =?us-ascii?Q?U8ACgvXf1gxknrC9n/PcwQFYI55U+qiNbvQLClSolSrNjfqgc+kF0TOsczn/?=
 =?us-ascii?Q?qiqExaKoAiT/FxTHD4ELJRJvvb32ojWNWcZ+8Zpxsrfs2glEACA9zknE3KxH?=
 =?us-ascii?Q?45E4DKyVrDR0FkrxJ5dquDgTyYrtzntBUNqXiqPDa+o+Weoro3Csmo0D1cGq?=
 =?us-ascii?Q?WJfx70ZaFtr/yQDoyNH8Als08NGn7Zhcoeo4zd9KoBzfHW4K8tDFdBVu27zw?=
 =?us-ascii?Q?vVo6qYDmJycm9Xqq0XwQO/Bl66lMoT+o+hiXCZj8SVwv6+svC2+TpqVce0n7?=
 =?us-ascii?Q?CVvv245sdJYuyfB2pYACWTE18r2hVcLeVh2MLox5pjxeDBjjhSRhfquclSIG?=
 =?us-ascii?Q?jcE42CFwq8j5cRw73B0Rg32s/cGQn9m2lf3EAXqNuT1di79RaELpuYpjDjgJ?=
 =?us-ascii?Q?pCIR2zekDWeYCdD5FpWuGGqXlW3ZaPw3QCTiYwExpGWEZyakdSeDORjWK/hz?=
 =?us-ascii?Q?AFRi2UupEeXQzt2IJgRgJyr8VjJbcXr6A7utOyq/Gq5exSaONgQXeTW2TjIU?=
 =?us-ascii?Q?KuzVIoHiVU8ELYRAa4MT5QJ1GuA9lfDxw/F5kyg2D+Y3iBJ7iI4DF+gJ1ItA?=
 =?us-ascii?Q?yLLOlzvKvSFu/jttrf928AB5YERHXIW6Vpq/2gr3dDvIRtVJwK/Tlqi+LiKw?=
 =?us-ascii?Q?78OvLoeGVTvwBm9RBC6IJCM/CoblTLI7aVkzzLkgsL9YU/PQcj+lfxgYaG8h?=
 =?us-ascii?Q?sl+RtWJuTA8bWVnt5Lv1EfNxlfR8VuBCKojQBDPdaRpInsSXn78aR4NwW94s?=
 =?us-ascii?Q?WHKpLevrxslaztcTaH/EXDDqOU0hrsyCQ3Loqo5vjwzlPTtESRXit8DXmDJ5?=
 =?us-ascii?Q?GpA3CYqFKUiduhKrPV4sGi/O6VTq2OfHMzmU2ANQ5TnpluRnrA70HtxDM0Sl?=
 =?us-ascii?Q?9Wnnd2+yfmQS/57EZ3azAYJ/9conSKowgxnEvTxQ8YllCZAJuCTllrXCuInL?=
 =?us-ascii?Q?kVJmJUvOyER7lqSF91v/z8BOXUirjbY1e74tF7xxC2d30LOYZf0zpuohcL9X?=
 =?us-ascii?Q?Tf4YUNTZidVxkpQgms5D0xYq6yV6iuNyE5w5tcoBZNx675xQvd8SS5ItDhfx?=
 =?us-ascii?Q?1FwarTq+QA=3D=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: a5ed12c2-c2f4-45a6-3e55-08da43f47bec
X-MS-Exchange-CrossTenant-AuthSource: MN2PR12MB4192.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Jun 2022 17:30:55.8225
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: efltRMaVMycoogPM/4gUWnWzlSetA9L/SoDKmyevzPmyO+EDNGRbWRXgToesi6we
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR12MB2677
X-Spam-Status: No, score=-2.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
	DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,RDNS_NONE,SPF_HELO_NONE,T_SCC_BODY_TEXT_LINE
	autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Wed, Jun 01, 2022 at 10:21:51AM -0600, Alex Williamson wrote:

> Some ioctls clearly cannot occur while the device is in low power, such
> as resets and interrupt control, but even less obvious things like
> getting region info require device access.  Migration also provides a
> channel to device access.  

I wonder what power management means in a case like that.

For the migration drivers they all rely on a PF driver that is not
VFIO, so it should be impossible for power management to cause the PF
to stop working.

I would expect any sane design of power management for a VF to not
cause any harm to the migration driver..

> I'm also still curious how we're going to handle devices that cannot
> return to low power such as the self-refresh mode on the GPU.  We can
> potentially prevent any wake-ups from the vfio device interface, but
> that doesn't preclude a wake-up via an external lspci.  I think we need
> to understand how we're going to handle such devices before we can
> really complete the design.  AIUI, we cannot disable the self-refresh
> sleep mode without imposing unreasonable latency and memory
> requirements on the guest and we cannot retrigger the self-refresh
> low-power mode without non-trivial device specific code.

It begs the question if power management should be something that only
a device-specific drivers should allow?

Jason