Received: by 2002:a05:6a10:2726:0:0:0:0 with SMTP id ib38csp887550pxb; Wed, 6 Apr 2022 03:16:47 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxd3S1jO2c6xCiCYuusoX5MvXA9dzoqzaXCPTTvekuxldb7Vq5siJ94ue/R72X8iZPoIFaG X-Received: by 2002:a17:90b:4b89:b0:1c8:105a:2262 with SMTP id lr9-20020a17090b4b8900b001c8105a2262mr9045066pjb.225.1649240207211; Wed, 06 Apr 2022 03:16:47 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1649240207; cv=none; d=google.com; s=arc-20160816; b=aIYIhhr69M7D0ZEvOuY7xrwuVmYwBiP9yDr+voV7f6XCJsyc+VHSqOdcw7fQdPQ5lr kVDpR5BebeVGQy5HtFkHoypcKX1qALmcoAg9r/nZkgajlmVAZA+oP87bzf4TeOBVFuJs Eyeqi2Kclxhx/pWzQl6pH/PcUe/7kHTvMVNDNT4TI17MfvweY17IIIWNBAaRf4jHljgN 3BG/tTX6KDdAw45C+lliw5dnF2xInvbtQlrysbBtfLwYiwhPZ5P/UuPygf7HMjHmlo4m oRXLHCLRmBJzqk7/mKtT+NFc2SxStix94LLWltYp1hBzLtKDMWHy68y6IHFnF6Ms6X7j IpLA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=aChCA0oHsLThqEV3Vl3yllqn6+d3V1LtJ+OYYhu8nCE=; b=v4aSUhtW9ymG+u/xytN64KRd00ogFtSHM6kgNaIDkK1C0Y1HIfDoiUuEO34vVER/kq LW36PEuZEdZH2LrEnnlpaG9vZvcunjpcxSXCr7RlePlvDYt0HBWD5XD1+7aIeVqU7iVH 5Aq/wvkgEzz/MOv7LjP+kojWZ0qrUQpuioCrwaJxXPb3zIfRA5w1fzqjmnzCec1tVpcO a7Lza2Ol5gh3YlwuVHMRm3ryLlUdcpcGjMYlzh00g/vgXK4SOEvhyknN9GO8TpLM+QPI uP92nRIlFsgIjkzMEGQ338c60CmGl7bzV8bj+o9gXCNXVl/nT/8AbPzgqvoYRacucIhu po0A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linuxfoundation.org header.s=korg header.b=gZbahMoY; spf=softfail (google.com: domain of transitioning linux-kernel-owner@vger.kernel.org does not designate 23.128.96.19 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linuxfoundation.org Return-Path: Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net. [23.128.96.19]) by mx.google.com with ESMTPS id i17-20020a170902c95100b00153b2d1640dsi15396221pla.21.2022.04.06.03.16.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 06 Apr 2022 03:16:47 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning linux-kernel-owner@vger.kernel.org does not designate 23.128.96.19 as permitted sender) client-ip=23.128.96.19; Authentication-Results: mx.google.com; dkim=pass header.i=@linuxfoundation.org header.s=korg header.b=gZbahMoY; spf=softfail (google.com: domain of transitioning linux-kernel-owner@vger.kernel.org does not designate 23.128.96.19 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linuxfoundation.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 8F96D57E26C; Wed, 6 Apr 2022 01:45:15 -0700 (PDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1451040AbiDFAFa (ORCPT + 99 others); Tue, 5 Apr 2022 20:05:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44246 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1349079AbiDEJtF (ORCPT ); Tue, 5 Apr 2022 05:49:05 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B957AA9950; Tue, 5 Apr 2022 02:40:14 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 79DF0B81B7F; Tue, 5 Apr 2022 09:40:13 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id DF78EC385A3; Tue, 5 Apr 2022 09:40:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1649151612; bh=0iRwUtqp+93FnhFuEnkpLF4y24/ADWGj5PK9wIfCrAU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=gZbahMoY7iOECwh4bU9MTSciCcPv3H812cSTSvJROP6AzWVTWTXn0wLCXEJwcJDcm AcNCTyCj/CLpER4gGArQ5PC4V9JK8SFl8OomoP6YyYqZPHhT6O3KWIggeAb9FATNRa ASbrtRSE1CYY4P2XgkKi5+63jClryDTIJMZQrFYw= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Abhishek Sahu , Alex Williamson , Sasha Levin Subject: [PATCH 5.15 478/913] vfio/pci: wake-up devices around reset functions Date: Tue, 5 Apr 2022 09:25:40 +0200 Message-Id: <20220405070354.182945646@linuxfoundation.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220405070339.801210740@linuxfoundation.org> References: <20220405070339.801210740@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RDNS_NONE,SPF_HELO_NONE,T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Abhishek Sahu [ Upstream commit 26a17b12d7f3dd8a7aa45a290e5b46e9cc775ddf ] If 'vfio_pci_core_device::needs_pm_restore' is set (PCI device does not have No_Soft_Reset bit set in its PMCSR config register), then the current PCI state will be saved locally in 'vfio_pci_core_device::pm_save' during D0->D3hot transition and same will be restored back during D3hot->D0 transition. For reset-related functionalities, vfio driver uses PCI reset API's. These API's internally change the PCI power state back to D0 first if the device power state is non-D0. This state change to D0 will happen without the involvement of vfio driver. Let's consider the following example: 1. The device is in D3hot. 2. User invokes VFIO_DEVICE_RESET ioctl. 3. pci_try_reset_function() will be called which internally invokes pci_dev_save_and_disable(). 4. pci_set_power_state(dev, PCI_D0) will be called first. 5. pci_save_state() will happen then. Now, for the devices which has NoSoftRst-, the pci_set_power_state() can trigger soft reset and the original PCI config state will be lost at step (4) and this state cannot be restored again. This original PCI state can include any setting which is performed by SBIOS or host linux kernel (for example LTR, ASPM L1 substates, etc.). When this soft reset will be triggered, then all these settings will be reset, and the device state saved at step (5) will also have this setting cleared so it cannot be restored. Since the vfio driver only exposes limited PCI capabilities to its user, so the vfio driver user also won't have the option to save and restore these capabilities state either and these original settings will be permanently lost. For pci_reset_bus() also, we can have the above situation. The other functions/devices can be in D3hot and the reset will change the power state of all devices to D0 without the involvement of vfio driver. So, before calling any reset-related API's, we need to make sure that the device state is D0. This is mainly to preserve the state around soft reset. For vfio_pci_core_disable(), we use __pci_reset_function_locked() which internally can use pci_pm_reset() for the function reset. pci_pm_reset() requires the device power state to be in D0, otherwise it returns error. This patch changes the device power state to D0 by invoking vfio_pci_set_power_state() explicitly before calling any reset related API's. Fixes: 51ef3a004b1e ("vfio/pci: Restore device state on PM transition") Signed-off-by: Abhishek Sahu Link: https://lore.kernel.org/r/20220217122107.22434-3-abhsahu@nvidia.com Signed-off-by: Alex Williamson Signed-off-by: Sasha Levin --- drivers/vfio/pci/vfio_pci_core.c | 48 ++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index 0c63091cc848..15d158bdcde0 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -335,6 +335,17 @@ void vfio_pci_core_disable(struct vfio_pci_core_device *vdev) /* For needs_reset */ lockdep_assert_held(&vdev->vdev.dev_set->lock); + /* + * This function can be invoked while the power state is non-D0. + * This function calls __pci_reset_function_locked() which internally + * can use pci_pm_reset() for the function reset. pci_pm_reset() will + * fail if the power state is non-D0. Also, for the devices which + * have NoSoftRst-, the reset function can cause the PCI config space + * reset without restoring the original state (saved locally in + * 'vdev->pm_save'). + */ + vfio_pci_set_power_state(vdev, PCI_D0); + /* Stop the device from further DMA */ pci_clear_master(pdev); @@ -934,6 +945,19 @@ long vfio_pci_core_ioctl(struct vfio_device *core_vdev, unsigned int cmd, return -EINVAL; vfio_pci_zap_and_down_write_memory_lock(vdev); + + /* + * This function can be invoked while the power state is non-D0. + * If pci_try_reset_function() has been called while the power + * state is non-D0, then pci_try_reset_function() will + * internally set the power state to D0 without vfio driver + * involvement. For the devices which have NoSoftRst-, the + * reset function can cause the PCI config space reset without + * restoring the original state (saved locally in + * 'vdev->pm_save'). + */ + vfio_pci_set_power_state(vdev, PCI_D0); + ret = pci_try_reset_function(vdev->pdev); up_write(&vdev->memory_lock); @@ -2077,6 +2101,18 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, } cur_mem = NULL; + /* + * The pci_reset_bus() will reset all the devices in the bus. + * The power state can be non-D0 for some of the devices in the bus. + * For these devices, the pci_reset_bus() will internally set + * the power state to D0 without vfio driver involvement. + * For the devices which have NoSoftRst-, the reset function can + * cause the PCI config space reset without restoring the original + * state (saved locally in 'vdev->pm_save'). + */ + list_for_each_entry(cur, &dev_set->device_list, vdev.dev_set_list) + vfio_pci_set_power_state(cur, PCI_D0); + ret = pci_reset_bus(pdev); err_undo: @@ -2130,6 +2166,18 @@ static bool vfio_pci_dev_set_try_reset(struct vfio_device_set *dev_set) if (!pdev) return false; + /* + * The pci_reset_bus() will reset all the devices in the bus. + * The power state can be non-D0 for some of the devices in the bus. + * For these devices, the pci_reset_bus() will internally set + * the power state to D0 without vfio driver involvement. + * For the devices which have NoSoftRst-, the reset function can + * cause the PCI config space reset without restoring the original + * state (saved locally in 'vdev->pm_save'). + */ + list_for_each_entry(cur, &dev_set->device_list, vdev.dev_set_list) + vfio_pci_set_power_state(cur, PCI_D0); + ret = pci_reset_bus(pdev); if (ret) return false; -- 2.34.1