Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
Message-ID: <70d0c0d4-093f-ae8a-9654-5a433285ab12@intel.com>
Date:   Thu, 12 May 2022 15:49:18 +0200
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Firefox/91.0 Thunderbird/91.9.0
Subject: Re: [PATCH v5 2/2] PCI/PM: Fix pci_pm_suspend_noirq() to disable PTM
Content-Language: en-US
To:     Rajvi Jingar <rajvi.jingar@intel.com>, <bhelgaas@google.com>
CC:     <david.e.box@linux.intel.com>, <linux-pci@vger.kernel.org>,
        <linux-kernel@vger.kernel.org>, <linux-pm@vger.kernel.org>
References: <20220429210538.2270472-1-rajvi.jingar@intel.com>
 <20220429210538.2270472-2-rajvi.jingar@intel.com>
From:   "Rafael J. Wysocki" <rafael.j.wysocki@intel.com>
Organization: Intel Technology Poland Sp. z o. o., KRS 101882, ul. Slowackiego
 173, 80-298 Gdansk
In-Reply-To: <20220429210538.2270472-2-rajvi.jingar@intel.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?NmFNeVRFY2RQclUxYWl2bXJhVWZGZVBUTmFFOVFhQzh3ZTBJT1MxVmpudSt5?=
 =?utf-8?B?Vk5DWVNOTTFkK2JSWHU3OGJOMVVESWhqaGhzdk1IVDBCdmNRZWs2QjArZ3Va?=
 =?utf-8?B?Zi81RUFOT0s5NmIwSVh6WENIdjRhYXhGdVB1ODFtT0h6UkVjeXU4NVM0M1ht?=
 =?utf-8?B?YzJlZzJsTUdNdVdhVlNSWUw5eHlTZXI0NTkwZmlZVUNtOWNiclN2NzVrZ1JY?=
 =?utf-8?B?NVNUU3dNaUxrVTNWSjgrdkdpR091cWIrR0ZqYnc3QWlIUnphSzFvNWVaZVor?=
 =?utf-8?B?VlM3QnJBODEvQU9GakZhM3FqdTBxU1N3WFNoKzdYU0QvMjg4WEgrZkZndDdx?=
 =?utf-8?B?Qjd1ZURrWGw4cnJxMy81cG13SzBEeXYvcW4xdFYzcDNPZmVINGtzSFFnUnhn?=
 =?utf-8?B?cURqQU9kUlBDejFSYXJVZTNyVGM3cnZZcjVkNjltMGZjMnBRaVRXSHZJdmVu?=
 =?utf-8?B?U1BpRytrWVhhU1BJUTlsc2RnT1d3MDBPQVVmRVV6ZTN3RmNzcWZONkR6OVN4?=
 =?utf-8?B?VTVoVlk5VTdaWVFaOGcrVnNxMDNKMlFYZ056eVZVMUc5R0JqNGFWcnlnZGdM?=
 =?utf-8?B?Ti9NRmVRTWNoem5rYkVDTGxkdEc3OUhnZnNRdHpjdjBmLzByRXRGUU1hZmkw?=
 =?utf-8?B?VnlTY0hEMXJUaVlLK2hjNHZEb01najh1czg0NWJiVkRWVkd1aVpHc0hUQUJO?=
 =?utf-8?B?a2tKN1NzK2h2YzlQSjRNbk13N2xtZERHQVR2SkZkUWZpTUI5LzJ3OTVQWG5H?=
 =?utf-8?B?Y3MrZUdNbXFPUjRaOUlBcmx6dTl0L1Y0YWNVY01pajNFaTRqeGF2NWRwdkU1?=
 =?utf-8?B?bW51bUtBbVlYcWgxM2ZuRkk2cWg5S1JmOTV5ZVh1MWxtQ0NaM3hId1FEaitj?=
 =?utf-8?B?andpdzB3bGQrZXluaFFxUFc3ZHlZT3h6eXZrL3hpbmwveSttWnd2ZGdqN3JP?=
 =?utf-8?B?dlFySkowUndWU3lsbFhmUlJSTHdTMWFveW9HRUMxMmI0Ujl4NzNuZjRVUC9u?=
 =?utf-8?B?Vm50RUE4VXFQT3BFSFQ5MGFIRithWktYNlNua3RYaG1KdFJnZkJpdFhIR2x6?=
 =?utf-8?B?MmxpM21QVm9nUEpIOUFvWlVSSDlIM2xiZjNFSjBzQWZFaU1Kd3BIcUtKaWdX?=
 =?utf-8?B?L1cyK1pZdzhuc1J4UUhWTTFFSWNYNksyZE9rVlg3blZ5VHcvVStXS2lsSW0w?=
 =?utf-8?B?ZTBUQlNyRjFIUk5mSXNXeG1odE1lRmY0a3U1Wi94Nm5lK3R0T2NIbE02YzNH?=
 =?utf-8?B?LzVPVXlVUUszZm02SlJIT1RWY0xTeENxaVlVa291RzFPQzZKMVd0Z29xNHo3?=
 =?utf-8?B?SnJnL2NUUmErWW9idEIzNHRhYmt1QVVrNEdsTTc3NVYwdHY0QjhyblhPRG5p?=
 =?utf-8?B?SnppT3l1VGo5TFBqdGpQNCsvNEl0T3QxMm5BcEZoeTU2cVRKNGVhUHVSNkFh?=
 =?utf-8?B?d29TQWI5NUR1NEV6NHpXZlJrd2Z6d3V1Zjc5QlVpWTZFZGJaYlZOMVFaTGV2?=
 =?utf-8?B?KzJ2eDQ0T0xoNkhaUlpJQnAwM2JDd014MXJqSFdKWXh0QlhnV1BrR3BZcVFa?=
 =?utf-8?B?eFZXcjlaMEhmSytTV25wTnQ5Tzl5Wis4ZUFBR0pwNnN0KzBKMzh4enNuSXZF?=
 =?utf-8?B?UklkRkE3RUtrY0YzcjRDL3kxT3NhK256UUFETGY0TXFvWEd4Q3diVUdkQVhD?=
 =?utf-8?B?L1NzOS9sc0ZPbkxVdkUyakJXL3prNzVBTnhQVWJCcklXbVZTYjRJNDZlaFNo?=
 =?utf-8?B?V0ZtVDFSK1Mvc1gySGhndFZSZDBXM3JrdzBBc2JMVzlXQWY2LzdYOGNNaHNx?=
 =?utf-8?B?OXdaYjFRWjdoMnhtVEwyWnc4eFFORkpQM0FLL2UyZ0w5QTNsZXU5S0ZUdlpV?=
 =?utf-8?B?OS9ab1JRb2NMd09WODE2SHVsMVpNWThkdUNIYk5JN29JZHorR3BBamJsT0xp?=
 =?utf-8?B?K2wrbDJ6T2RTa2Jldk0zc3lLQzFJcnF5dUl6UkEwZVlqdFROY05kSEh1N1Zl?=
 =?utf-8?B?ZkRrRHRBNGtjNFIzMEZQVDRTSTYvZm82U1dha2xXL1JaWUE2TDBmOVYrVjla?=
 =?utf-8?B?MDBYVWNzNzIrSHIxWVI2ZEdzaVAxUm94REtaWUg1R3A1emdEWC9OU2dObzBC?=
 =?utf-8?B?KzlHRHFQRjNNZVlSU0k0eHNTK2NBbWlQVmtrelJiKzdoWGZhR0g2REc3c0dI?=
 =?utf-8?B?ek51R2ZuRkMyRGxkVFlocGx6RHp3T3l6UitwVVI1elpySTY2ZU82NHdGSEN0?=
 =?utf-8?B?eklqczcyRHNXOXpkRXN0RzUvREJMcnRzY0gvTnJaZklTUG8vZENMNS94ZWNM?=
 =?utf-8?B?V2gxa0piMkYzdDliS054QzN6dHhVdFdZSmkzNytTS1JabzhHYTN1ck9YRVFO?=
 =?utf-8?Q?RKfJe3GVkTOxz7Bs=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 65146614-b0d1-4f5d-5233-08da341e39c3
X-MS-Exchange-CrossTenant-AuthSource: MW5PR11MB5810.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 May 2022 13:49:25.2392
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Rget8k/rdKhopoRiFHyM5TVWSqyfTsBcHFCjEGYr4wYUAymyJi7qiRmVP9oO3enRbAg0R847QbGykDxxF/2WXAx3oW0XNn4xyOQ3vf+jfds=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR11MB1267
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-5.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
	DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,NICE_REPLY_A,RDNS_NONE,SPF_HELO_NONE,
	T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no
	version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 4/29/2022 11:05 PM, Rajvi Jingar wrote:
> For the PCIe devices (like nvme) that do not go into D3 state still need to
> disable PTM to allow the port to enter a lower-power PM state and the SoC
> to reach a lower-power idle state as a whole. Move the pci_disable_ptm()
> out of pci_prepare_to_sleep() as this code path is not followed for devices
> that do not go into D3. This fixes the issue seen on Dell XPS 9300 with
> Ice Lake CPU and Dell Precision 5530 with Coffee Lake CPU platforms to get
> improved residency in low power idle states.
>
> Also, on receiving a PTM Request from a downstream device, if PTM is
> disabled on the root port, as per PCIe r6.0, sec 6.21.3, such a request
> would cause an Unsupported Request error. So it must first disable PTM in
> any downstream devices.
>
> Fixes: a697f072f5da ("PCI: Disable PTM during suspend to save power")
> Signed-off-by: Rajvi Jingar <rajvi.jingar@intel.com>
> Suggested-by: David E. Box <david.e.box@linux.intel.com>
> ---
>   v1 -> v2: add Fixes tag in commit message
>   v2 -> v3: move changelog after "---" marker
>   v3 -> v4: add "---" marker after changelog
>   v4 -> v5: move pci_disable_ptm() out of the pci_dev->state_saved check.
> 	   disable PTM for all devices, not just root ports.
> ---
>   drivers/pci/pci-driver.c | 28 +++++++++++++++++++---------
>   drivers/pci/pci.c        | 10 ----------
>   2 files changed, 19 insertions(+), 19 deletions(-)
>
> diff --git a/drivers/pci/pci-driver.c b/drivers/pci/pci-driver.c
> index 8b55a90126a2..400dd18a9cf5 100644
> --- a/drivers/pci/pci-driver.c
> +++ b/drivers/pci/pci-driver.c
> @@ -813,6 +813,7 @@ static int pci_pm_suspend_late(struct device *dev)
>   
>   static int pci_pm_suspend_noirq(struct device *dev)
>   {
> +	unsigned int dev_state_saved;
>   	struct pci_dev *pci_dev = to_pci_dev(dev);
>   	const struct dev_pm_ops *pm = dev->driver ? dev->driver->pm : NULL;
>   
> @@ -845,16 +846,25 @@ static int pci_pm_suspend_noirq(struct device *dev)
>   		}
>   	}
>   
> -	if (!pci_dev->state_saved) {
> +	dev_state_saved = pci_dev->state_saved;

If pci_dev->state_saved is set here, the device may be in D3cold already 
and disabling PTM for it will not work.  Of course, it is not necessary 
to disable PTM for it then, but this case need to be taken care of.

> +	if (!dev_state_saved)
>   		pci_save_state(pci_dev);
> -		/*
> -		 * If the device is a bridge with a child in D0 below it, it needs to
> -		 * stay in D0, so check skip_bus_pm to avoid putting it into a
> -		 * low-power state in that case.
> -		 */
> -		if (!pci_dev->skip_bus_pm && pci_power_manageable(pci_dev))
> -			pci_prepare_to_sleep(pci_dev);
> -	}
> +
> +	/*
> +	 * There are systems (for example, Intel mobile chips since Coffee
> +	 * Lake) where the power drawn while suspended can be significantly
> +	 * reduced by disabling PTM as this allows the SoC to reach a
> +	 * lower-power idle state as a whole.
> +	 */

Something like this should suffice IMV:

if (!dev_state_saved || pci_dev->current_state != PCI_D3cold)

         pci_disable_ptm(pci_dev);


> +	pci_disable_ptm(pci_dev);
> +
> +	/* If the device is a bridge with a child in D0 below it, it needs to
> +	 * stay in D0, so check skip_bus_pm to avoid putting it into a
> +	 * low-power state in that case.
> +	 */
> +	if (!dev_state_saved && !pci_dev->skip_bus_pm &&
> +	    pci_power_manageable(pci_dev))
> +		pci_prepare_to_sleep(pci_dev);
>   
>   	pci_dbg(pci_dev, "PCI PM: Suspend power state: %s\n",
>   		pci_power_name(pci_dev->current_state));
> diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
> index 9ecce435fb3f..f8768672c064 100644
> --- a/drivers/pci/pci.c
> +++ b/drivers/pci/pci.c
> @@ -2660,16 +2660,6 @@ int pci_prepare_to_sleep(struct pci_dev *dev)
>   	if (target_state == PCI_POWER_ERROR)
>   		return -EIO;
>   
> -	/*
> -	 * There are systems (for example, Intel mobile chips since Coffee
> -	 * Lake) where the power drawn while suspended can be significantly
> -	 * reduced by disabling PTM on PCIe root ports as this allows the
> -	 * port to enter a lower-power PM state and the SoC to reach a
> -	 * lower-power idle state as a whole.
> -	 */
> -	if (pci_pcie_type(dev) == PCI_EXP_TYPE_ROOT_PORT)
> -		pci_disable_ptm(dev);
> -
>   	pci_enable_wake(dev, target_state, wakeup);
>   
>   	error = pci_set_power_state(dev, target_state);

And the restoration of the PTM state on errors in pci_prepare_to_sleep() 
is not needed any more.