Received: by 2002:a25:31c3:0:0:0:0:0 with SMTP id x186csp916404ybx; Tue, 5 Nov 2019 07:30:50 -0800 (PST) X-Google-Smtp-Source: APXvYqwX3Bs5lXA1Ol2kktGsb6EVXysLvcC52Veh6f/TIWnFLtwvz6B8NY07/umv2qY0EWT+TRkn X-Received: by 2002:a17:906:245b:: with SMTP id a27mr30447510ejb.192.1572967850052; Tue, 05 Nov 2019 07:30:50 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1572967850; cv=none; d=google.com; s=arc-20160816; b=LzmnjAJShFBG7GhQo3nl/cBI+Dkv9mj+LvOWco4EJpJpUrrGPW6zGq0S4c/zbYS7r8 GrlVV36F1t6vAMVACeUsHEkXa6eWXzHrtTafMlnE6KPN6tbjWeR5cmVA6TZUp0ZYIO3r IqcmugM4NB8lsuv8rPCGnbR2sve+DB94Z3ki92Q/CLo6ZzZ22Le1ZhNsxUGJlsIAraGX v0iD0C/QaLAPaQ9yXwMdpqktX4KU9nPGthGas4iOcVUbHCHok+NfhOn/e8YW9q9RTI08 yBkNO/ZUdqGQy073jOW9JfbDVHzPGE4RjsCPjccD5lS8yX7I+AIe2crKbJWsy3ad/1Rp afJg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:user-agent:organization:in-reply-to :content-disposition:mime-version:references:message-id:subject:cc :to:from:date; bh=Aj6bppa1XCu4eohbyVHAbA2ggqw3F+f+fBuFWQ9g2+8=; b=GvUJas7GA1OX+/lWsBw0K6UbTHDLM0Toq+L29Oxj5PZ1FsWWxExh0DlaZy+Lpa/vTD DGc+dvi/4QckXdgy/X621gDF1TIm2x2KVcpUghYCUPW/F3ZAvBeJQFy4hGiX7mhUogTL NRf4WUZfnK3BjCCCX2vVTV9dHKPocm3ml+66pee7kskv04eJYg7f5VlRA8lDaW8v0Cbb WuoZgL/5oKBJWaT0r9FUEQkWr2Xd7SFmnNhaanLSgTm+jlZylJ4OMuRdpYuWK+51yz/u EKTS3VszDt+l0bTRxCp1WHmIeiDVSYsEkznJ3zuzs9Ak4BLUHUPiV2oUaRXTh8OT6dBc hPVQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id c29si9779478edb.223.2019.11.05.07.30.25; Tue, 05 Nov 2019 07:30:50 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2389909AbfKEP2i (ORCPT + 99 others); Tue, 5 Nov 2019 10:28:38 -0500 Received: from mga12.intel.com ([192.55.52.136]:20980 "EHLO mga12.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2389546AbfKEP2h (ORCPT ); Tue, 5 Nov 2019 10:28:37 -0500 X-Amp-Result: UNKNOWN X-Amp-Original-Verdict: FILE UNKNOWN X-Amp-File-Uploaded: False Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 05 Nov 2019 07:28:37 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.68,271,1569308400"; d="scan'208";a="212559786" Received: from lahna.fi.intel.com (HELO lahna) ([10.237.72.163]) by fmsmga001.fm.intel.com with SMTP; 05 Nov 2019 07:28:33 -0800 Received: by lahna (sSMTP sendmail emulation); Tue, 05 Nov 2019 17:28:32 +0200 Date: Tue, 5 Nov 2019 17:28:32 +0200 From: Mika Westerberg To: Bjorn Helgaas Cc: "Rafael J. Wysocki" , Len Brown , Lukas Wunner , Keith Busch , Alex Williamson , Alexandru Gagniuc , Kai-Heng Feng , Paul Menzel , Nicholas Johnson , linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH v2 2/2] PCI: Add missing link delays required by the PCIe spec Message-ID: <20191105152832.GC2552@lahna.fi.intel.com> References: <20191105095428.GR2552@lahna.fi.intel.com> <20191105150013.GA202873@google.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20191105150013.GA202873@google.com> Organization: Intel Finland Oy - BIC 0357606-4 - Westendinkatu 7, 02160 Espoo User-Agent: Mutt/1.12.1 (2019-06-15) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Nov 05, 2019 at 09:00:13AM -0600, Bjorn Helgaas wrote: > On Tue, Nov 05, 2019 at 11:54:28AM +0200, Mika Westerberg wrote: > > On Mon, Nov 04, 2019 at 06:00:00PM -0600, Bjorn Helgaas wrote: > > > > > If you think it is fine to do the delay before we have restored > > > > everything I can move it inside pci_power_up() or call it after > > > > pci_pm_default_resume_early() as above. I think at least we should make > > > > sure all the saved registers are restored before so that the link > > > > activation check actually works. > > > > > > What needs to be restored to make pcie_wait_for_link_delay() work? > > > > I'm not entirely sure. I think that pci_restore_state() at least should > > be called so that the PCIe capability gets restored. Maybe not even > > that because Data Link Layer Layer Active always reflects the DL_Active > > or not and it does not need to be enabled separately. > > > > > And what event does the restore need to be ordered with? > > > > Not sure I follow you here. > > You're suggesting that we should restore saved registers first so > pcie_wait_for_link_delay() works. If the link activation depends on > something being restored and we don't enforce an ordering, the > activation might succeed or fail depending on whether it happens > before or after the restore. So if there is a dependency, we should > make it explicit to avoid a race like that. OK thanks. By explicit you mean document it in the code, right? > But I'm not saying we *shouldn't* do the restore before the wait; only > that any dependency should be explicit. Even if there is no actual > dependency it probably makes sense to do the restore first so it can > overlap with the hardware link training, which may reduce the time > pcie_wait_for_link_delay() has to wait when we do call it, e.g., > > |-----------------| link activation > |-----| restore state > |--------| pcie_wait_for_link_delay() > > whereas if we do the restore after waiting for the link to come up, it > probably takes longer: > > |-----------------| link activation > |--------------| pcie_wait_for_link_delay() > |-----| restore state > > I actually suspect there *is* a dependency -- we should respect the > Target Link Speed and and width so the link resumes in the same > configuration it was before suspend. And I suspect that may require > an explicit retrain after restoring PCI_EXP_LNKCTL2. According the PCIe spec the PCI_EXP_LNKCTL2 Target Link Speed is marked as RWS (S for sticky) so I suspect its value is retained after reset in the same way as PME bits. Assuming I understood it correctly.