Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <44b33ce9-e0ed-976c-9a13-b154d32b200e@linux.intel.com>
Date:   Fri, 2 Sep 2022 14:11:12 -0700
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Firefox/91.0 Thunderbird/91.11.0
Subject: Re: [PATCH 1/4] PCI/PTM: Preserve PTM Root Select
Content-Language: en-US
To:     Bjorn Helgaas <helgaas@kernel.org>
Cc:     Kai-Heng Feng <kai.heng.feng@canonical.com>,
        Rajvi Jingar <rajvi.jingar@linux.intel.com>,
        "Rafael J . Wysocki" <rafael@kernel.org>,
        Koba Ko <koba.ko@canonical.com>,
        Mika Westerberg <mika.westerberg@linux.intel.com>,
        "David E . Box" <david.e.box@linux.intel.com>,
        linux-pci@vger.kernel.org, linux-pm@vger.kernel.org,
        linux-kernel@vger.kernel.org, Bjorn Helgaas <bhelgaas@google.com>
References: <20220902203848.GA370638@bhelgaas>
From:   Sathyanarayanan Kuppuswamy 
        <sathyanarayanan.kuppuswamy@linux.intel.com>
In-Reply-To: <20220902203848.GA370638@bhelgaas>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Precedence: bulk

Hi Bjorn,

On 9/2/22 1:38 PM, Bjorn Helgaas wrote:
> On Fri, Sep 02, 2022 at 10:24:05AM -0700, Sathyanarayanan Kuppuswamy wrote:
>> On 9/2/22 7:58 AM, Bjorn Helgaas wrote:
>>> From: Bjorn Helgaas <bhelgaas@google.com>
>>>
>>> When disabling PTM, there's no need to clear the Root Select bit.  We
>>> disable PTM during suspend, and we want to re-enable it during resume.
>>> Clearing Root Select here makes re-enabling more complicated.
>>
>> Currently, it looks like we disable PCI_PTM_CTRL_ROOT in pci_disable_ptm(),
>> but not enable it in pci_enable_ptm(). Do you know this did not trigger an
>> issue?
> 
> For Root Ports and Switches, we enable PTM (and set Root Select when
> appropriate) during enumeration in pci_ptm_init().  This is based on
> the assumption that enabling PTM in Root Ports and Switches is a no-op
> unless there's an Endpoint that generates PTM Requests.  (It turns out
> that's not quite true, because Kai-Heng's bug report [1] shows the
> 08:00.0 Switch sending PTM Requests even though no Endpoint even has a
> PTM Capability.)
> 
> If we didn't enable PTM in Root Ports and Switches during enumeration,
> we'd have to walk the whole path and enable them when enabling PTM for
> an Endpoint.
> 
> pci_enable_ptm() currently only works for Endpoints, which cannot be
> PTM Roots, so it never has to set PCI_PTM_CTRL_ROOT.
> 
> If we clear PCI_PTM_CTRL_ROOT in pci_disable_ptm(), it will never get
> set again unless we re-enumerate the Root Port.

Thanks for clarifying.

> 
> Thanks for asking this, because it reminds me why I didn't add
> pci_enable_ptm() calls in the resume paths!  That would make them
> parallel with the suspend paths, which would definitely be nice.  But
> we would have to rework pci_enable_ptm() to work for Root Ports and
> Switch Ports as well.  I think we *could* do that.  What do you think?

IMO, the code will look better if we keep the suspend and resume paths in
sync. Since we are calling pci_disable_ptm() in suspend path, it makes
sense to call pci_enable_ptm() in resume path.

Making the pci_enable_ptm() handle root and upstream ports should not
be very complicated, right?

> 
> Regardless of that question, I think it's unnecessary to clear
> PCI_PTM_CTRL_ROOT in pci_disable_ptm(), so we should leave it alone.

I agree with you. We should not touch PCI_PTM_CTRL_ROOT in pci_disable_ptm().

> 
> [1] https://bugzilla.kernel.org/show_bug.cgi?id=215453
> 
>> Also, you mentioned that it is complicated to enable it, can you add some
>> details?
>>
>>> Per PCIe r6.0, sec 7.9.15.3, "When set, if the PTM Enable bit is also Set,
>>> this Time Source is the PTM Root," so if PTM Enable is cleared, the value
>>> of Root Select should be irrelevant.
>>>
>>> Preserve Root Select to simplify re-enabling PTM.
>>>
>>> Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
>>> Cc: David E. Box <david.e.box@linux.intel.com>
>>> ---
>>>  drivers/pci/pcie/ptm.c | 2 +-
>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>
>>> diff --git a/drivers/pci/pcie/ptm.c b/drivers/pci/pcie/ptm.c
>>> index 368a254e3124..b6a417247ce3 100644
>>> --- a/drivers/pci/pcie/ptm.c
>>> +++ b/drivers/pci/pcie/ptm.c
>>> @@ -42,7 +42,7 @@ void pci_disable_ptm(struct pci_dev *dev)
>>>  		return;
>>>  
>>>  	pci_read_config_word(dev, ptm + PCI_PTM_CTRL, &ctrl);
>>> -	ctrl &= ~(PCI_PTM_CTRL_ENABLE | PCI_PTM_CTRL_ROOT);
>>> +	ctrl &= ~PCI_PTM_CTRL_ENABLE;
>>>  	pci_write_config_word(dev, ptm + PCI_PTM_CTRL, ctrl);
>>>  }
>>>  
>>
>> -- 
>> Sathyanarayanan Kuppuswamy
>> Linux Kernel Developer

-- 
Sathyanarayanan Kuppuswamy
Linux Kernel Developer