Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates
 216.228.112.34 as permitted sender) receiver=protection.outlook.com;
 client-ip=216.228.112.34; helo=mail.nvidia.com;
Subject: Re: [PATCH v4 2/2] PCI: Enable NO_BUS_RESET quirk for Nvidia GPUs
To:     Bjorn Helgaas <helgaas@kernel.org>
CC:     Alex Williamson <alex.williamson@redhat.com>,
        Bjorn Helgaas <bhelgaas@google.com>,
        <linux-pci@vger.kernel.org>, <linux-kernel@vger.kernel.org>,
        Sinan Kaya <okaya@kernel.org>,
        Vikram Sethi <vsethi@nvidia.com>,
        Amey Narkhede <ameynarkhede03@gmail.com>
References: <20210430170151.GA660969@bjorn-Precision-5520>
From:   Shanker R Donthineni <sdonthineni@nvidia.com>
Message-ID: <52c89d4e-6b26-6c56-d71e-508a715394ab@nvidia.com>
Date:   Fri, 30 Apr 2021 17:11:23 -0500
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101
 Thunderbird/68.10.0
MIME-Version: 1.0
In-Reply-To: <20210430170151.GA660969@bjorn-Precision-5520>
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
Content-Language: en-US
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Apr 2021 22:11:28.0145
 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: 1570c040-2cf1-4aef-c42b-08d90c24e6e6
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.112.34];Helo=[mail.nvidia.com]
X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT058.eop-nam11.prod.protection.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Anonymous
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB2651
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Thanks Bjorn for reviewing patch.

On 4/30/21 12:01 PM, Bjorn Helgaas wrote:
> External email: Use caution opening links or attachments
>
>
> On Wed, Apr 28, 2021 at 07:49:07PM -0500, Shanker Donthineni wrote:
>> On select platforms, some Nvidia GPU devices do not work with SBR.
>> Triggering SBR would leave the device inoperable for the current
>> system boot. It requires a system hard-reboot to get the GPU device
>> back to normal operating condition post-SBR. For the affected
>> devices, enable NO_BUS_RESET quirk to fix the issue.
> Since 1/2 adds _RST support, should I infer that _RST works on these
> Nvidia GPUs even though SBR does not?  If so, how does _RST do the
> reset?
Yes, _RST method works but not SBR. The _RST method in DSDT-AML uses
platform-specific initialization steps outside of the GPU BARs for resetting
the GPU device.
> Do you have a root cause for why SBR doesn't work?  
It is a hardware implementation specific issue. GPU end-point device
is inoperative after receiving SBR from the RP/SwitchPort. This quirk is
to prevent SBR.

> I'm not super
> confident that we perform resets correctly in general, and if the
> problem is an issue in Linux, it'd be nice to fix that.
We have not seen any issue with Linux SBR implementation.
>
>> This issue will be fixed in the next generation of hardware.
>>
>> Signed-off-by: Shanker Donthineni <sdonthineni@nvidia.com>
>> ---
>> Changes since v1:
>>  - Split patch into 2, code for handling _RST and SBR specific quirk
>>  - The RST based reset is called as a first-class mechanism in the reset code path
>>
>>  drivers/pci/quirks.c | 12 ++++++++++++
>>  1 file changed, 12 insertions(+)
>>
>> diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
>> index 8f47d139c381..e1216a8165df 100644
>> --- a/drivers/pci/quirks.c
>> +++ b/drivers/pci/quirks.c
>> @@ -3910,6 +3910,18 @@ static int delay_250ms_after_flr(struct pci_dev *dev, int probe)
>>       return 0;
>>  }
>>
>> +/*
>> + * Some Nvidia GPU devices do not work with bus reset, SBR needs to be
>> + * prevented for those affected devices.
>> + */
>> +static void quirk_nvidia_no_bus_reset(struct pci_dev *dev)
>> +{
>> +     if ((dev->device & 0xffc0) == 0x2340)
>> +             dev->dev_flags |= PCI_DEV_FLAGS_NO_BUS_RESET;
>> +}
>> +DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_NVIDIA, PCI_ANY_ID,
>> +                      quirk_nvidia_no_bus_reset);
> Can you move this next to the existing quirk_no_bus_reset(), and maybe
> even just call quirk_no_bus_reset(), e.g.,
>
>   if ((dev->device & 0xffc0) == 0x2340)
>     quirk_no_bus_reset(dev);
>
> It doesn't look connected to this spot.
>
Thanks, I will move next to the existing NO_BUS_RESET quirks.
>>  static const struct pci_dev_reset_methods pci_dev_reset_methods[] = {
>>       { PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_82599_SFP_VF,
>>                reset_intel_82599_sfp_virtfn },
>> --
>> 2.17.1
>>