Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   "Errabolu, Ramesh" <Ramesh.Errabolu@amd.com>
To:     Alex Deucher <alexdeucher@gmail.com>,
        "Kuehling, Felix" <Felix.Kuehling@amd.com>
CC:     amd-gfx list <amd-gfx@lists.freedesktop.org>,
        Maling list - DRI developers 
        <dri-devel@lists.freedesktop.org>,
        Bjorn Helgaas <bhelgaas@google.com>,
        Linux PCI <linux-pci@vger.kernel.org>,
        LKML <linux-kernel@vger.kernel.org>
Subject: RE: [PATCH] Whitelist AMD host bridge device(s) to enable P2P DMA
Thread-Topic: [PATCH] Whitelist AMD host bridge device(s) to enable P2P DMA
Thread-Index: AQHXjuSnWuaMWfMhHEmP//+Rviy3hKtusH+AgAAWvwCAAASPgIAAQzdw
Date:   Thu, 12 Aug 2021 01:09:20 +0000
Message-ID: <SA0PR12MB4399DE432D65D70E866E9FB6E3F99@SA0PR12MB4399.namprd12.prod.outlook.com>
References: <20210811191104.21919-1-Ramesh.Errabolu@amd.com>
 <CADnq5_OVA1fB5x6=CGrd_5O-i=P7snmoJaTyauF2RKuWjc8Gog@mail.gmail.com>
 <ba4ee532-d64a-72c5-7aab-2b86a75b8174@amd.com>
 <CADnq5_OmH6ZsL_Q_wO2B7dbaD_JFcaNbYFzd_Nv4kui1npkVPw@mail.gmail.com>
In-Reply-To: <CADnq5_OmH6ZsL_Q_wO2B7dbaD_JFcaNbYFzd_Nv4kui1npkVPw@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?Q?XXapuVccSSC2/OHYJUaGSz7RI0nzExWkmxvyVKSNdYu70qW21rCuUG111K?=
 =?iso-8859-1?Q?cnrNgf7no+t8jatD7c5IQ1tetzaC8rPHQM/5bJGRq1Zw5rEhw1ZZX+UgnT?=
 =?iso-8859-1?Q?O0CNnQYymElSpIEdQTrvvQHqYRodekYS75tKuhV62TOu/muMgD5zJM+63e?=
 =?iso-8859-1?Q?d7Rp57QO6pshj3zG6uJauu9qs3CdK8ooazn0hSCuC/UkB+BN9LxQJcz6D6?=
 =?iso-8859-1?Q?FZPYm8yS0Sr291FSBCdNc75PvWiTapSalQZrvxUP2MqCO5ANG9oLbhjENM?=
 =?iso-8859-1?Q?fx0IbXJOpS8VfZGnmGoPTQj9Gz6MfIYSizXQ9YhXv70+kzELRLD3gjm0/V?=
 =?iso-8859-1?Q?A9yuUrsbmNC6EMDt1kQksuGr4LkG15IVhXjLniHdeV9TO02Xh9EvzB72Rh?=
 =?iso-8859-1?Q?ObtnzkViHXelmxaiSvfaVxetHQpsz0g46GQSchE7uQp+EviPP7GOg/YHS7?=
 =?iso-8859-1?Q?3FV3AVgamH3tPpHzfA9XeqIqLo+QzgT89PUqW9bCTzLTLasbqHqJsLL4n0?=
 =?iso-8859-1?Q?s31qgE3ls3dDp38HC6gkjl8ReWdsbtOnttN9nCoX92kRGtPc0pNVwd7/jL?=
 =?iso-8859-1?Q?HfDhMc+wPvPbNpf/MWRHtGd1MtY6rPVcvG67CCezDDlLCTIsqq724kJzVR?=
 =?iso-8859-1?Q?Vuj9cK6zD++6Yz94wkQygSnbF9NdYeHQTYy7+PVD5qhcGcBGtsi6/Kb+VC?=
 =?iso-8859-1?Q?B0mh/4qBw+3qKL7ump5wS2HLg00tSfC0E4SzKuGZmCQJHNW1xIihcmggG3?=
 =?iso-8859-1?Q?HQ8YqmfYhNzpy+6A9EILCH3B96nFFkK95as5ayX4dhQWPqq9D6xmpf1cXj?=
 =?iso-8859-1?Q?xCiS4ae4XvU4ktdgQXItKvHfxkhVTBYR5YpidIsr0318sOOHhrAzP+24+P?=
 =?iso-8859-1?Q?Qcodlxup+SArMvYtSbsInKgYOK1f1xcnjIs6WbuerMhAWTUx8SAC9Y4/gR?=
 =?iso-8859-1?Q?3xK1d4Vnp4u4MTkD2kZ6Awn023JVOr76KgbgS1xR4AD5f1W631naDm32Z4?=
 =?iso-8859-1?Q?lxGeZvkyLIHvOURo99nWh0fi08dsrvzooO1vcSax1uA0QnINUfAp6KxsFw?=
 =?iso-8859-1?Q?alxFhfan+brnjfqGVrdQ6qkSUiy/kfKyIWk6+EfBOEBlj8j/Um78MwYKvI?=
 =?iso-8859-1?Q?m/C9LowwGQLv46zzxhjei7nwaPlkx6nJfETNqZK3gaaT1wRr/M+GItKiET?=
 =?iso-8859-1?Q?1vEQpmiSvw77MUksqrgMN7KJzi1gJbn+EN1WZqQdwk/TqO0NSmVgGhTozf?=
 =?iso-8859-1?Q?NB5PM0B76hlmeF9UY9celkGllEOnydQSFlz2ERQwvfyJ2op1PoVmPTX0BV?=
 =?iso-8859-1?Q?ibWUEmKg84JaZCXQ/AVpMhx7iYCACu6I9TeAYy7jIwqA8X0BKSjHqlt3To?=
 =?iso-8859-1?Q?IYCoG7CJvy?=
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: SA0PR12MB4399.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 22675c86-d427-4123-29b9-08d95d2dd0d3
X-MS-Exchange-CrossTenant-originalarrivaltime: 12 Aug 2021 01:09:20.7519
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: F17PEOjhrz3PjKnNCljbrbsEJ7oVEnBGsYRQWuihX3e5AWKaKbv9er7rcHrpGhTMhSOjcsdjGTLzkAveH17lCA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN6PR12MB2848
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

I will investigate it further. I am using DKMS-5.11 branch. The codebase I =
am using to build has the right definition i.e. allow P2PDMA for Zen CPU's.

Regards,
Ramesh

-----Original Message-----
From: Alex Deucher <alexdeucher@gmail.com>=20
Sent: Wednesday, August 11, 2021 4:07 PM
To: Kuehling, Felix <Felix.Kuehling@amd.com>
Cc: Errabolu, Ramesh <Ramesh.Errabolu@amd.com>; amd-gfx list <amd-gfx@lists=
.freedesktop.org>; Maling list - DRI developers <dri-devel@lists.freedeskto=
p.org>; Bjorn Helgaas <bhelgaas@google.com>; Linux PCI <linux-pci@vger.kern=
el.org>; LKML <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH] Whitelist AMD host bridge device(s) to enable P2P DMA

[CAUTION: External Email]

On Wed, Aug 11, 2021 at 4:50 PM Felix Kuehling <felix.kuehling@amd.com> wro=
te:
>
>
> Am 2021-08-11 um 3:29 p.m. schrieb Alex Deucher:
> > On Wed, Aug 11, 2021 at 3:11 PM Ramesh Errabolu <Ramesh.Errabolu@amd.co=
m> wrote:
> >> Current implementation will disallow P2P DMA if the participating=20
> >> devices belong to different root complexes. Implementation allows=20
> >> this default behavior to be overridden for whitelisted devices. The=20
> >> patch adds an AMD host bridge to be whitelisted
> > Why do we need this?  cpu_supports_p2pdma() should return true for=20
> > all AMD Zen CPUs.
>
> This is part of our on-going work to get P2P support upstream. We want=20
> to use pci_p2pdma_distance_many to determine whether P2P is possible=20
> between a pair of devices. This whitelist is used in this function.=20
> This will affect the P2P links reported in the topology and it will be=20
> double-checked in the BO mapping function to ensure a peer mapping is leg=
al.
>
> I think this change is a bit free of context at the moment, as we're=20
> still working on a few other loose ends for P2P support in our=20
> internal branch. I'm hoping we'll have a bigger patch series for=20
> upstreamable KFD P2P support ready in a few weeks. I also think we'll=20
> probably want to add a few more PCI IDs for other supported AMD root comp=
lexes.

We don't need to keep adding AMD root complexes.  You must be using an olde=
r kernel or something.  All root complexes on all Zen platforms support P2P=
 DMA.  See:

commit dea286bb71baded7d2fb4f090e3b9fd2c1ccac58
Author: Logan Gunthorpe <logang@deltatee.com>
Date:   Wed Jul 29 17:18:44 2020 -0600

    PCI/P2PDMA: Allow P2PDMA on AMD Zen and newer CPUs

    Allow P2PDMA if the CPU vendor is AMD and family is 0x17 (Zen) or great=
er.

    [bhelgaas: commit log, simplify #if/#else/#endif]
    Link: https://nam11.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F=
%2Flore.kernel.org%2Fr%2F20200729231844.4653-1-logang%40deltatee.com&amp;da=
ta=3D04%7C01%7CRamesh.Errabolu%40amd.com%7C7ffc96d473d4471889f808d95d0bf57b=
%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637643128220735445%7CUnknown%=
7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6M=
n0%3D%7C1000&amp;sdata=3DvmwJP2VwKPrXwl3ngUfwN%2BJu3JWeMP0ZZMDh29evW%2Bg%3D=
&amp;reserved=3D0
    Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
    Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
    Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
    Cc: Christian K=F6nig <christian.koenig@amd.com>
    Cc: Huang Rui <ray.huang@amd.com>

Alex


>
> Regards,
>   Felix
>
>
> >
> > Alex
> >
> >> Signed-off-by: Ramesh Errabolu <Ramesh.Errabolu@amd.com>
> >> ---
> >>  drivers/pci/p2pdma.c | 2 ++
> >>  1 file changed, 2 insertions(+)
> >>
> >> diff --git a/drivers/pci/p2pdma.c b/drivers/pci/p2pdma.c index=20
> >> 196382630363..7003bb9faf23 100644
> >> --- a/drivers/pci/p2pdma.c
> >> +++ b/drivers/pci/p2pdma.c
> >> @@ -305,6 +305,8 @@ static const struct pci_p2pdma_whitelist_entry {
> >>         {PCI_VENDOR_ID_INTEL,   0x2032, 0},
> >>         {PCI_VENDOR_ID_INTEL,   0x2033, 0},
> >>         {PCI_VENDOR_ID_INTEL,   0x2020, 0},
> >> +       /* AMD Host Bridge Devices */
> >> +       {PCI_VENDOR_ID_AMD,     0x1480, 0},
> >>         {}
> >>  };
> >>
> >> --
> >> 2.31.1
> >>