Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
From:   "Stephen  Bates" <sbates@raithlin.com>
To:     Eric Wehage <Eric.Wehage@huawei.com>,
        Bjorn Helgaas <helgaas@kernel.org>, yu <kkndyu@gmail.com>
CC:     "linux-pci@vger.kernel.org" <linux-pci@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        Logan Gunthorpe <logang@deltatee.com>,
        Jonathan Cameron <jonathan.cameron@huawei.com>,
        Alexander Duyck <alexander.h.duyck@linux.intel.com>
Subject: Re: How to force RC to forward p2p TLPs
Thread-Topic: How to force RC to forward p2p TLPs
Thread-Index: AQHUnx5WMU0tGDh35kWKmEWxpQy9mqWZ9ggAgAE3IhI=
Date:   Wed, 2 Jan 2019 01:07:27 +0000
Message-ID: <YQXPR0101MB07284F610E52FB61030913F1AA8C0@YQXPR0101MB0728.CANPRD01.PROD.OUTLOOK.COM>
References: <CADayL-pxr_dpJykyiTeQDO13XmKPoKUs544zVrB2taHTVEx9Pw@mail.gmail.com>
 <20181229022923.GA159477@google.com>,<DD8FB053C05D2D4DB8AA6C93326178785997FDDC@sjceml521-mbs.china.huawei.com>
In-Reply-To: <DD8FB053C05D2D4DB8AA6C93326178785997FDDC@sjceml521-mbs.china.huawei.com>
Accept-Language: en-CA, en-US
Content-Language: en-CA
received-spf: None (protection.outlook.com: raithlin.com does not designate
 permitted sender hosts)
spamdiagnosticoutput: 1:99
spamdiagnosticmetadata: NSPM
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: 8b7e7170-7abd-4f45-c5f6-08d6704eaa09
X-MS-Exchange-CrossTenant-originalarrivaltime: 02 Jan 2019 01:07:27.9687
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 18519031-7ff4-4cbb-bbcb-c3252d330f4b
X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQXPR0101MB2246
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


Hi Yu

As I understand it you are asking about the case where the two PCIe End Poi=
nts (EPs) are connected below the same PCIe switch that is connected via it=
's Upstream Port (UPS) to a Root Port (RP) that is part of a Root  Complex =
(RC). Do I understand this correctly?

The ECR Eric mentioned aims to implement a platform neutral method for dete=
rmining if P2P *between* RPs is viable in a system. However I think that is=
 a bit different to your case (since your EPs are connected to the same RP =
via a PCIe switch).

What I *think* should be happening in your case when ACS is enabled is the =
the TLPs should be routed to the RP and from there to some translation agen=
t (often an IOMMU). If the IOMMU has an entry for the addresses in  the TLP=
s then it should attempt to do a translation and forward on the TLP. If the=
 IOMMU does not have a translation (or does not exist) then the behaviour i=
s probably ill-defined and very specific to the platform you are working on=
.

If the IOMMU does exist and does have a translation entry it still may not =
support routing the TLP back to the PCI hierarchy. This again would be plat=
form specific.

I guess what I am saying is the same as Logan and Eric. Once TLPs leave the=
 PCIe Root Port and enter the system (CPU or SOC) you leave the world of th=
e PCI/PCIe specification and the behaviour of those TLPs will be dependant =
 on the specifics of your platform.

Do note that if your platform and your EPs=A0supports ATS then I think the =
behaviour can be different again because then the P2P TLPs should be able t=
o use a translated address and you can setup ACS to allow for translated  a=
ddresses. However we have not had access to any ATS enabled systems to work=
 on this aspect of p2pdma yet. It's on the roadmap.

Cheers

Stephen


From: Eric Wehage <Eric.Wehage@huawei.com>
Sent: December 31, 2018 11:20 PM
To: Bjorn Helgaas; yu
Cc: linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; Logan Gunthorp=
e; Stephen Bates; Jonathan Cameron; Alexander Duyck
Subject: RE: How to force RC to forward p2p TLPs
=A0=20

There is no method to force an RC to forward requests p2p. RC's are also no=
t required to support p2p and whether they do or not is implementation spec=
ific. There is also currently no PCIe based standard to determine whether a=
n RC supports  p2p or in what capacity. Also, even if p2p is supported, it =
might not be supported in a system that enables Virtualization (the Virtual=
ization would have to setup p2p tables to allow it).

There is currently a PCIe ECR that has been submitted, but is undergoing re=
view, that defines a PCIe capability for bridge devices (Root Ports and Swi=
tch Downstream Ports) that would allow them to advertise what level of p2p =
is supported by the device.

Eric Wehage
Huawei Principal Engineer, PCIe

-----Original Message-----
From: Bjorn Helgaas [mailto:helgaas@kernel.org]
Sent: Friday, December 28, 2018 6:29 PM
To: yu <kkndyu@gmail.com>
Cc: linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; Logan Gunthorp=
e <logang@deltatee.com>; Stephen Bates <sbates@raithlin.com>; Jonathan Came=
ron <jonathan.cameron@huawei.com>; Eric Wehage <Eric.Wehage@huawei.com>; Al=
exander Duyck <alexander.h.duyck@linux.intel.com>
Subject: Re: How to force RC to forward p2p TLPs

[+cc Logan, Stephen, Jonathan, Eric, Alex]

On Sat, Dec 22, 2018 at 12:50:19PM +0800, yu wrote:
> Hi all,
>=20
> We have a PCIE card which has a PEX8732 switch on-board,=A0 and there=20
> are two endpoint SOCs like graphic decoder behind the switch, and by=20
> default the ACS is enabled in 8732.
>=20
> We use the p2p DMA to transfer data between these two endpoint SOCs,=20
> and if the host server is not enable ACS in BIOS, the p2p works well,=20
> but when ACS is enabled in BIOS, the p2p is always failed. With the=20
> help of a protocol analyzer, we can see that the TLP is redirected to=20
> RC, and RC just discard it.
>=20
> I tried to find how to make RC forward redirected TLP to its original=20
> target, but nothing found, it seems this is highly related to the RC=20
> vendors.
>=20
> In the PCIE 4.0 spec, the section of the RC behavior of the p2p=20
> request redirect=A0 said that ''implementation-specific logic within the=
=20
> RC that determines whether the request is directed towards its=20
> original target, or blocked as an ACS Violation error. the algorithms=20
> and specific controls for making this determination are not=20
> architected by this spec''.
>=20
>=20
> So is there some spec or document to describe how to set the RC? Any=20
> suggestion is appreciated.

Not that I'm aware of, but the folks I cc'd would know a lot more about thi=
s area.

Bjorn
      =