Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
From:   Jason Gunthorpe <jgg@mellanox.com>
To:     Logan Gunthorpe <logang@deltatee.com>
CC:     Christoph Hellwig <hch@lst.de>, Jerome Glisse <jglisse@redhat.com>,
        "linux-mm@kvack.org" <linux-mm@kvack.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
        "Rafael J . Wysocki" <rafael@kernel.org>,
        Bjorn Helgaas <bhelgaas@google.com>,
        Christian Koenig <christian.koenig@amd.com>,
        Felix Kuehling <Felix.Kuehling@amd.com>,
        "linux-pci@vger.kernel.org" <linux-pci@vger.kernel.org>,
        "dri-devel@lists.freedesktop.org" <dri-devel@lists.freedesktop.org>,
        Marek Szyprowski <m.szyprowski@samsung.com>,
        Robin Murphy <robin.murphy@arm.com>,
        Joerg Roedel <jroedel@suse.de>,
        "iommu@lists.linux-foundation.org" <iommu@lists.linux-foundation.org>
Subject: Re: [RFC PATCH 3/5] mm/vma: add support for peer to peer to device
 vma
Thread-Topic: [RFC PATCH 3/5] mm/vma: add support for peer to peer to device
 vma
Thread-Index: AQHUt/rA/dLikqWEmEaIytHIBNLPlqXGkyOAgAAJwICAAAX+AIAAEreAgAAFCQCAAAk3gIAABX0AgAATFYCAAA25AIAAGRqAgAAykICAAD3dAIAAukqAgAAK3wCAAAOzAIAAEXyAgAANnoCAABFLgIAAnPCAgAC1FQCAAATigIAACeKA
Date:   Thu, 31 Jan 2019 19:54:59 +0000
Message-ID: <20190131195453.GE7548@mellanox.com>
References: <20190130080006.GB29665@lst.de>
 <20190130190651.GC17080@mellanox.com>
 <840256f8-0714-5d7d-e5f5-c96aec5c2c05@deltatee.com>
 <20190130195900.GG17080@mellanox.com>
 <35bad6d5-c06b-f2a3-08e6-2ed0197c8691@deltatee.com>
 <20190130215019.GL17080@mellanox.com>
 <07baf401-4d63-b830-57e1-5836a5149a0c@deltatee.com>
 <20190131081355.GC26495@lst.de> <20190131190202.GC7548@mellanox.com>
 <e4fd743f-61cb-a443-bc53-9a1c036ebe8c@deltatee.com>
In-Reply-To: <e4fd743f-61cb-a443-bc53-9a1c036ebe8c@deltatee.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-messagesentrepresentingtype: 1
x-originating-ip: [174.3.196.123]
x-ms-publictraffictype: Email
x-microsoft-exchange-diagnostics: 1;DBBPR05MB6538;6:kIapYD9pRMMtOltM8UsCrgNS5TK4Hcmo57gmQANKSQmzNiD8mv+1eUBhn71sWYqvN5Mnn7Gt7sIe51N5QXD/kWU5PerEcc4/tF0WCuAiH+g697tP0+1aoWorS+V2YDvhHtIB6keVht+njr3exuZTsRGrBwyFCmBVde9b4GgpT/1vaA2TaBNbCv70qdVyR23spjwsWWMdu2+5Kc3nj3ALcM5VraSvtzR9A1qCYpg7JRQ2SrX93qa5O3WF/bKKXUJBVcTU73Pp/tcvZ6DVQs8BUNLoHvLZgJL1hYyur2nHEai9FeBicXMORLlNITbPzNlKYMeoTTUhJf9kWZx6NhKH7ZVrNH3A1yF+2T42ERDgPeBQtG0j33Y/fI5Uc8TV+1Wpeuz696aCH7JMTJMQsu0YCX9Vp7gw+XseCeWNRNCHwIaYJF2tKPaBO286mAY5PdgSzfOBsdlxiaUzUutoCIQm6Q==;5:zp829Chukjpc/dmb3wsLoUVkEda2Y/Fz1kBEuLTFGja957sOy/28OEp1umctS9rLH7ExUCfp92ux7nSM5dwlVqscUuaq3+ifgPx3SaZ9ydQrcX5rE24D5SAY/V8GnVT67xwL0S4xOd7DjMbTm6JD2SkOjom0YHmM6u+7koSej+00LL6t+gbxDUbXl4+YgnAoOQFAwZFZEv+d7MyUm8h3GQ==;7:xXdu+z+rFdqAXTgU0TY/MLv7xKzp4VjQJP0a0d81KGHznfVLV5+E8oijAW/l2fRa7vTQnabkyMLa40ChKTfXFRyRSffm4tqiZ60nSFk7z9KKLhKCDGat3Mj+B5thzGNLQzWmLHrLJU9Wbv19ppJXSg==
x-ms-office365-filtering-correlation-id: 457401f2-83d9-483e-d36e-08d687b5fb41
x-ms-office365-filtering-ht: Tenant
x-microsoft-antispam: BCL:0;PCL:0;RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600110)(711020)(4605077)(4618075)(2017052603328)(7153060)(7193020);SRVR:DBBPR05MB6538;
x-ms-traffictypediagnostic: DBBPR05MB6538:
x-microsoft-antispam-prvs: <DBBPR05MB6538830010A5167680BC8A63CF910@DBBPR05MB6538.eurprd05.prod.outlook.com>
x-forefront-prvs: 09347618C4
x-forefront-antispam-report: SFV:NSPM;SFS:(10009020)(39860400002)(366004)(376002)(136003)(396003)(346002)(189003)(199004)(97736004)(6916009)(2906002)(76176011)(66066001)(93886005)(26005)(316002)(4326008)(33656002)(1076003)(53936002)(217873002)(36756003)(6246003)(52116002)(6116002)(3846002)(256004)(14444005)(86362001)(7416002)(71190400001)(71200400001)(6436002)(7736002)(478600001)(106356001)(476003)(229853002)(6506007)(2616005)(25786009)(11346002)(446003)(6486002)(54906003)(53546011)(68736007)(99286004)(186003)(486006)(81156014)(81166006)(14454004)(8676002)(102836004)(105586002)(8936002)(386003)(6512007)(305945005);DIR:OUT;SFP:1101;SCL:1;SRVR:DBBPR05MB6538;H:DBBPR05MB6426.eurprd05.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;MX:1;A:1;
received-spf: None (protection.outlook.com: mellanox.com does not designate
 permitted sender hosts)
x-ms-exchange-senderadcheck: 1
x-microsoft-antispam-message-info: wCWPMAz7RJ//53pPp8RHOUzbX4cRmPez8Ni7a5uPkvB9uoC+4oNzB2aZ4NR7R9WtFeEs9HnWhFR+HCBkWf2FR1Lrq0z1nsaGl1VoW5SKBCOD2G/tC6Kan/1bUEncuGLZApgQkeJbKM22Qpa6OzQ7AuNLOBNkXTNitpOHfzicD5Eiv/dw9aFAla8iDiBVpGuOKrjSZY6e8PYFv+3cZfsG+A/bNQ5pxwy07ZpkzdxJVfphyBb/t/u7Bg3mc71ZuVI/AhuflrJQrnttwT63nRQmX7GiobSiK1ocxOkoWwO0C4s/XyPMhDyuPSUk6aXSLHxdnoXc6jq9WSlqc2eEZPCa+ubHKzRVK6RSOgfQlI8nT1yZDKlYioidnJzs7SDZtnRuw0XyIyEKE7tqGfseXZk2pSkzPfSm4GYVD09+rbSVf3E=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <01E72E6BF04AD94ABCA85A4BF90D23C0@eurprd05.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: Mellanox.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 457401f2-83d9-483e-d36e-08d687b5fb41
X-MS-Exchange-CrossTenant-originalarrivaltime: 31 Jan 2019 19:54:59.3709
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DBBPR05MB6538
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Thu, Jan 31, 2019 at 12:19:31PM -0700, Logan Gunthorpe wrote:
>=20
>=20
> On 2019-01-31 12:02 p.m., Jason Gunthorpe wrote:
> > I still think the right direction is to build on what Logan has done -
> > realize that he created a DMA-only SGL - make that a formal type of
> > the kernel and provide the right set of APIs to work with this type,
> > without being forced to expose struct page.
>=20
> > Basically invert the API flow - the DMA map would be done close to
> > GUP, not buried in the driver. This absolutely doesn't work for every
> > flow we have, but it does enable the ones that people seem to care
> > about when talking about P2P.
> > It also does present a path to solve some cases of the O_DIRECT
> > problems if the block stack can develop some way to know if an IO will
> > go down a DMA-only IO path or not... This seems less challenging that
> > auditing every SGL user for iomem safety??
>=20
>=20
> The DMA-only SGL will work for some use cases, but I think it's going to
> be a challenge for others. We care most about NVMe and, therefore, the
> block layer.

The exercise here is not to enable O_DIRECT for P2P, it is to allow
certain much simpler users to use P2P. We should not be saying that
someone has to solve these complicated problems in the entire block
stack just to make RDMA work. :(

If the block stack can use a 'dma sgl' or not, I don't know.

However, it does look like it fits these RDMA, GPU and VFIO cases
fairly well, and looks better than the hacky
sgl-but-really-special-p2p hack we have in RDMA today.

Jason