Subject: Re: [RFC PATCH v4 0/2] RDMA/rxe: Add dma-buf support
To:     Shunsuke Mie <mie@igel.co.jp>
Cc:     Jason Gunthorpe <jgg@ziepe.ca>, Zhu Yanjun <zyjzyj2000@gmail.com>,
        Alex Deucher <alexander.deucher@amd.com>,
        Daniel Vetter <daniel.vetter@ffwll.ch>,
        Doug Ledford <dledford@redhat.com>,
        Jianxin Xiong <jianxin.xiong@intel.com>,
        Leon Romanovsky <leon@kernel.org>,
        Maor Gottlieb <maorg@nvidia.com>,
        Sean Hefty <sean.hefty@intel.com>,
        Sumit Semwal <sumit.semwal@linaro.org>,
        dri-devel@lists.freedesktop.org, linaro-mm-sig@lists.linaro.org,
        linux-media@vger.kernel.org,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        linux-rdma <linux-rdma@vger.kernel.org>,
        Damian Hobson-Garcia <dhobsong@igel.co.jp>,
        Takanari Hayama <taki@igel.co.jp>,
        Tomohito Esaki <etom@igel.co.jp>
References: <20211122110817.33319-1-mie@igel.co.jp>
 <CANXvt5oB8_2sDGccSiTMqeLYGi3Vuo-6NnHJ9PGgZZMv=fnUVw@mail.gmail.com>
 <20211207171447.GA6467@ziepe.ca>
 <CANXvt5rCayOcengPr7Z_aFmJaXwWj9VcWZbaHnuHj6=2CkPndA@mail.gmail.com>
 <20211210124204.GG6467@ziepe.ca>
 <880e25ad-4fe9-eacd-a971-993eaea37fc4@amd.com>
 <20211210132656.GH6467@ziepe.ca>
 <d25b2895-63b6-158d-ff73-f05e437e0f91@amd.com>
 <CANXvt5rzmEnF3Gph4U6NT-XzJhV6zqyay1g7dHkTgH=Aqc6Geg@mail.gmail.com>
From:   =?UTF-8?Q?Christian_K=c3=b6nig?= <christian.koenig@amd.com>
Message-ID: <51bcad64-8df8-b9a3-0aef-d88eb70fdbba@amd.com>
Date:   Tue, 14 Dec 2021 09:53:46 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.14.0
In-Reply-To: <CANXvt5rzmEnF3Gph4U6NT-XzJhV6zqyay1g7dHkTgH=Aqc6Geg@mail.gmail.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Content-Language: en-US
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?M3pLeTExY3ZvWUNRaTY0NXdlMnRZajFKV1hkdmtDTUxLL2YxTGlGY3ZlN1dr?=
 =?utf-8?B?RjRyRytiZGtaUXIyVFNycGJVOE1lak5lUmdZS21WYzgyc2N3N3JyU0RIY1hN?=
 =?utf-8?B?MGxJRDBmU1BSNTRHZTFOZ0tDQkdnbmlRZ0xFcTJWdnA2UDhFTXJHQkEwMWZW?=
 =?utf-8?B?L2RlaXZLVjEzVldNb05ib3pRc081V0NPRncyUzFDMDgxZEZpNjlIdk1pbVFL?=
 =?utf-8?B?L3h4QzZTR1VSbHRucmN1aUsyY095a2NCckEyME8yWmVSTWZ5dFZNd3Z1YzVI?=
 =?utf-8?B?ZC9taS9tOUJNM3NyZnpqL2h6OHdWTTRtR3dQc21OZTgyd3FlRWd4M3Z0aDlL?=
 =?utf-8?B?T0s0RWhnaml2L2tEQm1BbFpsTWJicmx0VnRwT1FDTlRrNGxZVWNJMnVMRFM5?=
 =?utf-8?B?RVNrcmJFWGhPWWpnRU5jVDBuUEV5OXlrOWpFWVF4RllXK3UyVVR4RVJvcGhV?=
 =?utf-8?B?Z0ViR05mbEkzMkVZblkvRzAySjJzYkdDblJkWVlzSTVyUEVqd01teDBpM3B5?=
 =?utf-8?B?NkhUNStpSEYyUC9Va2REdC9oZDFSQzQ1TXM4UTgvdER5YWRxZUJWbERFVjRv?=
 =?utf-8?B?cXVDNFJic1V3aUdSZVA3NXRrQ1ovalJ6U2R6anhqSGpTcDRPTWZybG5UZFkz?=
 =?utf-8?B?azJ6Yi93bUk4VmREZjZQOXBnRUZnTkc2czhqMEdyWDZjNUZrYmxReVdOWXZm?=
 =?utf-8?B?RWU2N1VDZnZGSlhhNXJlVlJ1L240WkxMTGtma1BYT1BvaTlSZUtrWWZLY0VF?=
 =?utf-8?B?OFUwaFptOU5kQzR3ZkVWT0RqYm1QSkh2cnBRM3pzZnJnV0pKRkxkMzZEOXA2?=
 =?utf-8?B?cTZjeWxjTlFUQWlGeXpyNFpuaHhYYjN2aFJ1bEo1dEhMZ242TGlxZXV5ck9Z?=
 =?utf-8?B?dWlmVjl3OGpQMmdRRk96aTV2bGtjMjVKQTN6K3ZJVzkrQ1NtZkFQMkRCbUMx?=
 =?utf-8?B?ZERER3ZXaW9jYXVOVGlxUlVVdjFmdURvYTdZdlYxQnlLMjFxUnYvRExuZVdE?=
 =?utf-8?B?d05WdjJjdVZoVS8yTzJTKzZEZFRSVHMveFJYYThPS3JIMXl0eUxZRm8xaWNJ?=
 =?utf-8?B?RUZ5bzBYUDFxRmpCcGJaOGtjOWFhZkF1YnFhMDBLbUlLc2R4c3FRcXNJUWRP?=
 =?utf-8?B?MlBrVXo4K2t3UExkNjBIdWJteHo1WWVDQ0tkQWRTYmp6MmwraVNkdUVra1or?=
 =?utf-8?B?cHM4ZTdjb3FOZStRQURSaWwzWDNUU0FVVnRMUzlMQysrcnBBc1NjRVQxdjJB?=
 =?utf-8?B?bkZMMXZRbG8zd0RBZTNIN09RNEM4d2hoY3hrL1BKZjIzcHQyamNGNFJXa0tP?=
 =?utf-8?B?Um9uTHRxTUtrVUFHSVVSb216UnkzSkhxVkJrYWJuQTZjdlI3aThScWl2Tmd1?=
 =?utf-8?B?NDZKWHJmK3ZpOWwwd28rVUtuM040TG5wanlYNzJ4M3E0UkpMd0hvTElnWVFa?=
 =?utf-8?B?MllOUGdFbnQ0VGhBRTBjNVFHRU90NVJGQW9mVndkUGw1Y2dubVNmeTkxdkxL?=
 =?utf-8?B?ZnpHMnc1VlVsNlIvMzc2ZFVYRlJzc0V3d3JUem1ZNkhqZm5zVnMvTk0zMHdk?=
 =?utf-8?B?L0U0bTFpa1lNZzcxMVA5VWxJcERseDh5ZmI1SEptQnNJdlJ5OHg3eXN4SERV?=
 =?utf-8?B?SWZ5K3EvOFhsT1JSaWtpV20wMW5Ka1llTFhkSnBEb2FzR2diZiszSUQvSkxB?=
 =?utf-8?B?ejdTc2wxc2RwU0hGdzd4UkQ1Mk95NzRMNGUzRWZqbm1tTHljZklDK2VMektx?=
 =?utf-8?B?Vy90Zm5CdnlzbGNnK2lSUG9vTFIzd3JZbFUvNmdvcGlZYnRWZThmQTNjMXli?=
 =?utf-8?B?cm9MZldSeUIwV09EQ2FUVXNpNXFCZUE1QTJsSzdwVmRwcERzZ2cwMklRc1Iv?=
 =?utf-8?B?ODIxOEFlWWxBQ2NZb1ErU2JGbUMybWpzTDBZaUFPS1FDNUhGRjVyREJ2QVNp?=
 =?utf-8?B?WU5aU0J6OTlKTEdBOFFSS2JHbG9MTzh5NHhhS1htUlo5d1N5ZzgwWFdzazVt?=
 =?utf-8?B?dkxWYlRtbU5wSlRMMStNK2ZLRHo1Y1JBNmRiSE9POCtnTjZScTAyQ2ZZUUY5?=
 =?utf-8?B?NFdpOXkwSEhib2MzVmRQUEd3ZEtJSlZrZzFjYTBMdmwvSDZNYTFvMEl0elpP?=
 =?utf-8?Q?Dg+w=3D?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 48504754-f042-4ebd-5184-08d9bedf43da
X-MS-Exchange-CrossTenant-AuthSource: MWHPR1201MB0192.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Dec 2021 08:53:57.6870
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 0ScayXQET05Ytt1iHx6H0HVoiilR0QRd2GhVDN0r4KEzHoEE5MqB9f1KRSjmEbnX
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR12MB1678
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Am 13.12.21 um 12:18 schrieb Shunsuke Mie:
> 2021年12月10日(金) 22:29 Christian König <christian.koenig@amd.com>:
>> Am 10.12.21 um 14:26 schrieb Jason Gunthorpe:
>>> On Fri, Dec 10, 2021 at 01:47:37PM +0100, Christian König wrote:
>>>> Am 10.12.21 um 13:42 schrieb Jason Gunthorpe:
>>>>> On Fri, Dec 10, 2021 at 08:29:24PM +0900, Shunsuke Mie wrote:
>>>>>> Hi Jason,
>>>>>> Thank you for replying.
>>>>>>
>>>>>> 2021年12月8日(水) 2:14 Jason Gunthorpe <jgg@ziepe.ca>:
>>>>>>> On Fri, Dec 03, 2021 at 12:51:44PM +0900, Shunsuke Mie wrote:
>>>>>>>> Hi maintainers,
>>>>>>>>
>>>>>>>> Could you please review this patch series?
>>>>>>> Why is it RFC?
>>>>>>>
>>>>>>> I'm confused why this is useful?
>>>>>>>
>>>>>>> This can't do copy from MMIO memory, so it shouldn't be compatible
>>>>>>> with things like Gaudi - does something prevent this?
>>>>>> I think if an export of the dma-buf supports vmap, CPU is able to access the
>>>>>> mmio memory.
>>>>>>
>>>>>> Is it wrong? If this is wrong, there is no advantages this changes..
>>>>> I don't know what the dmabuf folks did, but yes, it is wrong.
>>>>>
>>>>> IOMEM must be touched using only special accessors, some platforms
>>>>> crash if you don't do this. Even x86 will crash if you touch it with
>>>>> something like an XMM optimized memcpy.
>>>>>
>>>>> Christian? If the vmap succeeds what rules must the caller use to
>>>>> access the memory?
>>>> See dma-buf-map.h and especially struct dma_buf_map.
>>>>
>>>> MMIO memory is perfectly supported here and actually the most common case.
>>> Okay that looks sane, but this rxe RFC seems to ignore this
>>> completely. It stuffs the vaddr directly into a umem which goes to all
>>> manner of places in the driver.
>>>
>>> ??
>> Well, yes that can go boom pretty quickly.
> Sorry, I was wrong. The dma_buf_map treats both iomem and vaddr region, but
> this RFC only supports vaddr. Advantage of the partial support is we can use the
> vaddr dma-buf in RXE without changing a rxe data copy implementation.

Well that is most likely not a good idea.

For example buffers for GPU drivers can be placed in both MMIO memory 
and system memory.

If you don't want to provoke random failures you *MUST* be able to 
handle both if you want to use this.

Regards,
Christian.

>
> An example of a dma-buf pointing to a vaddr is some gpu drivers use RAM for
> VRAM and we can get dma-buf for the region that indicates vaddr regions.
> Specifically, the gpu driver using gpu/drm/drm_gem_cma_helper.c is one such
> example.
>
>> Not sure what they want to use this for.
> I'd like to use RDMA with RXE for that memory region.
>
> Best,
> Shunsuke
>> Christian.
>>
>>> Jason