Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <6287f5f8-d9af-e03d-a2c8-ea8ddcbdc0d8@amd.com>
Date:   Thu, 23 Jun 2022 13:10:59 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.9.1
Subject: Re: DMA-buf and uncached system memory
Content-Language: en-US
To:     Lucas Stach <l.stach@pengutronix.de>,
        Pekka Paalanen <ppaalanen@gmail.com>
Cc:     "Sharma, Shashank" <Shashank.Sharma@amd.com>,
        lkml <linux-kernel@vger.kernel.org>,
        dri-devel <dri-devel@lists.freedesktop.org>,
        Nicolas Dufresne <nicolas@ndufresne.ca>,
        linaro-mm-sig@lists.linaro.org,
        Sumit Semwal <sumit.semwal@linaro.org>,
        linux-media <linux-media@vger.kernel.org>
References: <91ff0bbb-ea3a-2663-3453-dea96ccd6dd8@amd.com>
 <YCuPhOT4GhY3RR/6@phenom.ffwll.local>
 <9178e19f5c0e141772b61b759abaa0d176f902b6.camel@ndufresne.ca>
 <CAPj87rPYQNkgVEdHECQcHcYe2nCpgF3RYQKk_=wwhvJSxwHXCg@mail.gmail.com>
 <c6e65ee1-531e-d72c-a6a6-da7149e34f18@amd.com>
 <20220623101326.18beeab3@eldfell>
 <954d0a9b-29ef-52ef-f6ca-22d7e6aa3f4d@amd.com>
 <4b69f9f542d6efde2190b73c87096e87fa24d8ef.camel@pengutronix.de>
 <adc626ec-ff5a-5c06-44ce-09111be450cd@amd.com>
 <fbb228cd78e9bebd7e7921c19e0c4c09d0891f23.camel@pengutronix.de>
 <e691bccc-171d-f674-2817-13a945970f4a@amd.com>
 <95cca943bbfda6af07339fb8d2dc7f4da3aa0280.camel@pengutronix.de>
 <05814ddb-4f3e-99d8-025a-c31db7b2c46b@amd.com>
 <708e27755317a7650ca08ba2e4c14691ac0d6ba2.camel@pengutronix.de>
From:   =?UTF-8?Q?Christian_K=c3=b6nig?= <christian.koenig@amd.com>
In-Reply-To: <708e27755317a7650ca08ba2e4c14691ac0d6ba2.camel@pengutronix.de>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?enF3SHNpTkVFeEdqeWFYVlZ3YkpjU3paY1BvWWRMblFSWWpzYU9jZlMxemdi?=
 =?utf-8?B?VVFhRm9BdjNlMDhZaWkxdXY2WHZMTTcraG96MUlXZmRFMFJqSU1SVnBSNys0?=
 =?utf-8?B?c1RORGJoVDBibjNZeThJU0lVbUUvcTA4Ui92NWs0R3pHeWNaTm5RZDk0ajR1?=
 =?utf-8?B?YVFhUG80ZGEvMGRzb2VlK3EzcUM1SXh6UzhTdmExTVV6cnNYSS9OTE1jZnV3?=
 =?utf-8?B?OGNic2c4NUx5c0pmd09Id2RqUjhhamZlRGp2d3hNRGVkb3U5d3E0KzBxcDJt?=
 =?utf-8?B?WWN2NktURjFSLy82ekQwRnV3RXdFd0JHWGRvVlBiZURQWlJmYVNsbXRWUHhi?=
 =?utf-8?B?L3NnWllPVXFmcEFjR2FJNkE2WktGQU5RV1E0aXM1ME5Gd3RzcUNyTXQvMUJz?=
 =?utf-8?B?TzNtcVYrb2psaTB2cE8vd2l6a0ZOczZGK3U3T3M4VHJ0T015Y0xlQ0IwWWx2?=
 =?utf-8?B?ZXhmZjBLSlRLKy9Jd0xJM2RDYzdQdHlhQ2h1L3BrTm9sWjdnbm44WFZOWUdC?=
 =?utf-8?B?TFRqelpYUWk1czVuYk1kcjQ3aGFObEVQRFJXU2x0QkRaejZsWlFxMDcwSzZN?=
 =?utf-8?B?UVMvVnNCVEZTT2xwQ05OYzZhZkZzdFV6VTBjazRlMk9YR3NCRkRYSWhISWVx?=
 =?utf-8?B?RjR6NGxrMkxETjU1OGtMTEwrTnlWRXFBRU5GYmRqSzh3QWJUQzVqelJONFpz?=
 =?utf-8?B?aEhYbldNVG41c1ZuTVFWakM2MFlxRmNyS1BERytRWjlldTV2dVFNdXB5TzE2?=
 =?utf-8?B?d0FqMFhWZ01VOHptZUNaQnlXRjIxUXpwaGo5c3BWaWVQeVBlSnVseXVNWm5y?=
 =?utf-8?B?WnMxWlZHTkVhRWZnQXI2MFNqaElqNjZrY3ViekFhUkpUbUNEM09XbERTNTN1?=
 =?utf-8?B?WkVhQ2ExeXZFTmgzRnpTc3hIWXBoUVJuY1BSWmprWkkxZUU3ZGRDZDVrdmFv?=
 =?utf-8?B?MG9NUXp2N2pqMzljdWVXdTc5aFlUR1lrR3dmcnNaUzV5Ri9PY3o3MERYbHpQ?=
 =?utf-8?B?QWljNWhLaEh5eDJmSmtOL1c0TWJ6OXkzaytlbzZLN2lhcXhTbUZjcmkxWmRX?=
 =?utf-8?B?TkZkY1ZwNURlS1IxZHFWRnUxcEo5YklDRFd6NWNicVVYZklZbkF0Qy8rcDl4?=
 =?utf-8?B?azJaZHRFZ1liVG0vU25uRTluU3lVMEtwYld0OTZOOVNpeVhWL1dXZVJHT1c2?=
 =?utf-8?B?MFU5Z3JQcHE3azFXT0k0b0RIUFZ4NnFBNUpFbDNVcWdRa1A2b3FFSXp5anhV?=
 =?utf-8?B?aGFEdmZUbUxWLzNHRkdKbFhtcDJRUVpjSkw2VmswZHhzN2Y5Sk1xWXNIRzRn?=
 =?utf-8?B?WjBNWS9DOVpWUTlNY0lpUEN0dmFrb0ViL2w2RjJBdDh1b2wwZU1zL29iZlNj?=
 =?utf-8?B?bjg1RlRNN1JIcE82VE5NL1BiVzJ2MTZUb3huclJlM1hONmJRdnhmY1RCRStC?=
 =?utf-8?B?VmVMejZ3Ly8wL1lXNTc5dFZzNkRHK09aMjB5S3htbHJ1TFlnSTR6emh0aVh2?=
 =?utf-8?B?QWxvMU5JdnBVa3pwbE5oRnBDMHo1VEZ0NkJnYzlyLzFSWEhVWWZTamVUMHpt?=
 =?utf-8?B?REVHYksyNzhDTnI1ME95YVplbDloOFlObkZKQWRuOGRib24xdWhoSWFwZlRn?=
 =?utf-8?B?Q05lb3NmZm5tYVFQeXd4cTZLTG1sNEJNNkV5OGN2YkJsdHFncDdyWko5V2sz?=
 =?utf-8?B?eURaajZYRWt0dkJJeWdndmk4elZPQkVOM0ZnNm1NOUh0d3d5VGU0My8vd3RZ?=
 =?utf-8?B?elNzQjNzWmhyZS9CMFNNcktFckprQkhnQzY1cmRKWEZib0xNaDBYd0g4OUhw?=
 =?utf-8?B?eXMwNU9aaVk5U3pxczhNOWY2UlFUQkMwNHlmeGYrSXlzbU5jNUdIMllCRjBW?=
 =?utf-8?B?enlvQnp5WGV5elNvK0QyOVRlcHdYZ1lMMDVzMTM5Z2IyNkl5MndjZHEyWTJl?=
 =?utf-8?B?K0tmVmMyY0lRbk01TzRFVFFxcmJuazN0aHZwQytpTm1NNGdvbWRqWmNPeEMw?=
 =?utf-8?B?NERWb1NhWnQ3M1E2MG5hbTdQb3NqMHlEU3pHdmxTWWZ6c0g1Z2J5SjBINVdo?=
 =?utf-8?B?WnBOUlExa1EvVEMzVkU5d3dxQlNuY05PRE1qTlZSNlJFRXhPdk5xMmNrU0U1?=
 =?utf-8?Q?njyMa5CXnnkMy31/5rgX3/z07?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: a482e23b-0270-4bae-2d9a-08da55091066
X-MS-Exchange-CrossTenant-AuthSource: BN8PR12MB3587.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Jun 2022 11:11:04.6888
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: vhMi7gSCjWF3f23AzBSmGTUiP04FF2A3E6L/5L5aC2v4WvIiV8Eayb2II58gOeLo
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4877
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_NONE,
        RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS,T_SCC_BODY_TEXT_LINE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Am 23.06.22 um 12:13 schrieb Lucas Stach:
> [SNIP]
>>> On most of the multimedia
>>> targeted ARM SoCs being unable to snoop the cache is the norm, not an
>>> exception.
>>>
>>>> See for example on AMD/Intel hardware most of the engines can perfectly
>>>> deal with cache coherent memory accesses. Only the display engines can't.
>>>>
>>>> So on import time we can't even say if the access can be coherent and
>>>> snoop the CPU cache or not because we don't know how the imported
>>>> DMA-buf will be used later on.
>>>>
>>> So for those mixed use cases, wouldn't it help to have something
>>> similar to the dma_sync in the DMA-buf API, so your scanout usage can
>>> tell the exporter that it's going to do non-snoop access and any dirty
>>> cache lines must be cleaned? Signaling this to the exporter would allow
>>> to skip the cache maintenance if the buffer is in CPU uncached memory,
>>> which again is a default case for the ARM SoC world.
>> Well for the AMD and Intel use cases we at least have the opportunity to
>> signal cache flushing, but I'm not sure if that counts for everybody.
>>
> Sure, all the non-coherent arches have some way to do the cache
> maintenance in some explicit way. Non coherent and no cache maintenance
> instruction would be a recipe for desaster. ;)
>
>> What we would rather do for those use cases is an indicator on the
>> DMA-buf if the underlying backing store is CPU cached or not. The
>> importer can then cleanly reject the use cases where it can't support
>> CPU cache snooping.
>>
>> This then results in the normal fallback paths which we have anyway for
>> those use cases because DMA-buf sharing is not always possible.
>>
> That's a very x86 centric world view you have there. 99% of DMA-buf
> uses on those cheap ARM SoCs is non-snooping. We can not do any
> fallbacks here, as the whole graphics world on those SoCs with their
> different IP cores mixed together depends on DMA-buf sharing working
> efficiently even when the SoC is mostly non coherent.
>
> In fact DMA-buf sharing works fine on most of those SoCs because
> everyone just assumes that all the accelerators don't snoop, so the
> memory shared via DMA-buf is mostly CPU uncached. It only falls apart
> for uses like the UVC cameras, where the shared buffer ends up being
> CPU cached.

Well then the existing DMA-buf framework is not what you want to use for 
this.

> Non-coherent without explicit domain transfer points is just not going
> to work. So why can't we solve the issue for DMA-buf in the same way as
> the DMA API already solved it years ago: by adding the equivalent of
> the dma_sync calls that do cache maintenance when necessary? On x86 (or
> any system where things are mostly coherent) you could still no-op them
> for the common case and only trigger cache cleaning if the importer
> explicitly says that is going to do a non-snooping access.

Because DMA-buf is a framework for buffer sharing between cache coherent 
devices which don't signal transitions.

We intentionally didn't implemented any of the dma_sync_* functions 
because that would break the intended use case.

You can of course use DMA-buf in an incoherent environment, but then you 
can't expect that this works all the time.

This is documented behavior and so far we have bluntly rejected any of 
the complains that it doesn't work on most ARM SoCs and I don't really 
see a way to do this differently.

Regards,
Christian.

>
> Regards,
> Lucas
>