Received-SPF: pass (google.com: domain of linux-kernel+bounces-42542-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161;
Date: Mon, 29 Jan 2024 11:32:08 +0100
From: Maxime Ripard <mripard@redhat.com>
To: Pavel Machek <pavel@ucw.cz>
Cc: Lucas Stach <dev@lynxeye.de>, 
	kernel list <linux-kernel@vger.kernel.org>, Milan Zamazal <mzamazal@redhat.com>, 
	Christoph Hellwig <hch@lst.de>, iommu@lists.linux.dev, Will Deacon <will@kernel.org>, 
	catalin.marinas@arm.com, Bryan O'Donoghue <bryan.odonoghue@linaro.org>, 
	Andrey Konovalov <andrey.konovalov.ynk@gmail.com>, Laurent Pinchart <laurent.pinchart@ideasonboard.com>, 
	kieran.bingham@ideasonboard.com, Hans de Goede <hdegoede@redhat.com>
Subject: Re: Re: Uncached buffers from CMA DMA heap on some Arm devices?
Message-ID: <xd5ofun26gfdgn7hig3gipj5rgojqyuouwmii7xecgrbzyliil@y6rufxtwdmrc>
References: <87bk9ahex7.fsf@redhat.com>
 <d2ff8df896d8a167e9abf447ae184ce2f5823852.camel@lynxeye.de>
 <Zbd8lOgVqfCrGUL7@duo.ucw.cz>
Precedence: bulk
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha512;
	protocol="application/pgp-signature"; boundary="rgbsaoykivldpxmx"
Content-Disposition: inline
In-Reply-To: <Zbd8lOgVqfCrGUL7@duo.ucw.cz>


--rgbsaoykivldpxmx
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Mon, Jan 29, 2024 at 11:23:16AM +0100, Pavel Machek wrote:
> Hi!
>=20
> > That's right and a reality you have to deal with on those small ARM
> > systems. The ARM architecture allows for systems that don't enforce
> > hardware coherency across the whole SoC and many of the small/cheap SoC
> > variants make use of this architectural feature.
> >=20
> > What this means is that the CPU caches aren't coherent when it comes to
> > DMA from other masters like the video capture units. There are two ways
> > to enforce DMA coherency on such systems:
> > 1. map the DMA buffers uncached on the CPU
> > 2. require explicit cache maintenance when touching DMA buffers with
> > the CPU
> >=20
> > Option 1 is what you see is happening in your setup, as it is simple,
> > straight-forward and doesn't require any synchronization points.
>=20
> Yeah, and it also does not work :-).
>=20
> Userspace gets the buffers, and it is not really equipped to work with
> them. For example, on pinephone, memcpy() crashes on uncached
> memory. I'm pretty sure user could have some kind of kernel-crashing
> fun if he passed the uncached memory to futex or something similar.

Uncached buffers are ubiquitous on arm/arm64 so there must be something
else going on. And there's nothing to equip for, it's just a memory
array you can access in any way you want (but very slowly).

How does it not work?

> > Option 2 could be implemented by allocating cached DMA buffers in the
> > V4L2 device and then executing the necessary cache synchronization in
> > qbuf/dqbuf when ownership of the DMA buffer changes between CPU and DMA
> > master. However this isn't guaranteed to be any faster, as the cache
> > synchronization itself is a pretty heavy-weight operation when you are
> > dealing with buffer that are potentially multi-megabytes in size.
>=20
> Yes, cache synchronization can be slow, but IIRC it was on order of
> milisecond in the worst case.. and copying megayte images is still
> slower than that.
>=20
> Note that it is faster to do read/write syscalls then deal with
> uncached memory. And userspace can't simply flush the caches and remap
> memory as cached.

You can't change the memory mapping, but you can flush the caches with
dma-buf. It's even required by the dma-buf documentation.

> v4l2 moved away from read/write "because it is slow" and switched to
> interface that is even slower than that. And libcamera exposes
> uncached memory to the user :-(.

There's also the number of copies to consider. If you were to use
read/write to display a frame on a framebuffer, you would use 4 copies
vs 2 with dma-buf.

Maxime

--rgbsaoykivldpxmx
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iHUEABYKAB0WIQRcEzekXsqa64kGDp7j7w1vZxhRxQUCZbd+qAAKCRDj7w1vZxhR
xTHhAP9A4AyvYSGs0JorWHjsfmCICZAeJOyYv/T4wk5MHI8IzQEAmMs2T3pXgUVp
aoZS4N2vWOpFF9+yrzmpVk/QoucFHgg=
=dyAt
-----END PGP SIGNATURE-----

--rgbsaoykivldpxmx--