On Mon, Jan 29, 2024 at 11:23:16AM +0100, Pavel Machek wrote:
> Hi!
>
> > That's right and a reality you have to deal with on those small ARM
> > systems. The ARM architecture allows for systems that don't enforce
> > hardware coherency across the whole SoC and many of the small/cheap SoC
> > variants make use of this architectural feature.
> >
> > What this means is that the CPU caches aren't coherent when it comes to
> > DMA from other masters like the video capture units. There are two ways
> > to enforce DMA coherency on such systems:
> > 1. map the DMA buffers uncached on the CPU
> > 2. require explicit cache maintenance when touching DMA buffers with
> > the CPU
> >
> > Option 1 is what you see is happening in your setup, as it is simple,
> > straight-forward and doesn't require any synchronization points.
>
> Yeah, and it also does not work :-).
>
> Userspace gets the buffers, and it is not really equipped to work with
> them. For example, on pinephone, memcpy() crashes on uncached
> memory. I'm pretty sure user could have some kind of kernel-crashing
> fun if he passed the uncached memory to futex or something similar.
Uncached buffers are ubiquitous on arm/arm64 so there must be something
else going on. And there's nothing to equip for, it's just a memory
array you can access in any way you want (but very slowly).
How does it not work?
> > Option 2 could be implemented by allocating cached DMA buffers in the
> > V4L2 device and then executing the necessary cache synchronization in
> > qbuf/dqbuf when ownership of the DMA buffer changes between CPU and DMA
> > master. However this isn't guaranteed to be any faster, as the cache
> > synchronization itself is a pretty heavy-weight operation when you are
> > dealing with buffer that are potentially multi-megabytes in size.
>
> Yes, cache synchronization can be slow, but IIRC it was on order of
> milisecond in the worst case.. and copying megayte images is still
> slower than that.
>
> Note that it is faster to do read/write syscalls then deal with
> uncached memory. And userspace can't simply flush the caches and remap
> memory as cached.
You can't change the memory mapping, but you can flush the caches with
dma-buf. It's even required by the dma-buf documentation.
> v4l2 moved away from read/write "because it is slow" and switched to
> interface that is even slower than that. And libcamera exposes
> uncached memory to the user :-(.
There's also the number of copies to consider. If you were to use
read/write to display a frame on a framebuffer, you would use 4 copies
vs 2 with dma-buf.
Maxime
On Mon, Jan 29, 2024 at 11:32:08AM +0100, Maxime Ripard wrote:
> On Mon, Jan 29, 2024 at 11:23:16AM +0100, Pavel Machek wrote:
> > Hi!
> >
> > > That's right and a reality you have to deal with on those small ARM
> > > systems. The ARM architecture allows for systems that don't enforce
> > > hardware coherency across the whole SoC and many of the small/cheap SoC
> > > variants make use of this architectural feature.
> > >
> > > What this means is that the CPU caches aren't coherent when it comes to
> > > DMA from other masters like the video capture units. There are two ways
> > > to enforce DMA coherency on such systems:
> > > 1. map the DMA buffers uncached on the CPU
> > > 2. require explicit cache maintenance when touching DMA buffers with
> > > the CPU
> > >
> > > Option 1 is what you see is happening in your setup, as it is simple,
> > > straight-forward and doesn't require any synchronization points.
> >
> > Yeah, and it also does not work :-).
> >
> > Userspace gets the buffers, and it is not really equipped to work with
> > them. For example, on pinephone, memcpy() crashes on uncached
> > memory. I'm pretty sure user could have some kind of kernel-crashing
> > fun if he passed the uncached memory to futex or something similar.
>
> Uncached buffers are ubiquitous on arm/arm64 so there must be something
> else going on. And there's nothing to equip for, it's just a memory
> array you can access in any way you want (but very slowly).
>
> How does it not work?
I agree, this should just work (albeit possibly slowly). A crash is a
sign something needs to be fixed.
> > > Option 2 could be implemented by allocating cached DMA buffers in the
> > > V4L2 device and then executing the necessary cache synchronization in
> > > qbuf/dqbuf when ownership of the DMA buffer changes between CPU and DMA
> > > master. However this isn't guaranteed to be any faster, as the cache
> > > synchronization itself is a pretty heavy-weight operation when you are
> > > dealing with buffer that are potentially multi-megabytes in size.
> >
> > Yes, cache synchronization can be slow, but IIRC it was on order of
> > milisecond in the worst case.. and copying megayte images is still
> > slower than that.
Those numbers are platform-specific, you can't assume this to be true
everywhere.
> > Note that it is faster to do read/write syscalls then deal with
> > uncached memory. And userspace can't simply flush the caches and remap
> > memory as cached.
>
> You can't change the memory mapping, but you can flush the caches with
> dma-buf. It's even required by the dma-buf documentation.
>
> > v4l2 moved away from read/write "because it is slow" and switched to
> > interface that is even slower than that. And libcamera exposes
> > uncached memory to the user :-(.
>
> There's also the number of copies to consider. If you were to use
> read/write to display a frame on a framebuffer, you would use 4 copies
> vs 2 with dma-buf.
--
Regards,
Laurent Pinchart
Hi!
> > Yeah, and it also does not work :-).
> >
> > Userspace gets the buffers, and it is not really equipped to work with
> > them. For example, on pinephone, memcpy() crashes on uncached
> > memory. I'm pretty sure user could have some kind of kernel-crashing
> > fun if he passed the uncached memory to futex or something similar.
>
> Uncached buffers are ubiquitous on arm/arm64 so there must be something
> else going on. And there's nothing to equip for, it's just a memory
> array you can access in any way you want (but very slowly).
Not really. Not on anything modern.
ll/sc will not work, for example, than's on ARM.
https://en.wikipedia.org/wiki/Load-link/store-conditional
Transactional memory will not work, that was on x86. Powerpc has
cacheline clearing instruction.
And that's design, I'm pretty sure there are also numerous CPU errata.
Best regards,
Pavel
--
People of Russia, stop Putin before his war on Ukraine escalates.