On Wed, Sep 13, 2023 at 11:04:00AM +0200, Yann Sionneau wrote:
> On 13/09/2023 03:03, Jan Bottorff wrote:
...
> > + /*
> > + * To guarantee data written by the current core is visible to
> > + * all cores, a write barrier is required. This needs to be
> > + * before an interrupt causes execution on another core.
> > + * For ARM processors, this needs to be a DSB barrier.
> > + */
> > + wmb();
>
> Apart from the commit message it looks good to me.
>
> If I understand correctly without this wmb() it is possible that the writes
> to dev->msg_write_idx , dev->msg_read_idx = 0 etc would not yet be visible
> to another CPU running the ISR handler right after enabling those.
If this is the case, shouldn't we rather use READ_ONCE()/WRITE_ONCE() where
appropriate?
--
With Best Regards,
Andy Shevchenko
On 13/09/2023 13:22, Andy Shevchenko wrote:
> On Wed, Sep 13, 2023 at 11:04:00AM +0200, Yann Sionneau wrote:
>> On 13/09/2023 03:03, Jan Bottorff wrote:
> ...
>
>>> + /*
>>> + * To guarantee data written by the current core is visible to
>>> + * all cores, a write barrier is required. This needs to be
>>> + * before an interrupt causes execution on another core.
>>> + * For ARM processors, this needs to be a DSB barrier.
>>> + */
>>> + wmb();
>> Apart from the commit message it looks good to me.
>>
>> If I understand correctly without this wmb() it is possible that the writes
>> to dev->msg_write_idx , dev->msg_read_idx = 0 etc would not yet be visible
>> to another CPU running the ISR handler right after enabling those.
> If this is the case, shouldn't we rather use READ_ONCE()/WRITE_ONCE() where
> appropriate?
>
To my knowledge the READ_ONCE()/WRITE_ONCE() only imply the use of
volatile to access memory thus preventing the compiler to do weird
optimizations like merging store/loads, moving store/loads, removing
them etc
They don't imply a memory barrier.
Some systems need a memory barrier, to emit a "fence" like instruction,
so that the pipeline stalls waiting for the store to "finish", for
systems where the writes are posted.
Regards,
--
Yann
On 13/09/2023 13:32, Yann Sionneau wrote:
>
> On 13/09/2023 13:22, Andy Shevchenko wrote:
>> On Wed, Sep 13, 2023 at 11:04:00AM +0200, Yann Sionneau wrote:
>>> On 13/09/2023 03:03, Jan Bottorff wrote:
>> ...
>>
>>>> + /*
>>>> + * To guarantee data written by the current core is visible to
>>>> + * all cores, a write barrier is required. This needs to be
>>>> + * before an interrupt causes execution on another core.
>>>> + * For ARM processors, this needs to be a DSB barrier.
>>>> + */
>>>> + wmb();
>>> Apart from the commit message it looks good to me.
>>>
>>> If I understand correctly without this wmb() it is possible that the
>>> writes
>>> to dev->msg_write_idx , dev->msg_read_idx = 0 etc would not yet be
>>> visible
>>> to another CPU running the ISR handler right after enabling those.
>> If this is the case, shouldn't we rather use READ_ONCE()/WRITE_ONCE()
>> where
>> appropriate?
>>
> To my knowledge the READ_ONCE()/WRITE_ONCE() only imply the use of
> volatile to access memory thus preventing the compiler to do weird
> optimizations like merging store/loads, moving store/loads, removing
> them etc
>
> They don't imply a memory barrier.
>
> Some systems need a memory barrier, to emit a "fence" like
> instruction, so that the pipeline stalls waiting for the store to
> "finish", for systems where the writes are posted.
>
> Regards,
>
Well, for the READ_ONCE() actually I'm wrong, it's overloaded for Alpha
and arm64 https://elixir.bootlin.com/linux/latest/C/ident/__READ_ONCE
--
Yann
On 9/13/2023 4:54 AM, Yann Sionneau wrote:
>
> On 13/09/2023 13:32, Yann Sionneau wrote:
>>
>> On 13/09/2023 13:22, Andy Shevchenko wrote:
...
>> To my knowledge the READ_ONCE()/WRITE_ONCE() only imply the use of
>> volatile to access memory thus preventing the compiler to do weird
>> optimizations like merging store/loads, moving store/loads, removing
>> them etc
>>
>> They don't imply a memory barrier.
>>
>> Some systems need a memory barrier, to emit a "fence" like
>> instruction, so that the pipeline stalls waiting for the store to
>> "finish", for systems where the writes are posted.
>>
>> Regards,
>>
> Well, for the READ_ONCE() actually I'm wrong, it's overloaded for Alpha
> and arm64 https://elixir.bootlin.com/linux/latest/C/ident/__READ_ONCE
>
Hi Yann,
I'll make a new submission with an improved commit message.
I looked at that ARM definition for READ_ONCE/WRITE_ONCE, and those
didn't really look like they they addressed this kind of cross-core
write visibility, it looked more like it was about avoiding read/write
tearing. Read acquire and write release can sometimes replace explicit
barrier instructions, but felt adding an explicit barrier was a simpler
solution. If this were highly performance critical code, then the
benefits of avoiding explicit barriers would be worth a lot more effort.
I know the ARM Barrier docs at
https://developer.arm.com/documentation/genc007826/latest/ have an
example under "Sending Interrupts and Barriers", page 19, that is very
specific that a DSB barrier is required in cases where you write to
normal memory on one core and then trigger an interrupt, and the normal
memory is read on another core. It explicitly says that even if a memory
write to strongly ordered memory (device memory) triggers the interrupt,
you would need a DSB barrier. That example seemed like a very good match
with what the DW i2c driver was doing.
I also found the video at https://www.youtube.com/watch?v=2I8OHacills
had some useful explanations of when barriers are needed. It was a
little puzzling in the video, it seemed to say a DMB barrier would be
sufficient for this case, even though the ARM barrier docs say a DSB is
needed. The DMB only stalls execution of memory accesses, the DSB stalls
execution of all instructions. The Linux wmb macro is a DSB barrier on ARM.
Jan