On Thu, May 12, 2022 at 2:18 PM Marco Elver <[email protected]> wrote:
>
> I guess the question is, is it the norm that per_cpu() retrieves data
> that can legally be modified concurrently, or not. If not, and in most
> cases it's a bug, the annotations should be here.
>
> Paul, was there any guidance/documentation on this, but I fail to find
> it right now? (access-marking.txt doesn't say much about per-CPU
> data.)
Normally, whenever we add a READ_ONCE(), we are supposed to add a comment.
We could make an exception for per_cpu_once(), because the comment
would be centralized
at per_cpu_once() definition.
We will be stuck with READ_ONCE() in places we are using
per_cpu_ptr(), for example
in dev_fetch_sw_netstats()
diff --git a/net/core/dev.c b/net/core/dev.c
index 1461c2d9dec8099a9a2d43a704b4c6cb0375f480..b66470291d7b7e6c33161093d71e40587f9ed838
100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -10381,10 +10381,13 @@ void dev_fetch_sw_netstats(struct
rtnl_link_stats64 *s,
stats = per_cpu_ptr(netstats, cpu);
do {
start = u64_stats_fetch_begin_irq(&stats->syncp);
- tmp.rx_packets = stats->rx_packets;
- tmp.rx_bytes = stats->rx_bytes;
- tmp.tx_packets = stats->tx_packets;
- tmp.tx_bytes = stats->tx_bytes;
+ /* These values can change under us.
+ * READ_ONCE() pair with too many write sides...
+ */
+ tmp.rx_packets = READ_ONCE(stats->rx_packets);
+ tmp.rx_bytes = READ_ONCE(stats->rx_bytes);
+ tmp.tx_packets = READ_ONCE(stats->tx_packets);
+ tmp.tx_bytes = READ_ONCE(stats->tx_bytes);
} while (u64_stats_fetch_retry_irq(&stats->syncp, start));
s->rx_packets += tmp.rx_packets;
On Thu, May 12, 2022 at 02:31:48PM -0700, Eric Dumazet wrote:
> On Thu, May 12, 2022 at 2:18 PM Marco Elver <[email protected]> wrote:
>
> >
> > I guess the question is, is it the norm that per_cpu() retrieves data
> > that can legally be modified concurrently, or not. If not, and in most
> > cases it's a bug, the annotations should be here.
> >
> > Paul, was there any guidance/documentation on this, but I fail to find
> > it right now? (access-marking.txt doesn't say much about per-CPU
> > data.)
>
> Normally, whenever we add a READ_ONCE(), we are supposed to add a comment.
I am starting to think that comments are even more necessary for unmarked
accesses to shared variables, with the comments setting out why the
compiler cannot mess things up. ;-)
> We could make an exception for per_cpu_once(), because the comment
> would be centralized
> at per_cpu_once() definition.
This makes a lot of sense to me.
> We will be stuck with READ_ONCE() in places we are using
> per_cpu_ptr(), for example
> in dev_fetch_sw_netstats()
If this is strictly statistics, data_race() is another possibility.
But it does not constrain the compiler at all.
Thanx, Paul
> diff --git a/net/core/dev.c b/net/core/dev.c
> index 1461c2d9dec8099a9a2d43a704b4c6cb0375f480..b66470291d7b7e6c33161093d71e40587f9ed838
> 100644
> --- a/net/core/dev.c
> +++ b/net/core/dev.c
> @@ -10381,10 +10381,13 @@ void dev_fetch_sw_netstats(struct
> rtnl_link_stats64 *s,
> stats = per_cpu_ptr(netstats, cpu);
> do {
> start = u64_stats_fetch_begin_irq(&stats->syncp);
> - tmp.rx_packets = stats->rx_packets;
> - tmp.rx_bytes = stats->rx_bytes;
> - tmp.tx_packets = stats->tx_packets;
> - tmp.tx_bytes = stats->tx_bytes;
> + /* These values can change under us.
> + * READ_ONCE() pair with too many write sides...
> + */
> + tmp.rx_packets = READ_ONCE(stats->rx_packets);
> + tmp.rx_bytes = READ_ONCE(stats->rx_bytes);
> + tmp.tx_packets = READ_ONCE(stats->tx_packets);
> + tmp.tx_bytes = READ_ONCE(stats->tx_bytes);
> } while (u64_stats_fetch_retry_irq(&stats->syncp, start));
>
> s->rx_packets += tmp.rx_packets;