LinuxLists.cc - [PATCH 00/11] Introduce Simple atomic and non-atomic counters

2020-09-25 23:50:50

Subject: [PATCH 00/11] Introduce Simple atomic and non-atomic counters

This patch series is a result of discussion at the refcount_t BOF
the Linux Plumbers Conference. In this discussion, we identified
a need for looking closely and investigating atomic_t usages in
the kernel when it is used strictly as a counter without it
controlling object lifetimes and state changes.

There are a number of atomic_t usages in the kernel where atomic_t api
is used strictly for counting and not for managing object lifetime. In
some cases, atomic_t might not even be needed.

The purpose of these counters is twofold: 1. clearly differentiate
atomic_t counters from atomic_t usages that guard object lifetimes,
hence prone to overflow and underflow errors. It allows tools that scan
for underflow and overflow on atomic_t usages to detect overflow and
underflows to scan just the cases that are prone to errors. 2. provides
non-atomic counters for cases where atomic isn't necessary.

Simple atomic and non-atomic counters api provides interfaces for simple
atomic and non-atomic counters that just count, and don't guard resource
lifetimes. Counters will wrap around to 0 when it overflows and should
not be used to guard resource lifetimes, device usage and open counts
that control state changes, and pm states.

Using counter_atomic to guard lifetimes could lead to use-after free
when it overflows and undefined behavior when used to manage state
changes and device usage/open states.

This patch series introduces Simple atomic and non-atomic counters.
Counter atomic ops leverage atomic_t and provide a sub-set of atomic_t
ops.

In addition this patch series converts a few drivers to use the new api.
The following criteria is used for select variables for conversion:

1. Variable doesn't guard object lifetimes, manage state changes e.g:
device usage counts, device open counts, and pm states.
2. Variable is used for stats and counters.
3. The conversion doesn't change the overflow behavior.

Changes since RFC:
-- Thanks for reviews and reviewed-by, and Acked-by tags. Updated
the patches with the tags.
-- Addressed Kees's comments:
1. Non-atomic counters renamed to counter_simple32 and counter_simple64
to clearly indicate size.
2. Added warning for counter_simple* usage and it should be used only
when there is no need for atomicity.
3. Renamed counter_atomic to counter_atomic32 to clearly indicate size.
4. Renamed counter_atomic_long to counter_atomic64 and it now uses
atomic64_t ops and indicates size.
5. Test updated for the API renames.
6. Added helper functions for test results printing
7. Verified that the test module compiles in kunit env. and test
module can be loaded to run the test.
8. Updated Documentation to reflect the intent to make the API
restricted so it can never be used to guard object lifetimes
and state management. I left _return ops for now, inc_return
is necessary for now as per the discussion we had on this topic.
-- Updated driver patches with API name changes.
-- We discussed if binder counters can be non-atomic. For now I left
them the same as the RFC patch - using counter_atomic32
-- Unrelated to this patch series:
The patch series review uncovered improvements could be made to
test_async_driver_probe and vmw_vmci/vmci_guest. I will track
these for fixing later.

Shuah Khan (11):
counters: Introduce counter_simple* and counter_atomic* counters
selftests:lib:test_counters: add new test for counters
drivers/base: convert deferred_trigger_count and probe_count to
counter_atomic32
drivers/base/devcoredump: convert devcd_count to counter_atomic32
drivers/acpi: convert seqno counter_atomic32
drivers/acpi/apei: convert seqno counter_atomic32
drivers/android/binder: convert stats, transaction_log to
counter_atomic32
drivers/base/test/test_async_driver_probe: convert to use
counter_atomic32
drivers/char/ipmi: convert stats to use counter_atomic32
drivers/misc/vmw_vmci: convert num guest devices counter to
counter_atomic32
drivers/edac: convert pci counters to counter_atomic32

Documentation/core-api/counters.rst | 174 +++++++++
MAINTAINERS | 8 +
drivers/acpi/acpi_extlog.c | 5 +-
drivers/acpi/apei/ghes.c | 5 +-
drivers/android/binder.c | 41 +--
drivers/android/binder_internal.h | 3 +-
drivers/base/dd.c | 19 +-
drivers/base/devcoredump.c | 5 +-
drivers/base/test/test_async_driver_probe.c | 23 +-
drivers/char/ipmi/ipmi_msghandler.c | 9 +-
drivers/char/ipmi/ipmi_si_intf.c | 9 +-
drivers/edac/edac_pci.h | 5 +-
drivers/edac/edac_pci_sysfs.c | 28 +-
drivers/misc/vmw_vmci/vmci_guest.c | 9 +-
include/linux/counters.h | 350 +++++++++++++++++++
lib/Kconfig | 10 +
lib/Makefile | 1 +
lib/test_counters.c | 276 +++++++++++++++
tools/testing/selftests/lib/Makefile | 1 +
tools/testing/selftests/lib/config | 1 +
tools/testing/selftests/lib/test_counters.sh | 5 +
21 files changed, 913 insertions(+), 74 deletions(-)
create mode 100644 Documentation/core-api/counters.rst
create mode 100644 include/linux/counters.h
create mode 100644 lib/test_counters.c
create mode 100755 tools/testing/selftests/lib/test_counters.sh

--
2.25.1

2020-09-25 23:50:55

by Shuah Khan

[permalink] [raw]

Subject: [PATCH 05/11] drivers/acpi: convert seqno counter_atomic32

2020-09-25 23:50:57

by Shuah Khan

[permalink] [raw]

Subject: [PATCH 10/11] drivers/misc/vmw_vmci: convert num guest devices counter to counter_atomic32

2020-09-25 23:52:08

by Shuah Khan

[permalink] [raw]

Subject: [PATCH 09/11] drivers/char/ipmi: convert stats to use counter_atomic32

counter_atomic* is introduced to be used when a variable is used as
a simple counter and doesn't guard object lifetimes. This clearly
differentiates atomic_t usages that guard object lifetimes.

counter_atomic* variables will wrap around to 0 when it overflows and
should not be used to guard resource lifetimes, device usage and
open counts that control state changes, and pm states.

atomic_t variables used for stats are atomic counters. Overflow will
wrap around and reset the stats and no change with the conversion.

Convert them to use counter_atomic32.

Signed-off-by: Shuah Khan <[email protected]>
---
drivers/char/ipmi/ipmi_msghandler.c | 9 +++++----
drivers/char/ipmi/ipmi_si_intf.c | 9 +++++----
2 files changed, 10 insertions(+), 8 deletions(-)

diff --git a/drivers/char/ipmi/ipmi_msghandler.c b/drivers/char/ipmi/ipmi_msghandler.c
index 737c0b6b24ea..36c0b1be22fb 100644
--- a/drivers/char/ipmi/ipmi_msghandler.c
+++ b/drivers/char/ipmi/ipmi_msghandler.c
@@ -34,6 +34,7 @@
#include <linux/uuid.h>
#include <linux/nospec.h>
#include <linux/vmalloc.h>
+#include <linux/counters.h>

#define IPMI_DRIVER_VERSION "39.2"

@@ -584,7 +585,7 @@ struct ipmi_smi {
struct ipmi_my_addrinfo addrinfo[IPMI_MAX_CHANNELS];
bool channels_ready;

- atomic_t stats[IPMI_NUM_STATS];
+ struct counter_atomic32 stats[IPMI_NUM_STATS];

/*
* run_to_completion duplicate of smb_info, smi_info
@@ -630,9 +631,9 @@ static LIST_HEAD(smi_watchers);
static DEFINE_MUTEX(smi_watchers_mutex);

#define ipmi_inc_stat(intf, stat) \
- atomic_inc(&(intf)->stats[IPMI_STAT_ ## stat])
+ counter_atomic32_inc(&(intf)->stats[IPMI_STAT_ ## stat])
#define ipmi_get_stat(intf, stat) \
- ((unsigned int) atomic_read(&(intf)->stats[IPMI_STAT_ ## stat]))
+ ((unsigned int) counter_atomic32_read(&(intf)->stats[IPMI_STAT_ ## stat]))

static const char * const addr_src_to_str[] = {
"invalid", "hotmod", "hardcoded", "SPMI", "ACPI", "SMBIOS", "PCI",
@@ -3448,7 +3449,7 @@ int ipmi_add_smi(struct module *owner,
INIT_LIST_HEAD(&intf->cmd_rcvrs);
init_waitqueue_head(&intf->waitq);
for (i = 0; i < IPMI_NUM_STATS; i++)
- atomic_set(&intf->stats[i], 0);
+ counter_atomic32_set(&intf->stats[i], 0);

mutex_lock(&ipmi_interfaces_mutex);
/* Look for a hole in the numbers. */
diff --git a/drivers/char/ipmi/ipmi_si_intf.c b/drivers/char/ipmi/ipmi_si_intf.c
index 77b8d551ae7f..0909a3461f05 100644
--- a/drivers/char/ipmi/ipmi_si_intf.c
+++ b/drivers/char/ipmi/ipmi_si_intf.c
@@ -43,6 +43,7 @@
#include "ipmi_si_sm.h"
#include <linux/string.h>
#include <linux/ctype.h>
+#include <linux/counters.h>

/* Measure times between events in the driver. */
#undef DEBUG_TIMING
@@ -237,7 +238,7 @@ struct smi_info {
bool dev_group_added;

/* Counters and things for the proc filesystem. */
- atomic_t stats[SI_NUM_STATS];
+ struct counter_atomic32 stats[SI_NUM_STATS];

struct task_struct *thread;

@@ -245,9 +246,9 @@ struct smi_info {
};

#define smi_inc_stat(smi, stat) \
- atomic_inc(&(smi)->stats[SI_STAT_ ## stat])
+ counter_atomic32_inc(&(smi)->stats[SI_STAT_ ## stat])
#define smi_get_stat(smi, stat) \
- ((unsigned int) atomic_read(&(smi)->stats[SI_STAT_ ## stat]))
+ ((unsigned int) counter_atomic32_read(&(smi)->stats[SI_STAT_ ## stat]))

#define IPMI_MAX_INTFS 4
static int force_kipmid[IPMI_MAX_INTFS];
@@ -2013,7 +2014,7 @@ static int try_smi_init(struct smi_info *new_smi)
atomic_set(&new_smi->req_events, 0);
new_smi->run_to_completion = false;
for (i = 0; i < SI_NUM_STATS; i++)
- atomic_set(&new_smi->stats[i], 0);
+ counter_atomic32_set(&new_smi->stats[i], 0);

new_smi->interrupt_disabled = true;
atomic_set(&new_smi->need_watch, 0);
--
2.25.1

2020-09-25 23:54:02

by Kees Cook

[permalink] [raw]

Subject: Re: [PATCH 00/11] Introduce Simple atomic and non-atomic counters

On Fri, Sep 25, 2020 at 05:47:14PM -0600, Shuah Khan wrote:
> -- Addressed Kees's comments:
> 1. Non-atomic counters renamed to counter_simple32 and counter_simple64
> to clearly indicate size.
> 2. Added warning for counter_simple* usage and it should be used only
> when there is no need for atomicity.
> 3. Renamed counter_atomic to counter_atomic32 to clearly indicate size.
> 4. Renamed counter_atomic_long to counter_atomic64 and it now uses
> atomic64_t ops and indicates size.
> 5. Test updated for the API renames.
> 6. Added helper functions for test results printing
> 7. Verified that the test module compiles in kunit env. and test
> module can be loaded to run the test.

Thanks for all of this!

> 8. Updated Documentation to reflect the intent to make the API
> restricted so it can never be used to guard object lifetimes
> and state management. I left _return ops for now, inc_return
> is necessary for now as per the discussion we had on this topic.

I still *really* do not want dec_return() to exist. That is asking for
trouble. I'd prefer inc_return() not exist either, but I can live with
it. ;)

--
Kees Cook

2020-09-26 00:15:03

by Shuah Khan

[permalink] [raw]

Subject: Re: [PATCH 00/11] Introduce Simple atomic and non-atomic counters

On 9/25/20 5:52 PM, Kees Cook wrote:
> On Fri, Sep 25, 2020 at 05:47:14PM -0600, Shuah Khan wrote:
>> -- Addressed Kees's comments:
>> 1. Non-atomic counters renamed to counter_simple32 and counter_simple64
>> to clearly indicate size.
>> 2. Added warning for counter_simple* usage and it should be used only
>> when there is no need for atomicity.
>> 3. Renamed counter_atomic to counter_atomic32 to clearly indicate size.
>> 4. Renamed counter_atomic_long to counter_atomic64 and it now uses
>> atomic64_t ops and indicates size.
>> 5. Test updated for the API renames.
>> 6. Added helper functions for test results printing
>> 7. Verified that the test module compiles in kunit env. and test
>> module can be loaded to run the test.
>
> Thanks for all of this!
>
>> 8. Updated Documentation to reflect the intent to make the API
>> restricted so it can never be used to guard object lifetimes
>> and state management. I left _return ops for now, inc_return
>> is necessary for now as per the discussion we had on this topic.
>
> I still *really* do not want dec_return() to exist. That is asking for
> trouble. I'd prefer inc_return() not exist either, but I can live with
> it. ;)
>

Thanks. I am equally concerned about adding anything that can be used to
guard object lifetimes. So I will make sure this set won't expand and
plan to remove dec_return() if we don't find any usages.

thanks,
-- Shuah

2020-09-26 00:19:16

by Corey Minyard

[permalink] [raw]

Subject: Re: [PATCH 09/11] drivers/char/ipmi: convert stats to use counter_atomic32

On Fri, Sep 25, 2020 at 05:47:23PM -0600, Shuah Khan wrote:
> counter_atomic* is introduced to be used when a variable is used as
> a simple counter and doesn't guard object lifetimes. This clearly
> differentiates atomic_t usages that guard object lifetimes.
>
> counter_atomic* variables will wrap around to 0 when it overflows and
> should not be used to guard resource lifetimes, device usage and
> open counts that control state changes, and pm states.
>
> atomic_t variables used for stats are atomic counters. Overflow will
> wrap around and reset the stats and no change with the conversion.
>
> Convert them to use counter_atomic32.
>
> Signed-off-by: Shuah Khan <[email protected]>

Reviewed-by: Corey Minyard <[email protected]>

I assume for this conversion that the plan is to eliminate atomic_t
completely and convert all atomic counters used for object lifetime to
struct kref? The new naming is certainly more clear and I'm happy with
this change.

-corey

> ---
> drivers/char/ipmi/ipmi_msghandler.c | 9 +++++----
> drivers/char/ipmi/ipmi_si_intf.c | 9 +++++----
> 2 files changed, 10 insertions(+), 8 deletions(-)
>
> diff --git a/drivers/char/ipmi/ipmi_msghandler.c b/drivers/char/ipmi/ipmi_msghandler.c
> index 737c0b6b24ea..36c0b1be22fb 100644
> --- a/drivers/char/ipmi/ipmi_msghandler.c
> +++ b/drivers/char/ipmi/ipmi_msghandler.c
> @@ -34,6 +34,7 @@
> #include <linux/uuid.h>
> #include <linux/nospec.h>
> #include <linux/vmalloc.h>
> +#include <linux/counters.h>
>
> #define IPMI_DRIVER_VERSION "39.2"
>
> @@ -584,7 +585,7 @@ struct ipmi_smi {
> struct ipmi_my_addrinfo addrinfo[IPMI_MAX_CHANNELS];
> bool channels_ready;
>
> - atomic_t stats[IPMI_NUM_STATS];
> + struct counter_atomic32 stats[IPMI_NUM_STATS];
>
> /*
> * run_to_completion duplicate of smb_info, smi_info
> @@ -630,9 +631,9 @@ static LIST_HEAD(smi_watchers);
> static DEFINE_MUTEX(smi_watchers_mutex);
>
> #define ipmi_inc_stat(intf, stat) \
> - atomic_inc(&(intf)->stats[IPMI_STAT_ ## stat])
> + counter_atomic32_inc(&(intf)->stats[IPMI_STAT_ ## stat])
> #define ipmi_get_stat(intf, stat) \
> - ((unsigned int) atomic_read(&(intf)->stats[IPMI_STAT_ ## stat]))
> + ((unsigned int) counter_atomic32_read(&(intf)->stats[IPMI_STAT_ ## stat]))
>
> static const char * const addr_src_to_str[] = {
> "invalid", "hotmod", "hardcoded", "SPMI", "ACPI", "SMBIOS", "PCI",
> @@ -3448,7 +3449,7 @@ int ipmi_add_smi(struct module *owner,
> INIT_LIST_HEAD(&intf->cmd_rcvrs);
> init_waitqueue_head(&intf->waitq);
> for (i = 0; i < IPMI_NUM_STATS; i++)
> - atomic_set(&intf->stats[i], 0);
> + counter_atomic32_set(&intf->stats[i], 0);
>
> mutex_lock(&ipmi_interfaces_mutex);
> /* Look for a hole in the numbers. */
> diff --git a/drivers/char/ipmi/ipmi_si_intf.c b/drivers/char/ipmi/ipmi_si_intf.c
> index 77b8d551ae7f..0909a3461f05 100644
> --- a/drivers/char/ipmi/ipmi_si_intf.c
> +++ b/drivers/char/ipmi/ipmi_si_intf.c
> @@ -43,6 +43,7 @@
> #include "ipmi_si_sm.h"
> #include <linux/string.h>
> #include <linux/ctype.h>
> +#include <linux/counters.h>
>
> /* Measure times between events in the driver. */
> #undef DEBUG_TIMING
> @@ -237,7 +238,7 @@ struct smi_info {
> bool dev_group_added;
>
> /* Counters and things for the proc filesystem. */
> - atomic_t stats[SI_NUM_STATS];
> + struct counter_atomic32 stats[SI_NUM_STATS];
>
> struct task_struct *thread;
>
> @@ -245,9 +246,9 @@ struct smi_info {
> };
>
> #define smi_inc_stat(smi, stat) \
> - atomic_inc(&(smi)->stats[SI_STAT_ ## stat])
> + counter_atomic32_inc(&(smi)->stats[SI_STAT_ ## stat])
> #define smi_get_stat(smi, stat) \
> - ((unsigned int) atomic_read(&(smi)->stats[SI_STAT_ ## stat]))
> + ((unsigned int) counter_atomic32_read(&(smi)->stats[SI_STAT_ ## stat]))
>
> #define IPMI_MAX_INTFS 4
> static int force_kipmid[IPMI_MAX_INTFS];
> @@ -2013,7 +2014,7 @@ static int try_smi_init(struct smi_info *new_smi)
> atomic_set(&new_smi->req_events, 0);
> new_smi->run_to_completion = false;
> for (i = 0; i < SI_NUM_STATS; i++)
> - atomic_set(&new_smi->stats[i], 0);
> + counter_atomic32_set(&new_smi->stats[i], 0);
>
> new_smi->interrupt_disabled = true;
> atomic_set(&new_smi->need_watch, 0);
> --
> 2.25.1
>

2020-09-26 02:07:02

by Shuah Khan

[permalink] [raw]

Subject: Re: [PATCH 09/11] drivers/char/ipmi: convert stats to use counter_atomic32

On 9/25/20 6:15 PM, Corey Minyard wrote:
> On Fri, Sep 25, 2020 at 05:47:23PM -0600, Shuah Khan wrote:
>> counter_atomic* is introduced to be used when a variable is used as
>> a simple counter and doesn't guard object lifetimes. This clearly
>> differentiates atomic_t usages that guard object lifetimes.
>>
>> counter_atomic* variables will wrap around to 0 when it overflows and
>> should not be used to guard resource lifetimes, device usage and
>> open counts that control state changes, and pm states.
>>
>> atomic_t variables used for stats are atomic counters. Overflow will
>> wrap around and reset the stats and no change with the conversion.
>>
>> Convert them to use counter_atomic32.
>>
>> Signed-off-by: Shuah Khan <[email protected]>
>
> Reviewed-by: Corey Minyard <[email protected]>
>

Thanks Corey.

> I assume for this conversion that the plan is to eliminate atomic_t
> completely and convert all atomic counters used for object lifetime to
> struct kref? The new naming is certainly more clear and I'm happy with
> this change.
>

No plans to replace or get rid of atomic_t/refcount_t ops. The reason is
to clearly differentiate atomic_t uses that don't guard object lifetimes
or state management, hence prone to overflow and underflow errors.

By clearly differentiating the ones guard the lifetimes and that don't
using this new counter interface, the existing tools that scan for
overflow/underflow conditions can filter out Counters API and look for
the variables that truly guard the lifetimes. Currently it is becoming
very hard to zero in on the errors with the noise.

Second reason is, atomic_t is overused. Non-atomic counters in this
API can be used for stats/counters that don't need atomocity.

Hope this helps.

thanks,
-- Shuah

2020-09-26 16:24:34

by Kees Cook

[permalink] [raw]

Subject: Re: [PATCH 00/11] Introduce Simple atomic and non-atomic counters

2020-09-26 16:33:46

by Kees Cook

[permalink] [raw]

Subject: Re: [PATCH 00/11] Introduce Simple atomic and non-atomic counters

On Fri, Sep 25, 2020 at 05:47:14PM -0600, Shuah Khan wrote:
> 7. Verified that the test module compiles in kunit env. and test
> module can be loaded to run the test.

I meant write it using KUnit interfaces (e.g. KUNIT_EXPECT*(),
kunit_test_suite(), etc):
https://www.kernel.org/doc/html/latest/dev-tools/kunit/

Though I see the docs are still not updated[1] to reflect the Kconfig
(CONFIG_foo_KUNIT_TEST) and file naming conventions (foo_kunit.c).

-Kees

[1] https://lore.kernel.org/lkml/[email protected]/

--
Kees Cook

2020-09-26 16:34:34

by Kees Cook

[permalink] [raw]

Subject: Re: [PATCH 00/11] Introduce Simple atomic and non-atomic counters

On Fri, Sep 25, 2020 at 06:13:37PM -0600, Shuah Khan wrote:
> On 9/25/20 5:52 PM, Kees Cook wrote:
> > On Fri, Sep 25, 2020 at 05:47:14PM -0600, Shuah Khan wrote:
> > > -- Addressed Kees's comments:
> > > 1. Non-atomic counters renamed to counter_simple32 and counter_simple64
> > > to clearly indicate size.
> > > 2. Added warning for counter_simple* usage and it should be used only
> > > when there is no need for atomicity.
> > > 3. Renamed counter_atomic to counter_atomic32 to clearly indicate size.
> > > 4. Renamed counter_atomic_long to counter_atomic64 and it now uses
> > > atomic64_t ops and indicates size.
> > > 5. Test updated for the API renames.
> > > 6. Added helper functions for test results printing
> > > 7. Verified that the test module compiles in kunit env. and test
> > > module can be loaded to run the test.
> >
> > Thanks for all of this!
> >
> > > 8. Updated Documentation to reflect the intent to make the API
> > > restricted so it can never be used to guard object lifetimes
> > > and state management. I left _return ops for now, inc_return
> > > is necessary for now as per the discussion we had on this topic.
> >
> > I still *really* do not want dec_return() to exist. That is asking for
> > trouble. I'd prefer inc_return() not exist either, but I can live with
> > it. ;)
> >
>
> Thanks. I am equally concerned about adding anything that can be used to
> guard object lifetimes. So I will make sure this set won't expand and
> plan to remove dec_return() if we don't find any usages.

I would like it much stronger than "if". dec_return() needs to be just
dec() and read(). It will not be less efficient (since they're both
inlines), but it _will_ create a case where the atomicity cannot be used
for ref counting. My point is that anything that _requires_ dec_return()
(or, frankly, inc_return()) is _not_ "just" a statistical counter. It
may not be a refcounter, but it relies on the inc/dec atomicity for some
reason beyond counting in once place and reporting it in another.

--
Kees Cook

2020-09-27 23:37:47

by Joel Fernandes

[permalink] [raw]

Subject: Re: [PATCH 00/11] Introduce Simple atomic and non-atomic counters

On Fri, Sep 25, 2020 at 05:47:14PM -0600, Shuah Khan wrote:
> This patch series is a result of discussion at the refcount_t BOF
> the Linux Plumbers Conference. In this discussion, we identified
> a need for looking closely and investigating atomic_t usages in
> the kernel when it is used strictly as a counter without it
> controlling object lifetimes and state changes.
>
> There are a number of atomic_t usages in the kernel where atomic_t api
> is used strictly for counting and not for managing object lifetime. In
> some cases, atomic_t might not even be needed.
>
> The purpose of these counters is twofold: 1. clearly differentiate
> atomic_t counters from atomic_t usages that guard object lifetimes,
> hence prone to overflow and underflow errors. It allows tools that scan
> for underflow and overflow on atomic_t usages to detect overflow and
> underflows to scan just the cases that are prone to errors. 2. provides
> non-atomic counters for cases where atomic isn't necessary.

Nice series :)

It appears there is no user of counter_simple in this series other than the
selftest. Would you be planning to add any conversions in the series itself,
for illustration of use? Sorry if I missed a usage.

Also how do we guard against atomicity of counter_simple RMW operations? Is
the implication that it should be guarded using other synchronization to
prevent lost-update problem?

Some more comments:

1. atomic RMW operations that have a return value are fully ordered. Would
you be adding support to counter_simple for such ordering as well, for
consistency?

2. I felt counter_atomic and counter_atomic64 would be nice equivalents to
the atomic and atomic64 naming currently used (i.e. dropping the '32').
However that is just my opinion and I am ok with either naming.

thanks!

- Joel

>
> Simple atomic and non-atomic counters api provides interfaces for simple
> atomic and non-atomic counters that just count, and don't guard resource
> lifetimes. Counters will wrap around to 0 when it overflows and should
> not be used to guard resource lifetimes, device usage and open counts
> that control state changes, and pm states.
>
> Using counter_atomic to guard lifetimes could lead to use-after free
> when it overflows and undefined behavior when used to manage state
> changes and device usage/open states.
>
> This patch series introduces Simple atomic and non-atomic counters.
> Counter atomic ops leverage atomic_t and provide a sub-set of atomic_t
> ops.
>
> In addition this patch series converts a few drivers to use the new api.
> The following criteria is used for select variables for conversion:
>
> 1. Variable doesn't guard object lifetimes, manage state changes e.g:
> device usage counts, device open counts, and pm states.
> 2. Variable is used for stats and counters.
> 3. The conversion doesn't change the overflow behavior.
>
> Changes since RFC:
> -- Thanks for reviews and reviewed-by, and Acked-by tags. Updated
> the patches with the tags.
> -- Addressed Kees's comments:
> 1. Non-atomic counters renamed to counter_simple32 and counter_simple64
> to clearly indicate size.
> 2. Added warning for counter_simple* usage and it should be used only
> when there is no need for atomicity.
> 3. Renamed counter_atomic to counter_atomic32 to clearly indicate size.
> 4. Renamed counter_atomic_long to counter_atomic64 and it now uses
> atomic64_t ops and indicates size.
> 5. Test updated for the API renames.
> 6. Added helper functions for test results printing
> 7. Verified that the test module compiles in kunit env. and test
> module can be loaded to run the test.
> 8. Updated Documentation to reflect the intent to make the API
> restricted so it can never be used to guard object lifetimes
> and state management. I left _return ops for now, inc_return
> is necessary for now as per the discussion we had on this topic.
> -- Updated driver patches with API name changes.
> -- We discussed if binder counters can be non-atomic. For now I left
> them the same as the RFC patch - using counter_atomic32
> -- Unrelated to this patch series:
> The patch series review uncovered improvements could be made to
> test_async_driver_probe and vmw_vmci/vmci_guest. I will track
> these for fixing later.
>
> Shuah Khan (11):
> counters: Introduce counter_simple* and counter_atomic* counters
> selftests:lib:test_counters: add new test for counters
> drivers/base: convert deferred_trigger_count and probe_count to
> counter_atomic32
> drivers/base/devcoredump: convert devcd_count to counter_atomic32
> drivers/acpi: convert seqno counter_atomic32
> drivers/acpi/apei: convert seqno counter_atomic32
> drivers/android/binder: convert stats, transaction_log to
> counter_atomic32
> drivers/base/test/test_async_driver_probe: convert to use
> counter_atomic32
> drivers/char/ipmi: convert stats to use counter_atomic32
> drivers/misc/vmw_vmci: convert num guest devices counter to
> counter_atomic32
> drivers/edac: convert pci counters to counter_atomic32
>
> Documentation/core-api/counters.rst | 174 +++++++++
> MAINTAINERS | 8 +
> drivers/acpi/acpi_extlog.c | 5 +-
> drivers/acpi/apei/ghes.c | 5 +-
> drivers/android/binder.c | 41 +--
> drivers/android/binder_internal.h | 3 +-
> drivers/base/dd.c | 19 +-
> drivers/base/devcoredump.c | 5 +-
> drivers/base/test/test_async_driver_probe.c | 23 +-
> drivers/char/ipmi/ipmi_msghandler.c | 9 +-
> drivers/char/ipmi/ipmi_si_intf.c | 9 +-
> drivers/edac/edac_pci.h | 5 +-
> drivers/edac/edac_pci_sysfs.c | 28 +-
> drivers/misc/vmw_vmci/vmci_guest.c | 9 +-
> include/linux/counters.h | 350 +++++++++++++++++++
> lib/Kconfig | 10 +
> lib/Makefile | 1 +
> lib/test_counters.c | 276 +++++++++++++++
> tools/testing/selftests/lib/Makefile | 1 +
> tools/testing/selftests/lib/config | 1 +
> tools/testing/selftests/lib/test_counters.sh | 5 +
> 21 files changed, 913 insertions(+), 74 deletions(-)
> create mode 100644 Documentation/core-api/counters.rst
> create mode 100644 include/linux/counters.h
> create mode 100644 lib/test_counters.c
> create mode 100755 tools/testing/selftests/lib/test_counters.sh
>
> --
> 2.25.1
>

2020-09-28 20:36:25

by Kees Cook

[permalink] [raw]

Subject: Re: [PATCH 00/11] Introduce Simple atomic and non-atomic counters

On Sun, Sep 27, 2020 at 07:35:26PM -0400, Joel Fernandes wrote:
> On Fri, Sep 25, 2020 at 05:47:14PM -0600, Shuah Khan wrote:
> > This patch series is a result of discussion at the refcount_t BOF
> > the Linux Plumbers Conference. In this discussion, we identified
> > a need for looking closely and investigating atomic_t usages in
> > the kernel when it is used strictly as a counter without it
> > controlling object lifetimes and state changes.
> >
> > There are a number of atomic_t usages in the kernel where atomic_t api
> > is used strictly for counting and not for managing object lifetime. In
> > some cases, atomic_t might not even be needed.
> >
> > The purpose of these counters is twofold: 1. clearly differentiate
> > atomic_t counters from atomic_t usages that guard object lifetimes,
> > hence prone to overflow and underflow errors. It allows tools that scan
> > for underflow and overflow on atomic_t usages to detect overflow and
> > underflows to scan just the cases that are prone to errors. 2. provides
> > non-atomic counters for cases where atomic isn't necessary.
>
> Nice series :)
>
> It appears there is no user of counter_simple in this series other than the
> selftest. Would you be planning to add any conversions in the series itself,
> for illustration of use? Sorry if I missed a usage.
>
> Also how do we guard against atomicity of counter_simple RMW operations? Is
> the implication that it should be guarded using other synchronization to
> prevent lost-update problem?
>
> Some more comments:
>
> 1. atomic RMW operations that have a return value are fully ordered. Would
> you be adding support to counter_simple for such ordering as well, for
> consistency?

No -- there is no atomicity guarantee for counter_simple. I would prefer
counter_simple not exist at all, specifically for this reason.

> 2. I felt counter_atomic and counter_atomic64 would be nice equivalents to
> the atomic and atomic64 naming currently used (i.e. dropping the '32').
> However that is just my opinion and I am ok with either naming.

I had asked that they be size-named to avoid any confusion (i.e. we're
making a new API).

--
Kees Cook

2020-09-28 21:18:36

by Joel Fernandes

[permalink] [raw]

Subject: Re: [PATCH 00/11] Introduce Simple atomic and non-atomic counters

On Mon, Sep 28, 2020 at 01:34:31PM -0700, Kees Cook wrote:
> On Sun, Sep 27, 2020 at 07:35:26PM -0400, Joel Fernandes wrote:
> > On Fri, Sep 25, 2020 at 05:47:14PM -0600, Shuah Khan wrote:
> > > This patch series is a result of discussion at the refcount_t BOF
> > > the Linux Plumbers Conference. In this discussion, we identified
> > > a need for looking closely and investigating atomic_t usages in
> > > the kernel when it is used strictly as a counter without it
> > > controlling object lifetimes and state changes.
> > >
> > > There are a number of atomic_t usages in the kernel where atomic_t api
> > > is used strictly for counting and not for managing object lifetime. In
> > > some cases, atomic_t might not even be needed.
> > >
> > > The purpose of these counters is twofold: 1. clearly differentiate
> > > atomic_t counters from atomic_t usages that guard object lifetimes,
> > > hence prone to overflow and underflow errors. It allows tools that scan
> > > for underflow and overflow on atomic_t usages to detect overflow and
> > > underflows to scan just the cases that are prone to errors. 2. provides
> > > non-atomic counters for cases where atomic isn't necessary.
> >
> > Nice series :)
> >
> > It appears there is no user of counter_simple in this series other than the
> > selftest. Would you be planning to add any conversions in the series itself,
> > for illustration of use? Sorry if I missed a usage.
> >
> > Also how do we guard against atomicity of counter_simple RMW operations? Is
> > the implication that it should be guarded using other synchronization to
> > prevent lost-update problem?
> >
> > Some more comments:
> >
> > 1. atomic RMW operations that have a return value are fully ordered. Would
> > you be adding support to counter_simple for such ordering as well, for
> > consistency?
>
> No -- there is no atomicity guarantee for counter_simple. I would prefer
> counter_simple not exist at all, specifically for this reason.

Yeah I am ok with it not existing, especially also as there are no examples
of its conversion/usage in the series.

> > 2. I felt counter_atomic and counter_atomic64 would be nice equivalents to
> > the atomic and atomic64 naming currently used (i.e. dropping the '32').
> > However that is just my opinion and I am ok with either naming.
>
> I had asked that they be size-named to avoid any confusion (i.e. we're
> making a new API).

Works for me.

Cheers,

- Joel

2020-09-28 23:16:08

by Kees Cook

[permalink] [raw]

Subject: Re: [PATCH 00/11] Introduce Simple atomic and non-atomic counters

On Mon, Sep 28, 2020 at 04:41:47PM -0600, Shuah Khan wrote:
> On 9/26/20 10:29 AM, Kees Cook wrote:
> > On Fri, Sep 25, 2020 at 05:47:14PM -0600, Shuah Khan wrote:
> > > 7. Verified that the test module compiles in kunit env. and test
> > > module can be loaded to run the test.
> >
> > I meant write it using KUnit interfaces (e.g. KUNIT_EXPECT*(),
> > kunit_test_suite(), etc):
> > https://www.kernel.org/doc/html/latest/dev-tools/kunit/
> >
> > Though I see the docs are still not updated[1] to reflect the Kconfig
> > (CONFIG_foo_KUNIT_TEST) and file naming conventions (foo_kunit.c).
> >
>
> I would like to be able to run this test outside Kunit env., hence the
> choice to go with a module and kselftest script. It makes it easier to
> test as part of my workflow as opposed to doing a kunit and build and
> running it that way.

It does -- you just load it normally like before and it prints out
everything just fine. This is how I use the lib/test_user_copy.c and
lib/test_overflow.c before/after their conversions.

--
Kees Cook

2020-09-28 23:23:16

by Shuah Khan

[permalink] [raw]

Subject: Re: [PATCH 00/11] Introduce Simple atomic and non-atomic counters

On 9/28/20 3:17 PM, Joel Fernandes wrote:
> On Mon, Sep 28, 2020 at 01:34:31PM -0700, Kees Cook wrote:
>> On Sun, Sep 27, 2020 at 07:35:26PM -0400, Joel Fernandes wrote:
>>> On Fri, Sep 25, 2020 at 05:47:14PM -0600, Shuah Khan wrote:
>>>> This patch series is a result of discussion at the refcount_t BOF
>>>> the Linux Plumbers Conference. In this discussion, we identified
>>>> a need for looking closely and investigating atomic_t usages in
>>>> the kernel when it is used strictly as a counter without it
>>>> controlling object lifetimes and state changes.
>>>>
>>>> There are a number of atomic_t usages in the kernel where atomic_t api
>>>> is used strictly for counting and not for managing object lifetime. In
>>>> some cases, atomic_t might not even be needed.
>>>>
>>>> The purpose of these counters is twofold: 1. clearly differentiate
>>>> atomic_t counters from atomic_t usages that guard object lifetimes,
>>>> hence prone to overflow and underflow errors. It allows tools that scan
>>>> for underflow and overflow on atomic_t usages to detect overflow and
>>>> underflows to scan just the cases that are prone to errors. 2. provides
>>>> non-atomic counters for cases where atomic isn't necessary.
>>>
>>> Nice series :)
>>>

Thanks.

>>> It appears there is no user of counter_simple in this series other than the
>>> selftest. Would you be planning to add any conversions in the series itself,
>>> for illustration of use? Sorry if I missed a usage.
>>>
>>> Also how do we guard against atomicity of counter_simple RMW operations? Is
>>> the implication that it should be guarded using other synchronization to
>>> prevent lost-update problem?
>>>
>>> Some more comments:
>>>
>>> 1. atomic RMW operations that have a return value are fully ordered. Would
>>> you be adding support to counter_simple for such ordering as well, for
>>> consistency?
>>
>> No -- there is no atomicity guarantee for counter_simple. I would prefer
>> counter_simple not exist at all, specifically for this reason.
>
> Yeah I am ok with it not existing, especially also as there are no examples
> of its conversion/usage in the series.
>

No. counter_simple is just for counting when there is no need for
atomicity with the premise that there might be some use-cases. You
are right that this patch series doesn't use these. My hunch is though
that atomic_t is overused and it isn't needed in all cases.

I will do some research to look for any places that can use
counter_simple before I spin v2. If I don't find any, I can drop them.

thanks,
-- Shuah

2020-09-28 23:23:26

by Shuah Khan

[permalink] [raw]

Subject: Re: [PATCH 00/11] Introduce Simple atomic and non-atomic counters

On 9/26/20 10:22 AM, Kees Cook wrote:
> On Fri, Sep 25, 2020 at 05:47:14PM -0600, Shuah Khan wrote:
>> This patch series is a result of discussion at the refcount_t BOF
>> the Linux Plumbers Conference. In this discussion, we identified
>> a need for looking closely and investigating atomic_t usages in
>> the kernel when it is used strictly as a counter without it
>> controlling object lifetimes and state changes.
>
> BTW, I realized the KSPP issue tracker hadn't broken this task out of
> the refcount_t conversion issue[1] into a separate issue, so I've created
> it now: https://github.com/KSPP/linux/issues/106
>

Cool. Thanks.

-- Shuah

2020-09-28 23:24:25

by Shuah Khan

[permalink] [raw]

Subject: Re: [PATCH 00/11] Introduce Simple atomic and non-atomic counters

On 9/26/20 10:29 AM, Kees Cook wrote:
> On Fri, Sep 25, 2020 at 05:47:14PM -0600, Shuah Khan wrote:
>> 7. Verified that the test module compiles in kunit env. and test
>> module can be loaded to run the test.
>
> I meant write it using KUnit interfaces (e.g. KUNIT_EXPECT*(),
> kunit_test_suite(), etc):
> https://www.kernel.org/doc/html/latest/dev-tools/kunit/
>
> Though I see the docs are still not updated[1] to reflect the Kconfig
> (CONFIG_foo_KUNIT_TEST) and file naming conventions (foo_kunit.c).
>

I would like to be able to run this test outside Kunit env., hence the
choice to go with a module and kselftest script. It makes it easier to
test as part of my workflow as opposed to doing a kunit and build and
running it that way.

I don't mind adding TEST_COUNTERS to kunit default configs though.

thanks,
-- Shuah

2020-09-28 23:30:52

by Shuah Khan

[permalink] [raw]

Subject: Re: [PATCH 00/11] Introduce Simple atomic and non-atomic counters

On 9/26/20 10:33 AM, Kees Cook wrote:
> On Fri, Sep 25, 2020 at 06:13:37PM -0600, Shuah Khan wrote:
>> On 9/25/20 5:52 PM, Kees Cook wrote:
>>> On Fri, Sep 25, 2020 at 05:47:14PM -0600, Shuah Khan wrote:
>>>> -- Addressed Kees's comments:
>>>> 1. Non-atomic counters renamed to counter_simple32 and counter_simple64
>>>> to clearly indicate size.
>>>> 2. Added warning for counter_simple* usage and it should be used only
>>>> when there is no need for atomicity.
>>>> 3. Renamed counter_atomic to counter_atomic32 to clearly indicate size.
>>>> 4. Renamed counter_atomic_long to counter_atomic64 and it now uses
>>>> atomic64_t ops and indicates size.
>>>> 5. Test updated for the API renames.
>>>> 6. Added helper functions for test results printing
>>>> 7. Verified that the test module compiles in kunit env. and test
>>>> module can be loaded to run the test.
>>>
>>> Thanks for all of this!
>>>
>>>> 8. Updated Documentation to reflect the intent to make the API
>>>> restricted so it can never be used to guard object lifetimes
>>>> and state management. I left _return ops for now, inc_return
>>>> is necessary for now as per the discussion we had on this topic.
>>>
>>> I still *really* do not want dec_return() to exist. That is asking for
>>> trouble. I'd prefer inc_return() not exist either, but I can live with
>>> it. ;)
>>>
>>

I didn't read this correctly the first time around.

>> Thanks. I am equally concerned about adding anything that can be used to
>> guard object lifetimes. So I will make sure this set won't expand and
>> plan to remove dec_return() if we don't find any usages.
>
> I would like it much stronger than "if". dec_return() needs to be just
> dec() and read(). It will not be less efficient (since they're both
> inlines), but it _will_ create a case where the atomicity cannot be used
> for ref counting. My point is that anything that _requires_ dec_return()
> (or, frankly, inc_return()) is _not_ "just" a statistical counter. It
> may not be a refcounter, but it relies on the inc/dec atomicity for some
> reason beyond counting in once place and reporting it in another.
>

I am not thinking about efficiency rather two calls instead of one if
an decrement needs to followed by return. In any case, I agree with you
that there is no need to add dec_return now without any use-cases.

I will update the patch series to remove it.

thanks,
-- Shuah

2020-10-06 15:23:40

by Shuah Khan

[permalink] [raw]

Subject: Re: [PATCH 00/11] Introduce Simple atomic and non-atomic counters

On 9/28/20 5:13 PM, Kees Cook wrote:
> On Mon, Sep 28, 2020 at 04:41:47PM -0600, Shuah Khan wrote:
>> On 9/26/20 10:29 AM, Kees Cook wrote:
>>> On Fri, Sep 25, 2020 at 05:47:14PM -0600, Shuah Khan wrote:
>>>> 7. Verified that the test module compiles in kunit env. and test
>>>> module can be loaded to run the test.
>>>
>>> I meant write it using KUnit interfaces (e.g. KUNIT_EXPECT*(),
>>> kunit_test_suite(), etc):
>>> https://www.kernel.org/doc/html/latest/dev-tools/kunit/
>>>
>>> Though I see the docs are still not updated[1] to reflect the Kconfig
>>> (CONFIG_foo_KUNIT_TEST) and file naming conventions (foo_kunit.c).
>>>
>>
>> I would like to be able to run this test outside Kunit env., hence the
>> choice to go with a module and kselftest script. It makes it easier to
>> test as part of my workflow as opposed to doing a kunit and build and
>> running it that way.
>
> It does -- you just load it normally like before and it prints out
> everything just fine. This is how I use the lib/test_user_copy.c and
> lib/test_overflow.c before/after their conversions.
>

I am not seeing any kunit links to either of these tests. I find the
lib/test_overflow.c very hard to read.

I am going to stick with what I have for now and handle conversion
later.

I think it might be a good idea to add tests for atomic_t and refcount_t
APIS as well at some point.

thanks,
-- Shuah