This provides kernel modules that can be used to test the error handling
of notifier call chain failures by injecting artifical errors to the
following notifier chain callbacks.
* CPU notifier
* PM notifier
* memory hotplug notifier
* powerpc pSeries reconfig notifier
Example: Inject CPU offline error (-1 == -EPERM)
# cd /sys/kernel/debug/notifier-error-inject/cpu
# echo -1 > actions/CPU_DOWN_PREPARE/error
# echo 0 > /sys/devices/system/cpu/cpu1/online
bash: echo: write error: Operation not permitted
There are also handy shell scripts to test CPU and memory hotplug notifier.
Note that these tests didn't detect error handling bugs on my machine but
I still think this feature is usefull to test the code path which is rarely
executed.
Changelog:
* v4 (It is about 11 months since v3)
- prefix all APIs with notifier_err_inject_*
- rearrange debugfs interface
(e.g. $DEBUGFS/cpu-notifier-error-inject/CPU_DOWN_PREPARE -->
$DEBUGFS/notifier-error-inject/cpu/actions/CPU_DOWN_PREPARE/error)
- update modules to follow new interface
- add -r option for memory-notifier.sh to specify percent of offlining
memory blocks
* v3
- rewrite to be kernel modules instead of initializing at late_initcall()s
(it makes the diffstat look different but most code remains unchanged)
- export err_inject_notifier_block_{init,cleanup} for modules
- export pSeries_reconfig_notifier_{,un}register symbols for a module
- notifier priority can be specified as a module parameter
- add testing scripts in tools/testing/fault-injection
* v2
- "PM: Improve error code of pm_notifier_call_chain()" is now in -next
- "debugfs: add debugfs_create_int" is dropped
- put a comment in err_inject_notifier_block_init()
- only allow valid errno to be injected (-MAX_ERRNO <= errno <= 0)
- improve Kconfig help text
- make CONFIG_PM_NOTIFIER_ERROR_INJECTION visible even if PM_DEBUG is disabled
- make CONFIG_PM_NOTIFIER_ERROR_INJECTION default if PM_DEBUG is enabled
Akinobu Mita (6):
fault-injection: notifier error injection
cpu: rewrite cpu-notifier-error-inject module
PM: PM notifier error injection module
memory: memory notifier error injection module
powerpc: pSeries reconfig notifier error injection module
fault-injection: add notifier error injection testing scripts
lib/Kconfig.debug | 91 ++++++++++-
lib/Makefile | 5 +
lib/cpu-notifier-error-inject.c | 63 +++-----
lib/memory-notifier-error-inject.c | 48 ++++++
lib/notifier-error-inject.c | 112 ++++++++++++++
lib/notifier-error-inject.h | 24 +++
lib/pSeries-reconfig-notifier-error-inject.c | 51 +++++++
lib/pm-notifier-error-inject.c | 49 ++++++
tools/testing/fault-injection/cpu-notifier.sh | 169 +++++++++++++++++++++
tools/testing/fault-injection/memory-notifier.sh | 176 ++++++++++++++++++++++
10 files changed, 748 insertions(+), 40 deletions(-)
create mode 100644 lib/memory-notifier-error-inject.c
create mode 100644 lib/notifier-error-inject.c
create mode 100644 lib/notifier-error-inject.h
create mode 100644 lib/pSeries-reconfig-notifier-error-inject.c
create mode 100644 lib/pm-notifier-error-inject.c
create mode 100755 tools/testing/fault-injection/cpu-notifier.sh
create mode 100755 tools/testing/fault-injection/memory-notifier.sh
Cc: Pavel Machek <[email protected]>
Cc: "Rafael J. Wysocki" <[email protected]>
Cc: [email protected]
Cc: Greg KH <[email protected]>
Cc: [email protected]
Cc: Benjamin Herrenschmidt <[email protected]>
Cc: Paul Mackerras <[email protected]>
Cc: [email protected]
Cc: Américo Wang <[email protected]>
Cc: Michael Ellerman <[email protected]>
--
1.7.10.2
Rewrite existing cpu-notifier-error-inject module to use debugfs based
new framework.
This change removes cpu_up_prepare_error and cpu_down_prepare_error module
parameters which were used to specify error code to be injected. We could
keep these module parameters for backward compatibility by module_param_cb
but it seems overkill for this module.
This provides the ability to inject artifical errors to CPU notifier chain
callbacks. It is controlled through debugfs interface under
/sys/kernel/debug/notifier-error-inject/cpu
If the notifier call chain should be failed with some events
notified, write the error code to "actions/<notifier event>/error".
Example1: inject CPU offline error (-1 == -EPERM)
# cd /sys/kernel/debug/notifier-error-inject/cpu
# echo -1 > actions/CPU_DOWN_PREPARE/error
# echo 0 > /sys/devices/system/cpu/cpu1/online
bash: echo: write error: Operation not permitted
Example2: inject CPU online error (-2 == -ENOENT)
# cd /sys/kernel/debug/notifier-error-inject/cpu
# echo -2 > actions/CPU_UP_PREPARE/error
# echo 1 > /sys/devices/system/cpu/cpu1/online
bash: echo: write error: No such file or directory
Signed-off-by: Akinobu Mita <[email protected]>
---
* v4
- update modules to follow new interface
lib/Kconfig.debug | 16 ++++++++--
lib/cpu-notifier-error-inject.c | 63 ++++++++++++++++-----------------------
2 files changed, 39 insertions(+), 40 deletions(-)
diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index c848758..be0c197 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -1097,10 +1097,22 @@ config NOTIFIER_ERROR_INJECTION
config CPU_NOTIFIER_ERROR_INJECT
tristate "CPU notifier error injection module"
- depends on HOTPLUG_CPU && DEBUG_KERNEL
+ depends on HOTPLUG_CPU && NOTIFIER_ERROR_INJECTION
help
This option provides a kernel module that can be used to test
- the error handling of the cpu notifiers
+ the error handling of the cpu notifiers by injecting artifical
+ errors to CPU notifier chain callbacks. It is controlled through
+ debugfs interface under /sys/kernel/debug/notifier-error-inject/cpu
+
+ If the notifier call chain should be failed with some events
+ notified, write the error code to "actions/<notifier event>/error".
+
+ Example: Inject CPU offline error (-1 == -EPERM)
+
+ # cd /sys/kernel/debug/notifier-error-inject/cpu
+ # echo -1 > actions/CPU_DOWN_PREPARE/error
+ # echo 0 > /sys/devices/system/cpu/cpu1/online
+ bash: echo: write error: Operation not permitted
To compile this code as a module, choose M here: the module will
be called cpu-notifier-error-inject.
diff --git a/lib/cpu-notifier-error-inject.c b/lib/cpu-notifier-error-inject.c
index 4dc2032..707ca24 100644
--- a/lib/cpu-notifier-error-inject.c
+++ b/lib/cpu-notifier-error-inject.c
@@ -1,58 +1,45 @@
#include <linux/kernel.h>
-#include <linux/cpu.h>
#include <linux/module.h>
-#include <linux/notifier.h>
+#include <linux/cpu.h>
-static int priority;
-static int cpu_up_prepare_error;
-static int cpu_down_prepare_error;
+#include "notifier-error-inject.h"
+static int priority;
module_param(priority, int, 0);
MODULE_PARM_DESC(priority, "specify cpu notifier priority");
-module_param(cpu_up_prepare_error, int, 0644);
-MODULE_PARM_DESC(cpu_up_prepare_error,
- "specify error code to inject CPU_UP_PREPARE action");
-
-module_param(cpu_down_prepare_error, int, 0644);
-MODULE_PARM_DESC(cpu_down_prepare_error,
- "specify error code to inject CPU_DOWN_PREPARE action");
-
-static int err_inject_cpu_callback(struct notifier_block *nfb,
- unsigned long action, void *hcpu)
-{
- int err = 0;
-
- switch (action) {
- case CPU_UP_PREPARE:
- case CPU_UP_PREPARE_FROZEN:
- err = cpu_up_prepare_error;
- break;
- case CPU_DOWN_PREPARE:
- case CPU_DOWN_PREPARE_FROZEN:
- err = cpu_down_prepare_error;
- break;
+static struct notifier_err_inject cpu_notifier_err_inject = {
+ .actions = {
+ { NOTIFIER_ERR_INJECT_ACTION(CPU_UP_PREPARE) },
+ { NOTIFIER_ERR_INJECT_ACTION(CPU_UP_PREPARE_FROZEN) },
+ { NOTIFIER_ERR_INJECT_ACTION(CPU_DOWN_PREPARE) },
+ { NOTIFIER_ERR_INJECT_ACTION(CPU_DOWN_PREPARE_FROZEN) },
+ {}
}
- if (err)
- printk(KERN_INFO "Injecting error (%d) at cpu notifier\n", err);
-
- return notifier_from_errno(err);
-}
-
-static struct notifier_block err_inject_cpu_notifier = {
- .notifier_call = err_inject_cpu_callback,
};
+static struct dentry *dir;
+
static int err_inject_init(void)
{
- err_inject_cpu_notifier.priority = priority;
+ int err;
+
+ dir = notifier_err_inject_init("cpu", notifier_err_inject_dir,
+ &cpu_notifier_err_inject, priority);
+ if (IS_ERR(dir))
+ return PTR_ERR(dir);
+
+ err = register_hotcpu_notifier(&cpu_notifier_err_inject.nb);
+ if (err)
+ debugfs_remove_recursive(dir);
- return register_hotcpu_notifier(&err_inject_cpu_notifier);
+ return err;
}
static void err_inject_exit(void)
{
- unregister_hotcpu_notifier(&err_inject_cpu_notifier);
+ unregister_hotcpu_notifier(&cpu_notifier_err_inject.nb);
+ debugfs_remove_recursive(dir);
}
module_init(err_inject_init);
--
1.7.10.2
This adds two testing scripts with notifier error injection
* tools/testing/fault-injection/cpu-notifier.sh is testing script for
CPU notifier error handling by using cpu-notifier-error-inject.ko.
1. Offline all hot-pluggable CPUs in preparation for testing
2. Test CPU hot-add error handling by injecting notifier errors
3. Online all hot-pluggable CPUs in preparation for testing
4. Test CPU hot-remove error handling by injecting notifier errors
* tools/testing/fault-injection/memory-notifier.sh is doing the similar
thing for memory hotplug notifier.
1. Offline 10% of hot-pluggable memory in preparation for testing
2. Test memory hot-add error handling by injecting notifier errors
3. Online all hot-pluggable memory in preparation for testing
4. Test memory hot-remove error handling by injecting notifier errors
Signed-off-by: Akinobu Mita <[email protected]>
Suggested-by: Andrew Morton <[email protected]>
Cc: Pavel Machek <[email protected]>
Cc: "Rafael J. Wysocki" <[email protected]>
Cc: [email protected]
Cc: Greg KH <[email protected]>
Cc: [email protected]
Cc: Benjamin Herrenschmidt <[email protected]>
Cc: Paul Mackerras <[email protected]>
Cc: [email protected]
Cc: Américo Wang <[email protected]>
---
* v4
- add -r option for memory-notifier.sh to specify percent of offlining
memory blocks
tools/testing/fault-injection/cpu-notifier.sh | 169 +++++++++++++++++++++
tools/testing/fault-injection/memory-notifier.sh | 176 ++++++++++++++++++++++
2 files changed, 345 insertions(+)
create mode 100755 tools/testing/fault-injection/cpu-notifier.sh
create mode 100755 tools/testing/fault-injection/memory-notifier.sh
diff --git a/tools/testing/fault-injection/cpu-notifier.sh b/tools/testing/fault-injection/cpu-notifier.sh
new file mode 100755
index 0000000..af93630
--- /dev/null
+++ b/tools/testing/fault-injection/cpu-notifier.sh
@@ -0,0 +1,169 @@
+#!/bin/bash
+
+#
+# list all hot-pluggable CPUs
+#
+hotpluggable_cpus()
+{
+ local state=${1:-.\*}
+
+ for cpu in /sys/devices/system/cpu/cpu*; do
+ if [ -f $cpu/online ] && grep -q $state $cpu/online; then
+ echo ${cpu##/*/cpu}
+ fi
+ done
+}
+
+hotplaggable_offline_cpus()
+{
+ hotpluggable_cpus 0
+}
+
+hotpluggable_online_cpus()
+{
+ hotpluggable_cpus 1
+}
+
+cpu_is_online()
+{
+ grep -q 1 /sys/devices/system/cpu/cpu$1/online
+}
+
+cpu_is_offline()
+{
+ grep -q 0 /sys/devices/system/cpu/cpu$1/online
+}
+
+add_cpu()
+{
+ echo 1 > /sys/devices/system/cpu/cpu$1/online
+}
+
+remove_cpu()
+{
+ echo 0 > /sys/devices/system/cpu/cpu$1/online
+}
+
+add_cpu_expect_success()
+{
+ local cpu=$1
+
+ if ! add_cpu $cpu; then
+ echo $FUNCNAME $cpu: unexpected fail >&2
+ elif ! cpu_is_online $cpu; then
+ echo $FUNCNAME $cpu: unexpected offline >&2
+ fi
+}
+
+add_cpu_expect_fail()
+{
+ local cpu=$1
+
+ if add_cpu $cpu 2> /dev/null; then
+ echo $FUNCNAME $cpu: unexpected success >&2
+ elif ! cpu_is_offline $cpu; then
+ echo $FUNCNAME $cpu: unexpected online >&2
+ fi
+}
+
+remove_cpu_expect_success()
+{
+ local cpu=$1
+
+ if ! remove_cpu $cpu; then
+ echo $FUNCNAME $cpu: unexpected fail >&2
+ elif ! cpu_is_offline $cpu; then
+ echo $FUNCNAME $cpu: unexpected offline >&2
+ fi
+}
+
+remove_cpu_expect_fail()
+{
+ local cpu=$1
+
+ if remove_cpu $cpu 2> /dev/null; then
+ echo $FUNCNAME $cpu: unexpected success >&2
+ elif ! cpu_is_online $cpu; then
+ echo $FUNCNAME $cpu: unexpected offline >&2
+ fi
+}
+
+if [ $UID != 0 ]; then
+ echo must be run as root >&2
+ exit 1
+fi
+
+error=-12
+priority=0
+
+while getopts e:hp: opt; do
+ case $opt in
+ e)
+ error=$OPTARG
+ ;;
+ h)
+ echo "Usage $0 [ -e errno ] [ -p notifier-priority ]"
+ exit
+ ;;
+ p)
+ priority=$OPTARG
+ ;;
+ esac
+done
+
+if ! [ "$error" -ge -4095 -a "$error" -lt 0 ]; then
+ echo "error code must be -4095 <= errno < 0" >&2
+ exit 1
+fi
+
+DEBUGFS=`mount -t debugfs | head -1 | awk '{ print $3 }'`
+
+if [ ! -d "$DEBUGFS" ]; then
+ echo debugfs is not mounted >&2
+ exit 1
+fi
+
+/sbin/modprobe -r cpu-notifier-error-inject
+/sbin/modprobe -q cpu-notifier-error-inject priority=$priority
+
+NOTIFIER_ERR_INJECT_DIR=$DEBUGFS/notifier-error-inject/cpu
+
+if [ ! -d $NOTIFIER_ERR_INJECT_DIR ]; then
+ echo cpu-notifier-error-inject module is not available >&2
+ exit 1
+fi
+
+#
+# Offline all hot-pluggable CPUs
+#
+echo 0 > $NOTIFIER_ERR_INJECT_DIR/actions/CPU_DOWN_PREPARE/error
+for cpu in `hotpluggable_online_cpus`; do
+ remove_cpu_expect_success $cpu
+done
+
+#
+# Test CPU hot-add error handling (offline => online)
+#
+echo $error > $NOTIFIER_ERR_INJECT_DIR/actions/CPU_UP_PREPARE/error
+for cpu in `hotplaggable_offline_cpus`; do
+ add_cpu_expect_fail $cpu
+done
+
+#
+# Online all hot-pluggable CPUs
+#
+echo 0 > $NOTIFIER_ERR_INJECT_DIR/actions/CPU_UP_PREPARE/error
+for cpu in `hotplaggable_offline_cpus`; do
+ add_cpu_expect_success $cpu
+done
+
+#
+# Test CPU hot-remove error handling (online => offline)
+#
+echo $error > $NOTIFIER_ERR_INJECT_DIR/actions/CPU_DOWN_PREPARE/error
+for cpu in `hotpluggable_online_cpus`; do
+ remove_cpu_expect_fail $cpu
+done
+
+echo 0 > $NOTIFIER_ERR_INJECT_DIR/actions/CPU_DOWN_PREPARE/error
+/sbin/modprobe -r cpu-notifier-error-inject
diff --git a/tools/testing/fault-injection/memory-notifier.sh b/tools/testing/fault-injection/memory-notifier.sh
new file mode 100755
index 0000000..843cba7
--- /dev/null
+++ b/tools/testing/fault-injection/memory-notifier.sh
@@ -0,0 +1,176 @@
+#!/bin/bash
+
+#
+# list all hot-pluggable memory
+#
+hotpluggable_memory()
+{
+ local state=${1:-.\*}
+
+ for memory in /sys/devices/system/memory/memory*; do
+ if grep -q 1 $memory/removable &&
+ grep -q $state $memory/state; then
+ echo ${memory##/*/memory}
+ fi
+ done
+}
+
+hotplaggable_offline_memory()
+{
+ hotpluggable_memory offline
+}
+
+hotpluggable_online_memory()
+{
+ hotpluggable_memory online
+}
+
+memory_is_online()
+{
+ grep -q online /sys/devices/system/memory/memory$1/state
+}
+
+memory_is_offline()
+{
+ grep -q offline /sys/devices/system/memory/memory$1/state
+}
+
+add_memory()
+{
+ echo online > /sys/devices/system/memory/memory$1/state
+}
+
+remove_memory()
+{
+ echo offline > /sys/devices/system/memory/memory$1/state
+}
+
+add_memory_expect_success()
+{
+ local memory=$1
+
+ if ! add_memory $memory; then
+ echo $FUNCNAME $memory: unexpected fail >&2
+ elif ! memory_is_online $memory; then
+ echo $FUNCNAME $memory: unexpected offline >&2
+ fi
+}
+
+add_memory_expect_fail()
+{
+ local memory=$1
+
+ if add_memory $memory 2> /dev/null; then
+ echo $FUNCNAME $memory: unexpected success >&2
+ elif ! memory_is_offline $memory; then
+ echo $FUNCNAME $memory: unexpected online >&2
+ fi
+}
+
+remove_memory_expect_success()
+{
+ local memory=$1
+
+ if ! remove_memory $memory; then
+ echo $FUNCNAME $memory: unexpected fail >&2
+ elif ! memory_is_offline $memory; then
+ echo $FUNCNAME $memory: unexpected offline >&2
+ fi
+}
+
+remove_memory_expect_fail()
+{
+ local memory=$1
+
+ if remove_memory $memory 2> /dev/null; then
+ echo $FUNCNAME $memory: unexpected success >&2
+ elif ! memory_is_online $memory; then
+ echo $FUNCNAME $memory: unexpected offline >&2
+ fi
+}
+
+if [ $UID != 0 ]; then
+ echo must be run as root >&2
+ exit 1
+fi
+
+error=-12
+priority=0
+ratio=10
+
+while getopts e:hp:r: opt; do
+ case $opt in
+ e)
+ error=$OPTARG
+ ;;
+ h)
+ echo "Usage $0 [ -e errno ] [ -p notifier-priority ] [ -r percent-of-memory-to-offline ]"
+ exit
+ ;;
+ p)
+ priority=$OPTARG
+ ;;
+ r)
+ ratio=$OPTARG
+ ;;
+ esac
+done
+
+if ! [ "$error" -ge -4095 -a "$error" -lt 0 ]; then
+ echo "error code must be -4095 <= errno < 0" >&2
+ exit 1
+fi
+
+DEBUGFS=`mount -t debugfs | head -1 | awk '{ print $3 }'`
+
+if [ ! -d "$DEBUGFS" ]; then
+ echo debugfs is not mounted >&2
+ exit 1
+fi
+
+/sbin/modprobe -r memory-notifier-error-inject
+/sbin/modprobe -q memory-notifier-error-inject priority=$priority
+
+NOTIFIER_ERR_INJECT_DIR=$DEBUGFS/notifier-error-inject/memory
+
+if [ ! -d $NOTIFIER_ERR_INJECT_DIR ]; then
+ echo memory-notifier-error-inject module is not available >&2
+ exit 1
+fi
+
+#
+# Offline $ratio percent of hot-pluggable memory
+#
+echo 0 > $NOTIFIER_ERR_INJECT_DIR/actions/MEM_GOING_OFFLINE/error
+for memory in `hotpluggable_online_memory`; do
+ if [ $((RANDOM % 100)) -lt $ratio ]; then
+ remove_memory_expect_success $memory
+ fi
+done
+
+#
+# Test memory hot-add error handling (offline => online)
+#
+echo $error > $NOTIFIER_ERR_INJECT_DIR/actions/MEM_GOING_ONLINE/error
+for memory in `hotplaggable_offline_memory`; do
+ add_memory_expect_fail $memory
+done
+
+#
+# Online all hot-pluggable memory
+#
+echo 0 > $NOTIFIER_ERR_INJECT_DIR/actions/MEM_GOING_ONLINE/error
+for memory in `hotplaggable_offline_memory`; do
+ add_memory_expect_success $memory
+done
+
+#
+# Test memory hot-remove error handling (online => offline)
+#
+echo $error > $NOTIFIER_ERR_INJECT_DIR/actions/MEM_GOING_OFFLINE/error
+for memory in `hotpluggable_online_memory`; do
+ remove_memory_expect_fail $memory
+done
+
+echo 0 > $NOTIFIER_ERR_INJECT_DIR/actions/MEM_GOING_OFFLINE/error
+/sbin/modprobe -r memory-notifier-error-inject
--
1.7.10.2
This provides the ability to inject artifical errors to pSeries reconfig
notifier chain callbacks. It is controlled through debugfs interface
under /sys/kernel/debug/notifier-error-inject/pSeries-reconfig
If the notifier call chain should be failed with some events
notified, write the error code to "actions/<notifier event>/error".
Signed-off-by: Akinobu Mita <[email protected]>
Cc: Benjamin Herrenschmidt <[email protected]>
Cc: Paul Mackerras <[email protected]>
Cc: [email protected]
---
* v4
- update modules to follow new interface
lib/Kconfig.debug | 17 +++++++++
lib/Makefile | 2 +
lib/pSeries-reconfig-notifier-error-inject.c | 51 ++++++++++++++++++++++++++
3 files changed, 70 insertions(+)
create mode 100644 lib/pSeries-reconfig-notifier-error-inject.c
diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index 7cceddc..8f8e226 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -1166,6 +1166,23 @@ config MEMORY_NOTIFIER_ERROR_INJECT
If unsure, say N.
+config PSERIES_RECONFIG_NOTIFIER_ERROR_INJECT
+ tristate "pSeries reconfig notifier error injection module"
+ depends on PPC_PSERIES && NOTIFIER_ERROR_INJECTION
+ help
+ This option provides the ability to inject artifical errors to
+ pSeries reconfig notifier chain callbacks. It is controlled
+ through debugfs interface under
+ /sys/kernel/debug/notifier-error-inject/pSeries-reconfig/
+
+ If the notifier call chain should be failed with some events
+ notified, write the error code to "actions/<notifier event>/error".
+
+ To compile this code as a module, choose M here: the module will
+ be called memory-notifier-error-inject.
+
+ If unsure, say N.
+
config FAULT_INJECTION
bool "Fault-injection framework"
depends on DEBUG_KERNEL
diff --git a/lib/Makefile b/lib/Makefile
index a867aa5..d055cb1 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -94,6 +94,8 @@ obj-$(CONFIG_NOTIFIER_ERROR_INJECTION) += notifier-error-inject.o
obj-$(CONFIG_CPU_NOTIFIER_ERROR_INJECT) += cpu-notifier-error-inject.o
obj-$(CONFIG_PM_NOTIFIER_ERROR_INJECT) += pm-notifier-error-inject.o
obj-$(CONFIG_MEMORY_NOTIFIER_ERROR_INJECT) += memory-notifier-error-inject.o
+obj-$(CONFIG_PSERIES_RECONFIG_NOTIFIER_ERROR_INJECT) += \
+ pSeries-reconfig-notifier-error-inject.o
lib-$(CONFIG_GENERIC_BUG) += bug.o
diff --git a/lib/pSeries-reconfig-notifier-error-inject.c b/lib/pSeries-reconfig-notifier-error-inject.c
new file mode 100644
index 0000000..7f7c98d
--- /dev/null
+++ b/lib/pSeries-reconfig-notifier-error-inject.c
@@ -0,0 +1,51 @@
+#include <linux/kernel.h>
+#include <linux/module.h>
+
+#include <asm/pSeries_reconfig.h>
+
+#include "notifier-error-inject.h"
+
+static int priority;
+module_param(priority, int, 0);
+MODULE_PARM_DESC(priority, "specify pSeries reconfig notifier priority");
+
+static struct notifier_err_inject reconfig_err_inject = {
+ .actions = {
+ { NOTIFIER_ERR_INJECT_ACTION(PSERIES_RECONFIG_ADD) },
+ { NOTIFIER_ERR_INJECT_ACTION(PSERIES_RECONFIG_REMOVE) },
+ { NOTIFIER_ERR_INJECT_ACTION(PSERIES_DRCONF_MEM_ADD) },
+ { NOTIFIER_ERR_INJECT_ACTION(PSERIES_DRCONF_MEM_REMOVE) },
+ {}
+ }
+};
+
+static struct dentry *dir;
+
+static int err_inject_init(void)
+{
+ int err;
+
+ dir = notifier_err_inject_init("pSeries-reconfig",
+ notifier_err_inject_dir, &reconfig_err_inject, priority);
+ if (IS_ERR(dir))
+ return PTR_ERR(dir);
+
+ err = pSeries_reconfig_notifier_register(&reconfig_err_inject.nb);
+ if (err)
+ debugfs_remove_recursive(dir);
+
+ return err;
+}
+
+static void err_inject_exit(void)
+{
+ pSeries_reconfig_notifier_unregister(&reconfig_err_inject.nb);
+ debugfs_remove_recursive(dir);
+}
+
+module_init(err_inject_init);
+module_exit(err_inject_exit);
+
+MODULE_DESCRIPTION("pSeries reconfig notifier error injection module");
+MODULE_LICENSE("GPL");
+MODULE_AUTHOR("Akinobu Mita <[email protected]>");
--
1.7.10.2
This provides the ability to inject artifical errors to memory hotplug
notifier chain callbacks. It is controlled through debugfs interface
under /sys/kernel/debug/notifier-error-inject/memory
If the notifier call chain should be failed with some events notified,
write the error code to "actions/<notifier event>/error".
Example: Inject memory hotplug offline error (-12 == -ENOMEM)
# cd /sys/kernel/debug/notifier-error-inject/memory
# echo -12 > actions/MEM_GOING_OFFLINE/error
# echo offline > /sys/devices/system/memory/memoryXXX/state
bash: echo: write error: Cannot allocate memory
Signed-off-by: Akinobu Mita <[email protected]>
Cc: Greg KH <[email protected]>
Cc: [email protected]
---
* v4
- update modules to follow new interface
lib/Kconfig.debug | 23 +++++++++++++++++
lib/Makefile | 1 +
lib/memory-notifier-error-inject.c | 48 ++++++++++++++++++++++++++++++++++++
3 files changed, 72 insertions(+)
create mode 100644 lib/memory-notifier-error-inject.c
diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index 246cea6..7cceddc 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -1143,6 +1143,29 @@ config PM_NOTIFIER_ERROR_INJECT
If unsure, say N.
+config MEMORY_NOTIFIER_ERROR_INJECT
+ tristate "Memory hotplug notifier error injection module"
+ depends on MEMORY_HOTPLUG_SPARSE && NOTIFIER_ERROR_INJECTION
+ help
+ This option provides the ability to inject artifical errors to
+ memory hotplug notifier chain callbacks. It is controlled through
+ debugfs interface under /sys/kernel/debug/notifier-error-inject/memory
+
+ If the notifier call chain should be failed with some events
+ notified, write the error code to "actions/<notifier event>/error".
+
+ Example: Inject memory hotplug offline error (-12 == -ENOMEM)
+
+ # cd /sys/kernel/debug/notifier-error-inject/memory
+ # echo -12 > actions/MEM_GOING_OFFLINE/error
+ # echo offline > /sys/devices/system/memory/memoryXXX/state
+ bash: echo: write error: Cannot allocate memory
+
+ To compile this code as a module, choose M here: the module will
+ be called memory-notifier-error-inject.
+
+ If unsure, say N.
+
config FAULT_INJECTION
bool "Fault-injection framework"
depends on DEBUG_KERNEL
diff --git a/lib/Makefile b/lib/Makefile
index 230a949..a867aa5 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -93,6 +93,7 @@ obj-$(CONFIG_FAULT_INJECTION) += fault-inject.o
obj-$(CONFIG_NOTIFIER_ERROR_INJECTION) += notifier-error-inject.o
obj-$(CONFIG_CPU_NOTIFIER_ERROR_INJECT) += cpu-notifier-error-inject.o
obj-$(CONFIG_PM_NOTIFIER_ERROR_INJECT) += pm-notifier-error-inject.o
+obj-$(CONFIG_MEMORY_NOTIFIER_ERROR_INJECT) += memory-notifier-error-inject.o
lib-$(CONFIG_GENERIC_BUG) += bug.o
diff --git a/lib/memory-notifier-error-inject.c b/lib/memory-notifier-error-inject.c
new file mode 100644
index 0000000..e6239bf
--- /dev/null
+++ b/lib/memory-notifier-error-inject.c
@@ -0,0 +1,48 @@
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/memory.h>
+
+#include "notifier-error-inject.h"
+
+static int priority;
+module_param(priority, int, 0);
+MODULE_PARM_DESC(priority, "specify memory notifier priority");
+
+static struct notifier_err_inject memory_notifier_err_inject = {
+ .actions = {
+ { NOTIFIER_ERR_INJECT_ACTION(MEM_GOING_ONLINE) },
+ { NOTIFIER_ERR_INJECT_ACTION(MEM_GOING_OFFLINE) },
+ {}
+ }
+};
+
+static struct dentry *dir;
+
+static int err_inject_init(void)
+{
+ int err;
+
+ dir = notifier_err_inject_init("memory", notifier_err_inject_dir,
+ &memory_notifier_err_inject, priority);
+ if (IS_ERR(dir))
+ return PTR_ERR(dir);
+
+ err = register_memory_notifier(&memory_notifier_err_inject.nb);
+ if (err)
+ debugfs_remove_recursive(dir);
+
+ return err;
+}
+
+static void err_inject_exit(void)
+{
+ unregister_memory_notifier(&memory_notifier_err_inject.nb);
+ debugfs_remove_recursive(dir);
+}
+
+module_init(err_inject_init);
+module_exit(err_inject_exit);
+
+MODULE_DESCRIPTION("memory notifier error injection module");
+MODULE_LICENSE("GPL");
+MODULE_AUTHOR("Akinobu Mita <[email protected]>");
--
1.7.10.2
The notifier error injection provides the ability to inject artifical
errors to specified notifier chain callbacks. It is useful to test the
error handling of notifier call chain failures.
This adds common basic functions to define which type of events can be
fail and to initialize the debugfs interface to control what error code
should be returned and which event should be failed.
Signed-off-by: Akinobu Mita <[email protected]>
Cc: Pavel Machek <[email protected]>
Cc: "Rafael J. Wysocki" <[email protected]>
Cc: [email protected]
Cc: Greg KH <[email protected]>
Cc: [email protected]
Cc: Benjamin Herrenschmidt <[email protected]>
Cc: Paul Mackerras <[email protected]>
Cc: [email protected]
Cc: Michael Ellerman <[email protected]>
---
* v4
- prefix all APIs with notifier_err_inject_*
- rearrange debugfs interface
(e.g. $DEBUGFS/cpu-notifier-error-inject/CPU_DOWN_PREPARE -->
$DEBUGFS/notifier-error-inject/cpu/actions/CPU_DOWN_PREPARE/error)
lib/Kconfig.debug | 11 +++++
lib/Makefile | 1 +
lib/notifier-error-inject.c | 112 +++++++++++++++++++++++++++++++++++++++++++
lib/notifier-error-inject.h | 24 ++++++++++
4 files changed, 148 insertions(+)
create mode 100644 lib/notifier-error-inject.c
create mode 100644 lib/notifier-error-inject.h
diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index ff5bdee..c848758 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -1084,6 +1084,17 @@ config LKDTM
Documentation on how to use the module can be found in
Documentation/fault-injection/provoke-crashes.txt
+config NOTIFIER_ERROR_INJECTION
+ tristate "Notifier error injection"
+ depends on DEBUG_KERNEL
+ select DEBUG_FS
+ help
+ This option provides the ability to inject artifical errors to
+ specified notifier chain callbacks. It is useful to test the error
+ handling of notifier call chain failures.
+
+ Say N if unsure.
+
config CPU_NOTIFIER_ERROR_INJECT
tristate "CPU notifier error injection module"
depends on HOTPLUG_CPU && DEBUG_KERNEL
diff --git a/lib/Makefile b/lib/Makefile
index 8c31a0c..23fba9e 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -90,6 +90,7 @@ obj-$(CONFIG_AUDIT_GENERIC) += audit.o
obj-$(CONFIG_SWIOTLB) += swiotlb.o
obj-$(CONFIG_IOMMU_HELPER) += iommu-helper.o
obj-$(CONFIG_FAULT_INJECTION) += fault-inject.o
+obj-$(CONFIG_NOTIFIER_ERROR_INJECTION) += notifier-error-inject.o
obj-$(CONFIG_CPU_NOTIFIER_ERROR_INJECT) += cpu-notifier-error-inject.o
lib-$(CONFIG_GENERIC_BUG) += bug.o
diff --git a/lib/notifier-error-inject.c b/lib/notifier-error-inject.c
new file mode 100644
index 0000000..44b92cb
--- /dev/null
+++ b/lib/notifier-error-inject.c
@@ -0,0 +1,112 @@
+#include <linux/module.h>
+
+#include "notifier-error-inject.h"
+
+static int debugfs_errno_set(void *data, u64 val)
+{
+ *(int *)data = clamp_t(int, val, -MAX_ERRNO, 0);
+ return 0;
+}
+
+static int debugfs_errno_get(void *data, u64 *val)
+{
+ *val = *(int *)data;
+ return 0;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(fops_errno, debugfs_errno_get, debugfs_errno_set,
+ "%lld\n");
+
+static struct dentry *debugfs_create_errno(const char *name, mode_t mode,
+ struct dentry *parent, int *value)
+{
+ return debugfs_create_file(name, mode, parent, value, &fops_errno);
+}
+
+static int notifier_err_inject_callback(struct notifier_block *nb,
+ unsigned long val, void *p)
+{
+ int err = 0;
+ struct notifier_err_inject *err_inject =
+ container_of(nb, struct notifier_err_inject, nb);
+ struct notifier_err_inject_action *action;
+
+ for (action = err_inject->actions; action->name; action++) {
+ if (action->val == val) {
+ err = action->error;
+ break;
+ }
+ }
+ if (err)
+ pr_info("Injecting error (%d) to %s\n", err, action->name);
+
+ return notifier_from_errno(err);
+}
+
+struct dentry *notifier_err_inject_dir;
+EXPORT_SYMBOL_GPL(notifier_err_inject_dir);
+
+struct dentry *notifier_err_inject_init(const char *name, struct dentry *parent,
+ struct notifier_err_inject *err_inject, int priority)
+{
+ struct notifier_err_inject_action *action;
+ mode_t mode = S_IFREG | S_IRUSR | S_IWUSR;
+ struct dentry *dir;
+ struct dentry *actions_dir;
+
+ err_inject->nb.notifier_call = notifier_err_inject_callback;
+ err_inject->nb.priority = priority;
+
+ dir = debugfs_create_dir(name, parent);
+ if (!dir)
+ return ERR_PTR(-ENOMEM);
+
+ actions_dir = debugfs_create_dir("actions", dir);
+ if (!actions_dir)
+ goto fail;
+
+ for (action = err_inject->actions; action->name; action++) {
+ struct dentry *action_dir;
+
+ action_dir = debugfs_create_dir(action->name, actions_dir);
+ if (!action_dir)
+ goto fail;
+
+ /*
+ * Create debugfs r/w file containing action->error. If
+ * notifier call chain is called with action->val, it will
+ * fail with the error code
+ */
+ if (!debugfs_create_errno("error", mode, action_dir,
+ &action->error))
+ goto fail;
+ }
+ return dir;
+fail:
+ debugfs_remove_recursive(dir);
+ return ERR_PTR(-ENOMEM);
+}
+EXPORT_SYMBOL_GPL(notifier_err_inject_init);
+
+static int __init err_inject_init(void)
+{
+ notifier_err_inject_dir =
+ debugfs_create_dir("notifier-error-inject", NULL);
+
+ if (!notifier_err_inject_dir)
+ return -ENOMEM;
+
+ return 0;
+}
+
+static void __exit err_inject_exit(void)
+{
+ debugfs_remove_recursive(notifier_err_inject_dir);
+}
+
+module_init(err_inject_init);
+module_exit(err_inject_exit);
+
+MODULE_DESCRIPTION("Notifier error injection module");
+MODULE_LICENSE("GPL");
+MODULE_AUTHOR("Akinobu Mita <[email protected]>");
diff --git a/lib/notifier-error-inject.h b/lib/notifier-error-inject.h
new file mode 100644
index 0000000..99b3b6f
--- /dev/null
+++ b/lib/notifier-error-inject.h
@@ -0,0 +1,24 @@
+#include <linux/atomic.h>
+#include <linux/debugfs.h>
+#include <linux/notifier.h>
+
+struct notifier_err_inject_action {
+ unsigned long val;
+ int error;
+ const char *name;
+};
+
+#define NOTIFIER_ERR_INJECT_ACTION(action) \
+ .name = #action, .val = (action),
+
+struct notifier_err_inject {
+ struct notifier_block nb;
+ struct notifier_err_inject_action actions[];
+ /* The last slot must be terminated with zero sentinel */
+};
+
+extern struct dentry *notifier_err_inject_dir;
+
+extern struct dentry *notifier_err_inject_init(const char *name,
+ struct dentry *parent, struct notifier_err_inject *err_inject,
+ int priority);
--
1.7.10.2
This provides the ability to inject artifical errors to PM notifier chain
callbacks. It is controlled through debugfs interface under
/sys/kernel/debug/notifier-error-inject/pm
Each of the files in "error" directory represents an event which can be
failed and contains the error code. If the notifier call chain should
be failed with some events notified, write the error code to the files.
If the notifier call chain should be failed with some events notified,
write the error code to "actions/<notifier event>/error".
Example: Inject PM suspend error (-12 = -ENOMEM)
# cd /sys/kernel/debug/notifier-error-inject/pm
# echo -12 > actions/PM_SUSPEND_PREPARE/error
# echo mem > /sys/power/state
bash: echo: write error: Cannot allocate memory
Signed-off-by: Akinobu Mita <[email protected]>
Cc: Pavel Machek <[email protected]>
Cc: "Rafael J. Wysocki" <[email protected]>
Cc: [email protected]
---
* v4
- update modules to follow new interface
lib/Kconfig.debug | 24 ++++++++++++++++++++
lib/Makefile | 1 +
lib/pm-notifier-error-inject.c | 49 ++++++++++++++++++++++++++++++++++++++++
3 files changed, 74 insertions(+)
create mode 100644 lib/pm-notifier-error-inject.c
diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index be0c197..246cea6 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -1119,6 +1119,30 @@ config CPU_NOTIFIER_ERROR_INJECT
If unsure, say N.
+config PM_NOTIFIER_ERROR_INJECT
+ tristate "PM notifier error injection module"
+ depends on PM && NOTIFIER_ERROR_INJECTION
+ default m if PM_DEBUG
+ help
+ This option provides the ability to inject artifical errors to
+ PM notifier chain callbacks. It is controlled through debugfs
+ interface /sys/kernel/debug/notifier-error-inject/pm
+
+ If the notifier call chain should be failed with some events
+ notified, write the error code to "actions/<notifier event>/error".
+
+ Example: Inject PM suspend error (-12 = -ENOMEM)
+
+ # cd /sys/kernel/debug/notifier-error-inject/pm/
+ # echo -12 > actions/PM_SUSPEND_PREPARE/error
+ # echo mem > /sys/power/state
+ bash: echo: write error: Cannot allocate memory
+
+ To compile this code as a module, choose M here: the module will
+ be called pm-notifier-error-inject.
+
+ If unsure, say N.
+
config FAULT_INJECTION
bool "Fault-injection framework"
depends on DEBUG_KERNEL
diff --git a/lib/Makefile b/lib/Makefile
index 23fba9e..230a949 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -92,6 +92,7 @@ obj-$(CONFIG_IOMMU_HELPER) += iommu-helper.o
obj-$(CONFIG_FAULT_INJECTION) += fault-inject.o
obj-$(CONFIG_NOTIFIER_ERROR_INJECTION) += notifier-error-inject.o
obj-$(CONFIG_CPU_NOTIFIER_ERROR_INJECT) += cpu-notifier-error-inject.o
+obj-$(CONFIG_PM_NOTIFIER_ERROR_INJECT) += pm-notifier-error-inject.o
lib-$(CONFIG_GENERIC_BUG) += bug.o
diff --git a/lib/pm-notifier-error-inject.c b/lib/pm-notifier-error-inject.c
new file mode 100644
index 0000000..c094b2d
--- /dev/null
+++ b/lib/pm-notifier-error-inject.c
@@ -0,0 +1,49 @@
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/suspend.h>
+
+#include "notifier-error-inject.h"
+
+static int priority;
+module_param(priority, int, 0);
+MODULE_PARM_DESC(priority, "specify PM notifier priority");
+
+static struct notifier_err_inject pm_notifier_err_inject = {
+ .actions = {
+ { NOTIFIER_ERR_INJECT_ACTION(PM_HIBERNATION_PREPARE) },
+ { NOTIFIER_ERR_INJECT_ACTION(PM_SUSPEND_PREPARE) },
+ { NOTIFIER_ERR_INJECT_ACTION(PM_RESTORE_PREPARE) },
+ {}
+ }
+};
+
+static struct dentry *dir;
+
+static int err_inject_init(void)
+{
+ int err;
+
+ dir = notifier_err_inject_init("pm", notifier_err_inject_dir,
+ &pm_notifier_err_inject, priority);
+ if (IS_ERR(dir))
+ return PTR_ERR(dir);
+
+ err = register_pm_notifier(&pm_notifier_err_inject.nb);
+ if (err)
+ debugfs_remove_recursive(dir);
+
+ return err;
+}
+
+static void err_inject_exit(void)
+{
+ unregister_pm_notifier(&pm_notifier_err_inject.nb);
+ debugfs_remove_recursive(dir);
+}
+
+module_init(err_inject_init);
+module_exit(err_inject_exit);
+
+MODULE_DESCRIPTION("PM notifier error injection module");
+MODULE_LICENSE("GPL");
+MODULE_AUTHOR("Akinobu Mita <[email protected]>");
--
1.7.10.2
On Sat, 23 Jun 2012 23:58:22 +0900
Akinobu Mita <[email protected]> wrote:
> This adds two testing scripts with notifier error injection
Can we move these into tools/testing/selftests/, so that a "make
run_tests" runs these tests?
Also, I don't think it's appropriate that "fault-injection" be in the
path - that's an implementation detail. What we're testing here is
memory hotplug, pm, cpu hotplug, etc. So each test would go into, say,
tools/testing/selftests/cpu-hotplug.
Now, your cpu-hotplug test only tests a tiny part of the cpu-hotplug
code. But it is a start, and creates the place where additional tests
will be placed in the future.
If the kernel configuration means that the tests cannot be run, the
attempt should succeed so that other tests are not disrupted. I guess
that printing a warning in this case is useful.
Probably the selftests will require root permissions - we haven't
really thought about that much. If these tests require root (I assume
they do?) then a sensible approach would be to check for that and to
emit a warning and return "success".
My overall take on the fault-injection code is that there has been a
disappointing amount of uptake: I don't see many developers using them
for whitebox testing their stuff. I guess this patchset addresses
that, in a way.
On Tue, Jun 26, 2012 at 04:31:47PM -0700, Andrew Morton wrote:
> My overall take on the fault-injection code is that there has been a
> disappointing amount of uptake: I don't see many developers using them
> for whitebox testing their stuff. I guess this patchset addresses
> that, in a way.
I added support for make-it-fail to my syscall fuzzer a while ago.
(if the file exists, the child processes set it before calling the fuzzed syscall).
I've not had a chance to really play with it, because I find enough problems
already even without it.
Dave
2012/6/27 Andrew Morton <[email protected]>:
> On Sat, 23 Jun 2012 23:58:22 +0900
> Akinobu Mita <[email protected]> wrote:
>
>> This adds two testing scripts with notifier error injection
>
> Can we move these into tools/testing/selftests/, so that a "make
> run_tests" runs these tests?
>
> Also, I don't think it's appropriate that "fault-injection" be in the
> path - that's an implementation detail. ?What we're testing here is
> memory hotplug, pm, cpu hotplug, etc. ?So each test would go into, say,
> tools/testing/selftests/cpu-hotplug.
>
> Now, your cpu-hotplug test only tests a tiny part of the cpu-hotplug
> code. ?But it is a start, and creates the place where additional tests
> will be placed in the future.
>
>
> If the kernel configuration means that the tests cannot be run, the
> attempt should succeed so that other tests are not disrupted. ?I guess
> that printing a warning in this case is useful.
>
> Probably the selftests will require root permissions - we haven't
> really thought about that much. ?If these tests require root (I assume
> they do?) then a sensible approach would be to check for that and to
> emit a warning and return "success".
Thanks for your advice.
I'm going to make the following changes on these scripts
1. Change these paths to:
tools/testing/selftests/{cpu,memory}-hotplug/on-off-test.sh
2. Skip tests and exit(0) with a warning if no root or no sysfs
so that a "make run_tests" doesn't stop.
3. Add tests that simply online and offline cpus (or memory blocks)
and then tests with this notifier error injection features if the
kernel supports.
> My overall take on the fault-injection code is that there has been a
> disappointing amount of uptake: I don't see many developers using them
> for whitebox testing their stuff. ?I guess this patchset addresses
> that, in a way.
I hope so. the impact of notifier error injection is restricted to
the particular kernel functionarity and these scripts are easy to run.
On the other hand, fault injection like failslab has a huge impact
on any kernel components and it often results catastrophe to userspace
even if no kernel bug. I am confident that I can find a certain amount
of kernel bugs with failslab but it requires enough spare time.
On Saturday, June 23, 2012, Akinobu Mita wrote:
> This provides the ability to inject artifical errors to PM notifier chain
> callbacks. It is controlled through debugfs interface under
> /sys/kernel/debug/notifier-error-inject/pm
>
> Each of the files in "error" directory represents an event which can be
> failed and contains the error code. If the notifier call chain should
> be failed with some events notified, write the error code to the files.
>
> If the notifier call chain should be failed with some events notified,
> write the error code to "actions/<notifier event>/error".
>
> Example: Inject PM suspend error (-12 = -ENOMEM)
>
> # cd /sys/kernel/debug/notifier-error-inject/pm
> # echo -12 > actions/PM_SUSPEND_PREPARE/error
> # echo mem > /sys/power/state
> bash: echo: write error: Cannot allocate memory
>
> Signed-off-by: Akinobu Mita <[email protected]>
> Cc: Pavel Machek <[email protected]>
> Cc: "Rafael J. Wysocki" <[email protected]>
> Cc: [email protected]
OK, I have no objections.
Thanks,
Rafael
> ---
> * v4
> - update modules to follow new interface
>
> lib/Kconfig.debug | 24 ++++++++++++++++++++
> lib/Makefile | 1 +
> lib/pm-notifier-error-inject.c | 49 ++++++++++++++++++++++++++++++++++++++++
> 3 files changed, 74 insertions(+)
> create mode 100644 lib/pm-notifier-error-inject.c
>
> diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
> index be0c197..246cea6 100644
> --- a/lib/Kconfig.debug
> +++ b/lib/Kconfig.debug
> @@ -1119,6 +1119,30 @@ config CPU_NOTIFIER_ERROR_INJECT
>
> If unsure, say N.
>
> +config PM_NOTIFIER_ERROR_INJECT
> + tristate "PM notifier error injection module"
> + depends on PM && NOTIFIER_ERROR_INJECTION
> + default m if PM_DEBUG
> + help
> + This option provides the ability to inject artifical errors to
> + PM notifier chain callbacks. It is controlled through debugfs
> + interface /sys/kernel/debug/notifier-error-inject/pm
> +
> + If the notifier call chain should be failed with some events
> + notified, write the error code to "actions/<notifier event>/error".
> +
> + Example: Inject PM suspend error (-12 = -ENOMEM)
> +
> + # cd /sys/kernel/debug/notifier-error-inject/pm/
> + # echo -12 > actions/PM_SUSPEND_PREPARE/error
> + # echo mem > /sys/power/state
> + bash: echo: write error: Cannot allocate memory
> +
> + To compile this code as a module, choose M here: the module will
> + be called pm-notifier-error-inject.
> +
> + If unsure, say N.
> +
> config FAULT_INJECTION
> bool "Fault-injection framework"
> depends on DEBUG_KERNEL
> diff --git a/lib/Makefile b/lib/Makefile
> index 23fba9e..230a949 100644
> --- a/lib/Makefile
> +++ b/lib/Makefile
> @@ -92,6 +92,7 @@ obj-$(CONFIG_IOMMU_HELPER) += iommu-helper.o
> obj-$(CONFIG_FAULT_INJECTION) += fault-inject.o
> obj-$(CONFIG_NOTIFIER_ERROR_INJECTION) += notifier-error-inject.o
> obj-$(CONFIG_CPU_NOTIFIER_ERROR_INJECT) += cpu-notifier-error-inject.o
> +obj-$(CONFIG_PM_NOTIFIER_ERROR_INJECT) += pm-notifier-error-inject.o
>
> lib-$(CONFIG_GENERIC_BUG) += bug.o
>
> diff --git a/lib/pm-notifier-error-inject.c b/lib/pm-notifier-error-inject.c
> new file mode 100644
> index 0000000..c094b2d
> --- /dev/null
> +++ b/lib/pm-notifier-error-inject.c
> @@ -0,0 +1,49 @@
> +#include <linux/kernel.h>
> +#include <linux/module.h>
> +#include <linux/suspend.h>
> +
> +#include "notifier-error-inject.h"
> +
> +static int priority;
> +module_param(priority, int, 0);
> +MODULE_PARM_DESC(priority, "specify PM notifier priority");
> +
> +static struct notifier_err_inject pm_notifier_err_inject = {
> + .actions = {
> + { NOTIFIER_ERR_INJECT_ACTION(PM_HIBERNATION_PREPARE) },
> + { NOTIFIER_ERR_INJECT_ACTION(PM_SUSPEND_PREPARE) },
> + { NOTIFIER_ERR_INJECT_ACTION(PM_RESTORE_PREPARE) },
> + {}
> + }
> +};
> +
> +static struct dentry *dir;
> +
> +static int err_inject_init(void)
> +{
> + int err;
> +
> + dir = notifier_err_inject_init("pm", notifier_err_inject_dir,
> + &pm_notifier_err_inject, priority);
> + if (IS_ERR(dir))
> + return PTR_ERR(dir);
> +
> + err = register_pm_notifier(&pm_notifier_err_inject.nb);
> + if (err)
> + debugfs_remove_recursive(dir);
> +
> + return err;
> +}
> +
> +static void err_inject_exit(void)
> +{
> + unregister_pm_notifier(&pm_notifier_err_inject.nb);
> + debugfs_remove_recursive(dir);
> +}
> +
> +module_init(err_inject_init);
> +module_exit(err_inject_exit);
> +
> +MODULE_DESCRIPTION("PM notifier error injection module");
> +MODULE_LICENSE("GPL");
> +MODULE_AUTHOR("Akinobu Mita <[email protected]>");
>