Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754168AbeALAa4 (ORCPT + 1 other); Thu, 11 Jan 2018 19:30:56 -0500 Received: from mail-pf0-f194.google.com ([209.85.192.194]:45131 "EHLO mail-pf0-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753186AbeALAay (ORCPT ); Thu, 11 Jan 2018 19:30:54 -0500 X-Google-Smtp-Source: ACJfBotOmCNu7xnQswQCmSFQyRXZVApbndR/vvAuk2AmV+2C0dz8DZAGffcsAZ+56VlakBD4mDqWmQFkWxY7TcIqoaI= MIME-Version: 1.0 In-Reply-To: <20180112011552.afc97df0651d1f1c7f919363@kernel.org> References: <151563173430.628.5070705610243278195.stgit@devbox> <151563188390.628.13138319905985348016.stgit@devbox> <20180112011552.afc97df0651d1f1c7f919363@kernel.org> From: Akinobu Mita Date: Fri, 12 Jan 2018 09:30:32 +0900 Message-ID: Subject: Re: [PATCH bpf-next v4 5/5] error-injection: Support fault injection framework To: Masami Hiramatsu Cc: Alexei Starovoitov , Josef Bacik , Steven Rostedt , Ingo Molnar , "David S. Miller" , netdev , LKML , ast@kernel.org, kernel-team@fb.com, daniel@iogearbox.net, linux-btrfs@vger.kernel.org, darrick.wong@oracle.com, Josef Bacik Content-Type: text/plain; charset="UTF-8" Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Return-Path: 2018-01-12 1:15 GMT+09:00 Masami Hiramatsu : > On Thu, 11 Jan 2018 23:44:57 +0900 > Akinobu Mita wrote: > >> 2018-01-11 9:51 GMT+09:00 Masami Hiramatsu : >> > Support in-kernel fault-injection framework via debugfs. >> > This allows you to inject a conditional error to specified >> > function using debugfs interfaces. >> > >> > Here is the result of test script described in >> > Documentation/fault-injection/fault-injection.txt >> > >> > =========== >> > # ./test_fail_function.sh >> > 1+0 records in >> > 1+0 records out >> > 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227404 s, 46.1 MB/s >> > btrfs-progs v4.4 >> > See http://btrfs.wiki.kernel.org for more information. >> > >> > Label: (null) >> > UUID: bfa96010-12e9-4360-aed0-42eec7af5798 >> > Node size: 16384 >> > Sector size: 4096 >> > Filesystem size: 1001.00MiB >> > Block group profiles: >> > Data: single 8.00MiB >> > Metadata: DUP 58.00MiB >> > System: DUP 12.00MiB >> > SSD detected: no >> > Incompat features: extref, skinny-metadata >> > Number of devices: 1 >> > Devices: >> > ID SIZE PATH >> > 1 1001.00MiB /dev/loop2 >> > >> > mount: mount /dev/loop2 on /opt/tmpmnt failed: Cannot allocate memory >> > SUCCESS! >> > =========== >> > >> > >> > Signed-off-by: Masami Hiramatsu >> > Reviewed-by: Josef Bacik >> > --- >> > Changes in v3: >> > - Check and adjust error value for each target function >> > - Clear kporbe flag for reuse >> > - Add more documents and example >> > --- >> > Documentation/fault-injection/fault-injection.txt | 62 ++++++ >> > kernel/Makefile | 1 >> > kernel/fail_function.c | 217 +++++++++++++++++++++ >> > lib/Kconfig.debug | 10 + >> > 4 files changed, 290 insertions(+) >> > create mode 100644 kernel/fail_function.c >> > >> > diff --git a/Documentation/fault-injection/fault-injection.txt b/Documentation/fault-injection/fault-injection.txt >> > index 918972babcd8..4aecbceef9d2 100644 >> > --- a/Documentation/fault-injection/fault-injection.txt >> > +++ b/Documentation/fault-injection/fault-injection.txt >> > @@ -30,6 +30,12 @@ o fail_mmc_request >> > injects MMC data errors on devices permitted by setting >> > debugfs entries under /sys/kernel/debug/mmc0/fail_mmc_request >> > >> > +o fail_function >> > + >> > + injects error return on specific functions, which are marked by >> > + ALLOW_ERROR_INJECTION() macro, by setting debugfs entries >> > + under /sys/kernel/debug/fail_function. No boot option supported. >> > + >> > Configure fault-injection capabilities behavior >> > ----------------------------------------------- >> > >> > @@ -123,6 +129,24 @@ configuration of fault-injection capabilities. >> > default is 'N', setting it to 'Y' will disable failure injections >> > when dealing with private (address space) futexes. >> > >> > +- /sys/kernel/debug/fail_function/inject: >> > + >> > + specifies the target function of error injection by name. >> > + >> > +- /sys/kernel/debug/fail_function/retval: >> > + >> > + specifies the "error" return value to inject to the given >> > + function. >> > + >> >> Is it possible to inject errors into multiple functions at the same time? > > Yes, it is. > >> If so, it will be more useful to support it in the fault injection, too. >> Because some kind of bugs are caused by the combination of errors. >> (e.g. another error in an error path) >> >> I suggest the following interface. >> >> - /sys/kernel/debug/fail_function/inject: >> >> specifies the target function of error injection by name. >> /sys/kernel/debug/fail_function// directory will be created. >> >> - /sys/kernel/debug/fail_function/uninject: >> >> specifies the target function of error injection by name that is >> currently being injected. /sys/kernel/debug/fail_function// >> directory will be removed. >> >> - /sys/kernel/debug/fail_function//retval: >> >> specifies the "error" return value to inject to the given function. > > OK, it is easy to make it. But also we might need to consider using bpf > if we do such complex error injection. > > BTW, would we need "uninject" file? or just make inject file accept > "!function" syntax to remove function as ftrace does? It also sounds good. Either way is fine with me.