Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <a53e8c6d-b0e5-72bf-1aba-07609c9cf1bf@meta.com>
Date:   Tue, 22 Nov 2022 14:51:08 -0500
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0)
 Gecko/20100101 Thunderbird/102.5.0
Subject: Re: [PATCH] error-injection: Add prompt for function error injection
Content-Language: en-US
To:     Steven Rostedt <rostedt@goodmis.org>
Cc:     Borislav Petkov <bp@alien8.de>,
        Alexei Starovoitov <alexei.starovoitov@gmail.com>,
        LKML <linux-kernel@vger.kernel.org>,
        Linus Torvalds <torvalds@linux-foundation.org>,
        Masami Hiramatsu <mhiramat@kernel.org>,
        Andrew Morton <akpm@linux-foundation.org>,
        Peter Zijlstra <peterz@infradead.org>,
        Kees Cook <keescook@chromium.org>,
        Josh Poimboeuf <jpoimboe@redhat.com>,
        KP Singh <kpsingh@kernel.org>,
        Mark Rutland <mark.rutland@arm.com>,
        Florent Revest <revest@chromium.org>,
        Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
        Christoph Hellwig <hch@infradead.org>
References: <20221121104403.1545f9b5@gandalf.local.home>
 <Y3vSQo85ofkfD/L8@zn.tnic>
 <CAADnVQLJFnu6gARdZ7ckgxeGaSv70jHBiQo+W=zDKO0-ELFGcQ@mail.gmail.com>
 <Y3ymzAYRyH5IWEe/@zn.tnic> <3fa8ec60-dd96-c41f-ea46-8856bf855949@meta.com>
 <20221122132905.12a8d5ad@gandalf.local.home>
From:   Chris Mason <clm@meta.com>
In-Reply-To: <20221122132905.12a8d5ad@gandalf.local.home>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?OXpSekV5VEVyTURjYVgwNmNLRVc2TkpBa2tsZWFWYXc5YWlob0phbHM4Wncr?=
 =?utf-8?B?VmwxWDA5V3lxMWxYUm9ZS2dySzFUQyt6djhqTm9EdGprWk5Lc1JQMWxJZVVv?=
 =?utf-8?B?S1ZEczNUNXh3TmRNbUpJNTFOR0lXY1JVL0QwZWZZcHZUNWRZOHNyMytqV3Nx?=
 =?utf-8?B?UU40QzJQSEdoVVhQcG9NZnNBcEtTZ0dvNC85RGgzVCtvQ3Noajl5MG56R0ov?=
 =?utf-8?B?cmdPNVlOYUtleHNMUVpJa1hvLzlXeDhhelp3bnlCcnRtTlNQZU4rUFNUZTNK?=
 =?utf-8?B?ckZqTzlMSnJScXVjdlY3dG90MFNZM1pIYWVzVUZ2b2V6bWlHNGJkS0srT29p?=
 =?utf-8?B?YlROdk9SMk9MeittVDhqc0Z6c0R0TUgzUnFRWDBlOHVFZ2xVTnpzQmNLYlU1?=
 =?utf-8?B?anpXZFNLSXFaV21uS28rMUlEN2hGdmkwbGF2d29LcHo1VzV1czNNcm1kTkRa?=
 =?utf-8?B?T1psSTlrOG1zbkcwNjJ3eVhKYTBuYm1UaUNHbk9nU2ZwWWR1Z1hyb1krWVBt?=
 =?utf-8?B?SWIrUDU2WFFXeExUc2E4aXo3QWhmKzUxZXJQNG1pQm5ySU4wUHl2cjlPMUZo?=
 =?utf-8?B?dnR0TzJOMmg2RFBNdkFNRmNyd08wT3NRS25lOVBGUDJlWU9WdjdyWmQrM0tW?=
 =?utf-8?B?K1ZjV3RDaVM1UXEzMTVGNU1JUXZ1SG0rL0lWUXdXVm9haUlPRWlqMDdhOHhZ?=
 =?utf-8?B?WXlEVERkRlNxeUd0MFJTek5rRERmdnlDdk5nZ2xJNGtIYXBMSXY4L2Z2WXFY?=
 =?utf-8?B?dm40WkE3aklvcUxpRVM4WVZHSFFlajBLR3MvQ2p4d09rNlNCVnhqL0liMWJi?=
 =?utf-8?B?VmxxSmlna2FzUXFVVS9LazhMU3dNNC9WOGkrTFlVNDJZQTd5ZXFucHM1RjBp?=
 =?utf-8?B?ZnppdnlCZHYyT1VKUkxDOE05cVdLMzBGNndBT2E5S0U5VGNnWHVJeXlJcTJG?=
 =?utf-8?B?UnZRdE5pQTVyRVV4NlNIN2pyWWZnUmVPVzJVZTRpQktmYzIreGMwK3dRUElO?=
 =?utf-8?B?ZXVzcno0YzhpMHZjbjdtRmRFa3VYUHlZSWIvZWpDOHF3M0V4Z09Ja2lTWEhV?=
 =?utf-8?B?NlVXTUFJZU04dGEzeFliQUVKeWxSMVovVmpFZGpPRzVndnBoQlJlNXhUYmFi?=
 =?utf-8?B?UE1wVGQzQVlEdzlKY2tjOUx5eHFsM2pRUExCOVcrYVJNZVRBV2JRNmI2emU2?=
 =?utf-8?B?cXlXOWlQVUUyMzRjaFB5VlFkczY1MjE2OUc3d3pEV3VpWEVWMUZDQ0I1T2RG?=
 =?utf-8?B?ZFpQcVc3TUdqZUN4ZTJGNVJ1SUY5b2NTTkMvYWZaQlJMSmZGNUdpZDVzQ0l4?=
 =?utf-8?B?aVRnWFFlN1dseW4vWWNRZG5seDNnTFpOOE1IL3JmN3Z6dUdyVG54bEhwYThI?=
 =?utf-8?B?MGxCSDRaZms2cnlaUEYxOVpOd1FvSS9kTWFsQnBPbXVMTTh5QWljMlptTTUz?=
 =?utf-8?B?YVlGQVJUblE0RTBYOEtKUFBjNU52SzNxeEJWY1c3TEF6ZURoYk4wSHdJVDdE?=
 =?utf-8?B?UCs5ZmlxcjM0aFhPQk5Jdll4d3p4RjgrUng4MG1YOHJ1UFVhRjFtNC84Q0NT?=
 =?utf-8?B?T21ENmZRbFEyaDBia1hRbFQ2cTF2WSt6MDJISTFOOXphMzRyNmNiUnVDanI2?=
 =?utf-8?B?bWt1WE1DWFI1UVVJeHM4SEJQSnV5WENPV3VTLzhibzA3OXZxaVBFbWZkdWo3?=
 =?utf-8?B?YUNrOGhmRlMyTVNWQkVVUHF6a2QreC9WR2VGMjhYRlVia0VFKzF3MlNlTyty?=
 =?utf-8?B?NnZIeXhzbHBxT2JadmdrMWZJUnBMUGNTdVE5cU9HR3lRNnlUUldCVDczK3kv?=
 =?utf-8?B?UTd2S2QrU2pneEhITmE2MW8xeG5QZXVoSk5IZ0NOdURVQ1hQbGVBcHIwd0dU?=
 =?utf-8?B?M1JBNkoyR1V1TWlIYWhCc0V0NHQ5K0VlbTI3d0duYUtLdHpaNWF5bll5aUhl?=
 =?utf-8?B?dGhVNjFEdWFsVVBGb2NkbHc3enpPVmFnQmRjdytuMElQVHA3SXI2T1IyVUZ0?=
 =?utf-8?B?c2QvcUFMT3ZWYWpjVitTeUNId1k0azZ0Q29pRzYwYlU4N0ozc29NTWIxQU5l?=
 =?utf-8?B?UVZHbmljaXExOGdickJEa2ZyeTZUOFJQOVp3TXJpSnJISlh6WlUvSWIzTGJI?=
 =?utf-8?Q?BvDY=3D?=
X-OriginatorOrg: meta.com
X-MS-Exchange-CrossTenant-Network-Message-Id: bd7a527f-3cfa-4677-7650-08daccc2e753
X-MS-Exchange-CrossTenant-AuthSource: MN2PR15MB4287.namprd15.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Nov 2022 19:51:10.4949
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 8ae927fe-1255-47a7-a2af-5f3a069daaa2
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: oF9SZWfKC4yKvJLpQU3H4EiBCI4iheXVuoVX+XclEJ9t4/uN9e4mT2rpSQntRyI/
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR15MB4136
X-Proofpoint-ORIG-GUID: AxTuKf1Bp88dkRVYsdl7ms78o5QfYpQy
X-Proofpoint-GUID: AxTuKf1Bp88dkRVYsdl7ms78o5QfYpQy
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.219,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1
 definitions=2022-11-22_11,2022-11-18_01,2022-06-22_01
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 11/22/22 1:29 PM, Steven Rostedt wrote:
> On Tue, 22 Nov 2022 12:42:33 -0500
> Chris Mason <clm@meta.com> wrote:
> 
>> On 11/22/22 5:39 AM, Borislav Petkov wrote:
>>> On Mon, Nov 21, 2022 at 03:36:08PM -0800, Alexei Starovoitov wrote:  
>>>> The commit log is bogus and the lack of understanding what  
>>>
>>> You mean that:
>>>
>>> Documentation/fault-injection/fault-injection.rst
>>>
>>> ?
>>>
>>> I don't want any of that possible in production setups. And until you
>>> give me a sane argument why it is good to have in production setups
>>> generically, this is end of story.
>>>   
>>
>> I think there are a few different sides to this:
>>
>> - it makes total sense that we all have wildly different ideas about
>> which tools should be available in prod.  Making this decision more fine
>> grained seems reasonable.
>>
>> - fault injection for testing: we have a stage of qualification that
>> does error injection against the prod kernel.  It helps to have this
>> against the debug kernel too, but that misses some races etc.  I always
>> just assumed distros and partners did some fault injection tests against
>> the prod kernel builds?
>>
>> - fault injection for debugging:  it doesn't happen often but at some
>> point we run out of ideas and start making different functions fail in
>> prod to figure out why we're not prodding.
> 
> As you have stated, we have different ideas for production. Your POV is
> cloud based (as is with other parts of my company). But my POV is
> Chromebooks where production means what's on a user's device. There's no
> reason to ever have fault injection enabled in such cases. I would assume
> that distributions are the same. But having kprobes for visibility can also
> be useful for debugging purposes, even in the field.
> 

Yeah, I definitely don't have opinions on the right way to build a
chromebook, and replying to Boris, only slightly better at distros.
Josef's original intent was this be easy to turn off.

>>
>> - overriding return values for security fixes: also not a common thing,
>> but it's a tool we've used.  There are usually better long term fixes,
>> but it happens.
>>
>> Stepping back to the big picture of debugging systems with bpf in use, I
>> love hearing (and telling) stories of debugging difficult problems.  As
>> far as I know, BPF telling lies hasn't really been a problem for us, so
>> even though it's a huge tangent, if you have specific examples of
>> problems you've seen, I'm really interested in hearing more.
>>
>> When I talk about production, both overall stability and validating new
>> kernels, if I compare the BPF subsystem with MM, filesystems, cgroups,
>> the scheduler, networking, and all things Jens, the systems BPF
>> developers put in place are working really well for me.
>>
>> If I expand the discussion to the BPF programs themselves, there have
>> been rare issues.   Still completely on par with the rest of the kernel
>> subsystems and within the noise in comparison with hardware failures.
>>
>> In other words, I really do care about the concerns you're expressing
>> here, and I'm usually first in line to complain when random people make
>> my job harder.  I'm just not seeing these issues with BPF, and I see
>> them actively trying to increase safety over time.
> 
> I'm sure you are not seeing theses issues with BPF, as the main developers
> and you have the same focus areas.
> 
> I have no problem with the concept of BPF. My concern is mostly the
> development side of it. As you can basically attach functionality to
> arbitrary points in the kernel via BPF programs, the perception is that
> anything that is available is fair game. BPF tends to expand features
> beyond their intended usage. Heck, look at the name itself. "extended
> Berkeley Packet Filter", were eBPF has nothing to do with packet filtering
> anymore. Perhaps it should be renamed to CUST (Compiled Use Space
> Trampoline) ;-)

Developers in general tend to stretch interfaces a lot.  At some point
the friction of using the interface is worse than the friction of
changing it, and things get redone.  At the end of the day, BPF
developers are still kernel developers and we end up with relatively
sane feedback loops.

> 
> Alexei said it's "sad" about my expression of BPF and error injection. If
> it has to do with security, then I would like to see more collaboration
> with the security folks and perhaps have BPF integrate with their
> infrastructure.

Now is a great time to grab KP and hear all about BPF LSM.

> But the usual response is "that's not fast enough for me"
> and then something is done from scratch without working with that
> subsystem to make it fast enough. Yes, it takes more time to collaborate
> than just doing it on your own. But that's the nature of an open source
> *community*.

One of the awkward and wonderful parts of our community is that none of
us have the same goals or needs.  Going back to the original thread, ARM
has either one or two different live patching subsystems in use in the
industry, and neither is upstream.

One reason you end up having these arguments often with BPF is because
they stick around and work with the community to upstream their work.
The tradeoffs, compromises and decisions aren't always what you want,
but we all show up every day and keep engaging.

-chris