Received: by 2002:a05:6a10:d5a5:0:0:0:0 with SMTP id gn37csp400050pxb; Thu, 30 Sep 2021 08:24:54 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyIuAvGmEGa4CZzbxFvBZtFHSIhpBvnbQC8v6h98fB3kInLmtaw3KWcs+HmPhTvdpJCdSW4 X-Received: by 2002:a17:903:32ce:b0:13e:2245:6fc8 with SMTP id i14-20020a17090332ce00b0013e22456fc8mr4600330plr.67.1633015493863; Thu, 30 Sep 2021 08:24:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1633015493; cv=none; d=google.com; s=arc-20160816; b=Yw5FFPlU/Cv95ejp91mFQrDvz0wAQmhpiPiu4Npws8VazVqfRZ3hFm1huqG1ISElqZ kvt1/i6iOSpdS/PqWYJBNm5iwPvT6yhfAn0hh9j5ljGWGxoBNnfwWH7BgRcBs1GSHrNk 1KRDN0eS5t2/ORS/efkZYlD0AHMIw5k2qutraG6vGNbIfQpxW8GAabNDk7Fmolpw/DiJ 1gZ2wDYDXx7nJHsU7WqlHnnuvsxCek3VsxxmCxpqA2R/YCuMDpJ6Y3SR8oQaG/7Dhatw MqFUT2iSFHdDA8GIK2XCjQlzwPm7n4NAiueIVpuGeJuoSAZTrrG7aj8gAOe5uyHt0dZK hP7A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:content-language :in-reply-to:mime-version:user-agent:date:message-id:from:references :cc:to:subject:reply-to; bh=afuuRoWEct2MfL+JBgtz3aEAzNrXRi3zezmIOBkaE24=; b=BXB1TN1tG3NtrGTnAGylkc2UVHMhUa+lFWRULzhX8lca+rjFDJcsqDRIpH6g34gZEx 49pJCIqLDtgctu78/XJQQd/OzlQ1Gs6JTZxTRsnPn5Kg8yPKVgXfh+3uCKoWYWrXrvRJ S+ppFYWkFn2onTjOoXvzxa8lgx7Dr60C3xc3D82FLwGP0jPYJbbBFhkOfSGJ0p00tBQK l3EriFgjIsVM2VoPumgjD1izPLY0scjiPS2RruAtGEoWzS1/qS7fspOpdK6vlFmpDNe0 Sn2v6Pm3W00EHahYt3ZKJqwCfG9kXGaV2vn04fjNdqg2cpTAey9onmnUEowN9dVyr0GS X72g== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id v64si2853789pgd.272.2021.09.30.08.24.40; Thu, 30 Sep 2021 08:24:53 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244451AbhI3PKf (ORCPT + 99 others); Thu, 30 Sep 2021 11:10:35 -0400 Received: from mail-ed1-f49.google.com ([209.85.208.49]:40656 "EHLO mail-ed1-f49.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239919AbhI3PKY (ORCPT ); Thu, 30 Sep 2021 11:10:24 -0400 Received: by mail-ed1-f49.google.com with SMTP id g8so23567170edt.7; Thu, 30 Sep 2021 08:08:41 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:subject:to:cc:references:from :message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=afuuRoWEct2MfL+JBgtz3aEAzNrXRi3zezmIOBkaE24=; b=cjkHe6Z9RmQhXpASuDZVgqkAhu5NNqeN2F9KqpQIbdKFEW9uF/dAycbLqvzd3EhjcH n+EHwCpPXMJuwopjuP5hvostD4oODPap0CFDubgmIw+3xu3rKCv6wk7YkjKHyLPSmNXB umekLb3wjbOGqLHlRFBBdyS9cHD0cLFvgsPbXM3EEFbRNW1l/WMkxUB5NlviSORz0EV3 sTY+I6ZwD/mAgskbL/xXBQTywGJWT7g5bn1e8FU49ueBPF4NtTu60L9eH9HMswS652FR ibBL3cD8X1z2EJsLyVcKot77EtGUDvoRLEkgp19yHrz/j9OOQRSDNPIoR6bvW9SH0UdS pkGg== X-Gm-Message-State: AOAM530FmcQECKLjJO2JBtMpNpLYQLB5ucGvXQrnft50SE2CGExuFIoC SV5pB0p5DCmVqE9eM2LtTLw= X-Received: by 2002:a17:906:f6cf:: with SMTP id jo15mr7441915ejb.244.1633014360379; Thu, 30 Sep 2021 08:06:00 -0700 (PDT) Received: from [10.9.0.26] ([46.166.133.199]) by smtp.gmail.com with ESMTPSA id v22sm992164eds.20.2021.09.30.08.05.56 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 30 Sep 2021 08:05:59 -0700 (PDT) Reply-To: alex.popov@linux.com Subject: Re: [PATCH] Introduce the pkill_on_warn boot parameter To: Petr Mladek , "Paul E. McKenney" Cc: Jonathan Corbet , Andrew Morton , Thomas Gleixner , Peter Zijlstra , Joerg Roedel , Maciej Rozycki , Muchun Song , Viresh Kumar , Robin Murphy , Randy Dunlap , Lu Baolu , Kees Cook , Luis Chamberlain , Wei Liu , John Ogness , Andy Shevchenko , Alexey Kardashevskiy , Christophe Leroy , Jann Horn , Greg Kroah-Hartman , Mark Rutland , Andy Lutomirski , Dave Hansen , Steven Rostedt , Will Deacon , David S Miller , Borislav Petkov , kernel-hardening@lists.openwall.com, linux-hardening@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, notify@kernel.org, Linus Torvalds , Dmitry Vyukov References: <20210929185823.499268-1-alex.popov@linux.com> <20210929194924.GA880162@paulmck-ThinkPad-P17-Gen-1> From: Alexander Popov Message-ID: Date: Thu, 30 Sep 2021 18:05:54 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 30.09.2021 12:15, Petr Mladek wrote: > On Wed 2021-09-29 12:49:24, Paul E. McKenney wrote: >> On Wed, Sep 29, 2021 at 10:01:33PM +0300, Alexander Popov wrote: >>> On 29.09.2021 21:58, Alexander Popov wrote: >>>> Currently, the Linux kernel provides two types of reaction to kernel >>>> warnings: >>>> 1. Do nothing (by default), >>>> 2. Call panic() if panic_on_warn is set. That's a very strong reaction, >>>> so panic_on_warn is usually disabled on production systems. > > Honestly, I am not sure if panic_on_warn() or the new pkill_on_warn() > work as expected. I wonder who uses it in practice and what is > the experience. > > The problem is that many developers do not know about this behavior. > They use WARN() when they are lazy to write more useful message or when > they want to see all the provided details: task, registry, backtrace. > > Also it is inconsistent with pr_warn() behavior. Why a single line > warning would be innocent and full info WARN() cause panic/pkill? > > What about pr_err(), pr_crit(), pr_alert(), pr_emerg()? They inform > about even more serious problems. Why a warning should cause panic/pkill > while an alert message is just printed? That's a good question. I guess various kernel continuous integration (CI) systems have panic_on_warn enabled. [Adding Dmitry Vyukov to this discussion] If we look at the syzbot dashboard [1] with the results of Linux kernel fuzzing, we see the issues that appear as various kernel crashes and warnings. We don't see anything from pr_err(), pr_crit(), pr_alert(), pr_emerg(). Maybe these situations are not considered as kernel bugs that require fixing. Anyway, from a security point of view, a kernel warning output is interesting for attackers as an infoleak. The messages printed by pr_err(), pr_crit(), pr_alert(), pr_emerg() provide less information. [1]: https://syzkaller.appspot.com/upstream > It somehow reminds me the saga with %pK. We were not able to teach > developers to use it correctly for years and ended with hashed > pointers. > > Well, this might be different. Developers might learn this the hard > way from bug reports. But there will be bug reports only when > anyone really enables this behavior. They will enable it only > when it works the right way most of the time. > > >>>> From a safety point of view, the Linux kernel misses a middle way of >>>> handling kernel warnings: >>>> - The kernel should stop the activity that provokes a warning, >>>> - But the kernel should avoid complete denial of service. >>>> >>>> From a security point of view, kernel warning messages provide a lot of >>>> useful information for attackers. Many GNU/Linux distributions allow >>>> unprivileged users to read the kernel log, so attackers use kernel >>>> warning infoleak in vulnerability exploits. See the examples: >>>> https://a13xp0p0v.github.io/2020/02/15/CVE-2019-18683.html >>>> https://a13xp0p0v.github.io/2021/02/09/CVE-2021-26708.html >>>> >>>> Let's introduce the pkill_on_warn boot parameter. >>>> If this parameter is set, the kernel kills all threads in a process >>>> that provoked a kernel warning. This behavior is reasonable from a safety >>>> point of view described above. It is also useful for kernel security >>>> hardening because the system kills an exploit process that hits a >>>> kernel warning. >>>> >>>> Signed-off-by: Alexander Popov >>> >>> This patch was tested using CONFIG_LKDTM. >>> The kernel kills a process that performs this: >>> echo WARNING > /sys/kernel/debug/provoke-crash/DIRECT >>> >>> If you are fine with this approach, I will prepare a patch adding the >>> pkill_on_warn sysctl. >> >> I suspect that you need a list of kthreads for which you are better >> off just invoking panic(). RCU's various kthreads, for but one set >> of examples. > > I wonder if kernel could survive killing of any kthread. I have never > seen a code that would check whether a kthread was killed and > restart it. The do_group_exit() function calls do_exit() from kernel/exit.c, which is also called during a kernel oops. This function cares about a lot of special cases depending on the current task_struct. Is it fine? Best regards, Alexander