Received: by 2002:ac0:bc90:0:0:0:0:0 with SMTP id a16csp4379096img; Tue, 26 Mar 2019 08:18:07 -0700 (PDT) X-Google-Smtp-Source: APXvYqyU8EicIm7d3zA2kRz5HYDVy1LtFtLL0yEpNgU54kiBh8ThQ6rElevwE8+83wAZinl4Sy26 X-Received: by 2002:a17:902:f08a:: with SMTP id go10mr18697670plb.121.1553613487473; Tue, 26 Mar 2019 08:18:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1553613487; cv=none; d=google.com; s=arc-20160816; b=yh4AK57B1eNxdWgOXcj7MlOma+LZl6/lPFtHjtpwJRAKJWP79cD0Yl/Jz4SFkOGGvw tdFgSyY65jInMSDZ6/pwTYCPrYm8hQfOXQUpHO6qtjPjrGAEXj05CrpHcWxpXCWWl/yC 35si4G74iefkYEha+J30uJyzRBVo15J6dFrcLwsKYIYWj8e8DvE3YvSuYATb61EA/3BA 9eIKku8biWR2QqB6cnWzIEjR9zJM5n6fvezp7TEHSUyvz7eQMwFmmMXmfdFYv5ivWbUI Zi5SBcs98K1w9ZV7MhtCzH9UvDcZ2LIliHpGtTBzAedcq1rfXOC5LnGjEfLZxRcVy+6c 99SQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:user-agent:in-reply-to :content-disposition:mime-version:references:message-id:subject:cc :to:from:date; bh=XCTT4HeI24lZ4s3NPPhbU2k8cL7vAe8DX3JDmVZbazM=; b=Id6+t9XdAiKcZlTEfxAci3y+iyaKB1dzaWyZv2Mkx0VMUv1JcvVr8FaH+kJP4F1mgv HWjMshQyvRJlGFnAeUD6lBNDJK+TQ6gtK52HkB1y4ePtmRdgomQcLdfaJ7AcgswMpLFQ 8srAtEnkWFCzBju7mjhx4r0qnC6xQlLndJ0T5bvmfrYe0lT+IltfhyEXQ0UKgeMs4iDr gcaxP7EJz8FizPabLL1TCzXNl0RPc8tpNnsvz0noCz5nVcIqG4r2q/hd9Dmg2HW9C2l/ VWu+C7aD6xspgT/SVj0cp7V/NGUOTD8qUIbL0OC8XTDO6UHu91mj76C0xpqAuJqccz7h E8Qw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=canonical.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id g2si13574534pgc.225.2019.03.26.08.17.51; Tue, 26 Mar 2019 08:18:07 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=canonical.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731943AbfCZPRS (ORCPT + 99 others); Tue, 26 Mar 2019 11:17:18 -0400 Received: from youngberry.canonical.com ([91.189.89.112]:42183 "EHLO youngberry.canonical.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726266AbfCZPRR (ORCPT ); Tue, 26 Mar 2019 11:17:17 -0400 Received: from mail-wm1-f71.google.com ([209.85.128.71]) by youngberry.canonical.com with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.76) (envelope-from ) id 1h8npE-0002v5-0t for linux-kernel@vger.kernel.org; Tue, 26 Mar 2019 15:17:16 +0000 Received: by mail-wm1-f71.google.com with SMTP id 16so6663282wme.0 for ; Tue, 26 Mar 2019 08:17:16 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=XCTT4HeI24lZ4s3NPPhbU2k8cL7vAe8DX3JDmVZbazM=; b=Hri7cngtoKyYsqmAxqjtCirWHBTwU+pf8B8IuFgBU+4YKcC85p6ouMyM9Tr/e0GCOR s/6wpPCbNhJZFDfKWT3HReAjVUNLPPwRx1dg3Eh7qSJW/U9N34klBXwF1tAs+sG4+vJP SztMtsx3MRZIrYR+RCLN8sdiIFybJ1iuBlz8eX/Sb1mm6K5I79OE60niMACf4O+4onV3 00ZU773mXQkuRYlmmmPMQ6oThc70/mYjkGJYDVWgq7zZuQjc8/e8KTPkNzsa5UKsHNoZ FMRRhdDnP07NklgifMyyVcRAwN06gwd3PaaqxFj9yLfrX+la6jLDUO0ixN5a8lqdh4nC 8MzQ== X-Gm-Message-State: APjAAAXUf41X7nDW6E8XynnCVt8+7iW2hGKLWf19/FBzqilpyp+lsQnr K5n+aDuNAflUNsWCHon2Q7djC0kkvlc4p4KZ2n4//Nnzyu7YXpa50sOHyVPfUZyd8ospB0yuJz8 KPk2Xxjxu/fwynRuqL893mmGTnJTpepKMsEYRKni7qw== X-Received: by 2002:a5d:4e82:: with SMTP id e2mr21012580wru.164.1553613435338; Tue, 26 Mar 2019 08:17:15 -0700 (PDT) X-Received: by 2002:a5d:4e82:: with SMTP id e2mr21012571wru.164.1553613435168; Tue, 26 Mar 2019 08:17:15 -0700 (PDT) Received: from localhost (host118-125-dynamic.3-87-r.retail.telecomitalia.it. [87.3.125.118]) by smtp.gmail.com with ESMTPSA id x192sm21091566wmf.48.2019.03.26.08.17.14 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 26 Mar 2019 08:17:14 -0700 (PDT) Date: Tue, 26 Mar 2019 16:17:13 +0100 From: Andrea Righi To: Masami Hiramatsu Cc: Steven Rostedt , Ingo Molnar , peterz@infradead.org, Mathieu Desnoyers , linux-kernel Subject: Re: [PATCH -tip v3 04/10] x86/kprobes: Prohibit probing on IRQ handlers directly Message-ID: <20190326151713.GB27897@xps-13> References: <154998785011.31052.1475728497912659748.stgit@devbox> <154998796400.31052.8406236614820687840.stgit@devbox> <20190325172334.559f8c5b@gandalf.local.home> <20190326235052.793b81b121021020fb3b1e93@kernel.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20190326235052.793b81b121021020fb3b1e93@kernel.org> User-Agent: Mutt/1.10.1 (2018-07-13) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Mar 26, 2019 at 11:50:52PM +0900, Masami Hiramatsu wrote: > On Mon, 25 Mar 2019 17:23:34 -0400 > Steven Rostedt wrote: > > > On Wed, 13 Feb 2019 01:12:44 +0900 > > Masami Hiramatsu wrote: > > > > > Prohibit probing on IRQ handlers in irqentry_text because > > > if it interrupts user mode, at that point we haven't changed > > > to kernel space yet and which eventually leads a double fault. > > > E.g. > > > > > > # echo p apic_timer_interrupt > kprobe_events > > > > Hmm, this breaks one of my tests (which I probe on do_IRQ). > > OK, it seems this patch is a bit redundant, because > I found that these interrupt handler issue has been fixed > by Andrea's commit before merge this patch. > > commit a50480cb6d61d5c5fc13308479407b628b6bc1c5 > Author: Andrea Righi > Date: Thu Dec 6 10:56:48 2018 +0100 > > kprobes/x86: Blacklist non-attachable interrupt functions > > These interrupt functions are already non-attachable by kprobes. > Blacklist them explicitly so that they can show up in > /sys/kernel/debug/kprobes/blacklist and tools like BCC can use this > additional information. > > This description is a bit odd (maybe his patch is after mine?) I think > while updating this series, the patches were merged out of order. > Anyway, with above patch, the core problematic probe points are blacklisted. This is the previous thread when I posted my patch (not sure if it helps to figure out what happened - maybe it was just an out of order merge issue, like you said): https://lkml.org/lkml/2018/12/6/212 > > > > > It's been working for years. > > > > > > > # echo 1 > events/kprobes/enable > > > PANIC: double fault, error_code: 0x0 > > > CPU: 1 PID: 814 Comm: less Not tainted 4.20.0-rc3+ #30 > > > Hardware name: QEMU Standard PC (i440FX + PIIX, 1996) > > > RIP: 0010:error_entry+0x12/0xf0 > > > [snip] > > > Call Trace: > > > > > > ? native_iret+0x7/0x7 > > > ? async_page_fault+0x8/0x30 > > > ? trace_hardirqs_on_thunk+0x1c/0x1c > > > ? error_entry+0x7c/0xf0 > > > ? async_page_fault+0x8/0x30 > > > ? native_iret+0x7/0x7 > > > ? int3+0xa/0x20 > > > ? trace_hardirqs_on_thunk+0x1c/0x1c > > > ? error_entry+0x7c/0xf0 > > > ? int3+0xa/0x20 > > > ? apic_timer_interrupt+0x1/0x20 > > > > > > Kernel panic - not syncing: Machine halted. > > > Kernel Offset: disabled > > > > I'm not able to reproduce this (by removing this commit). > > I ensured that if I revert both of this patch and Andrea's patch, > I can reproduce this with probing on apic_timer_interrupt(). > > > I'm thinking something else may have changed, as I've been tracing > > interrupt entries for years, and interrupting userspace while doing > > this. > > > > I've even added probes where ftrace isn't (where it uses an int3) and > > still haven't hit a problem. > > > > I think this patch is swatting a symptom of a bug and not addressing > > the bug itself. Can you send me the config that triggers this? > > Yes, it seems you're right. Andrea's commit specifically fixed the > issue and mine is redundant. (I'm not sure why do_IRQ is in > __irqentry_text...) Not sure if there are specific reasons for that, but do_IRQ is part of __irqentry_text because it's explicitly marked with __irq_entry. > > So, Ingo, please revert this, since this bug already has been fixed by > commit a50480cb6d61 ("kprobes: x86_64: blacklist non-attachable interrupt > functions") > > BTW, for further error investigation, I attached my kconfig which is > usually I'm testing (some options can be changed) on Qemu. > I'm using my mini-container shellscript ( https://github.com/mhiramat/mincs > ) which supports qemu-container. > > > Thank you, > > -- > Masami Hiramatsu Thanks, -Andrea