Received: by 2002:a05:6a10:8c0a:0:0:0:0 with SMTP id go10csp923480pxb; Sat, 6 Mar 2021 23:53:45 -0800 (PST) X-Google-Smtp-Source: ABdhPJwNmA1aP3kypcJ33tVe4B9bBn1o+ez4+6MjzSlSEnOhKgNYVKfXJSTNNYnTRMn31a2Pra0K X-Received: by 2002:a05:6402:215:: with SMTP id t21mr16944034edv.363.1615103624995; Sat, 06 Mar 2021 23:53:44 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1615103624; cv=pass; d=google.com; s=arc-20160816; b=UAokVcPMpql5x9bo3A19hsncgyixTp2YOzaThaYxzvT/XjifDb1BshxxZbKHrca/Mb Li+GNP3xwPpR6Vyr8rDK8aziOhlwlfv21Xibuq5tLFVfELrSKx2bffCbbPugyIeY6bdi 2WCQAicDjUWuNwR+wXqSRCNZeWD1diHVls8AKDJmVGVMt2+ZjK6bSy8scsqN8K0mqlsq pgUF7zVs7dntoSwe1Tmrn+z4z7QF6J+2/tbTVEa/jBDLy2GpPUPtMwF6Xutf1RekIWki EQWT9jmjVwzCaIvEKMNhd4qFeo2dKVYBMECY/Rl4XgpCrwpjEQzflSCcbdIbBEEwQxVj e7mg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:to:from:dkim-signature; bh=HEEEysoWkKp/UIS/qSkdJPSpBjqYkSPRs79w/tJEBfk=; b=VHL6cbJWNjKrt2aTpjCHxLIeB/20LCfdCWBBW23qXzjBOcqrisBX+zpkozxArREhhv KH3HJbKBIIOLDpmGJzmx/3csVvIpTYMQ1tL+nOStx4+mMgVrlVRtQ7WPO6IFsLChEevr Ijj2fJsCVOvHtHA233aoe/fSIJVYF3ev92B5ISn4pGqVnPzLEr7Y2SgjXfdZCbQ0r5cF ZcNngdUtRFRySoCR0vzbBKNHnPkiKKZjg4D1M/WWLrhIpUxse+65IZKnfxxu65MOZoRL 5JA+T/FwpTFTsVosunlWuu5MPn7bK8Bwa+antPDyYXdlWC+2mWK2CVt1G5yz5QOIKr2R Farg== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@schoebel-theuer.de header.s=strato-dkim-0002 header.b=iPXfNWHV; arc=pass (i=1); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id c13si4754650eje.45.2021.03.06.23.53.22; Sat, 06 Mar 2021 23:53:44 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@schoebel-theuer.de header.s=strato-dkim-0002 header.b=iPXfNWHV; arc=pass (i=1); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230390AbhCGH1J (ORCPT + 99 others); Sun, 7 Mar 2021 02:27:09 -0500 Received: from mo4-p01-ob.smtp.rzone.de ([85.215.255.53]:12897 "EHLO mo4-p01-ob.smtp.rzone.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230070AbhCGH0v (ORCPT ); Sun, 7 Mar 2021 02:26:51 -0500 ARC-Seal: i=1; a=rsa-sha256; t=1615101994; cv=none; d=strato.com; s=strato-dkim-0002; b=p80Nk2wCYoGWK7X6PbZ1QUKdxi7BvGaVHD9QvqwDPEb9UuYKqJQpK5deeAxG5zxQdC tSbagLAM3WZ/+ZpCbopamOudP/gtJGj6oEaoxw1OodpgHj1tL9iNnXkhNiVoODwHxxO+ YA4r/R+WMWarKwi7JCqLkpGkqf33PURWWbgmU9BJc29sT6HG6HW93zzzWZ8lISDCfrHR eQCPl5adVuWEWsuMLgOKsd/GndBI8JF2LEUpfBkUXhZBx2IhC1U7nAKHOV74n+QiBv9f u+cvf4OH4H3+61Nnq5eMJQ7fTkQQ/QCy6NUHBBRYWjL8i6VrIA7K+W+JAZEAqDVuW0Ni 3A1A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; t=1615101994; s=strato-dkim-0002; d=strato.com; h=Message-Id:Date:Subject:To:From:Cc:Date:From:Subject:Sender; bh=HEEEysoWkKp/UIS/qSkdJPSpBjqYkSPRs79w/tJEBfk=; b=BNLUThCiIMO1RGCx4muOvw0W6AgpymeHESx1EjfBY/Z1nYCS60NaGRYTNOONIUX4G/ JAvIvARrnBPyB+8vQMEaOk28qxqI3HD7AamZT1ACdFKqp4hw7A648Vgtu243u1SuTw7c wyRuvQPF8M1fbdmKx+rB3oWr780pdYkSM0dejpcI2J3fNxQi5njKJzPtAt3XkIJw+Nz6 X+fzrJ2NlxksJ0MsfRGq8EJZUcmNOafjpAddmfxbVWDi/yIVChIrl9+ZbZeLYTr8l5vA BVj+q+Y0YVOMEVgNOpRjwXredfYkcvwTnog7KmniA4FTnKXufOExWD39wTyrk+4YgYQZ nUjw== ARC-Authentication-Results: i=1; strato.com; dkim=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; t=1615101994; s=strato-dkim-0002; d=schoebel-theuer.de; h=Message-Id:Date:Subject:To:From:Cc:Date:From:Subject:Sender; bh=HEEEysoWkKp/UIS/qSkdJPSpBjqYkSPRs79w/tJEBfk=; b=iPXfNWHVfqofq1T4Y5OdJm37Yp8wC1HGYf6dnMJ7IsI8cGTHHK4YRwrwKe/q5nSBE4 JujBFNzUFR51GVeRhpDGPY5Fyo47VZ42AA1JbpAx3IkKlay2DIi6xHO8VkiHpBXaJtPt yDfbuJ96JEacr3KHFSAFNwOaWK+l2ZhQQi/W5vjjBUJcgYVS+pfVF3pXKVzZ/Y/5+hNV deaS3DA+fD6pjjD9gwSlVQHSto5A3rg4iO7QroZlHji48qpTJt8W8fqH0B7Ut5r+uoYM tj+Yck8NcEE6nu4oXx82/OCuOseehrusHVbOtzT7adfQOQzg5P/qDNDJ1M03X96eNChl GeAg== Authentication-Results: strato.com; dkim=none X-RZG-AUTH: ":OH8QVVOrc/CP6za/qRmbF3BWedPGA1vjs2e0bDjfg8SjapJoMy/ngEsCKWYLcLtzTI+/at5T6HYnAg==" X-RZG-CLASS-ID: mo00 Received: from localhost.localdomain by smtp.strato.de (RZmta 47.20.3 DYNA|AUTH) with ESMTPSA id 6007d4x277QX8fL (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256 bits)) (Client did not present a certificate); Sun, 7 Mar 2021 08:26:33 +0100 (CET) From: Thomas Schoebel-Theuer To: tst@schoebel-theuer.de, stable@vger.kernel.org, Greg Kroah-Hartman , Lee Jones , Peter Zijlstra , Thomas Gleixner , Ingo Molnar , Thomas Schoebel-Theuer , Arnd Bergmann , linux-kernel@vger.kernel.org (open list) Subject: [PATCH RESEND v2 STABLE 4.4] futex: fix irq self-deadlock and satisfy assertion Date: Sun, 7 Mar 2021 08:26:29 +0100 Message-Id: <20210307072629.5955-1-tst@schoebel-theuer.de> X-Mailer: git-send-email 2.26.2 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Thomas Schoebel-Theuer This patch and problem analysis is specific for 4.4 LTS, due to incomplete backporting of other fixes. Later LTS series have different backports. Since v4.4.257 when CONFIG_PROVE_LOCKING=y the following triggers right after reboot of our pre-life systems which equal our production setup: Mar 03 11:27:33 icpu-test-bap10 kernel: ================================= Mar 03 11:27:33 icpu-test-bap10 kernel: [ INFO: inconsistent lock state ] Mar 03 11:27:33 icpu-test-bap10 kernel: 4.4.259-rc1-grsec+ #730 Not tainted Mar 03 11:27:33 icpu-test-bap10 kernel: --------------------------------- Mar 03 11:27:33 icpu-test-bap10 kernel: inconsistent {IN-HARDIRQ-W} -> {HARDIRQ-ON-W} usage. Mar 03 11:27:33 icpu-test-bap10 kernel: apache2-ssl/9310 [HC0[0]:SC0[0]:HE1:SE1] takes: Mar 03 11:27:33 icpu-test-bap10 kernel: (&p->pi_lock){?.-.-.}, at: [] pi_state_update_owner+0x51/0xd7 Mar 03 11:27:33 icpu-test-bap10 kernel: {IN-HARDIRQ-W} state was registered at: Mar 03 11:27:33 icpu-test-bap10 kernel: [] __lock_acquire+0x3a7/0xe4a Mar 03 11:27:33 icpu-test-bap10 kernel: [] lock_acquire+0x18d/0x1bc Mar 03 11:27:33 icpu-test-bap10 kernel: [] _raw_spin_lock_irqsave+0x3e/0x50 Mar 03 11:27:33 icpu-test-bap10 kernel: [] try_to_wake_up+0x2c/0x210 Mar 03 11:27:33 icpu-test-bap10 kernel: [] default_wake_function+0xd/0xf Mar 03 11:27:33 icpu-test-bap10 kernel: [] autoremove_wake_function+0x11/0x35 Mar 03 11:27:33 icpu-test-bap10 kernel: [] __wake_up_common+0x48/0x7c Mar 03 11:27:33 icpu-test-bap10 kernel: [] __wake_up+0x34/0x46 Mar 03 11:27:33 icpu-test-bap10 kernel: [] megasas_complete_int_cmd+0x31/0x33 Mar 03 11:27:33 icpu-test-bap10 kernel: [] megasas_complete_cmd+0x570/0x57b Mar 03 11:27:33 icpu-test-bap10 kernel: [] complete_cmd_fusion+0x23e/0x33d Mar 03 11:27:33 icpu-test-bap10 kernel: [] megasas_isr_fusion+0x67/0x74 Mar 03 11:27:33 icpu-test-bap10 kernel: [] handle_irq_event_percpu+0x134/0x311 Mar 03 11:27:33 icpu-test-bap10 kernel: [] handle_irq_event+0x33/0x51 Mar 03 11:27:33 icpu-test-bap10 kernel: [] handle_edge_irq+0xa3/0xc2 Mar 03 11:27:33 icpu-test-bap10 kernel: [] handle_irq+0xf9/0x101 Mar 03 11:27:33 icpu-test-bap10 kernel: [] do_IRQ+0x80/0xf5 Mar 03 11:27:33 icpu-test-bap10 kernel: [] ret_from_intr+0x0/0x20 Mar 03 11:27:33 icpu-test-bap10 kernel: [] arch_cpu_idle+0xa/0xc Mar 03 11:27:33 icpu-test-bap10 kernel: [] default_idle_call+0x1e/0x20 Mar 03 11:27:33 icpu-test-bap10 kernel: [] cpu_startup_entry+0x141/0x22f Mar 03 11:27:33 icpu-test-bap10 kernel: [] rest_init+0x135/0x13b Mar 03 11:27:33 icpu-test-bap10 kernel: [] start_kernel+0x3fa/0x40a Mar 03 11:27:33 icpu-test-bap10 kernel: [] x86_64_start_reservations+0x2a/0x2c Mar 03 11:27:33 icpu-test-bap10 kernel: [] x86_64_start_kernel+0x11f/0x12c Mar 03 11:27:33 icpu-test-bap10 kernel: irq event stamp: 1457 Mar 03 11:27:33 icpu-test-bap10 kernel: hardirqs last enabled at (1457): [] get_user_pages_fast+0xeb/0x14f Mar 03 11:27:33 icpu-test-bap10 kernel: hardirqs last disabled at (1456): [] get_user_pages_fast+0x5f/0x14f Mar 03 11:27:33 icpu-test-bap10 kernel: softirqs last enabled at (1446): [] release_sock+0x142/0x14d Mar 03 11:27:33 icpu-test-bap10 kernel: softirqs last disabled at (1444): [] release_sock+0x34/0x14d Mar 03 11:27:33 icpu-test-bap10 kernel: other info that might help us debug this: Mar 03 11:27:33 icpu-test-bap10 kernel: Possible unsafe locking scenario: Mar 03 11:27:33 icpu-test-bap10 kernel: CPU0 Mar 03 11:27:33 icpu-test-bap10 kernel: ---- Mar 03 11:27:33 icpu-test-bap10 kernel: lock(&p->pi_lock); Mar 03 11:27:33 icpu-test-bap10 kernel: Mar 03 11:27:33 icpu-test-bap10 kernel: lock(&p->pi_lock); Mar 03 11:27:33 icpu-test-bap10 kernel: *** DEADLOCK *** Mar 03 11:27:33 icpu-test-bap10 kernel: 2 locks held by apache2-ssl/9310: Mar 03 11:27:33 icpu-test-bap10 kernel: #0: (&(&(__futex_data.queues)[i].lock)->rlock){+.+...}, at: [] do Mar 03 11:27:33 icpu-test-bap10 kernel: #1: (&lock->wait_lock){+.+...}, at: [] do_futex+0x639/0x809 Mar 03 11:27:33 icpu-test-bap10 kernel: stack backtrace: Mar 03 11:27:33 icpu-test-bap10 kernel: CPU: 13 PID: 9310 UID: 99 Comm: apache2-ssl Not tainted 4.4.259-rc1-grsec+ #730 Mar 03 11:27:33 icpu-test-bap10 kernel: Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.11.0 11/02/2019 Mar 03 11:27:33 icpu-test-bap10 kernel: 0000000000000000 ffff883fb79bfc00 ffffffff816f8fc2 ffff883ffa66d300 Mar 03 11:27:33 icpu-test-bap10 kernel: ffffffff8eaa71f0 ffff883fb79bfc50 ffffffff81088484 0000000000000000 Mar 03 11:27:33 icpu-test-bap10 kernel: 0000000000000001 0000000000000001 0000000000000002 ffff883ffa66db58 Mar 03 11:27:33 icpu-test-bap10 kernel: Call Trace: Mar 03 11:27:33 icpu-test-bap10 kernel: [] dump_stack+0x94/0xca Mar 03 11:27:33 icpu-test-bap10 kernel: [] print_usage_bug+0x1bc/0x1d1 Mar 03 11:27:33 icpu-test-bap10 kernel: [] ? check_usage_forwards+0x98/0x98 Mar 03 11:27:33 icpu-test-bap10 kernel: [] mark_lock+0x10c/0x203 Mar 03 11:27:33 icpu-test-bap10 kernel: [] __lock_acquire+0x416/0xe4a Mar 03 11:27:33 icpu-test-bap10 kernel: [] ? pi_state_update_owner+0x51/0xd7 Mar 03 11:27:33 icpu-test-bap10 kernel: [] lock_acquire+0x18d/0x1bc Mar 03 11:27:33 icpu-test-bap10 kernel: [] ? lock_acquire+0x18d/0x1bc Mar 03 11:27:33 icpu-test-bap10 kernel: [] ? pi_state_update_owner+0x51/0xd7 Mar 03 11:27:33 icpu-test-bap10 kernel: [] _raw_spin_lock+0x2a/0x39 Mar 03 11:27:33 icpu-test-bap10 kernel: [] ? pi_state_update_owner+0x51/0xd7 Mar 03 11:27:33 icpu-test-bap10 kernel: [] pi_state_update_owner+0x51/0xd7 Mar 03 11:27:33 icpu-test-bap10 kernel: [] do_futex+0x6ae/0x809 Mar 03 11:27:33 icpu-test-bap10 kernel: [] SyS_futex+0x133/0x143 Mar 03 11:27:33 icpu-test-bap10 kernel: [] ? syscall_trace_enter_phase2+0x1a2/0x1bb Mar 03 11:27:33 icpu-test-bap10 kernel: [] tracesys_phase2+0x90/0x95 Bisecting detects 47e452fcf2f in the above specific scenario using apache-ssl, but apparently the missing *_irq() was introduced in 34c8e1c2c02. However, just reverting the old _irq() variants to a similar status than before 34c8e1c2c02, or using _irqsave() / _irqrestore() as some other backports are doing in various places, would not really help. The fundamental problem is the following violation of the assertion lockdep_assert_held(&pi_state->pi_mutex.wait_lock) in pi_state_update_owner(): Mar 03 12:50:03 icpu-test-bap10 kernel: ------------[ cut here ]------------ Mar 03 12:50:03 icpu-test-bap10 kernel: WARNING: CPU: 37 PID: 8488 at kernel/futex.c:844 pi_state_update_owner+0x3d/0xd7() Mar 03 12:50:03 icpu-test-bap10 kernel: Modules linked in: xt_time xt_connlimit xt_connmark xt_NFLOG xt_limit xt_hashlimit veth ip_set_bitmap_port xt_DSCP xt_multiport ip_set_hash_ip xt_owner xt_set ip_set_hash_net xt_state xt_conntrack nf_conntrack_ftp mars lz4_decompress lz4_compress ipmi_devintf x86_pkg_temp_thermal coretemp crct10dif_pclmul crc32_pclmul hed ipmi_si ipmi_msghandler processor crc32c_intel ehci_pci ehci_hcd usbcore i40e usb_common Mar 03 12:50:03 icpu-test-bap10 kernel: CPU: 37 PID: 8488 UID: 99 Comm: apache2-ssl Not tainted 4.4.259-rc1-grsec+ #737 Mar 03 12:50:03 icpu-test-bap10 kernel: Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.11.0 11/02/2019 Mar 03 12:50:03 icpu-test-bap10 kernel: 0000000000000000 ffff883f863f7c70 ffffffff816f9002 0000000000000000 Mar 03 12:50:03 icpu-test-bap10 kernel: 0000000000000009 ffff883f863f7ca8 ffffffff8104cda2 ffffffff810abac7 Mar 03 12:50:03 icpu-test-bap10 kernel: ffff883ffbfe5e80 0000000000000000 ffff883f82ed4bc0 00007fc01c9bf000 Mar 03 12:50:03 icpu-test-bap10 kernel: Call Trace: Mar 03 12:50:03 icpu-test-bap10 kernel: [] dump_stack+0x94/0xca Mar 03 12:50:03 icpu-test-bap10 kernel: [] warn_slowpath_common+0x94/0xad Mar 03 12:50:03 icpu-test-bap10 kernel: [] ? pi_state_update_owner+0x3d/0xd7 Mar 03 12:50:03 icpu-test-bap10 kernel: [] warn_slowpath_null+0x15/0x17 Mar 03 12:50:03 icpu-test-bap10 kernel: [] pi_state_update_owner+0x3d/0xd7 Mar 03 12:50:03 icpu-test-bap10 kernel: [] free_pi_state+0x2d/0x73 Mar 03 12:50:03 icpu-test-bap10 kernel: [] unqueue_me_pi+0x1d/0x31 Mar 03 12:50:03 icpu-test-bap10 kernel: [] futex_lock_pi+0x27a/0x2e8 Mar 03 12:50:03 icpu-test-bap10 kernel: [] ? __lock_acquire+0x327/0xe4a Mar 03 12:50:03 icpu-test-bap10 kernel: [] do_futex+0x784/0x809 Mar 03 12:50:03 icpu-test-bap10 kernel: [] ? seccomp_phase1+0xde/0x1e7 Mar 03 12:50:03 icpu-test-bap10 kernel: [] ? current_kernel_time64+0xb/0x31 Mar 03 12:50:03 icpu-test-bap10 kernel: [] ? current_kernel_time+0xb/0xf Mar 03 12:50:03 icpu-test-bap10 kernel: [] SyS_futex+0x133/0x143 Mar 03 12:50:03 icpu-test-bap10 kernel: [] ? syscall_trace_enter_phase2+0x1a2/0x1bb Mar 03 12:50:03 icpu-test-bap10 kernel: [] tracesys_phase2+0x90/0x95 Mar 03 12:50:03 icpu-test-bap10 kernel: ---[ end trace 968f95a458dea951 ]--- In order to both (1) prevent the self-deadlock, and (2) to satisfy the assertion at pi_state_update_owner(), some locking with irq disable is needed, at least in the specific call stack. Interestingly, there existed a suchalike locking just before f08a4af5ccb. This is just a quick hotfix, resurrecting some previous locks at the old places, but now using ->wait_lock in place of the previous ->pi_lock (which was in place before f08a4af5ccb). The ->pi_lock is now also taken, by the new code which had been introduced in 34c8e1c2c02. When this patch is applied, both the above splats are no longer triggering at my prelife machines. Without this patch, I cannot ensure stable production at 1&1 Ionos. Hint for further work: I have not yet tested other call paths, since I am under time pressure for security reasons. Hint for further hardening of 4.4.y and probably some more LTS series: Probably some more systematic testing with CONFIG_PROVE_LOCKING (and probably some more options) should be invested in order to make the 4.4 LTS series really "stable" again. Signed-off-by: Thomas Schoebel-Theuer Cc: Thomas Gleixner Cc: Lee Jones Cc: Greg Kroah-Hartman Fixes: f08a4af5ccb Fixes: 34c8e1c2c02 --- kernel/futex.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/kernel/futex.c b/kernel/futex.c index 4a707bc7cceb..0c42d2313660 100644 --- a/kernel/futex.c +++ b/kernel/futex.c @@ -874,7 +874,9 @@ static void free_pi_state(struct futex_pi_state *pi_state) * and has cleaned up the pi_state already */ if (pi_state->owner) { + raw_spin_lock_irq(&pi_state->pi_mutex.wait_lock); pi_state_update_owner(pi_state, NULL); + raw_spin_unlock_irq(&pi_state->pi_mutex.wait_lock); rt_mutex_proxy_unlock(&pi_state->pi_mutex); } -- 2.26.2