Received: by 2002:a25:b323:0:0:0:0:0 with SMTP id l35csp1800586ybj; Sun, 22 Sep 2019 12:12:37 -0700 (PDT) X-Google-Smtp-Source: APXvYqwuwNCHne6hvsUpkDmsBgkCLhbPLcgPw4SRyzE350p6ME6FnxgmqX2wP7pJVToQ06iRt0W7 X-Received: by 2002:a17:906:c5b:: with SMTP id t27mr26729196ejf.180.1569179557608; Sun, 22 Sep 2019 12:12:37 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1569179557; cv=none; d=google.com; s=arc-20160816; b=Zq+Oe3lX6vHKXyJkFlX1urhB1F7ehY88F13bMCkDZZDwUcJfCDZRllhzAkTs8045f+ Nq7fjMzfYB2iF1wMMUHzLJBCfv67k5HC/fg64t+dits6WBQzeP3BGeghji4Qf9LkQmpf O8XBVLQuRnDFgHgY4Y04UMoDRfmCsc/93JotXQo6ho2J2x3beYB+FLMEFNHPpEPw5dDO Ab/PHGGDJRDJM7O+80tyl+qcKe0GT5/4b8NBqX6+yix6FRjtkRWCehtBW8gX1JLE84uv qq8pHqf1WhxGJTyY0I7QZBHBeIDlgvBoj8gpVpKmjLN7LuYCmdeXQgS+sGfW1dF9qKLU SkAQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:cc:to:subject:message-id:date:from :references:in-reply-to:mime-version:dkim-signature; bh=6FUJB9iC+bvrTtNriAO1ARI5RLS9whH28zugV601iio=; b=GX2wpJMgBU0ZmJqjo5z8V4yrD510+pP5EEHpchls6blrgqy0Jc7szzxYQ9XXw4CsuW 8Dk6/G1B6fMYqzqzOTddVe1I2JsaXhOoQYkNm0kRra/BX1ob9Y18o7/th5LQLlql4NdD cc11v4wODOGXXc1VD4lIx8BT/hBcGNHCsi/9RQl9guh1N++vzIPtEQcnS+aD+qXdGgbJ QH0/2ztPcVmN/GpGcgM8yT2zlW/eNp0GwWM05k7YQis4tuJgChahhfXvjpw/IM0guUzl 3HvkexcytLB3/suKmUgmNDGhzw4URIJVUWgPjWt30udTDVXM1VxdntuDpsSSAwXInsnP +i+w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@draconx-ca.20150623.gappssmtp.com header.s=20150623 header.b=SU6Xnngd; spf=pass (google.com: best guess record for domain of linux-nfs-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-nfs-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id f21si5300400edb.379.2019.09.22.12.12.11; Sun, 22 Sep 2019 12:12:37 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-nfs-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@draconx-ca.20150623.gappssmtp.com header.s=20150623 header.b=SU6Xnngd; spf=pass (google.com: best guess record for domain of linux-nfs-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-nfs-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2436997AbfIUBy3 (ORCPT + 99 others); Fri, 20 Sep 2019 21:54:29 -0400 Received: from mail-lj1-f193.google.com ([209.85.208.193]:40735 "EHLO mail-lj1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2436996AbfIUBy3 (ORCPT ); Fri, 20 Sep 2019 21:54:29 -0400 Received: by mail-lj1-f193.google.com with SMTP id 7so8756280ljw.7 for ; Fri, 20 Sep 2019 18:54:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=draconx-ca.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=6FUJB9iC+bvrTtNriAO1ARI5RLS9whH28zugV601iio=; b=SU6Xnngd2bMY4WuCnPThYobViQS/n90H3Y4wRz1VD7MnEZSh4dIoiArZHgVk0bwsop g0D/18YIqKKYtfltsk1zhFeG94ov+vTZAXQD2xU0iZvEdsCu1G3isURpq/QGCbXwzk7r 6NJh01KDU96fiFppPgcf8ycm9e4wcTDq1eIuHEyAsjxrNldG4tWROh8jaNEiPje92Vd/ DPJTvH7HCnZRxLLoHQ6/nLPoqjBY/39blm0ERcZxcHOgoe/oPP9Pm8NFla13KJlEWSMc A5TG6vwe1Ebh3BTf69/kIh4ZaaXSGot4qfqE3d0lTFjaEF4V0Nu6tr6jeoZea/f3awN9 Qqkw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=6FUJB9iC+bvrTtNriAO1ARI5RLS9whH28zugV601iio=; b=kazLyPJFqXY7S4pclRZfbN7IFQP9Oo2wjbnAULE0PzEpYoqH8hjtmbf8VBwwM2uYAH 2tU8Qycw2UvFWS4asg64iEfOU3aBijKSvg356TKzbsgB5rTavClemql1XdBZuchSHNni AzreKXf1QU1TSboXn3X90mpxqmyCtfEnhd5QxbEqN60Nkd/BzqJX9cI5u6sSEpAARLYc hE8yrkxHFuCA87sD42tSqAl4DIO5481d6/GwdBfKtb6eWxjekSzi+rLXfVtYX/THeTBO QUDGmJwa9YxsGkoE0WY8RiObAvspscg3RbChliPUvn4JDPVPVGP2rYgHMtf6oZmi0w1s 5v3A== X-Gm-Message-State: APjAAAXLOa8G0eRux6uREZ/grYFq2LHidrLK7PAj+cMRJOQpbCYAWVRk T/K18BSh5SKvZqmyE+Dpgz1DLhOVfITV5hY6Dz6DWxCq5px0FA== X-Received: by 2002:a2e:5b9a:: with SMTP id m26mr10711008lje.90.1569030866534; Fri, 20 Sep 2019 18:54:26 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a2e:534c:0:0:0:0:0 with HTTP; Fri, 20 Sep 2019 18:54:25 -0700 (PDT) X-Originating-IP: [24.53.243.131] In-Reply-To: References: From: Nick Bowler Date: Fri, 20 Sep 2019 21:54:25 -0400 Message-ID: Subject: Re: PROBLEM: nfs? crash in Linux 5.3 (possible regression) To: Trond Myklebust Cc: "linux-nfs@vger.kernel.org" , "keyrings@vger.kernel.org" , "linux-kernel@vger.kernel.org" Content-Type: text/plain; charset="UTF-8" Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org On 9/20/19, Trond Myklebust wrote: > On Fri, 2019-09-20 at 14:23 -0400, Nick Bowler wrote: >> Not sure how reproducible this is. Since I've never seen a crash >> like this before it may be a regression compared to, say, Linux 4.19 >> but I am not certain because this particular machine is brand new so >> I don't have experience with older kernels on it... So it actually seems pretty reliably reproducible, 4 attempts to compile Linux on Linux 5.3 and all four crash the same way, although there's definitely some randomness here... On the other hand, I cannot reproduce if I install Linux 5.2 so it does seem like a regression in 5.3. I will see how well bisecting goes... >> [ 796.050025] BUG: kernel NULL pointer dereference, address: >> 0000000000000014 >> [ 796.051280] #PF: supervisor read access in kernel mode >> [ 796.053063] #PF: error_code(0x0000) - not-present page >> [ 796.054636] PGD 0 P4D 0 >> [ 796.055688] Oops: 0000 [#1] PREEMPT SMP >> [ 796.056768] CPU: 2 PID: 190 Comm: kworker/2:2 Tainted: G W >> 5.3.0 #6 >> [ 796.057953] Hardware name: To Be Filled By O.E.M. To Be Filled By >> O.E.M./B450 Gaming-ITX/ac, BIOS P3.30 05/17/2019 >> [ 796.059329] Workqueue: events key_garbage_collector >> [ 796.060623] RIP: 0010:keyring_gc_check_iterator+0x27/0x30 > > That would be the keyring garbage collector, not NFS. > > Cced keyrings@vger.kernel.org > > >> [ 796.061845] Code: 44 00 00 48 83 e7 fc b8 01 00 00 00 f6 87 80 00 >> 00 00 21 75 19 48 8b 57 58 48 39 16 7c 05 48 85 d2 7f 0b 48 8b 87 a0 >> 00 00 00 <0f> b6 40 14 c3 0f 1f 40 00 48 83 e7 fc e9 27 eb ff ff 0f >> 1f >> 80 00 >> [ 796.064638] RSP: 0018:ffffb40fc0757df8 EFLAGS: 00010282 >> [ 796.066058] RAX: 0000000000000000 RBX: ffffa14338caed80 RCX: >> ffffb40fc0757e40 >> [ 796.067531] RDX: ffffa1433ae85558 RSI: ffffb40fc0757e40 RDI: >> ffffa1433ae85500 >> [ 796.069014] RBP: ffffb40fc0757e40 R08: 0000000000000000 R09: >> 000000000000000f >> [ 796.070513] R10: 8080808080808080 R11: 0000000000000001 R12: >> ffffffffa4cd6180 >> [ 796.072025] R13: ffffa14338caee10 R14: ffffa14338caedf0 R15: >> ffffa1433ffeff00 >> [ 796.073567] FS: 0000000000000000(0000) GS:ffffa14340480000(0000) >> knlGS:0000000000000000 >> [ 796.075171] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >> [ 796.076785] CR2: 0000000000000014 CR3: 0000000747ce6000 CR4: >> 00000000003406e0 >> [ 796.078445] Call Trace: >> [ 796.080091] assoc_array_subtree_iterate+0x55/0x100 >> [ 796.081770] keyring_gc+0x3f/0x80 >> [ 796.083447] key_garbage_collector+0x330/0x3d0 >> [ 796.085155] process_one_work+0x1cb/0x320 >> [ 796.086869] worker_thread+0x28/0x3c0 >> [ 796.088603] ? process_one_work+0x320/0x320 >> [ 796.090335] kthread+0x106/0x120 >> [ 796.092053] ? kthread_create_on_node+0x40/0x40 >> [ 796.093810] ret_from_fork+0x1f/0x30 >> [ 796.095569] Modules linked in: sha1_ssse3 sha1_generic cbc cts >> rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd grace ext4 crc16 mbcache >> jbd2 iwlmvm mac80211 libarc4 amdgpu iwlwifi snd_hda_codec_realtek >> snd_hda_codec_generic kvm_amd gpu_sched kvm snd_hda_codec_hdmi >> drm_kms_helper irqbypass k10temp syscopyarea sysfillrect sysimgblt >> fb_sys_fops video ttm cfg80211 snd_hda_intel snd_hda_codec drm >> snd_hwdep rfkill snd_hda_core backlight snd_pcm evdev snd_timer snd >> soundcore efivarfs dm_crypt hid_generic igb hwmon i2c_algo_bit sr_mod >> cdrom sunrpc dm_mod >> [ 796.104033] CR2: 0000000000000014 >> [ 796.106304] ---[ end trace 695aee10f9202347 ]--- >> [ 796.108585] RIP: 0010:keyring_gc_check_iterator+0x27/0x30 >> [ 796.110894] Code: 44 00 00 48 83 e7 fc b8 01 00 00 00 f6 87 80 00 >> 00 00 21 75 19 48 8b 57 58 48 39 16 7c 05 48 85 d2 7f 0b 48 8b 87 a0 >> 00 00 00 <0f> b6 40 14 c3 0f 1f 40 00 48 83 e7 fc e9 27 eb ff ff 0f >> 1f >> 80 00 >> [ 796.115773] RSP: 0018:ffffb40fc0757df8 EFLAGS: 00010282 >> [ 796.118209] RAX: 0000000000000000 RBX: ffffa14338caed80 RCX: >> ffffb40fc0757e40 >> [ 796.120683] RDX: ffffa1433ae85558 RSI: ffffb40fc0757e40 RDI: >> ffffa1433ae85500 >> [ 796.123176] RBP: ffffb40fc0757e40 R08: 0000000000000000 R09: >> 000000000000000f >> [ 796.125668] R10: 8080808080808080 R11: 0000000000000001 R12: >> ffffffffa4cd6180 >> [ 796.128104] R13: ffffa14338caee10 R14: ffffa14338caedf0 R15: >> ffffa1433ffeff00 >> [ 796.130493] FS: 0000000000000000(0000) GS:ffffa14340480000(0000) >> knlGS:0000000000000000 >> [ 796.132923] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >> [ 796.135266] CR2: 0000000000000014 CR3: 0000000747ce6000 CR4: >> 00000000003406e0 >