Received: by 2002:a05:6358:4e97:b0:b3:742d:4702 with SMTP id ce23csp1999515rwb; Sun, 14 Aug 2022 18:31:25 -0700 (PDT) X-Google-Smtp-Source: AA6agR7utCwetb5emiWYEfrwSBuiAeGahbq3hIJHsmvuM9XZVV+yYXx0FRRNISL0Jf7oDWDE9TJO X-Received: by 2002:a63:f708:0:b0:426:9c51:fdbf with SMTP id x8-20020a63f708000000b004269c51fdbfmr6698859pgh.576.1660527085071; Sun, 14 Aug 2022 18:31:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1660527085; cv=none; d=google.com; s=arc-20160816; b=LK6cGdtPGuQi6F0mKYYHrm1AX4H2b6ZoWWXq6Hrsrngt8UUaJ52cNFSg7FuyZRBV+n YfAhsulyK7AEqRRcaGrqe2XQ/WhUXMql+aWdJenwGYcEj7FORBeLxVkijZXD/WRq2SCF r6MixxCSRUHGIvKwkUYPgd/QRWtwPFi0w4SAj+MkDhnjawy5wi9bGI3MK0mk3nG0etQq ckdUzQj/O8e5xlEpwgG9UStbBf0wdMCw+uXQL0Z5E/EZxKslCtIxS4+ufmDDCZRimGU7 v/IfilML0BfTuurVuz3lFPx+Ng4CGcLbw+B1aMXxnVlI10MGzaG1qTJmNXbbNMEnr16h WA4Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:in-reply-to:content-disposition:mime-version :references:message-id:subject:cc:to:from:date:sender:dkim-signature; bh=oWRN18BRqoVUNoHBjmPoDmMnYJ9ulZd+9c70hfw7qos=; b=ImKqaaVz99jNS8kJ3Vwu7y+jp4+3EgY6Yn/Etw1HpUk8E5cggwXSe/81bxMfJJcvuW MkkgSGwXDwYNNpRnjb/4ng1TCv1HqMO7uCTncxb3MUe/Gt8yABHD6Eq7OX0RxQTOpGqz TAkaxJBPZcUPayXgYNBC+yq95OV44A2ojxWkb4zU8gQexaEGwv7I+Srtq/hytCv2XNbb HSqoLCtMUdLWVEC0SrBkniPVu3QK65mLSbefNXDke2/spr9mGFjlne/Z68f4ZQHKp5Js C21HoOFL0Sj6XWVL4Y1EF5FStL7PiP7L817cUBcKpUXi+R42LWOP9FgvhEGBG379zHza 6PyQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=K+ReV53P; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id c10-20020a170902f30a00b0016f196209bbsi7593695ple.220.2022.08.14.18.31.14; Sun, 14 Aug 2022 18:31:25 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=K+ReV53P; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229919AbiHOBRY (ORCPT + 99 others); Sun, 14 Aug 2022 21:17:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39182 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229612AbiHOBRX (ORCPT ); Sun, 14 Aug 2022 21:17:23 -0400 Received: from mail-pg1-x536.google.com (mail-pg1-x536.google.com [IPv6:2607:f8b0:4864:20::536]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2F1E911C26 for ; Sun, 14 Aug 2022 18:17:22 -0700 (PDT) Received: by mail-pg1-x536.google.com with SMTP id s206so5406359pgs.3 for ; Sun, 14 Aug 2022 18:17:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:sender:from:to:cc; bh=oWRN18BRqoVUNoHBjmPoDmMnYJ9ulZd+9c70hfw7qos=; b=K+ReV53PTma63xWZKK8qKolyBLIW0Prrfaa/RgHWMlUEXpZRMSo8LjRe7PiIanpROb +WiyPi4QTRLy/sI3XsWfX3j+/80UTZesjCh7xumGFmKb/tAhYOtcwjWFQp1z9iTuGWtj WfLs+LhxR2S68VAiHcJB2M/gnixIxRGBtO9Pfkq91haidmW8S2LAxyG22Blb4VTAHfyD R1rapl9UWET2DkqmqZraMHwiryv0s+3st8TeDhHRg5sH2Vw6Njo3GAzDxvvvWehtxDSH DaHlciMnaflbxJkgCmNO9EhzJ1ngvRNAQC+2WfAC75XXhUDiQcIM5+tD7JKHifGO8pri YQ6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:sender:x-gm-message-state:from:to:cc; bh=oWRN18BRqoVUNoHBjmPoDmMnYJ9ulZd+9c70hfw7qos=; b=DmrNwQGbrp8nFTJ8iJY1TTXponFQHmH6nsrAbp1VEsJ44jfFzfd8tMfEIv9Uocepse gsBVHo9JiT1nZVeceid1uQHPfECDBjjaZZLUHz+QGiLhBWA//4DJcHUqcQEjclRNa+wG qvtogKDXHw1RsFQwlmSd9euacocsw/YhvpGSKKxArDUUUV3Pbnsvv8gks9LfZCi3aNmg /kpC/kUBQBpfUC+2qijputIYyrX6GxgxvrVGVbyOBcuCWk9XVSx7Xv6GTkyh8l8vVvdR TFgOYqSVQHAnVX+oY/ON4GBjJ9Ob29MHGZC0kOm86g9PNMinOQBS9sKq+p8Uf/PJKzx3 bzaA== X-Gm-Message-State: ACgBeo0uYlDQvmXrK4wTEmziWU7y5lxw/pUgakAhzl4dqho1R//TTqLh VF9/WcbxYRBRdSvgZLG7l08= X-Received: by 2002:a63:154a:0:b0:421:5af6:6247 with SMTP id 10-20020a63154a000000b004215af66247mr11948738pgv.353.1660526241603; Sun, 14 Aug 2022 18:17:21 -0700 (PDT) Received: from server.roeck-us.net ([2600:1700:e321:62f0:329c:23ff:fee3:9d7c]) by smtp.gmail.com with ESMTPSA id d7-20020a170903230700b0016efc27ca98sm5982406plh.169.2022.08.14.18.17.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 14 Aug 2022 18:17:20 -0700 (PDT) Sender: Guenter Roeck Date: Sun, 14 Aug 2022 18:17:18 -0700 From: Guenter Roeck To: Linus Torvalds Cc: Andres Freund , Jens Axboe , James Bottomley , "Martin K. Petersen" , linux-kernel@vger.kernel.org, Greg KH Subject: Re: upstream kernel crashes Message-ID: <20220815011718.GA3694888@roeck-us.net> References: <20220814212610.GA3690074@roeck-us.net> <20220814223743.26ebsbnrvrjien4f@awork3.anarazel.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Spam-Status: No, score=-1.3 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,FREEMAIL_ENVFROM_END_DIGIT, FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Sun, Aug 14, 2022 at 03:47:44PM -0700, Linus Torvalds wrote: > On Sun, Aug 14, 2022 at 3:37 PM Andres Freund wrote: > > > > That range had different symptoms, I think (networking not working, but not > > oopsing). I hit similar issues when tried to reproduce the issue > > interactively, to produce more details, and unwisely did git pull instead of > > checking out the precise revision, ending up with aea23e7c464b. That's when > > symptoms look similar to the above. So it'd be 69dac8e431af..aea23e7c464b > > that I'd be more suspicious off in the context of this thread. > > Ok. > > > Which would make me look at the following first: > > e140f731f980 Merge tag 'scsi-misc' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi > > abe7a481aac9 Merge tag 'block-6.0-2022-08-12' of git://git.kernel.dk/linux-block > > 1da8cf961bb1 Merge tag 'io_uring-6.0-2022-08-13' of git://git.kernel.dk/linux-block > > All right, that maks sense.The reported oopses seem to be about block > requests. Some of them were scsi in particular. > > Let's bring in Jens and the SCSI people. Maybe that host reference > counting? There's quite a lot of "move freeing around" in that late > scsi pull, even if it was touted as "mostly small bug fixes and > trivial updates". > I may be wrong, but I don't think it is SCSI. The crashes are all over the place. Here is another one that just came in. I can dig up more tomorrow. Guenter --- list_add corruption. next->prev should be prev (ffff8881401c0a00), but was ffff000000000000. (next=ffff88801fb50308). ------------[ cut here ]------------ kernel BUG at lib/list_debug.c:27 invalid opcode: 0000 [#1] PREEMPT SMP KASAN CPU: 0 PID: 24 Comm: kdevtmpfs Not tainted 5.19.0-syzkaller-14374-g5d6a0f4da927 #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 07/22/2022 RIP: 0010:__list_add_valid.cold+0xf/0x58 lib/list_debug.c:27 Code: 48 c7 c6 00 ec 48 8a 48 89 ef 49 c7 c7 ea ff ff ff e8 5b 63 05 00 e9 c2 7d b6 fa 4c 89 e1 48 c7 c7 a0 f2 48 8a e8 95 f2 f0 ff <0f> 0b 48 c7 c7 40 f2 48 8a e8 87 f2 f0 ff 0f 0b 48 c7 c7 a0 f1 48 RSP: 0018:ffffc900001efc10 EFLAGS: 00010286 RAX: 0000000000000075 RBX: ffff8881401c0000 RCX: 0000000000000000 RDX: ffff888012620000 RSI: ffffffff8161f148 RDI: fffff5200003df74 RBP: ffff88801db8b588 R08: 0000000000000075 R09: 0000000000000000 R10: 0000000080000001 R11: 0000000000000000 R12: ffff88801fb50308 R13: ffff88801fb50308 R14: ffff8881401c0000 R15: ffff88801db8b588 FS: 0000000000000000(0000) GS:ffff8880b9a00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: ffff88823ffff000 CR3: 000000000bc8e000 CR4: 0000000000350ef0 Call Trace: __list_add include/linux/list.h:69 [inline] list_add include/linux/list.h:88 [inline] inode_sb_list_add fs/inode.c:495 [inline] new_inode+0x114/0x270 fs/inode.c:1049 shmem_get_inode+0x19b/0xe00 mm/shmem.c:2306 shmem_mknod+0x5a/0x1f0 mm/shmem.c:2873 vfs_mknod+0x4d2/0x7e0 fs/namei.c:3892 handle_create+0x340/0x4b3 drivers/base/devtmpfs.c:226 handle drivers/base/devtmpfs.c:391 [inline] devtmpfs_work_loop drivers/base/devtmpfs.c:406 [inline] devtmpfsd+0x1a4/0x2a3 drivers/base/devtmpfs.c:448 kthread+0x2e4/0x3a0 kernel/kthread.c:376 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:306 Modules linked in: ---[ end trace 0000000000000000 ]--- RIP: 0010:__list_add_valid.cold+0xf/0x58 lib/list_debug.c:27 Code: 48 c7 c6 00 ec 48 8a 48 89 ef 49 c7 c7 ea ff ff ff e8 5b 63 05 00 e9 c2 7d b6 fa 4c 89 e1 48 c7 c7 a0 f2 48 8a e8 95 f2 f0 ff <0f> 0b 48 c7 c7 40 f2 48 8a e8 87 f2 f0 ff 0f 0b 48 c7 c7 a0 f1 48 RSP: 0018:ffffc900001efc10 EFLAGS: 00010286 RAX: 0000000000000075 RBX: ffff8881401c0000 RCX: 0000000000000000 RDX: ffff888012620000 RSI: ffffffff8161f148 RDI: fffff5200003df74 RBP: ffff88801db8b588 R08: 0000000000000075 R09: 0000000000000000 R10: 0000000080000001 R11: 0000000000000000 R12: ffff88801fb50308 R13: ffff88801fb50308 R14: ffff8881401c0000 R15: ffff88801db8b588 FS: 0000000000000000(0000) GS:ffff8880b9a00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: ffff88823ffff000 CR3: 000000000bc8e000 CR4: 0000000000350ef0