Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S941666AbcJZXUj (ORCPT ); Wed, 26 Oct 2016 19:20:39 -0400 Received: from mx0b-00082601.pphosted.com ([67.231.153.30]:33880 "EHLO mx0a-00082601.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S932898AbcJZXUY (ORCPT ); Wed, 26 Oct 2016 19:20:24 -0400 Date: Wed, 26 Oct 2016 19:19:51 -0400 From: Chris Mason To: Jens Axboe CC: Linus Torvalds , Dave Jones , Andy Lutomirski , "Andy Lutomirski" , Al Viro , Josef Bacik , David Sterba , linux-btrfs , Linux Kernel , Dave Chinner Subject: Re: bio linked list corruption. Message-ID: <20161026231950.GB15247@clm-mbp.thefacebook.com> Mail-Followup-To: Chris Mason , Jens Axboe , Linus Torvalds , Dave Jones , Andy Lutomirski , Andy Lutomirski , Al Viro , Josef Bacik , David Sterba , linux-btrfs , Linux Kernel , Dave Chinner References: <20161026184201.6ofblkd3j5uxystq@codemonkey.org.uk> <488f9edc-6a1c-2c68-0d33-d3aa32ece9a4@fb.com> <20161026224025.mou27kki4bslftli@codemonkey.org.uk> <2bdc068d-afd5-7a78-f334-26970c91aaca@fb.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Content-Disposition: inline In-Reply-To: <2bdc068d-afd5-7a78-f334-26970c91aaca@fb.com> User-Agent: Mutt/1.5.24 (2015-08-30) X-Originating-IP: [2620:10d:c091:180::1:5709] X-ClientProxiedBy: MWHPR11CA0033.namprd11.prod.outlook.com (10.175.56.147) To BN6PR15MB1233.namprd15.prod.outlook.com (10.172.206.7) X-MS-Office365-Filtering-Correlation-Id: 275ad9b1-3bd5-4c1a-53b2-08d3fdf69d1a X-Microsoft-Exchange-Diagnostics: 1;BN6PR15MB1233;2:jRr7gBaI/uPB4a7yJw+ik4m5FuX44MPoB1iuLgCtGjURhAsoutgCurGjRmcEVhGlZZbEMhWwtEBtZKnrXtAUcfq97nCWNIW6Ngu8XBrO6kqzwouCSklnlcHELLI2cNdBr4G3cvkXOMAjtbre5wcfxwN0lh8D2WNB+6aaDl4AX3nprvQxfbtTeufUR89tNnGVmKvOPClm3r1aIk3T6rmtbA==;3:iI6Gf1okJRWiTGu1MGs+o1HeeYAkMsmnyOB9cuzYwodki+O3pHV0gGoC3SjaQB54SyUNRfIgnS/i21FShHZlKEGqmOYFJje93OlSSq1AaY3Khb657prPMT/kw9Cwggjgi6wCWO4A1ebRcKQNdqJY6g== X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:BN6PR15MB1233; X-Microsoft-Exchange-Diagnostics: 1;BN6PR15MB1233;25:CHfcBh1fo+JDC3hJ/aPvScX0ZXR5mBMCAnaAzTimU6C7K8scZ6b3MrI0MWMHP41OIm2blDL8ABy69cFGM1K/rNqvn+mRhgml/pwTkPbP7kRFXxxXDodp9J7DGsR3Sk3UfL1LFvEQ0x/nlRYpYZComNZ3etw7rEJ49UDIoRVS7VRnvXVV+/tFxXMorrFDo39sus4zBEYOvTP+qF/dLw6kutzDJY6xsGMJqVbPvV916lCt8T93h4BIrRMK0uNZy3GQ7naZvh3EL5nyAxNd1bhdSEzncCnHmUouzUnlqdB8mXqncWBksNMYqr1sgzxXxpBBLZaxWBrG3JpPq4JUYiqmlHOFemw68GdDaNbuD69LHwrAHlhNVbpOMUzcE2JLT0vd2+nDhSiSzBb5YLRFIaRbw8yV/x9A9tbKYHT8Cllg5uPQK1kJ2FQghtEhmq6D6S0nV43QBzKGLzNclEqeV6BG2U+x99N8OlfVYZW7+uGRl9ExrmHw3CBG6eoHEeTc00XBbIKJUOd86Twg2RKaofs3j/n4SaNpU4cWYYCRBIgde5bq8SyZ8WOcQFdnZxO9+Qxch0Q9j3jOpchOQat+UkDFVmACGlbQ0lPm6C7QII3lbO/q0tK8SpwJHqa7bFB/p020Ok5WjmJBlvEdOftoOrmI4Ba6CT/uVALbD68AKihR0c93t033G674QAtYdb8GmWHo9YD9qU6NUYsnfDYfiAZ7spRS/NYqb7rMH5G9tTzcmxhBupElHlrRS72milxRthxIF+L+5GDMGar+zInrq1Qm7g== X-Microsoft-Exchange-Diagnostics: 1;BN6PR15MB1233;31:z6i28KvPRdgfqoHaqJjtQhI+kK/s5NJKIqw0hTCnfwdX1PQzcNk7XqffpRjtL0vpSgxcjttO49R/CIerK1LHKcSklRVdlXAiFa/F5QE80amKcOXm5nleqL7MdIGveBO1wlOYirwr9pVIIwxzLmDn9pKVe4VRDJd3rdK2Rbzj8sK5ZRwpAS66xJ3wR1obCF2qb8zYC9kLtd/DzKwtXltOa3TnI4Zpm4loH+z9+u4bJemUR3C7LUnRO2IGJePPrRwv;20:LHpLx1cdIExrHmsFi9pskHQnuNg9jpepdBSYYF4LzD/dJrRCXkQn/ZxH/+3sZiT5ABnnyz6g5WOLw72R3NZ+Tlh8TTOrvEGSqfY4uNMrc65CDtKieWr4FLecr8E6wTCddID/c3Al8EvclSUtXdXYjY7TSFHy1NOJVQN7LZIMnoU=;4:wcYpxMOFFjXRkP34owbCR7GGpQYemEZNm98idC3gaB9EHGH1Pn5SxqdsBEOVU3dNR2+yXcVzH4UngV81y8ZzS9CWdrvWxGllQbhFuhxi/21w7IFZvpCy0W134ohUZNbbqiqldg+YA1AUsMMzOKyFSR+v6TG045njYjBFvoLJeHBvV4jjXvwOE4A/hBr8rUzOXy13ImqkKT4onCZdm5Ty+3iAWC+s54jEu2rQurcDaNka/qT58DQvm/9qiyp+LprCQmdNRXDqW/wd4xfF/wGGHXWJCkNaM5IhCrzGa9fC8o/hx4+3BcxoaBepFY13P9VaP5dO7cgOTOQIfM5nm12/Mim6M+Hjxo8cg+gvFtHFIAc8QXjuyxi9ye2xGnehtyN/OAufRKGUuKpeXEwIyLFjMg== X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040176)(601004)(2401047)(5005006)(8121501046)(10201501046)(3002001);SRVR:BN6PR15MB1233;BCL:0;PCL:0;RULEID:;SRVR:BN6PR15MB1233; X-Forefront-PRVS: 0107098B6C X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10019020)(4630300001)(6009001)(7916002)(189002)(377454003)(24454002)(199003)(586003)(2906002)(83506001)(105586002)(42186005)(92566002)(23726003)(189998001)(1076002)(6116002)(77096005)(93886004)(86362001)(575784001)(110136003)(106356001)(81166006)(33656002)(81156014)(8676002)(4326007)(9686002)(5660300001)(53416004)(50466002)(3480700004)(305945005)(6636002)(6666003)(101416001)(4001350100001)(7846002)(69596002)(19580395003)(19580405001)(97736004)(50986999)(68736007)(2950100002)(54356999)(76176999)(47776003)(5890100001)(7736002)(18370500001)(42262002);DIR:OUT;SFP:1102;SCL:1;SRVR:BN6PR15MB1233;H:clm-mbp.thefacebook.com;FPR:;SPF:None;PTR:InfoNoRecords;MX:1;A:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1;BN6PR15MB1233;23:OmwbpXWj0Xk6ICtRe9cdQ6rgHxQ4a9Ktx2ZVVvv3L?= =?us-ascii?Q?CaOytEBASSz4rxMw9mTlLdSq7HFYUkBCjIP6YKL5SxuBaIyNVNtwIuutyDhg?= =?us-ascii?Q?ry00uVCLDDRz1LTmEljNVUn1CYHaHYzqZ7+uuAQmPCWTwHNQjMOrmqOdAJJZ?= =?us-ascii?Q?W8wPzxZW1bXI5EyIdtaS9EuTFi/1XiREb9UNXuhGJRWMCDe8SXqCxeeA2Y5K?= =?us-ascii?Q?+2ZrFWVQot4bXgn1+JBPol9V3mgn/biZrsZJUf6Jt/0s+3v8TxRqJSvlEZT+?= =?us-ascii?Q?A9qGJI32uX+NeNVI31JBmB1X/rUppDOkmznxbyiKDgqkC1SXfAfbxSP0QGw3?= =?us-ascii?Q?hMXBdPAo9pG/hO5m+4Q3GlgIGrx692zlelMtE/ac9D2WMyYyDNHIctY3qzOM?= =?us-ascii?Q?apCmL4dnertsZLzxVc/vgmvpzybunrzoBEbDl/0ZZNxItfnplQbM2FAWjatA?= =?us-ascii?Q?A9XHhSLjBLD/SIDtnbNiE6MQ0wdI27/3pQ/eoIHYwYXBeCXmBdzxF6Ngu61N?= =?us-ascii?Q?Vm98G+lmzQVG3MdR5lBncRGieKLBvRhEFdST+JqYrEsoEt0ZUTncuKHGBc62?= =?us-ascii?Q?wgC0QKQpvkXJ3xeVnQkveGives2OSOseNCPLqTt3bTjNVzLIFBisARz9wsRQ?= =?us-ascii?Q?1D/46Bx3eIZLi0ec+U2qOdlP06c4SnowTiLsvRH6RIkQnQgn6okZpBz2lL4C?= =?us-ascii?Q?dAbBi5XE87gmxqF8hNTaSyeUsHmDYFa/6IpYDXNwYhp+mTuXH+oMAVSY+4x+?= =?us-ascii?Q?XxXPkvXLZ9b1HHWNiVkelYTKSkeqCAUK7g4cUB7pl8oBoWGAABC+s1sGF1U6?= =?us-ascii?Q?T1PLnu07UHOt5HFlRlzdcLxQksx1tl1fl84c0OYSoe3Eb+KP4RUiA3uh11De?= =?us-ascii?Q?GhBF4gwIrtCr/u/5vFx31inBPOKzyHWJUN5ciufrCR/f/ulFYkJLaMA6lDUX?= =?us-ascii?Q?Lj0Hamw9R14A0eLGpqxJN1V35k0bkU++rZwrig3ESCyKZFSAVtN9SnV5rM5U?= =?us-ascii?Q?mMyGdj0QbBfpT/stDw0vxpAM5qR5yJRYKlv7L5kWnwf5Xv6iQclcHr/4H6ta?= =?us-ascii?Q?FZzNNWdmwfT1XxQoV3qCx6cDL02GSInE8XMmphWIwJYBNmVznYADAQI42EG6?= =?us-ascii?Q?5SKNw37MDFgGNm6biF3ZyYJKvkTvLDZZgbgpO+Gsis1m1iIXDEhg4KsvXoIv?= =?us-ascii?Q?UxX7lNbauqleOheMEQX8cctao1oypFR/VeLXhnh7ACgDhqqLfBRj8xmqBiiF?= =?us-ascii?Q?904etRNq8HpMq3i3HkmuggsGJl0Towj2AU0LFCrWHBl26eFtmrbMj1v1zZrO?= =?us-ascii?Q?i6/tiwlNsgcnNAuKAcRrPI=3D?= X-Microsoft-Exchange-Diagnostics: 1;BN6PR15MB1233;6:mQg3dlABhUYNmTh+MeItc6gCfyOtseYI4Ion6GrL/f8FD3lywBAXxicJc6ki0HCC/9yWpmy6GeT3ANpvItNtj5rAnHcgaxUg+3wnghRdq2+vRMWhsNQio+UIs8X2+QLmPggaLi5Qy3FldPXYPlHs+W63DUTDNF/KFS9aFiNgbeVlwBCXb5klAwS53CizGw37jll++QpSbvcuKX9M+M9OKLjPXSI7O7SYBkXiSir+YDYy8nMq/iUk45FUxSk+ieL25NVBEqZQ5EYRXHX/l++IqZabrN4wKFTwuFoSDqNVq/cSrFuld5TYB6uiFBKZVsbK;5:UaYtir3yICDNrzgseeI39HK0HNfPMSPA2SMYJv/gpfmQhoNudFlc8z3btOz2GwvtnyELwVtMSBpPTQw9iTmpivbMOomicYzqPTnV1qQna/FgCHNbfEnMBk2NKVf+cbVxfVKTpfuUlwiiwZazX/luHw==;24:jQuKGOfRzQ9JKV4v8g3DMnUMoSsrRC2K59BgjGaLckpKyZfDuYYE8KTZeRXzat6rFeA4HrUuDpl1gnOVbtBtx/bUlp2v3P+1uUhPs91Zio8=;7:rg0iq4pduAAB7xx6/Y/0QgRMxADReiggbIjktgvcjgdTfLs+hNy0/96MvsoqujNe3uo+jv0n5LWDtj1dJOIyd7cyn/E1ZN3BdhIvFOPUSOn/WYQPPe323RGfzP5ccjhiJc4y6lrTE+5CqmpzrdfK2jEaxtpq0z78VBDcQYfaXnUTEDPQ26UmlxZHRvI3kldukeTmoJskt8nfYpV15j+ghi5PQWcaa0ZBgHChw1zJOZ/oeiKiJhzCFJeqHdarPDdvuJLBKmvlNWLY5bZBJf1fLEaQhMaN/CvXhTua2Zg/Un6SK3/SLtxVyj+KfECO5UaVwdJmUUBeMI7jClMFcZXW0Zgb5lzimxAPb3dMQNmwu2Y= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;BN6PR15MB1233;20:IeiAlP3anOfVp6wjWw9g3uxBHDD8rOGMJ/id5gDVWGViLWLmC93kbzgETKbRknP5rEfM7PQtZvXcpnzuCEf8gIUpZqr9rjsXD+Bo+5giZEBsR8fWiZ6bXLrJ8dvlDn3bVyT612o1Rq5Jgeucq04n3y9XMhSrV+mD7T3oMgE5WnQ= X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Oct 2016 23:19:59.4803 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR15MB1233 X-OriginatorOrg: fb.com X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2016-10-26_14:,, signatures=0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 5586 Lines: 125 On Wed, Oct 26, 2016 at 05:03:45PM -0600, Jens Axboe wrote: >On 10/26/2016 04:58 PM, Linus Torvalds wrote: >>On Wed, Oct 26, 2016 at 3:51 PM, Linus Torvalds >> wrote: >>> >>>Dave: it might be a good idea to split that "WARN_ON_ONCE()" in >>>blk_mq_merge_queue_io() into two >> >>I did that myself too, since Dave sees this during boot. >> >>But I'm not getting the warning ;( >> >>Dave gets it with ext4, and thats' what I have too, so I'm not sure >>what the required trigger would be. > >Actually, I think I see what might trigger it. You are on nvme, iirc, >and that has a deep queue. Dave, are you testing on a sata drive or >something similar with a shallower queue depth? If we end up sleeping >for a request, I think we could trigger data->ctx being different. > >Dave, can you hit the warnings with this? Totally untested... Confirmed, totally untested ;) Don't try this one at home folks (working this out with Jens offlist) G: unable to handle kernel paging request at 0000000002411200 IP: [] _raw_spin_lock+0x22/0x40 PGD 12840a067 PUD 128446067 PMD 0 Oops: 0002 [#1] PREEMPT SMP Modules linked in: virtio_blk(+) CPU: 4 PID: 125 Comm: modprobe Not tainted 4.9.0-rc2-00041-g811d54d-dirty #320 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.0-1.fc24 04/01/2014 task: ffff88013849aac0 task.stack: ffff8801293d8000 RIP: 0010:[] [] _raw_spin_lock+0x22/0x40 RSP: 0018:ffff8801293db278 EFLAGS: 00010246 RAX: 0000000000000000 RBX: 0000000002411200 RCX: 0000000000000000 RDX: 0000000000000001 RSI: ffff88013a5c1048 RDI: 0000000000000000 RBP: ffff8801293db288 R08: 0000000000000005 R09: ffff880128449380 R10: 0000000000000000 R11: 0000000000000008 R12: 0000000000000000 R13: 0000000000000001 R14: 0000000000000076 R15: ffff8801293b6a80 FS: 00007f1a2a9cdb40(0000) GS:ffff88013fd00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000002411200 CR3: 000000013a5d1000 CR4: 00000000000406e0 Stack: ffff8801293db2d0 ffff880128488000 ffff8801293db348 ffffffff814debff 00ff8801293db2c8 ffff8801293db338 ffff8801284888c0 ffff8801284888b8 000060fec00004f9 0000000002411200 ffff880128f810c0 ffff880128f810c0 Call Trace: [] blk_sq_make_request+0x34f/0x580 [] ? mempool_alloc_slab+0x15/0x20 [] generic_make_request+0x104/0x200 [] submit_bio+0x65/0x130 [] submit_bh_wbc+0x16e/0x210 [] submit_bh+0x13/0x20 [] block_read_full_page+0x205/0x3d0 [] ? I_BDEV+0x20/0x20 [] ? lru_cache_add+0xe/0x10 [] ? add_to_page_cache_lru+0x92/0xf0 [] ? __page_cache_alloc+0xd1/0xe0 [] blkdev_readpage+0x18/0x20 [] do_read_cache_page+0x1c6/0x380 [] ? blkdev_writepages+0x10/0x10 [] ? alloc_pages_current+0xb2/0x1c0 [] read_cache_page+0x12/0x20 [] read_dev_sector+0x31/0xb0 [] read_lba+0xbd/0x130 [] find_valid_gpt+0xa2/0x580 [] ? find_valid_gpt+0x580/0x580 [] efi_partition+0x67/0x3d0 [] ? vsnprintf+0x2aa/0x470 [] ? snprintf+0x34/0x40 [] ? find_valid_gpt+0x580/0x580 [] check_partition+0x106/0x1e0 [] rescan_partitions+0x8c/0x270 [] __blkdev_get+0x328/0x3f0 [] blkdev_get+0x54/0x320 [] ? unlock_new_inode+0x5a/0x80 [] ? bdget+0xff/0x110 [] device_add_disk+0x3c6/0x450 [] ? ioread8+0x1a/0x40 [] ? vp_get+0x4e/0x70 [] virtblk_probe+0x460/0x708 [virtio_blk] [] ? vp_finalize_features+0x36/0x50 [] virtio_dev_probe+0x132/0x1e0 [] driver_probe_device+0x1a9/0x2d0 [] ? mutex_lock+0x24/0x50 [] __driver_attach+0xbd/0xc0 [] ? driver_probe_device+0x2d0/0x2d0 [] ? driver_probe_device+0x2d0/0x2d0 [] bus_for_each_dev+0x8a/0xb0 [] driver_attach+0x1e/0x20 [] bus_add_driver+0x1b6/0x230 [] driver_register+0x60/0xe0 [] register_virtio_driver+0x20/0x40 [] init+0x57/0x81 [virtio_blk] [] ? 0xffffffffa0004000 [] ? 0xffffffffa0004000 [] do_one_initcall+0x46/0x150 [] do_init_module+0x6a/0x210 [] ? vfree+0x37/0x90 [] load_module+0x1638/0x1860 [] ? do_free_init+0x30/0x30 [] ? kernel_read_file_from_fd+0x54/0x90 [] SYSC_finit_module+0xc2/0xd0 [] SyS_finit_module+0xe/0x10 [] entry_SYSCALL_64_fastpath+0x13/0x94 Code: 89 df e8 a2 52 70 ff eb e6 55 48 89 e5 53 48 83 ec 08 66 66 66 66 90 48 89 fb bf 01 00 00 00 e8 95 53 6e ff 31 c0 ba 01 00 00 00 0f b1 13 85 c0 75 07 48 83 c4 08 5b c9 c3 89 c6 48 89 df e8 RIP [] _raw_spin_lock+0x22/0x40 RSP CR2: 0000000002411200 ---[ end trace e8cb117e64947621 ]--- Kernel panic - not syncing: Fatal exception Kernel Offset: disabled ---[ end Kernel panic - not syncing: Fatal exception -chris