Received: by 2002:a25:ab43:0:0:0:0:0 with SMTP id u61csp5986824ybi; Wed, 29 May 2019 00:42:48 -0700 (PDT) X-Google-Smtp-Source: APXvYqyw8AcJ5q/wlCA10tLr6uh7Ukv+9f1ihh9SNhYSMa6i2bktZpudeAE1PoTZuA2fDMjHWfcY X-Received: by 2002:aa7:8d81:: with SMTP id i1mr122018664pfr.244.1559115768259; Wed, 29 May 2019 00:42:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559115768; cv=none; d=google.com; s=arc-20160816; b=oQcuCzvOK9QHP/UZTl2X19C5t6M0hKXEvDJcS6AveX7Y4j71vV1AOGEkPM3U74ZUBU U7t8JfT42r2O2X4Xvn89Hr7wizZraRcka5twVjpw8Axs2jPEgz17xNL+2GIaMNcm+LAj ioJ+UR5K2vECthQHse8wGTrCEZS3crIYD6qBzXJcy7WbzdZ+sPWWZ9ZQjTqcxy7w1JUX aMpkx+OG+YYWdjJXSViwmJvziP59KgPkRV4nfpK5pS3x7JEmSQKxoj4bj+q0p2L3tk8T EfzsupDTZkrvlhmseyjSzysWbjm+/KF1eR838hhTra23/9KsSv5vO6Whoo5y2togBftw T4PQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:to:cc:in-reply-to:date:subject :mime-version:message-id:from:dkim-signature; bh=SOy3B69zClHnFybJMv8XI3Olw2skCEmEA1+tMUtIH0I=; b=MIcIdOhhpnbtc+KC3yjrdyFSp75LBLVgDNX66eTczTpiJZDfsaeay8eZPKO3FYr3rb Te150aRUcz5T3L+/ZW9OX67U9wjvbhOuEXOrhg8ofbOEC6SO2whbXt/5J8cdL67eIHqH hcULpEGW1vNe0tZKsnJEfRJG7YRg5JN2JUPMnqz7m6/xB9+fe2kowjaBqNu9fm1TmmeU rPU70Z8NlncavnoKNW25YMWl99tpLIm1igZYlQs5W7MAuDMF/MvymdNfFS4XBRYzsWVx Gqu0lF9iZQNnfCSC1vYPG69cDMcgdbMQR6EIm1/jy7j2hSInuR8DjV82XnSjbSx8D0IR Ta8g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=safvrxif; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id n13si24812415pgv.304.2019.05.29.00.42.32; Wed, 29 May 2019 00:42:48 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=safvrxif; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726857AbfE2HlY (ORCPT + 99 others); Wed, 29 May 2019 03:41:24 -0400 Received: from mail-wm1-f68.google.com ([209.85.128.68]:35621 "EHLO mail-wm1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726101AbfE2HlX (ORCPT ); Wed, 29 May 2019 03:41:23 -0400 Received: by mail-wm1-f68.google.com with SMTP id w9so812422wmi.0 for ; Wed, 29 May 2019 00:41:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:message-id:mime-version:subject:date:in-reply-to:cc:to :references; bh=SOy3B69zClHnFybJMv8XI3Olw2skCEmEA1+tMUtIH0I=; b=safvrxifA82sUE1Dw036tt/ISyC+sByTQSLTH6TjRNc5bASOjOtJ1y+wJz+6vw1xiz CgsT0QA9bFy5sZihY8HgVo2P5g3m00WFcb4r6xpp/cPgtcYvR+i3Ht83XLdP/mLXv4jH jqWhFYa2817Jud+CFJAzieFknry94eWp8oX3BcoRWyevRB/Ad7iT/tzFy6wRrunjKe63 FxlIO3FXcMgCVMq8yuiBxONoMchIaf0OctxP5Vulmn2MjPj7b0iG1EspRiQHjI3Sj7Gg 9u80ALo7aOFuS3FfZPXBYXDm30VBqlOHj0v+CdXUCBJ+JGQhR94y+AwxLRdfb4i2/sMn YtNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:message-id:mime-version:subject:date :in-reply-to:cc:to:references; bh=SOy3B69zClHnFybJMv8XI3Olw2skCEmEA1+tMUtIH0I=; b=bCmemxVFKxciNC9PoNC02jAYSr4Y/nd1RMScd0aoTyQKcUAQ0jw5zysbUqj6dAuHLX QorXAammFRFwpsPY/380EkNIcnjcXjjfELoRXqdKr0fSVRX8UtuwFwzeIuqT8+nb3CZP Mj6hfba4ABmCV6mpwjVmRhUE+ZDSm84ruVqxaUaIDPuv4sg8r3NFBYfhLZfUY+fDZN+N zTlmUhj0oDgGGMfGZVNy9zXDzrhLEfvrV5NVqSsuzueNIVEOiHGtV7+QGh/ehZE6hGfP vM0mbtfKQEHuLNVCoCHEjj7BEOlAO63d6QKXg3TVz+PAN7LnECuilXuBUZXtrJv4EU8G vscw== X-Gm-Message-State: APjAAAVFD/IB155X5LsZ62/DNAbb+bShcPeu4zC5D5Bc8ugt4lKbC/Se R13BCIF+H873zRJIDk+qsNzmTA== X-Received: by 2002:a1c:5f09:: with SMTP id t9mr6183006wmb.112.1559115680026; Wed, 29 May 2019 00:41:20 -0700 (PDT) Received: from [192.168.0.100] (146-241-114-97.dyn.eolo.it. [146.241.114.97]) by smtp.gmail.com with ESMTPSA id n10sm21383362wrr.11.2019.05.29.00.41.18 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 29 May 2019 00:41:19 -0700 (PDT) From: Paolo Valente Message-Id: Content-Type: multipart/signed; boundary="Apple-Mail=_88874A7E-B5F0-4892-B8A9-16FC260C00C1"; protocol="application/pgp-signature"; micalg=pgp-sha256 Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.8\)) Subject: Re: CFQ idling kills I/O performance on ext4 with blkio cgroup controller Date: Wed, 29 May 2019 09:41:17 +0200 In-Reply-To: <5b71028c-72f0-73dd-0cd5-f28ff298a0a3@csail.mit.edu> Cc: linux-fsdevel@vger.kernel.org, linux-block , linux-ext4@vger.kernel.org, cgroups@vger.kernel.org, kernel list , Jens Axboe , Jan Kara , jmoyer@redhat.com, Theodore Ts'o , amakhalov@vmware.com, anishs@vmware.com, srivatsab@vmware.com To: "Srivatsa S. Bhat" References: <8d72fcf7-bbb4-2965-1a06-e9fc177a8938@csail.mit.edu> <1812E450-14EF-4D5A-8F31-668499E13652@linaro.org> <46c6a4be-f567-3621-2e16-0e341762b828@csail.mit.edu> <07D11833-8285-49C2-943D-E4C1D23E8859@linaro.org> <5B6570A2-541A-4CF8-98E0-979EA6E3717D@linaro.org> <2CB39B34-21EE-4A95-A073-8633CF2D187C@linaro.org> <0e3fdf31-70d9-26eb-7b42-2795d4b03722@csail.mit.edu> <686D6469-9DE7-4738-B92A-002144C3E63E@linaro.org> <01d55216-5718-767a-e1e6-aadc67b632f4@csail.mit.edu> <6FE0A98F-1E3D-4EF6-8B38-2C85741924A4@linaro.org> <2A58C239-EF3F-422B-8D87-E7A3B500C57C@linaro.org> <5b71028c-72f0-73dd-0cd5-f28ff298a0a3@csail.mit.edu> X-Mailer: Apple Mail (2.3445.104.8) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org --Apple-Mail=_88874A7E-B5F0-4892-B8A9-16FC260C00C1 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii > Il giorno 29 mag 2019, alle ore 03:09, Srivatsa S. Bhat = ha scritto: >=20 > On 5/23/19 11:51 PM, Paolo Valente wrote: >>=20 >>> Il giorno 24 mag 2019, alle ore 01:43, Srivatsa S. Bhat = ha scritto: >>>=20 >>> When trying to run multiple dd tasks simultaneously, I get the = kernel >>> panic shown below (mainline is fine, without these patches). >>>=20 >>=20 >> Could you please provide me somehow with a list = *(bfq_serv_to_charge+0x21) ? >>=20 >=20 > Hi Paolo, >=20 > Sorry for the delay! Here you go: >=20 > (gdb) list *(bfq_serv_to_charge+0x21) > 0xffffffff814bad91 is in bfq_serv_to_charge = (./include/linux/blkdev.h:919). > 914 > 915 extern unsigned int blk_rq_err_bytes(const struct request *rq); > 916 > 917 static inline unsigned int blk_rq_sectors(const struct request = *rq) > 918 { > 919 return blk_rq_bytes(rq) >> SECTOR_SHIFT; > 920 } > 921 > 922 static inline unsigned int blk_rq_cur_sectors(const struct = request *rq) > 923 { > (gdb) >=20 >=20 > For some reason, I've not been able to reproduce this issue after > reporting it here. (Perhaps I got lucky when I hit the kernel panic > a bunch of times last week). >=20 > I'll test with your fix applied and see how it goes. >=20 Great! the offending line above gives me hope that my fix is correct. If no more failures occur, then I'm eager (and a little worried ...) to see how it goes with throughput :) Thanks, Paolo > Thank you! >=20 > Regards, > Srivatsa >=20 >>=20 >>> [ 568.232231] BUG: kernel NULL pointer dereference, address: = 0000000000000024 >>> [ 568.232257] #PF: supervisor read access in kernel mode >>> [ 568.232273] #PF: error_code(0x0000) - not-present page >>> [ 568.232289] PGD 0 P4D 0 >>> [ 568.232299] Oops: 0000 [#1] SMP PTI >>> [ 568.232312] CPU: 0 PID: 1029 Comm: dd Tainted: G E = 5.1.0-io-dbg-4+ #6 >>> [ 568.232334] Hardware name: VMware, Inc. VMware Virtual = Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016 >>> [ 568.232388] RIP: 0010:bfq_serv_to_charge+0x21/0x50 >>> [ 568.232404] Code: ff e8 c3 5e bc ff 0f 1f 00 0f 1f 44 00 00 48 8b = 86 20 01 00 00 55 48 89 e5 53 48 89 fb a8 40 75 09 83 be a0 01 00 00 01 = 76 09 <8b> 43 24 c1 e8 09 5b 5d c3 48 8b 7e 08 e8 5d fd ff ff 84 c0 75 = ea >>> [ 568.232473] RSP: 0018:ffffa73a42dab750 EFLAGS: 00010002 >>> [ 568.232489] RAX: 0000000000001052 RBX: 0000000000000000 RCX: = ffffa73a42dab7a0 >>> [ 568.232510] RDX: ffffa73a42dab657 RSI: ffff8b7b6ba2ab70 RDI: = 0000000000000000 >>> [ 568.232530] RBP: ffffa73a42dab758 R08: 0000000000000000 R09: = 0000000000000001 >>> [ 568.232551] R10: 0000000000000000 R11: ffffa73a42dab7a0 R12: = ffff8b7b6aed3800 >>> [ 568.232571] R13: 0000000000000000 R14: 0000000000000000 R15: = ffff8b7b6aed3800 >>> [ 568.232592] FS: 00007fb5b0724540(0000) GS:ffff8b7b6f800000(0000) = knlGS:0000000000000000 >>> [ 568.232615] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >>> [ 568.232632] CR2: 0000000000000024 CR3: 00000004266be002 CR4: = 00000000001606f0 >>> [ 568.232690] Call Trace: >>> [ 568.232703] bfq_select_queue+0x781/0x1000 >>> [ 568.232717] bfq_dispatch_request+0x1d7/0xd60 >>> [ 568.232731] ? = bfq_bfqq_handle_idle_busy_switch.isra.36+0x2cd/0xb20 >>> [ 568.232751] blk_mq_do_dispatch_sched+0xa8/0xe0 >>> [ 568.232765] blk_mq_sched_dispatch_requests+0xe3/0x150 >>> [ 568.232783] __blk_mq_run_hw_queue+0x56/0x100 >>> [ 568.232798] __blk_mq_delay_run_hw_queue+0x107/0x160 >>> [ 568.232814] blk_mq_run_hw_queue+0x75/0x190 >>> [ 568.232828] blk_mq_sched_insert_requests+0x7a/0x100 >>> [ 568.232844] blk_mq_flush_plug_list+0x1d7/0x280 >>> [ 568.232859] blk_flush_plug_list+0xc2/0xe0 >>> [ 568.232872] blk_finish_plug+0x2c/0x40 >>> [ 568.232886] ext4_writepages+0x592/0xe60 >>> [ 568.233381] ? ext4_mark_iloc_dirty+0x52b/0x860 >>> [ 568.233851] do_writepages+0x3c/0xd0 >>> [ 568.234304] ? ext4_mark_inode_dirty+0x1a0/0x1a0 >>> [ 568.234748] ? do_writepages+0x3c/0xd0 >>> [ 568.235197] ? __generic_write_end+0x4e/0x80 >>> [ 568.235644] __filemap_fdatawrite_range+0xa5/0xe0 >>> [ 568.236089] ? __filemap_fdatawrite_range+0xa5/0xe0 >>> [ 568.236533] ? ext4_da_write_end+0x13c/0x280 >>> [ 568.236983] file_write_and_wait_range+0x5a/0xb0 >>> [ 568.237407] ext4_sync_file+0x11e/0x3e0 >>> [ 568.237819] vfs_fsync_range+0x48/0x80 >>> [ 568.238217] ext4_file_write_iter+0x234/0x3d0 >>> [ 568.238610] ? _cond_resched+0x19/0x40 >>> [ 568.238982] new_sync_write+0x112/0x190 >>> [ 568.239347] __vfs_write+0x29/0x40 >>> [ 568.239705] vfs_write+0xb1/0x1a0 >>> [ 568.240078] ksys_write+0x89/0xc0 >>> [ 568.240428] __x64_sys_write+0x1a/0x20 >>> [ 568.240771] do_syscall_64+0x5b/0x140 >>> [ 568.241115] entry_SYSCALL_64_after_hwframe+0x49/0xbe >>> [ 568.241456] RIP: 0033:0x7fb5b02325f4 >>> [ 568.241787] Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b3 0f = 1f 80 00 00 00 00 48 8d 05 09 11 2d 00 8b 00 85 c0 75 13 b8 01 00 00 00 = 0f 05 <48> 3d 00 f0 ff ff 77 54 f3 c3 66 90 41 54 55 49 89 d4 53 48 89 = f5 >>> [ 568.242842] RSP: 002b:00007ffcb12e2968 EFLAGS: 00000246 ORIG_RAX: = 0000000000000001 >>> [ 568.243220] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: = 00007fb5b02325f4 >>> [ 568.243616] RDX: 0000000000000200 RSI: 000055698f2ad000 RDI: = 0000000000000001 >>> [ 568.244026] RBP: 0000000000000200 R08: 0000000000000004 R09: = 0000000000000003 >>> [ 568.244401] R10: 00007fb5b04feca0 R11: 0000000000000246 R12: = 000055698f2ad000 >>> [ 568.244775] R13: 0000000000000000 R14: 0000000000000000 R15: = 000055698f2ad000 >>> [ 568.245154] Modules linked in: xt_MASQUERADE(E) = nf_conntrack_netlink(E) nfnetlink(E) xfrm_user(E) xfrm_algo(E) = xt_addrtype(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) = vmw_vsock_vmci_transport(E) vsock(E) ip6table_filter(E) ip6_tables(E) = xt_conntrack(E) iptable_mangle(E) iptable_nat(E) nf_nat(E) = iptable_filter >>> [ 568.248651] CR2: 0000000000000024 >>> [ 568.249142] ---[ end trace 0ddd315e0a5bdfba ]--- >>>=20 --Apple-Mail=_88874A7E-B5F0-4892-B8A9-16FC260C00C1 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEpYoduex+OneZyvO8OAkCLQGo9oMFAlzuN50ACgkQOAkCLQGo 9oOyVxAAiJh/RlNWNerXrJ2E+eEQCMmQPRoqJEZ3Jr93GqJ1Y8/XqR7efG2rX90H rgqYU/rzcZgANERdk99Z2wJHF/W2tO+f7OZ2iGB3NetMvny9YS+8R0bcaH6b8uZQ pRwOrC2wbGNCApvbPP9Gm85SgJggEdeFWNkN9UcMgJwfsWvR0tIWUWiYppDMsuQs JcTfX4glwOm5RUjXAYslIqNm2bYRpBWbYM0zr+YrjvoipKwj/gkzur3B3htn3Pqj PNBAH1O0684ZkRTOMaDM/OQxNv5BlgDXvEuAoih0kS3aSk+H/JPiacJ5iUaa+j1u d7UR3/OPsq5zM351WmcCvkBbLSqqQlBeNo0Nktm1/CAQ4I+F7Z+IYWgLtCSqgXPw BG4mdupvQnFK6wb08xBd9C7CfSO2rQyGYn95Gg+eFpmcwZ+VWQPZyKqmDxN/0zW9 5ERe0EHJUePOmHtrpmfcqcltT2pkvNNqGnz6F0cY7u9vZcHxjRSKaLltgUJzRjwZ hJzjif8W1GR3wrSV/mICFU2RBujBPnIaR0XMG+UAsTl0PzFPM5ICLV5ws5wnSnFk WZaOYPmQgI64j/N0B5BV0EccxVUstWizDuKtDDFMWvfNs9rjeQDaShUzL7r0g2gx IRVDdj9POK11csHsato4TZzMDXj3nzMtMRbj8b8dim3DIS0zjHM= =DNH0 -----END PGP SIGNATURE----- --Apple-Mail=_88874A7E-B5F0-4892-B8A9-16FC260C00C1--