Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S933485AbdLRJZi (ORCPT ); Mon, 18 Dec 2017 04:25:38 -0500 Received: from mail-qt0-f174.google.com ([209.85.216.174]:33741 "EHLO mail-qt0-f174.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S933457AbdLRJZf (ORCPT ); Mon, 18 Dec 2017 04:25:35 -0500 X-Google-Smtp-Source: ACJfBotv6gm4xF/EicXlX8D2YMt3kX3x8pO0uh9e4/m3GztnciYSDJR3YVDtDIhjKE9efYMKIlj+LZWvIMyPXFFt59Q= MIME-Version: 1.0 In-Reply-To: <20171201213322.GW692@shells.gnugeneration.com> References: <20171129183919.GQ692@shells.gnugeneration.com> <20171201213322.GW692@shells.gnugeneration.com> From: Enric Balletbo Serra Date: Mon, 18 Dec 2017 10:25:33 +0100 Message-ID: Subject: Re: [REGRESSION] (>= v4.12) IO w/dmcrypt causing audio underruns To: vcaputo@pengaru.com Cc: linux-kernel , timmurray@google.com, tj@kernel.org Content-Type: text/plain; charset="UTF-8" Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 4180 Lines: 96 Hi Vito, 2017-12-01 22:33 GMT+01:00 : > On Wed, Nov 29, 2017 at 10:39:19AM -0800, vcaputo@pengaru.com wrote: >> Hello, >> >> Recently I noticed substantial audio dropouts when listening to MP3s in >> `cmus` while doing big and churny `git checkout` commands in my linux git >> tree. >> >> It's not something I've done much of over the last couple months so I >> hadn't noticed until yesterday, but didn't remember this being a problem in >> recent history. >> >> As there's quite an accumulation of similarly configured and built kernels >> in my grub menu, it was trivial to determine approximately when this began: >> >> 4.11.0: no dropouts >> 4.12.0-rc7: dropouts >> 4.14.0-rc6: dropouts (seem more substantial as well, didn't investigate) >> >> Watching top while this is going on in the various kernel versions, it's >> apparent that the kworker behavior changed. Both the priority and quantity >> of running kworker threads is elevated in kernels experiencing dropouts. >> >> Searching through the commit history for v4.11..v4.12 uncovered: >> >> commit a1b89132dc4f61071bdeaab92ea958e0953380a1 >> Author: Tim Murray >> Date: Fri Apr 21 11:11:36 2017 +0200 >> >> dm crypt: use WQ_HIGHPRI for the IO and crypt workqueues >> >> Running dm-crypt with workqueues at the standard priority results in IO >> competing for CPU time with standard user apps, which can lead to >> pipeline bubbles and seriously degraded performance. Move to using >> WQ_HIGHPRI workqueues to protect against that. >> >> Signed-off-by: Tim Murray >> Signed-off-by: Enric Balletbo i Serra >> Signed-off-by: Mike Snitzer >> >> --- >> >> Reverting a1b8913 from 4.14.0-rc6, my current kernel, eliminates the >> problem completely. >> >> Looking at the diff in that commit, it looks like the commit message isn't >> even accurate; not only is the priority of the dmcrypt workqueues being >> changed - they're also being made "CPU intensive" workqueues as well. >> >> This combination appears to result in both elevated scheduling priority and >> greater quantity of participant worker threads effectively starving any >> normal priority user task under periods of heavy IO on dmcrypt volumes. >> >> I don't know what the right solution is here. It seems to me we're lacking >> the appropriate mechanism for charging CPU resources consumed on behalf of >> user processes in kworker threads to the work-causing process. >> >> What effectively happens is my normal `git` user process is able to >> greatly amplify what share of CPU it takes from the system by generating IO >> on what happens to be a high-priority CPU-intensive storage volume. >> >> It looks potentially complicated to fix properly, but I suspect at its core >> this may be a fairly longstanding shortcoming of the page cache and its >> asynchronous design. Something that has been exacerbated substantially by >> the introduction of CPU-intensive storage subsystems like dmcrypt. >> >> If we imagine the whole stack simplified, where all the IO was being done >> synchronously in-band, and the dmcrypt kernel code simply ran in the >> IO-causing process context, it would be getting charged to the calling >> process and scheduled accordingly. The resource accounting and scheduling >> problems all emerge with the page cache, buffered IO, and async background >> writeback in a pool of unrelated worker threads, etc. That's how it >> appears to me anyways... >> >> The system used is a X61s Thinkpad 1.8Ghz with 840 EVO SSD, lvm on dmcrypt. >> The kernel .config is attached in case it's of interest. >> >> Thanks, >> Vito Caputo > > > > Ping... > > Could somebody please at least ACK receiving this so I'm not left wondering > if my mails to lkml are somehow winding up flagged as spam, thanks! Sorry I did not notice your email before you ping me directly. It's interesting that issue, though we didn't notice this problem. It's a bit far since I tested this patch but I'll setup the environment again and do more tests to understand better what is happening. Thanks, Enric