Received: by 2002:ac0:946b:0:0:0:0:0 with SMTP id j40csp3082081imj; Mon, 11 Feb 2019 13:35:11 -0800 (PST) X-Google-Smtp-Source: AHgI3Ia4a1hrhJxupzIEvOENkUIBqf6DWDGWbTM9RBYtr323Rhrx33Ky77Wf9KbMLdhdSHw8ZvIg X-Received: by 2002:a17:902:241:: with SMTP id 59mr373092plc.72.1549920911679; Mon, 11 Feb 2019 13:35:11 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1549920911; cv=none; d=google.com; s=arc-20160816; b=cV/2Nd1eEffYA/f2eTFKjhYJ5nRe5dCD9mtqkw8YPuooB4kfufhNmP0SAMniepYCyA 0XNRulQpHdo/CXq5tR3TJmyqzMGe+Kc/1/OiyDhXHLgDJauZwqWIXsYHeUR+RmUykCLx Td9HOv3clIHDS3Qj/ZPFH3kUlniSy6Ej29KOJbdaaxi2PWrTeCjOmSEdzcQxmXCQDbln qRn12i6FZORPRz75mNNlGP1gGz4QKW5gqv8bCe64mIWsSa13HRSyXZLmxPxNc/+Y7xTt /nMJquavYaPWTJZBAZMkSAouccZmE8c+/wlsqmRJyCIBhOuns2Urak9zrwfTrvWSaKMq aigA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:user-agent:in-reply-to :content-disposition:mime-version:references:message-id:subject:cc :to:from:date:dkim-signature; bh=hVKwAqoHVFgMa2AWbKba1qi0i0vrDDmcwy0nd3S4ZKE=; b=kxZBdt+bpNeJlcf16C5ZQYsWGiBwy3uCPv1gTqqVTaqxZsz+bW3l+zo5McXW6Kxepn b+WuOyhl6iFVgajajE2UY40TGLmqDuVFf0G8vLVcwgZxpedFGz5zww7GLfHAieMzw8Xy vEMyhjFGaRd43/mG2WDQNoVgVRg1FhNvG8DEUtTtUqDepiIwrfveTUQSZ3Q46MR88r1u ON8agRAdJEjNcKfDZrIfMDMjhBSAQ9ONQZF0CTUPBh+Q3uFk+nyxCDEMFDwody1Hs91f 5hcdm5OFdySTT6STMzV5oPt7U12Mr6BZqXk5xYXdoiinhbfp0KJ45vvnfk4qeyMb761T ireQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@toxicpanda-com.20150623.gappssmtp.com header.s=20150623 header.b=n4kLctD7; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id 61si10758173plc.364.2019.02.11.13.34.55; Mon, 11 Feb 2019 13:35:11 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@toxicpanda-com.20150623.gappssmtp.com header.s=20150623 header.b=n4kLctD7; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727585AbfBKVdz (ORCPT + 99 others); Mon, 11 Feb 2019 16:33:55 -0500 Received: from mail-yw1-f67.google.com ([209.85.161.67]:34584 "EHLO mail-yw1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727212AbfBKVdz (ORCPT ); Mon, 11 Feb 2019 16:33:55 -0500 Received: by mail-yw1-f67.google.com with SMTP id u205so185875ywe.1 for ; Mon, 11 Feb 2019 13:33:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=hVKwAqoHVFgMa2AWbKba1qi0i0vrDDmcwy0nd3S4ZKE=; b=n4kLctD7UE5F10ODXcqZ39+kzrDLdbwMD/i67pe02a+359+Umq6pnwTJxeC682tS0I DMwKL3L153uyB0BH8cvIlBADZ3lTY7S1EQfotC+bLPGgmtetWYD+UumpJZSpgLQhqSo2 8nYsYOJQ41/OkZSt4eRCegqlBeGQWMA4PW+FYjDueMH/uRS853aNPooWC57pxrU/2RBm R1pHvY+LRehia/wmB9yRGoXGDUq1Nr1TxgSsHro9uaYwenJ/P5pjqiDiVWHtCrhJ6DF0 +OKywYzSNvOJIeLEXL8HofmqrDSBmWrcFd+3M9exT9kFuM30WqqtjPEZDNLrq4k6K274 SaAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=hVKwAqoHVFgMa2AWbKba1qi0i0vrDDmcwy0nd3S4ZKE=; b=YPGzLjA29Q3YVxh0uXovyRt4xcf8dwqMtczKO8EzLKw9LubTV3CFy6mLkZBjBHrUuw KwQGDNr2ga4OiUs0HkQo8nH2XqOkyjY/AEI4V/0Y37jtHE/1rcJdVOFlMokeePObiWDm R5kmDS61U6hb1SSZWrYrZ8ISxChbBXk+rMOkdIbq78Ke51o5u8GC7aA7g2VU/vwTUvQA fELVz/Rql8bbpvYzR4Ztu7udrkHVH50CdIpTekjZERoXovMBiz+mjK8Q39A2V9EJzpe8 qPb+NS9dY5/YMNxp8NYUZFrg0/NNDlmW3WxtWR3PjPS0kRnQoIgHTPbImPU9L4JcwYRd 45yQ== X-Gm-Message-State: AHQUAuZknlmoUze1z3cgOdK6E8tQS4H2omAB90O7nPTPpMOnUKEFRidR mfX3JHSUxIYqD20BntiGxww8/Q== X-Received: by 2002:a0d:d58d:: with SMTP id x135mr248358ywd.488.1549920834359; Mon, 11 Feb 2019 13:33:54 -0800 (PST) Received: from localhost ([2620:10d:c091:200::5:67a5]) by smtp.gmail.com with ESMTPSA id o4sm1571505ywe.102.2019.02.11.13.33.53 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 11 Feb 2019 13:33:53 -0800 (PST) Date: Mon, 11 Feb 2019 16:34:19 -0500 From: Josef Bacik To: Andrea Righi Cc: Josef Bacik , Paolo Valente , Tejun Heo , Li Zefan , Johannes Weiner , Jens Axboe , Vivek Goyal , Dennis Zhou , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: Re: [RFC PATCH v2] blkcg: prevent priority inversion problem during sync() Message-ID: <20190211213417.uhfiz5iqwkfrvk25@macbook-pro-91.dhcp.thefacebook.com> References: <20190209140749.GB1910@xps-13> <20190211153933.p26pu5jmbmisbkos@macbook-pro-91.dhcp.thefacebook.com> <20190211204029.GB1520@xps-13> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20190211204029.GB1520@xps-13> User-Agent: NeoMutt/20180716 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, Feb 11, 2019 at 09:40:29PM +0100, Andrea Righi wrote: > On Mon, Feb 11, 2019 at 10:39:34AM -0500, Josef Bacik wrote: > > On Sat, Feb 09, 2019 at 03:07:49PM +0100, Andrea Righi wrote: > > > This is an attempt to mitigate the priority inversion problem of a > > > high-priority blkcg issuing a sync() and being forced to wait the > > > completion of all the writeback I/O generated by any other low-priority > > > blkcg, causing massive latencies to processes that shouldn't be > > > I/O-throttled at all. > > > > > > The idea is to save a list of blkcg's that are waiting for writeback: > > > every time a sync() is executed the current blkcg is added to the list. > > > > > > Then, when I/O is throttled, if there's a blkcg waiting for writeback > > > different than the current blkcg, no throttling is applied (we can > > > probably refine this logic later, i.e., a better policy could be to > > > adjust the throttling I/O rate using the blkcg with the highest speed > > > from the list of waiters - priority inheritance, kinda). > > > > > > This topic has been discussed here: > > > https://lwn.net/ml/cgroups/20190118103127.325-1-righi.andrea@gmail.com/ > > > > > > But we didn't come up with any definitive solution. > > > > > > This patch is not a definitive solution either, but it's an attempt to > > > continue addressing this issue and handling the priority inversion > > > problem with sync() in a better way. > > > > > > Signed-off-by: Andrea Righi > > > > Talked with Tejun about this some and we agreed the following is probably the > > best way forward > > First of all thanks for the update! > > > > > 1) Track the submitter of the wb work to the writeback code. > > Are we going to track the cgroup that originated the dirty pages (or > maybe dirty inodes) or do you have any idea in particular? > The guy doing the sync(), so that way we can accomplish #3. But really this is an implementation detail, however you want to accomplish it is fine by me. > > 2) Sync() defaults to the root cg, and and it writes all the things as the root > > cg. > > OK. > > > 3) Add a flag to the cgroups that would make sync()'ers in that group only be > > allowed to write out things that belong to its group. > > So, IIUC, when this flag is enabled a cgroup that is doing sync() would > trigger the writeback of the pages that belong to that cgroup only and > it waits only for these pages to be sync-ed, right? In this case > writeback can still go at cgroup's speed. > > Instead when the flag is disabled, sync() would trigger writeback I/O > globally, as usual, and it goes at full speed (root cgroup's speed). > > Am I understanding correctly? > Yup that's exactly it. Thanks, Josef