Received-SPF: pass (google.com: domain of linux-kernel+bounces-180233-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
Message-ID: <31cef5e2-cf6e-42df-a6d2-10ecd04e60d7@redhat.com>
Date: Wed, 15 May 2024 13:51:41 -0400
Precedence: bulk
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: cgroup io.stat propagation regression
To: Dan Schatzberg <schatzberg.dan@gmail.com>
Cc: Tejun Heo <tj@kernel.org>, linux-block@vger.kernel.org,
 cgroups@vger.kernel.org, linux-kernel@vger.kernel.org
References: <ZkO6l/ODzadSgdhC@dschatzberg-fedora-PF3DHTBV>
 <3ed32279-904a-411d-91a4-a62f4ca2dde2@redhat.com>
 <11b8c1e4-45a7-4895-a1f3-6626744cee1e@redhat.com>
 <ZkToxDuKxSPEg5aP@dschatzberg-fedora-PF3DHTBV>
Content-Language: en-US
From: Waiman Long <longman@redhat.com>
In-Reply-To: <ZkToxDuKxSPEg5aP@dschatzberg-fedora-PF3DHTBV>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

On 5/15/24 12:54, Dan Schatzberg wrote:
> On Wed, May 15, 2024 at 10:26:31AM -0400, Waiman Long wrote:
>> On 5/14/24 23:59, Waiman Long wrote:
>>> On 5/14/24 15:25, Dan Schatzberg wrote:
>>>> Hi Waiman,
>>>>
>>>> I've noticed that on recent kernels io.stat metrics don't propagate
>>>> all the way up the hierarchy. Specifically, io.stat metrics of some
>>>> leaf cgroup will be propagated to the parent, but not its grandparent.
>>>>
>>>> For a simple repro, run the following:
>>>>
>>>> systemd-run --slice test-test dd if=/dev/urandom of=/tmp/test
>>>> bs=4096 count=1
>>>>
>>>> Then:
>>>>
>>>> cat /sys/fs/cgroup/test.slice/test-test.slice/io.stat
>>>>
>>>> Shows the parent cgroup stats and I see wbytes=4096 but the
>>>> grandparent cgroup:
>>>>
>>>> cat /sys/fs/cgroup/test.slice/io.stat
>>>>
>>>> shows no writes.
>>>>
>>>> I believe this was caused by the change in "blk-cgroup: Optimize
>>>> blkcg_rstat_flush()". When blkcg_rstat_flush is called on the parent
>>>> cgroup, it exits early because the lockless list is empty since the
>>>> parent cgroup never issued writes itself (e.g. in
>>>> blk_cgroup_bio_start). However, in doing so it never propagated stats
>>>> to its parent.
>>>>
>>>> Can you confirm if my understanding of the logic here is correct and
>>>> advise on a fix?
>>> Yes, I believe your analysis is correct. Thanks for spotting this iostat
>>> propagation problem.
>>>
>>> I am working on a fix to address this problem and will post a patch once
>>> I have finished my testing.
>> Actually, I can only reproduce the issue with a 3-level
>> (child-parent-grandparent) cgroup hierarchy below the root cgroup. The dd
>> command is run test.slice/test-test.slice. So both test.slice/io.stat and
>> test.slice/test-test.slice/io.stat are properly updated.
> That's correct, this repros with a 3-level cgroup hierarchy (or
> more). systemd-run should create an ephemeral .scope cgroup under
> test-test.slice and then delete it when the dd command finishes. So
> test.slice/test-test.slice was the parent (2nd level) and test.slice
> is the grandparent.

OK, I didn't get the .scope sub-cgroup when I ran the above systemd-run 
command. Perhaps it's due to a difference in configuration. Anyway, I 
was able to reproduce the problem and devise the fix accordingly.

Thanks,
Longman