Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Subject: Re: [V2][PATCH] cgroup: fix memory leak caused by missing
 cgroup_bpf_offline
To:     Roman Gushchin <guro@fb.com>
Cc:     Tejun Heo <tj@kernel.org>, Zefan Li <lizefan.x@bytedance.com>,
        Johannes Weiner <hannes@cmpxchg.org>,
        Alexei Starovoitov <ast@kernel.org>,
        Daniel Borkmann <daniel@iogearbox.net>,
        Andrii Nakryiko <andrii@kernel.org>,
        Jens Axboe <axboe@kernel.dk>, Ming Lei <ming.lei@redhat.com>,
        mkoutny@suse.com, cgroups@vger.kernel.org,
        linux-kernel@vger.kernel.org, netdev@vger.kernel.org,
        bpf@vger.kernel.org
References: <20211018075623.26884-1-quanyang.wang@windriver.com>
 <YXIUMJWrXUcrvZf5@carbon.DHCP.thefacebook.com>
From:   Quanyang Wang <quanyang.wang@windriver.com>
Message-ID: <35e9e89f-d92f-06f9-b919-ef956d99d7df@windriver.com>
Date:   Fri, 22 Oct 2021 19:55:50 +0800
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.13.0
In-Reply-To: <YXIUMJWrXUcrvZf5@carbon.DHCP.thefacebook.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?NTROU3B1MjJuTmNTK0QySWNvZ0NLSElDSFJtVDlGTWZJOXJTQ25UaGZLSURU?=
 =?utf-8?B?YmlZc3Z1cDB4bW9Ma0FhWk9iejRxaEVDTmhRcWxTc0RnVVB6Z2dibjZlckJQ?=
 =?utf-8?B?U3BEck8waXp1MlBXc2QrZkFsZWdWNXdONWJiU3RPTlpuN2ZnWmJJSGl1cWp3?=
 =?utf-8?B?N1RQeGpBQ1ZLTVJER1E5SHEzQmhTenNtUUhrYlV0b1hGVFBiUzlNdUhMSzJ4?=
 =?utf-8?B?QW4rWDIvaEp6YzNTckJEYWtUQjY0NFNaQ0VGMDcramQvTmZ0RFRBSTN0b2JH?=
 =?utf-8?B?RXFtTkp6Qi9XOXNlL3RwdVpOaFFaYzVaTitBLzhvdW5WeUppYW9lVitpK3pR?=
 =?utf-8?B?aGhFYWVNUEpCZkpna3k5dGJWeTlaQkYrL3BJTHpRbDVPazdmK0NvMmN0UldG?=
 =?utf-8?B?cVpIbUgwVlYrK3AxUGxIMEllMzJtOGN0REVIY3dGVitIUmdSbEtvbVg3Ly9C?=
 =?utf-8?B?YlhDcW40cUYxZ2FLNStZclYxUnVvL2prMnNCTE9McE5XR01iNit1eUdraFQr?=
 =?utf-8?B?RWFOVk1QRjFKaFFwQi85cExBM3F5cTBDWTA3OEp5U08wbkZrSmZWRkdnS2x4?=
 =?utf-8?B?Rm5Id2U4bkJ2N0VUemlqY1pLY0FPUkY3Y1lFWXpyVXljUWhLTHRJOTlzczh2?=
 =?utf-8?B?b2ZpUnN6QVMyNTJhSld1Um1JY2drbnZ5NlZaQk1IS2NZSGhlYW1pRkxkaGov?=
 =?utf-8?B?cDNRQ2EzYlJMbThHMm02QURPUkRZS3hBMjl0NmgyeENSL3AvRFVIdytCOXd2?=
 =?utf-8?B?Ymlad1p4YU1qZlBITG1WU2swUys1Ykt4TFRPNkhUeUxTQWQ3bVhWejJDaVNn?=
 =?utf-8?B?NzVtRVd4U3BPWEhzemJZV1BIaGhMMUNLdGdseEwzcEY2R3JHTlF6Y0kwUlFR?=
 =?utf-8?B?SmF6ZWRpVC9HdmMxNzZVMjJFYW9vd0Y2Q3pQaVRGaCtKMUxQM1ZvWjMyVGV4?=
 =?utf-8?B?QWMxZEVyK0VhRjR3OWZOVG9sZ0FGM2xLckxicFgzTXdmR29KY3ZwSU8wcHN2?=
 =?utf-8?B?SjIrbU5qRTR3YkczKzFub0hJZGxnQkhWSlhuY01sTWNKZXhFMjVnWjJiNVh4?=
 =?utf-8?B?WDJQL09lYkJDenRKMmlzVjBTM1JuSkZUb2p5RTg5SlBCd3Njb25sTjhtTW01?=
 =?utf-8?B?c01CVXk5NGczbXJjSnVScmd3aTBNRFFWcFhqTWpwUUNPaUthY0RCcUxCZUM5?=
 =?utf-8?B?bFhuRldwN2tJa3BGNUVuWHlERFE0SzUxZ2ZZMmpmaFJRaEZCRkpiNC96QU91?=
 =?utf-8?B?R3FsSEUyald6aHdGa2JFeFVTRjlXTHJzcXVoQmVJd21GTGdCWTdTdGNJTzF1?=
 =?utf-8?B?dUtoODJBMUhBcitiamd4dmkza2VjVDlQbm0zcG52Q2I3TzFjeHRLQktpOWZW?=
 =?utf-8?B?aHB1cVh3L1kxQUdyNWxrV3h2RmJKaVpMMEQ2em9XejhDYXl2UlVXdUZjZk1J?=
 =?utf-8?B?QzZnNno0U1lzZTEwNm0rQ0FiVDE5NHBCcU00bmpSQ21XRVNnNEFOeno3NzVP?=
 =?utf-8?B?Y3YwWlpJWWhYN05KZDR0c2oyUmloU2ZrVFVlNDAwYUw5WGVNQUYwM1ZBZkti?=
 =?utf-8?B?cEtLSVY5OEJzRndha2ZOalp4eG4xNjdZeVAzeUFueVZ6T3Y0V0VrU21rSmRD?=
 =?utf-8?B?dU5KSzI4SGxURmRzeWhhT20ycGRYQ3pXUDJva1c4M3hzanhLRHhKMmVleWNY?=
 =?utf-8?B?aHptUjIrTnVFTENFenpyN3dXNlp6Q3NUckRNTk9BRGJZQThhcVVHY3dJbXVF?=
 =?utf-8?B?dENpTC9qbU9YcEw5M0gwemtmaGxJMjZyc25vS3NXQ1NDM0I1dGFOcVV5cTN6?=
 =?utf-8?B?TENxNWZKdk9vcWpQL24vU1AyR3I4Tm1QZEY0R0FiTTRPU3JHVndhTU1lQlQ3?=
 =?utf-8?B?VlpwWGo3SlM2Y2FsQzNTVjMxNitMY1k1TTlDdmttOVpoQTV2d3c5L1BTQms2?=
 =?utf-8?Q?E/XuHbtfQPVrYxwGakjsgu4yufL+hYqB?=
X-OriginatorOrg: windriver.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 5b27e513-685e-44f4-8ea8-08d99552ea02
X-MS-Exchange-CrossTenant-AuthSource: PH7PR11MB5819.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Oct 2021 11:55:59.6683
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 8ddb2873-a1ad-4a18-ae4e-4644631433be
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: quanyang.wang@windriver.com
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR11MB5079
X-Proofpoint-ORIG-GUID: tcCL2iwepCNZnbn30kZrOukyDroWkb3H
X-Proofpoint-GUID: tcCL2iwepCNZnbn30kZrOukyDroWkb3H
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.182.1,Aquarius:18.0.790,Hydra:6.0.425,FMLib:17.0.607.475
 definitions=2021-10-22_03,2021-10-21_02,2020-04-07_01
X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 impostorscore=0
 malwarescore=0 lowpriorityscore=0 priorityscore=1501 bulkscore=0
 suspectscore=0 mlxlogscore=999 adultscore=0 clxscore=1015 mlxscore=0
 spamscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2109230001 definitions=main-2110220067
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Roman,

On 10/22/21 9:30 AM, Roman Gushchin wrote:
> On Mon, Oct 18, 2021 at 03:56:23PM +0800, quanyang.wang@windriver.com wrote:
>> From: Quanyang Wang <quanyang.wang@windriver.com>
>>
>> When enabling CONFIG_CGROUP_BPF, kmemleak can be observed by running
>> the command as below:
>>
>>      $mount -t cgroup -o none,name=foo cgroup cgroup/
>>      $umount cgroup/
>>
>> unreferenced object 0xc3585c40 (size 64):
>>    comm "mount", pid 425, jiffies 4294959825 (age 31.990s)
>>    hex dump (first 32 bytes):
>>      01 00 00 80 84 8c 28 c0 00 00 00 00 00 00 00 00  ......(.........
>>      00 00 00 00 00 00 00 00 6c 43 a0 c3 00 00 00 00  ........lC......
>>    backtrace:
>>      [<e95a2f9e>] cgroup_bpf_inherit+0x44/0x24c
>>      [<1f03679c>] cgroup_setup_root+0x174/0x37c
>>      [<ed4b0ac5>] cgroup1_get_tree+0x2c0/0x4a0
>>      [<f85b12fd>] vfs_get_tree+0x24/0x108
>>      [<f55aec5c>] path_mount+0x384/0x988
>>      [<e2d5e9cd>] do_mount+0x64/0x9c
>>      [<208c9cfe>] sys_mount+0xfc/0x1f4
>>      [<06dd06e0>] ret_fast_syscall+0x0/0x48
>>      [<a8308cb3>] 0xbeb4daa8
>>
>> This is because that since the commit 2b0d3d3e4fcf ("percpu_ref: reduce
>> memory footprint of percpu_ref in fast path") root_cgrp->bpf.refcnt.data
>> is allocated by the function percpu_ref_init in cgroup_bpf_inherit which
>> is called by cgroup_setup_root when mounting, but not freed along with
>> root_cgrp when umounting. Adding cgroup_bpf_offline which calls
>> percpu_ref_kill to cgroup_kill_sb can free root_cgrp->bpf.refcnt.data in
>> umount path.
>>
>> This patch also fixes the commit 4bfc0bb2c60e ("bpf: decouple the lifetime
>> of cgroup_bpf from cgroup itself"). A cgroup_bpf_offline is needed to do a
>> cleanup that frees the resources which are allocated by cgroup_bpf_inherit
>> in cgroup_setup_root.
>>
>> And inside cgroup_bpf_offline, cgroup_get() is at the beginning and
>> cgroup_put is at the end of cgroup_bpf_release which is called by
>> cgroup_bpf_offline. So cgroup_bpf_offline can keep the balance of
>> cgroup's refcount.
>>
>> Fixes: 2b0d3d3e4fcf ("percpu_ref: reduce memory footprint of percpu_ref in fast path")
>> Fixes: 4bfc0bb2c60e ("bpf: decouple the lifetime of cgroup_bpf from cgroup itself")
>> Signed-off-by: Quanyang Wang <quanyang.wang@windriver.com>
>> ---
>> V1 ---> V2:
>> 1. As per Daniel's suggestion, add description to commit msg about the
>> balance of cgroup's refcount in cgroup_bpf_offline.
>> 2. As per Michal's suggestion, add tag "Fixes: 4bfc0bb2c60e" and add
>> description about it.
>> 3. Fix indentation on the percpu_ref_is_dying line.
> 
> Acked-by: Roman Gushchin <guro@fb.com>
> 
> The fix looks correct, two fixes tag are fine too, if only it won't
> confuse scripts picking up patches for stable backports.
> 
> In fact, it's a very cold path, which is arguably never hit in the real
> life. On cgroup v2 it's not an issue. I'm not sure we need a stable
> backport at all, only if it creates a noise for some automation tests.
> 
> Quanyang, out of curiosity, how did you find it?
I ran ltp testsuite to find this.

./runltp -f controllers -s cgroup

Thanks,
Quanyang
> 
> Anyway, thanks for catching and fixing it!
> 
> Roman
>