Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <3f3ffe0e-d2ac-c868-a1bf-cdf1b58fd666@fb.com>
Date:   Thu, 21 Jul 2022 09:15:07 -0700
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0)
 Gecko/20100101 Thunderbird/91.11.0
Subject: Re: [PATCH bpf-next v3 4/8] bpf: Introduce cgroup iter
Content-Language: en-US
To:     Hao Luo <haoluo@google.com>
Cc:     Yosry Ahmed <yosryahmed@google.com>,
        Alexei Starovoitov <ast@kernel.org>,
        Daniel Borkmann <daniel@iogearbox.net>,
        Andrii Nakryiko <andrii@kernel.org>,
        Martin KaFai Lau <kafai@fb.com>,
        Song Liu <songliubraving@fb.com>, Tejun Heo <tj@kernel.org>,
        Zefan Li <lizefan.x@bytedance.com>,
        Johannes Weiner <hannes@cmpxchg.org>,
        Shuah Khan <shuah@kernel.org>,
        Michal Hocko <mhocko@kernel.org>,
        KP Singh <kpsingh@kernel.org>,
        Benjamin Tissoires <benjamin.tissoires@redhat.com>,
        John Fastabend <john.fastabend@gmail.com>,
        =?UTF-8?Q?Michal_Koutn=c3=bd?= <mkoutny@suse.com>,
        Roman Gushchin <roman.gushchin@linux.dev>,
        David Rientjes <rientjes@google.com>,
        Stanislav Fomichev <sdf@google.com>,
        Greg Thelen <gthelen@google.com>,
        Shakeel Butt <shakeelb@google.com>,
        linux-kernel@vger.kernel.org, netdev@vger.kernel.org,
        bpf@vger.kernel.org, cgroups@vger.kernel.org
References: <20220709000439.243271-1-yosryahmed@google.com>
 <20220709000439.243271-5-yosryahmed@google.com>
 <370cb480-a427-4d93-37d9-3c6acd73b967@fb.com>
 <a6d048b8-d017-ea7e-36f0-1c4f88fc4399@fb.com>
 <CA+khW7gmVmXMg4YP4fxTtgqNyAr4mQqnXbP=z0nUeQ8=hfGC3g@mail.gmail.com>
 <2a26b45d-6fab-b2a2-786e-5cb4572219ea@fb.com>
 <CA+khW7jp+0AadVagqCcV8ELNRphP47vJ6=jGyuMJGnTtYynF+Q@mail.gmail.com>
From:   Yonghong Song <yhs@fb.com>
In-Reply-To: <CA+khW7jp+0AadVagqCcV8ELNRphP47vJ6=jGyuMJGnTtYynF+Q@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?aEtSTEFZZ0ZZeXBEME9kYzFnYk53N1dUVmphaTBCOEc3dGhQTG9TT09uRVdI?=
 =?utf-8?B?ZUk4MnZyVzZsZjRJWW0yV0lyQTVmWHIrdnpvNCtzeWovMW5Wd0g2TEg0S3Ey?=
 =?utf-8?B?M1U2OHJqR1pCYkJNWnVma0hNSHAzd0RpZ2JwcXVOWm85QTlUYVQxUUtSZHBr?=
 =?utf-8?B?YzRVbXVqRnhIV01PaWNvM0RkeWpzTklPWkRKYTdsZnlJQWlmYUhqVjgwaWlX?=
 =?utf-8?B?V0dUM2srczg4K2hEaE5Wc09kRVV6bmllYXM3K29lNTR0MzRTVVFxWHlMb3M2?=
 =?utf-8?B?cVM1VkxES2UzNHFtTjcwcytpR2xZSjNBQXNaZDN1WjMrMXRYQTVvR081c0pn?=
 =?utf-8?B?dUZSdm5aY3NJK0RmUlVpVjBOUEdYVzE1RW92OUFsNXRETzhyamJNNFRXWTBi?=
 =?utf-8?B?TTlKZEJoNFhsc3E1Vmh2VVFNTFpsdktyVkJxQmN1QzdTVm04THdwbzB6Mk5h?=
 =?utf-8?B?SFB4OEFqWHNJTFJDMGwrWHI4UnN1MVFkdmVBTWM2UW5JQ1JYVHJxNndEaVlD?=
 =?utf-8?B?VnhMM3dla25OalhvNm9rN1RtNkZkcUZybGk1ZzUvMXM2UmljcFNPT2ROTlVL?=
 =?utf-8?B?eUtiUGNtK0dyb2lhWk9rOFVBQUhtMC9Zc0NmcjFJWHRoZVNuMm1oMFhlcXZh?=
 =?utf-8?B?NWM4MFNrcHdGL1ZjS2t4b21nSXU2blJmYTRiQUhEb1p3aGxFckRUVUwxT1dx?=
 =?utf-8?B?V2J5ZUloTTZZMU9HeWk1c2ptNE1LdjZXRkhkclF2VWpKckZ5cC96NWRhYjFw?=
 =?utf-8?B?M2RiUWhiWVZvZENoSW9vb3pTWjY2VmZHakFJRHhKZXEvYnVER0lEQ0dFVnlR?=
 =?utf-8?B?ZmphUERycTlrYVB4eXdpNGgrUEVSRTRVblgwOVF6Mk1GRXVEM25jNE1EL0VR?=
 =?utf-8?B?REluUWNKR2dxMEs4dGpBTGFDVXJ1YXBkeXRneityVTN6WW9TM3dYdVJvYldv?=
 =?utf-8?B?NlA2NWdqL09pWEZ3U3pzVlNDZVpRSFpFMHVhejdReCtRNWlXYzcydkx3cHM5?=
 =?utf-8?B?MnhWVXBUZUxQOTdLcG5vREFQd3VINnNxYTQ5Q0xvR3J5MjFtcFcxNlpXbU1p?=
 =?utf-8?B?VW9ucFBHZ0Rpcm1saGw5YXV6cUF1ZncrK1hqNjFvUERadHRnM1RUU1o2WUFJ?=
 =?utf-8?B?WGkwWFNnc25DNXpBRTcySWhLaytJMGxINWtsRTFKREtoRzdxUldDeDFiR1l4?=
 =?utf-8?B?QjFlMGJaanIzZjV0UW9xMk9JZGhmVEVqd0c0UExzS3pNVmVhTFFKdWU2REt0?=
 =?utf-8?B?Z3J5dEM1emJNUU5aZ0Nwby9LdE9qQStLelRzYnJzK1hTSTRRRTc0dXJwZFB0?=
 =?utf-8?B?QmdJL1UwR0xJQm4wdTlDcUtQZ2pvTXRmWmdaZ2xHcmFYTTVndzZCdFVaNlc2?=
 =?utf-8?B?R0tLNUhFbzU4bVpKUlpyRWxIY3NhRnBMYXc3RmYrd2d6eTZQeVM0M2JieGt4?=
 =?utf-8?B?ZlNEUkNWSUQybklSNi90Z3pVTVhkL3VUQzBVcGhFY1lZY1grTkw0dUVhRnFF?=
 =?utf-8?B?Zmc2ZzRqK3NobHRjM1Q1OFNWaWo4UEZhKzNCU2xPYVRmYmxYVnoxTXowbTZD?=
 =?utf-8?B?RmxiMHNBSHpKK0VBcUV6bTRXRjAxWSs1aHZ1a05PdUttTU96dFU0ZDhOR1or?=
 =?utf-8?B?c3RBOEc4Q1N6VjdsZEhVWXN2dlRqZUpiOWY0TWx0ZFc2ZEVkQklhdXY4SGFX?=
 =?utf-8?B?NkFwcGVyK25teXNLU1hVN2Zjc1RuVGZmcHUyZ3c2M0RxVVcxTXBDaFVEZEwz?=
 =?utf-8?B?S0pVV2UyR0xpc1hXazVrdFhGSEl2UVVFN01ZZ2NxVHA0N2hnZmZtYTBic2pN?=
 =?utf-8?B?RFZ3Y29lYWZYOUFNcWc3c2RzZWlPSGNkWGhoZTIvKzgvcFg5YlFkT2Y1OGpm?=
 =?utf-8?B?QnhtWlZYNjVyU2xTL1RHb0NZd015dlFjaU1HM2J0bTdNUVlYUDAvU2cyM0Y5?=
 =?utf-8?B?a0NVWTRYWXp0bklsTkMrZ0xCRzR6aE9lVERTVWk3QkJxMVVPSENYc2Z1V1lH?=
 =?utf-8?B?c040c2s5QzlURXMwbTJNMkdzMXFvc0tkRFREVVFTa0NmMkdOK2N2ODZqeHpU?=
 =?utf-8?B?eUZTTFZyc3ozbW00bHM0ZExkdTF1ZTVjZ1JXaEZ1U3M3Tm0ySERVdjB3T3Jh?=
 =?utf-8?B?QWhlS0MzdXZsS05sMFpyOXQrRzlBdUhMVG14YXNsZENJQ29WWFBYdERaY1Ux?=
 =?utf-8?B?bXc9PQ==?=
X-OriginatorOrg: fb.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 9ff52a17-8544-4fc1-8403-08da6b3430e5
X-MS-Exchange-CrossTenant-AuthSource: SN6PR1501MB2064.namprd15.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Jul 2022 16:15:13.1978
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 8ae927fe-1255-47a7-a2af-5f3a069daaa2
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 1XE0xIRprjcgbDHDVyc/YRBd1/g7FtBfov0/lMf9HFSFmlZJr6Rmfgvj61WKqKpl
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR15MB3021
X-Proofpoint-GUID: r5rLMkKdlMpBTkQyv1DQzB9nLwV7_RUR
X-Proofpoint-ORIG-GUID: r5rLMkKdlMpBTkQyv1DQzB9nLwV7_RUR
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.205,Aquarius:18.0.883,Hydra:6.0.517,FMLib:17.11.122.1
 definitions=2022-07-21_22,2022-07-20_01,2022-06-22_01
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,
        SPF_NONE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 7/20/22 5:40 PM, Hao Luo wrote:
> On Mon, Jul 11, 2022 at 8:45 PM Yonghong Song <yhs@fb.com> wrote:
>>
>> On 7/11/22 5:42 PM, Hao Luo wrote:
> [...]
>>>>>> +
>>>>>> +static void *cgroup_iter_seq_start(struct seq_file *seq, loff_t *pos)
>>>>>> +{
>>>>>> +    struct cgroup_iter_priv *p = seq->private;
>>>>>> +
>>>>>> +    mutex_lock(&cgroup_mutex);
>>>>>> +
>>>>>> +    /* support only one session */
>>>>>> +    if (*pos > 0)
>>>>>> +        return NULL;
>>>>>
>>>>> This might be okay. But want to check what is
>>>>> the practical upper limit for cgroups in a system
>>>>> and whether we may miss some cgroups. If this
>>>>> happens, it will be a surprise to the user.
>>>>>
>>>
>>> Ok. What's the max number of items supported in a single session?
>>
>> The max number of items (cgroups) in a single session is determined
>> by kernel_buffer_size which equals to 8 * PAGE_SIZE. So it really
>> depends on how much data bpf program intends to send to user space.
>> If each bpf program run intends to send 64B to user space, e.g., for
>> cpu, memory, cpu pressure, mem pressure, io pressure, read rate, write
>> rate, read/write rate. Then each session can support 512 cgroups.
>>
> 
> Hi Yonghong,
> 
> Sorry about the late reply. It's possible that the number of cgroup
> can be large, 1000+, in our production environment. But that may not
> be common. Would it be good to leave handling large number of cgroups
> as follow up for this patch? If it turns out to be a problem, to
> alleviate it, we could:
> 
> 1. tell users to write program to skip a certain uninteresting cgroups.
> 2. support requesting large kernel_buffer_size for bpf_iter, maybe as
> a new bpf_iter flag.

Currently if we intend to support multiple read() for cgroup_iter,
the following is a very inefficient approach:

in seq_file private data structure, remember the last cgroup visited
and for the second read() syscall, do the traversal again (but not 
calling bpf program) until the last cgroup and proceed from there.
This is inefficient and probably works. But if the last cgroup is
gone from the hierarchy, that the above approach won't work. One
possibility is to rememobe the last two cgroups. If the last cgroup
is gone, check the 'next' cgroup based on the one before the last
cgroup. If both are gone, we return NULL.

But in any case, if there are additional cgroups not visited,
in the second read(), we should not return NULL which indicates
done with all cgroups. We may return EOPNOTSUPP to indicate there
are missing cgroups due to not supported.

Once users see EOPNOTSUPP which indicates there are missing
cgroups, they can do more filtering in bpf program to avoid
large data volume to user space.

To provide a way to truely visit *all* cgroups,
we can either use bpf_iter link_create->flags
to increase the buffer size as your suggested in the above so
user can try to allocate more kernel buffer size. Or implement
proper second read() traversal which I don't have a good idea
how to do it efficiently.
> 
> Hao
> 
>>>
> [...]
>>>>> [...]