Date:   Fri, 10 Dec 2021 17:56:56 -0800
From:   Martin KaFai Lau <kafai@fb.com>
To:     Pavel Begunkov <asml.silence@gmail.com>
CC:     <netdev@vger.kernel.org>, <bpf@vger.kernel.org>,
        Alexei Starovoitov <ast@kernel.org>,
        Daniel Borkmann <daniel@iogearbox.net>,
        Andrii Nakryiko <andrii@kernel.org>,
        Song Liu <songliubraving@fb.com>,
        <linux-kernel@vger.kernel.org>
Subject: Re: [BPF PATCH for-next] cgroup/bpf: fast path for not loaded skb
 BPF filtering
Message-ID: <20211211015656.tvufcnh5k4rrc7sw@kafai-mbp.dhcp.thefacebook.com>
References: <d77b08bf757a8ea8dab3a495885c7de6ff6678da.1639102791.git.asml.silence@gmail.com>
 <20211211003838.7u4lcqghcq2gqvho@kafai-mbp.dhcp.thefacebook.com>
 <5f7d2f60-b833-04e5-7710-fdd2ef3b6f67@gmail.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <5f7d2f60-b833-04e5-7710-fdd2ef3b6f67@gmail.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?AwJui23Y4+c+ALkwMOwsrz+iH5B73RcruPXRGz7qB6XRxcCE4PDoKPgYg9JF?=
 =?us-ascii?Q?b5s4stbxbKRILKPWqIVgYYTiM5qGobFcy/mGFF4b887Dn7pyW84AEmtDTTTc?=
 =?us-ascii?Q?2TtL2vy4F69pDpC7lZ45l/VVW28xQ+HOGfIJTv1nFMH6wWNjkt60ZgtqgtIx?=
 =?us-ascii?Q?Uu6GSlaLW62gRcja+hwMUDmxeYEKPH+GhMFiwjxC/96zMPPUJLJi2jrHS3+M?=
 =?us-ascii?Q?ee/UPGPwZqtMwfWos1IfobRaovMYbIXmSvk/hhPjbhtkB6K9xWaPppbBO/rV?=
 =?us-ascii?Q?2BKI2pvMJ8eqkdEWYKKJ0VbzJQbbR0VvzHrQ3R2EnBmgqDnbICdkuZCYw8KX?=
 =?us-ascii?Q?SFjS2GqDUmM8f4e1tiNNYkkm3hDwmuZKo1+5dl3bGH22UwWJC7fvOr3EEa0m?=
 =?us-ascii?Q?ZZWxNwvZ2vqbUmd9K22yDhkqVHv5mt4+MRGLv7eU2F8AMgaGZuzeUZq4ePvR?=
 =?us-ascii?Q?d+lnSypoziT9+bXzMiaBAHLuqounB575rSapsVsaTdeo6Twl9Rd8TtiAs+WI?=
 =?us-ascii?Q?DKgXlz0isTh00a+6LzBKaDuXyfK7Ypq0wF7Mvxf2vtAwxohFfwwo5IwMY1pV?=
 =?us-ascii?Q?HBCSzdG1/MaYpNLoHQy8bYEfUI10Orp7peKU3oV4cJLEovK6UpIC16gG0j29?=
 =?us-ascii?Q?ch2lFk5U5OkiCXWn+ADq5IRO9LCDvpYKiTzDc6XWr7I23EyJLI0lCuIa0XAr?=
 =?us-ascii?Q?U6DnBREaBWUy0R7QgV3VXaejmns0AHVsyaZpoSeBz7U0kZFPVWXziUjWd45A?=
 =?us-ascii?Q?dH7ZOs8WyAYeQXmGagw+P524UyZL0pjAcMkHE/fiiwLyLrrq+ozFmWIUaWvu?=
 =?us-ascii?Q?CWfJRTzUdi+auWQghqB059vVLT1VpLz40c2huOxKYy3QE8t3oR/Y2ISMWAjI?=
 =?us-ascii?Q?DIhVIoTExG1XFkkagnwLSbmi+TZcIUeWR5oS74FakwigBb7GQkwEHNK6EjTg?=
 =?us-ascii?Q?RZFup/vA6+pUtQ7vcbZStBcsNyE/XfLHq6TrZn8Ni+j+Wv9OQNjHaBAPg1Mu?=
 =?us-ascii?Q?N4NXUGADLcLV4oqCOGzvHaWh4ji4wcrwWkWspHEaOSTewF8U0B3FZbvLGUup?=
 =?us-ascii?Q?+NmtCNBMa4+MX/cYJyRXWWrtx5zdV3eKG6TTvTS2nd2YIcnXd28zFyE7wTqb?=
 =?us-ascii?Q?ExKRgQuziM0Yk9jvkYhhH7AKg5uYwqlv8tQFsjVZE1OH0Sjx4IyD+oel1H/j?=
 =?us-ascii?Q?zCwGRb3YGms+P0sWMezLM8+keZX3oMGqkvQ3h/YnBLOWDT+N52pgSjjOCGDk?=
 =?us-ascii?Q?eNpIfKll2klpyi5Fuc45NtYNkUhgnmYqvswibQh8zeaxiNcTYNP4ty0h9thG?=
 =?us-ascii?Q?xoL+BmdGEDh17VFRzJsSyZuYkcHNQt1HYJyxEKHL5XnwItJVXFjus9l96bBT?=
 =?us-ascii?Q?e5bwSWbT7bWE/V4RmGK105uXHIWrL5lEawpcZrvwnFi9nqY4RTz/VN41jEVl?=
 =?us-ascii?Q?M+LrrEWrm2eL9h/DKU29qt6kvGJ7T6+/XpZFXjexY3qHdKuXLokOsft0hFzc?=
 =?us-ascii?Q?7D867IWKu7kMvoaqh5qZIv3rYbFQOUO6ECpXGa3Jjtnk3gVvz2JQ6/IaZsPk?=
 =?us-ascii?Q?47+RTTxYKhhKsd4MY9v6aF3YEmx2dMB3kT2Qz7nJ?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 04260dfc-ed0b-4fdd-c0ea-08d9bc4984fb
X-MS-Exchange-CrossTenant-AuthSource: SA1PR15MB5016.namprd15.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Dec 2021 01:57:00.2328
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 8ae927fe-1255-47a7-a2af-5f3a069daaa2
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Zp3TIU716/x4n8NsTluU8hA85VLvF2YxpT93awhO7OTdGelzz4tMBtNd6NCmMvGp
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR15MB5046
X-OriginatorOrg: fb.com
X-Proofpoint-ORIG-GUID: pkyl2SU2Oz3A6swOS0vBLZkg61Z01o4E
X-Proofpoint-GUID: pkyl2SU2Oz3A6swOS0vBLZkg61Z01o4E
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.205,Aquarius:18.0.790,Hydra:6.0.425,FMLib:17.11.62.513
 definitions=2021-12-10_09,2021-12-10_01,2021-12-02_01
X-Proofpoint-Spam-Details: rule=fb_outbound_notspam policy=fb_outbound score=0 clxscore=1015
 malwarescore=0 phishscore=0 priorityscore=1501 impostorscore=0
 suspectscore=0 spamscore=0 mlxscore=0 mlxlogscore=446 lowpriorityscore=0
 bulkscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2110150000 definitions=main-2112110009
X-FB-Internal: deliver
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Sat, Dec 11, 2021 at 01:15:05AM +0000, Pavel Begunkov wrote:
> On 12/11/21 00:38, Martin KaFai Lau wrote:
> > On Fri, Dec 10, 2021 at 02:23:34AM +0000, Pavel Begunkov wrote:
> > > cgroup_bpf_enabled_key static key guards from overhead in cases where
> > > no cgroup bpf program of a specific type is loaded in any cgroup. Turn
> > > out that's not always good enough, e.g. when there are many cgroups but
> > > ones that we're interesting in are without bpf. It's seen in server
> > > environments, but the problem seems to be even wider as apparently
> > > systemd loads some BPF affecting my laptop.
> > > 
> > > Profiles for small packet or zerocopy transmissions over fast network
> > > show __cgroup_bpf_run_filter_skb() taking 2-3%, 1% of which is from
> > > migrate_disable/enable(), and similarly on the receiving side. Also
> > > got +4-5% of t-put for local testing.
> > > 
> > > Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
> > > ---
> > >   include/linux/bpf-cgroup.h | 24 +++++++++++++++++++++---
> > >   kernel/bpf/cgroup.c        | 23 +++++++----------------
> > >   2 files changed, 28 insertions(+), 19 deletions(-)
> > > 
> > > diff --git a/include/linux/bpf-cgroup.h b/include/linux/bpf-cgroup.h
> > > index 11820a430d6c..99b01201d7db 100644
> > > --- a/include/linux/bpf-cgroup.h
> > > +++ b/include/linux/bpf-cgroup.h
> > > @@ -141,6 +141,9 @@ struct cgroup_bpf {
> > >   	struct list_head progs[MAX_CGROUP_BPF_ATTACH_TYPE];
> > >   	u32 flags[MAX_CGROUP_BPF_ATTACH_TYPE];
> > > +	/* for each type tracks whether effective prog array is not empty */
> > > +	unsigned long enabled_mask;
> > > +
> > >   	/* list of cgroup shared storages */
> > >   	struct list_head storages;
> > > @@ -219,11 +222,25 @@ int bpf_percpu_cgroup_storage_copy(struct bpf_map *map, void *key, void *value);
> > >   int bpf_percpu_cgroup_storage_update(struct bpf_map *map, void *key,
> > >   				     void *value, u64 flags);
> > > +static inline bool __cgroup_bpf_type_enabled(struct cgroup_bpf *cgrp_bpf,
> > > +					     enum cgroup_bpf_attach_type atype)
> > > +{
> > > +	return test_bit(atype, &cgrp_bpf->enabled_mask);
> > > +}
> > > +
> > > +#define CGROUP_BPF_TYPE_ENABLED(sk, atype)				       \
> > > +({									       \
> > > +	struct cgroup *__cgrp = sock_cgroup_ptr(&(sk)->sk_cgrp_data);	       \
> > > +									       \
> > > +	__cgroup_bpf_type_enabled(&__cgrp->bpf, (atype));		       \
> > > +})
> > I think it should directly test if the array is empty or not instead of
> > adding another bit.
> > 
> > Can the existing __cgroup_bpf_prog_array_is_empty(cgrp, ...) test be used instead?
> 
> That was the first idea, but it's still heavier than I'd wish. 0.3%-0.7%
> in profiles, something similar in reqs/s. rcu_read_lock/unlock() pair is
> cheap but anyway adds 2 barrier()s, and with bitmasks we can inline
> the check.
It sounds like there is opportunity to optimize
__cgroup_bpf_prog_array_is_empty().

How about using rcu_access_pointer(), testing with &empty_prog_array.hdr,
and then inline it?  The cgroup prog array cannot be all
dummy_bpf_prog.prog.  If that could be the case, it should be replaced
with &empty_prog_array.hdr earlier, so please check.