Received-SPF: pass (google.com: domain of linux-kernel+bounces-171953-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Date: Tue, 7 May 2024 14:06:10 -0400
From: "Liam R. Howlett" <Liam.Howlett@oracle.com>
To: Andrii Nakryiko <andrii.nakryiko@gmail.com>
Cc: Greg KH <gregkh@linuxfoundation.org>, Andrii Nakryiko <andrii@kernel.org>,
        linux-fsdevel@vger.kernel.org, brauner@kernel.org,
        viro@zeniv.linux.org.uk, akpm@linux-foundation.org,
        linux-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-mm@kvack.org,
        Suren Baghdasaryan <surenb@google.com>,
        Matthew Wilcox <willy@infradead.org>
Subject: Re: [PATCH 5/5] selftests/bpf: a simple benchmark tool for
 /proc/<pid>/maps APIs
Message-ID: <qa3ffj62mrdrskqg33atupnphc6il6ygdzbtknpky4xfhilqg2@mqojpw2vwbul>
Mail-Followup-To: "Liam R. Howlett" <Liam.Howlett@oracle.com>, 
	Andrii Nakryiko <andrii.nakryiko@gmail.com>, Greg KH <gregkh@linuxfoundation.org>, 
	Andrii Nakryiko <andrii@kernel.org>, linux-fsdevel@vger.kernel.org, brauner@kernel.org, 
	viro@zeniv.linux.org.uk, akpm@linux-foundation.org, linux-kernel@vger.kernel.org, 
	bpf@vger.kernel.org, linux-mm@kvack.org, Suren Baghdasaryan <surenb@google.com>, 
	Matthew Wilcox <willy@infradead.org>
References: <20240504003006.3303334-1-andrii@kernel.org>
 <20240504003006.3303334-6-andrii@kernel.org>
 <2024050425-setting-enhance-3bcd@gregkh>
 <CAEf4BzbiTQk6pLPQj=p9d18YW4fgn9k2V=zk6nUYAOK975J=xg@mail.gmail.com>
 <cgpi2vaxveiytrtywsd4qynxnm3qqur3xlmbzcqqgoap6oxcjv@wjxukapfjowc>
 <CAEf4BzZQexjTvROUMkNb2MMB2scmjJHNRunA-NqeNzfo-yYh9g@mail.gmail.com>
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
In-Reply-To: <CAEf4BzZQexjTvROUMkNb2MMB2scmjJHNRunA-NqeNzfo-yYh9g@mail.gmail.com>
User-Agent: NeoMutt/20231103
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?utf-8?B?RVZ0dkVCYWZkMnBRc3ZFTWdCdDYrUnZ6WEQxMHVOR1g1SEM3UnhBZzNUTVF0?=
 =?utf-8?B?OXVqMWVUU0drNXdTeSttT1RjUmhuOWp2azluUGRwalVnVjZkSDlpZUp0bmY2?=
 =?utf-8?B?WFd3eVZyd1Q4S21wRzFTd2VkaXV6VG50VGM2QmMycGpyeGxyYndqTmlYbkVr?=
 =?utf-8?B?Mk9ZVHROalpDVzlxT0FuYzduWnZOQ3hCSzB1S2NLemdNYjRGNEZMNXZuaTNp?=
 =?utf-8?B?S3UrQmhvd09PYm94QkhPMnY1QlRjYlVoM3FWM0JwZFpIb1k0SGdjNFg1dno3?=
 =?utf-8?B?STBpcTRxcHdHeGl0VFB2UkhiSXY0cktOTFV3TU1namVjdlNUMC8rNEtWMmhV?=
 =?utf-8?B?Ky9nSUZjRlMydTVkZHl5d3lNM2F2MlVhS3l2ZXV5OThRY1pQeDRuZkhBdjFp?=
 =?utf-8?B?Vkt3L3RweFhGdGRLQWZxN0FrakxCa2xIL01vTDRUZmJ3blBQbkRaa1ZDTWpM?=
 =?utf-8?B?TGkyVGxWWE1QWm5sYnB0UnovUEYvNlVBTmt5WTBrRG1DQzc2MTY3Wk1xMkhh?=
 =?utf-8?B?cDdGQzFrMUFPUis3bFdxdDVYS09BdVgyZUJsL21GSEM5eEdVa0RFTzZNNDZR?=
 =?utf-8?B?OHRsU1IydzMxbm5XVXFwMGNZTE5IM2JjMFNVcUZKaWh6NUlOVGovczJIYzZV?=
 =?utf-8?B?TlZRaTA5M3JWeUhndzZxN2NTbmdTMTdQYnFuVGJQazFOdlRHamM3SThSQ1g1?=
 =?utf-8?B?S1liRTk3SEVralNaUFZFRGwvZ3B4RTZzOHNvdEo5Yzd1RVREcy9yaVFkNkdS?=
 =?utf-8?B?WDQvZGJHQU9kWG90YVlleGQxTTRsTGhHWnJYbmd1M1JPZHlpU0JiSWZuVVIz?=
 =?utf-8?B?MUJYZG52aFR0T3VrakZMUVN2VTQrYXI3OHU2ckh0N3p1Wjk5OHBsN2c2WDhz?=
 =?utf-8?B?dE1aQ09TRnF4aG9QaXJFbTE0ZFpJVEdtMGMrLyt1QUl1UGZpamNUeEU5eit3?=
 =?utf-8?B?OVd3b2I5Yld6MktZWmhWdlBlQXlzK3F5UWVpNTNWUmFCak1kUTBnNGZCam0r?=
 =?utf-8?B?aWlaVkVNbmZseU1qaDNxTWVrQlFFd2JENjFKL3lKaEE3WThtMCt6QkRnMjV3?=
 =?utf-8?B?OUpaUHE4QXpNcWd1RlAxVXFtTXhBZmk3a3JIUFExMTB2WE5zd3hILzBBdWhp?=
 =?utf-8?B?Yy82Z201cEdXQS8wMU5LMUIyQmQxQkZuaU1zdlliQk9KRDlONCtwbitoRDhk?=
 =?utf-8?B?NHFHcGF1WGpGQlBwQmI4Y3puc2RkZVdNdUhVaDViZDBrdWswWmdwa0pUVVJy?=
 =?utf-8?B?NWVqTnBXNzhkOTl5Sm5pMVJKYzJsQm4wdUsvZ0R3aW04dnFKSkdpaHhGd2c1?=
 =?utf-8?B?cndZNXN4RWpBZ1g4VWRFOEgzTjVqNFZBUy84cDc3ekJuWmI4MDlJYVE4V25m?=
 =?utf-8?B?cTRSemtqUUEzanRHS0lLVlVjdExUaWlzRjd2U0FHT3EybU1QMXR0TElUMXJV?=
 =?utf-8?B?OTVCTURreWl3aTRBOXNWQVBtVzdrZ0lmd0NWdzNRWHlqZ0Y1dWl1TTVOWHVX?=
 =?utf-8?B?b0hBY09FdnVmVFRRQThwYklFSzlPUHFGNW1YeEtaSFByMmpEcHdJVXVpOXlK?=
 =?utf-8?B?cjV0U25CazQ0azFobk1LUjlrcHhDY0FZT0ZTOU03MFdhV0tlbHVsTm5CM3Q5?=
 =?utf-8?B?SzhTOUJpT0FpK2hRLytaNnd0UzJhWDBIRDN6UDFYUFFJME5Oa3YvSnQ2K2Vw?=
 =?utf-8?B?eXlISDNNeVVUS3Qrb0pScjdMdEp3QVIrTUwyTUd0Zm5sV0ZOM1IzM3BhVHdh?=
 =?utf-8?B?K3dPdmMvaG1uU25PdURSc2NXK2ZBRjJLK2lGcGVwQm5BR0RIL29lZ2RGZ2Zz?=
 =?utf-8?B?ODZxV1hGQUN3M1JCYlhrRUpQSGprWGhFa0V6NzBYcXFHeklDTU92b1VlK09I?=
 =?utf-8?B?LzUvZ1VKaEVQSUVXbVRITXhXd3N6dGdYTG9PTnk4OXhFK0JWNkNKK2lzTlZk?=
 =?utf-8?B?aGE4S2JMQSsvbXRQUFBQUTJFQ1RNckM2T1o4ZG1tZDJnUUUyMUpoazQyTTNF?=
 =?utf-8?B?YUJMN1Q5MWVDcStkUE01THM1cHlmNTdISC90bWpLOUwvMC9HVE8reGpVY3Jp?=
 =?utf-8?B?c1NrR3lvbjVPOVdSME5pYmJENjd2QWY1ZEdTWk1RWm1QMXNXY1RGMmZDY0VG?=
 =?utf-8?B?eTRjWm5CSnM5TUlGMXkyZGxGWVB3cEtCaGNIaGhWNW5WOTJaVzdiU2pIUVpL?=
 =?utf-8?B?RVE9PQ==?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	XQ/2h4+yPlDcybLGvsy6x2RQw7mNe8utbBlZmXNZJW5Uf3W68l+K9TUAAjYwRNGwPlqpXg4pImTRAlVt7XQlCeZoS4Ocx+j21aHQzJOo6aASjqy7OIZNxPiGi84Q2qfTkXQbr60PvCrEOtN0PB3GslrgJkvr5VvMeLwWuC60T/l/TizkbkLJh9wdSShOVcXZt1rlfR0beLVqzOOkdkHLSY3vdsV8bdY3uQ983WEuENGiEICjaPBLhqWhrpCcPmGPdPub9vp1f+diaCCJR4xq7PpLyNs/rclwN5a1sXJ6Wu57N4/tGdIEN1+0W9Debo6Lhil0wQnc3YixEshX8OkoEzg9onUNn47AXuaqP80nMnSf+8rvB0oSRx9eTzXwp9K35QPehTbsA0KESthWoRjquDjvxCidx3LXOsCjM2BdK67Ar70hDDfB7oK0zSO0e0hC/LJMIhSBFv7kwj7pJdlcw33k78zP8cvRtF91wsPi96gBN81CxVVhU0Px7Ky+P+bY7NV9oZw2bY9OHLA5F00Aozo8L5Y+Ect2z8f3+xFMvpQj0uUIDfjGZJVjsEA5UCKl0Yz5+NP22U5e9Fx1vQrGLXlx8DleLlOqseKX7P8hWMM=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 59b593db-c736-40ba-6162-08dc6ec06148
X-MS-Exchange-CrossTenant-AuthSource: DS0PR10MB7933.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 May 2024 18:06:12.5868
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 8beP/GMIhv4f9W9oC/ZTdRFGlut3juAzkRg2uFx1fLS8+c343Zu7JGthrgv4GW5d8YtzQfYySCnNtoFeuIqx1A==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB5630
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.650,FMLib:17.11.176.26
 definitions=2024-05-07_10,2024-05-06_02,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 mlxlogscore=999
 bulkscore=0 suspectscore=0 spamscore=0 malwarescore=0 mlxscore=0
 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2404010000 definitions=main-2405070126
X-Proofpoint-GUID: aLifSHoZIe8smT9ThtJrqRJKb_Y5kiIr
X-Proofpoint-ORIG-GUID: aLifSHoZIe8smT9ThtJrqRJKb_Y5kiIr

* Andrii Nakryiko <andrii.nakryiko@gmail.com> [240507 12:28]:
> On Tue, May 7, 2024 at 8:49=E2=80=AFAM Liam R. Howlett <Liam.Howlett@orac=
le.com> wrote:
> >
> > .. Adding Suren & Willy to the Cc
> >
> > * Andrii Nakryiko <andrii.nakryiko@gmail.com> [240504 18:14]:
> > > On Sat, May 4, 2024 at 8:32=E2=80=AFAM Greg KH <gregkh@linuxfoundatio=
n.org> wrote:
> > > >
> > > > On Fri, May 03, 2024 at 05:30:06PM -0700, Andrii Nakryiko wrote:
> > > > > I also did an strace run of both cases. In text-based one the too=
l did
> > > > > 68 read() syscalls, fetching up to 4KB of data in one go.
> > > >
> > > > Why not fetch more at once?
> > > >
> > >
> > > I didn't expect to be interrogated so much on the performance of the
> > > text parsing front, sorry. :) You can probably tune this, but where i=
s
> > > the reasonable limit? 64KB? 256KB? 1MB? See below for some more
> > > production numbers.
> >
> > The reason the file reads are limited to 4KB is because this file is
> > used for monitoring processes.  We have a significant number of
> > organisations polling this file so frequently that the mmap lock
> > contention becomes an issue. (reading a file is free, right?)  People
> > also tend to try to figure out why a process is slow by reading this
> > file - which amplifies the lock contention.
> >
> > What happens today is that the lock is yielded after 4KB to allow time
> > for mmap writes to happen.  This also means your data may be
> > inconsistent from one 4KB block to the next (the write may be around
> > this boundary).
> >
> > This new interface also takes the lock in do_procmap_query() and does
> > the 4kb blocks as well.  Extending this size means more time spent
> > blocking mmap writes, but a more consistent view of the world (less
> > "tearing" of the addresses).
>=20
> Hold on. There is no 4KB in the new ioctl-based API I'm adding. It
> does a single VMA look up (presumably O(logN) operation) using a
> single vma_iter_init(addr) + vma_next() call on vma_iterator.

Sorry, I read this:

+	if (usize > PAGE_SIZE)
+		return -E2BIG;

And thought you were going to return many vmas in that buffer.  I see
now that you are doing one copy at a time.

>=20
> As for the mmap_read_lock_killable() (is that what we are talking
> about?), I'm happy to use anything else available, please give me a
> pointer. But I suspect given how fast and small this new API is,
> mmap_read_lock_killable() in it is not comparable to holding it for
> producing /proc/<pid>/maps contents.

Yes, mmap_read_lock_killable() is the mmap lock (formally known as the
mmap sem).

You can see examples of avoiding the mmap lock by use of rcu in
mm/memory.c lock_vma_under_rcu() which is used in the fault path.
userfaultfd has an example as well. But again, remember that not all
archs have this functionality, so you'd need to fall back to full mmap
locking.

Certainly a single lookup and copy will be faster than a 4k buffer
filling copy, but you will be walking the tree O(n) times, where n is
the vma count.  This isn't as efficient as multiple lookups in a row as
we will re-walk from the top of the tree. You will also need to contend
with the fact that the chance of the vmas changing between calls is much
higher here too - if that's an issue. Neither of these issues go away
with use of the rcu locking instead of the mmap lock, but we can be
quite certain that we won't cause locking contention.

Thanks,
Liam