Received: by 2002:a05:7412:8d1c:b0:fa:4c10:6cad with SMTP id bj28csp199341rdb; Tue, 16 Jan 2024 21:55:59 -0800 (PST) X-Google-Smtp-Source: AGHT+IHaxyuFJTBUVU4c9fO7LbRhuHQgH2iou3FQUrHFAhRswejc6eU+8VCdd72JRV2MsNNZooy7 X-Received: by 2002:a05:620a:3891:b0:781:5aaf:3017 with SMTP id qp17-20020a05620a389100b007815aaf3017mr8222225qkn.142.1705470958691; Tue, 16 Jan 2024 21:55:58 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1705470958; cv=pass; d=google.com; s=arc-20160816; b=sbCR/w4N+x7uhvcLuEjR0QAd7u9g6r7sKlhJ2hyX4c60Zww/WBMfp2o/kOLKtRP5U+ t5K3qek38wAD09ZC51X1TEh8tEqTn521zTEnrTxBuMmLg5fWQYt62r1wO3mNNxYDXPXS o2UsxhSzmCDU40WEvhYIaKtNnanvDPKm9uWBzKxztCreDnkhGoux7ZIX8T7QhOUb7r5e ryncw3S4VU0y/iDFZHS051zeAJ/8nh+ugoyQwe6VEQQSjcTX/tDr7Sr46kpNZyYrX/Eq mj6l9K9xda2GIOTHIJ1VHEYpoRif5IZW8PJNK4xj+FSjBZpAGjuay0t2qlzI76QRVk6o fLuA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=in-reply-to:content-disposition:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:message-id:subject:cc :to:from:date:dkim-signature; bh=E6Z/eQkBMXHSee05+WcpGSIz4ByCJGYdInYrVxwiAVk=; fh=XjW7rVDrvFkiaqtS+lR8QDehwcuooLq8VA1GK/zsYoU=; b=ExzF/9ur1dXKjm1lQE4YxlL1S1EY73/2gJV8363j1advZHZmmTvVCu3kAnOZxJaN8H QLOZqPIiulrGbfiEF8/qPuFi0hZLXzKFG+knHuI1r/yGU6rfE5G1imjF+Dz7Y9RP3YeH Yi8ksrBn/plFYIeB15OmlDySv86eCfv7KttBCIpbm9rcMIDbAdTY6QnUfLfPLIJqKtyr 0f8Na0Lk5HCqH0p7N6phjBB/n27lHEvVoP59k9tnpxrzevR/MYI4zAXeCjr7CZc8mc1n lngc5hOk71tdf5Kfje8z5s+GiwJVRGhhlQUeg17uEaalofShP5J9+wZakJ7atJXd0ml7 CWmA== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@mit.edu header.s=outgoing header.b=lCTi0ygw; arc=pass (i=1 spf=pass spfdomain=mit.edu dkim=pass dkdomain=mit.edu dmarc=pass fromdomain=mit.edu); spf=pass (google.com: domain of linux-kernel+bounces-28549-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-28549-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=mit.edu Return-Path: Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id 15-20020a05620a048f00b0077f88d7a779si11467542qkr.426.2024.01.16.21.55.58 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Jan 2024 21:55:58 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-28549-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@mit.edu header.s=outgoing header.b=lCTi0ygw; arc=pass (i=1 spf=pass spfdomain=mit.edu dkim=pass dkdomain=mit.edu dmarc=pass fromdomain=mit.edu); spf=pass (google.com: domain of linux-kernel+bounces-28549-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-28549-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=mit.edu Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 653501C23BBC for ; Wed, 17 Jan 2024 05:55:58 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 93A068F44; Wed, 17 Jan 2024 05:55:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=mit.edu header.i=@mit.edu header.b="lCTi0ygw" Received: from outgoing.mit.edu (outgoing-auth-1.mit.edu [18.9.28.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1F3C779D8 for ; Wed, 17 Jan 2024 05:55:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=18.9.28.11 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1705470945; cv=none; b=H77Z+4mADAjx+Cc12+w7+Jcph6QIvc2IwwwsdzYInaip6tp8oazE8TxsEJIAiCzyYUDkB+RyKH3WNlMs66qBtSR0lsJPb4W2f9tw+URxJbLrAvEyS/0VXsuXnrI4XfAnx4sqqb08icJQ8HKdk6sWPbJEOeAQOB3+68MUzqU2jfA= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1705470945; c=relaxed/simple; bh=Cajldxp5L4q+bgoRIKnlBgXzSAKMtR7sfecEvOfpwAs=; h=Received:DKIM-Signature:Received:Date:From:To:Cc:Subject: Message-ID:References:MIME-Version:Content-Type: Content-Disposition:In-Reply-To; b=Fz9gP3OnA7t4awspExXIn5PzJ2FmmPm9BtEpihu50MzarIinTxhvl82CVK4dJ2Vwq1c2L0B7ZBZEMuHhHKjsv+lqXw/2mxYAkdXZGTZnjLFmPr/34ZiIQdOP7RVSTYrMj3crk+AyyRHhKfbJuUQPBkS7atZYxkl0Qxr0vvO6aqc= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=mit.edu; spf=pass smtp.mailfrom=mit.edu; dkim=pass (2048-bit key) header.d=mit.edu header.i=@mit.edu header.b=lCTi0ygw; arc=none smtp.client-ip=18.9.28.11 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=mit.edu Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=mit.edu Received: from cwcc.thunk.org (pool-173-48-112-211.bstnma.fios.verizon.net [173.48.112.211]) (authenticated bits=0) (User authenticated as tytso@ATHENA.MIT.EDU) by outgoing.mit.edu (8.14.7/8.12.4) with ESMTP id 40H5sv3E016205 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 17 Jan 2024 00:54:58 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mit.edu; s=outgoing; t=1705470901; bh=E6Z/eQkBMXHSee05+WcpGSIz4ByCJGYdInYrVxwiAVk=; h=Date:From:Subject:Message-ID:MIME-Version:Content-Type; b=lCTi0ygwe47zegAHZboHFyqL3Sd5496YzTJRXwjVrME86O0SGElAaUC1FFlOL73QF C2WX0qpUXggTBxcZrTHCnD+b7q4VV5CHVW5em+Tm3VCWP8B7y8bkMr2LrC/208YnUJ HByA2MwQztUOiFI+Y6tkvXLd5nFtoNgFTt87OoxpVZkV3UK36kgNaakybCEELm8L8i ebttMKfsy7qs+mqakIxV4DuOpAkRlnyigPymnoUORF2A14qORY0nwM8SDVW1yCjAoa OpJdDQZzJl+nkzuj16Lllw4DAnHaUvhCIXyBUBfIGiSbC7khdXDosihq9XwXXDBsMJ GNZLM844ywkxg== Received: by cwcc.thunk.org (Postfix, from userid 15806) id 89D5315C0278; Wed, 17 Jan 2024 00:54:57 -0500 (EST) Date: Wed, 17 Jan 2024 00:54:57 -0500 From: "Theodore Ts'o" To: Kent Overstreet Cc: Greg KH , Mark Brown , Neal Gompa , Kees Cook , Linus Torvalds , linux-bcachefs@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-hardening@vger.kernel.org, Nikolai Kondrashov , Philip Li , Luis Chamberlain Subject: Re: [GIT PULL] bcachefs updates for 6.8 Message-ID: <20240117055457.GL911245@mit.edu> References: <40bcbbe5-948e-4c92-8562-53e60fd9506d@sirena.org.uk> <2uh4sgj5mqqkuv7h7fjlpigwjurcxoo6mqxz7cjyzh4edvqdhv@h2y6ytnh37tj> <2024011532-mortician-region-8302@gregkh> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: On Tue, Jan 16, 2024 at 11:41:25PM -0500, Kent Overstreet wrote: > > > No, it's a leadership/mentorship thing. > > > > > > And this is something that's always been lacking in kernel culture. > > > Witness the kind of general grousing that goes on at maintainer summits; > > > maintainers complain about being overworked and people not stepping up > > > to help with the grungy responsibilities, while simultaneously we still > > > Tests and test infrastructure fall into the necessary but not fun > > > category, so they languish. > > > > No, they fall into the "no company wants to pay someone to do the work" > > category, so it doesn't get done. > > > > It's not a "leadership" issue, what is the "leadership" supposed to do > > here, refuse to take any new changes unless someone ponys up and does > > the infrastructure and testing work first? That's not going to fly, for > > valid reasons. Greg is absolutely right about this. > But good tools are important beacuse they affect the rate of everyday > development; they're a multiplier on the money everone is spending on > salaries. Alas, companies don't see it that way. They take the value that get from Linux for granted, and they only care about the multipler effect of their employees salaries (and sometimes not even that). They most certainly care about the salutary effects on the entire ecosyustem. At least, I haven't seen any company make funding decisions on that basis. It's easy enough for you to blame "leadership", but the problem is the leaders at the VP and SVP level who control the budgets, not the leadership of the maintainers, who are overworked, and who often invest in testing themselves, on their own personal time, because they don't get adequate support from others. It's also for that reason why we try to prove that people won't just stick around enough for their pet feature (or in the case of ntfs, their pet file system) gets into the kernel --- and then disappear. For too often, this is what happens, either because they have their itch scratched, or their company reassigns them to some other project that is important for their company's bottom-line. If that person is willing their own personal time, long after work hours, to steward their contribution in the absence of corporate support, great. But we need to have that proven to us, or at the very least, make sure the feature's long-term maintenace burden is as low possible, to mitigate the likelihood that we won't see the new engineer after their feature lands upstream. > Having one common way of running all our functional VM tests, and a > common collection of those tests would be a huge win for productivity > because _way_ too many developers are still using slow ad hoc testing > methods, and a good test runner (ktest) gets the edit/compile/test cycle > down to < 1 minute, with the same tests framework for local development > and automated testing in the big test cloud... I'm going to call bullshit on this assertion. The fact that we have multiple ways of running our tests is not the reason why testing takes a long time. If you are going to run stress tests, which is critical for testing real file systems, that's going to take at least an hour; more if you want to test muliple file system features. The full regression set for ext4, using the common fstests testt suite, takes about 25 hours of VM time; and about 2.5 hours of wall clock time since I shard it across a dozen VM's. Yes, w could try to add some unit tests which take much less time running tests where fstests is creating a file system, mounting it, exercising the code through userspace functions, and then unmounting the file system and then checking the file system. Even if that were an adequate replacement for some of the existing fstests, (a) it's not a replacement for stress testing, and (b) this would require a vast amount of file system specific software engineering investment, and where is that going from? The bottom line is that problem is that having a one common way of running our functional VM tests is not even *close* to root cause of the problem. - Ted