Received: by 2002:a25:4158:0:0:0:0:0 with SMTP id o85csp5383676yba; Wed, 8 May 2019 12:23:16 -0700 (PDT) X-Google-Smtp-Source: APXvYqxOeu+Xij51fn+POEuwUkkqWpKNDGLgO4JdZ2VnmZx8N//XADQJElZEDiVs2FF60lgAbcHO X-Received: by 2002:a17:902:263:: with SMTP id 90mr50549629plc.257.1557343396457; Wed, 08 May 2019 12:23:16 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1557343396; cv=none; d=google.com; s=arc-20160816; b=M8J6dyQGIWn+Tfy4gWQTwybpLoBe/GGH/7T2ubrY3emmq7jC1Osi5fpB3rtG+d7Q/0 ZeMOz0Q2oGegKFGnq3G2UoFPMRNcDUD2Sr1S9G/rQmU/QUGcYQ3SPF8HQEbEbV42yTKP 2Y4dkzc5X1/WKbg8EQ2xLQRee4PEaBsiz+5YCxZnokZOaPlDTHrMJusepmGpfooeN30R riQ4mwd59AECr4yDbZdhZ0iRMvmB8Po0bThyYY4WDEnjq2oOCNTjVSOxq2i0fQXZ6hxb 5nOdhqJwyhTz+LutualQ/7iHHHNECnoXoIX60JFU2ZIFIKmlHHs/Y3liZFwQFEMVUIi0 9VUg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:to:subject:message-id:date:from :in-reply-to:references:mime-version:dkim-signature; bh=x37pcn3M6vPyg8cwbr5kr7DA8TIGTpCINTXyPVgD+Ik=; b=Y48mJR4XXGRrxtukUx7DuGqB15kCH79/sA1VpU0znQmCbYosWBZXydTnCaRNSsXw0n kzguDUqhz/RFMFISSUS2QZrRXagSUzI4zG/ZEQLkHgQWH6nGgEEP5qX1ff5UmhkCbO0V UXRvYeRp8ZKN1w+maZFymarD/Cwibz0KJ61SVGIVT03zNuxdFQVzO26/fvhsCk/ZsRWq yiSzbvJa+lduPMBRcniF8fpzxLK+m4uhMVwYoYzLgcRXoZkCQ/nM3Mif1RtKNi2uvEJF EQwV1abz+m23ZPCLXHP731TtmMZsUff2GMywhTth/lmXzIBBStjbnxnlTht5V7A2/X+M n69w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=Tx5sqWDX; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id t13si22984485pgh.548.2019.05.08.12.23.00; Wed, 08 May 2019 12:23:16 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=Tx5sqWDX; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728041AbfEHTSN (ORCPT + 99 others); Wed, 8 May 2019 15:18:13 -0400 Received: from mail-ot1-f65.google.com ([209.85.210.65]:35893 "EHLO mail-ot1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727970AbfEHTSM (ORCPT ); Wed, 8 May 2019 15:18:12 -0400 Received: by mail-ot1-f65.google.com with SMTP id c3so6050otr.3 for ; Wed, 08 May 2019 12:18:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=x37pcn3M6vPyg8cwbr5kr7DA8TIGTpCINTXyPVgD+Ik=; b=Tx5sqWDXdN8o4nGLji4/tassr/7PVDxGlOko7CN/z8i4N9ZMaHczzcrn/j+3bWaIHg iOhpr/qiQfk83FIrSpg8oOQ/3ZDOOWZK1iGRN1Ddaj0rjVUqqAdg+akK6Az9iUT/HWCE +RbgLGBoM1KXmGzzFIdkSL/IP4jH5qxrIacDRusA9GI4pmte2X+oIf0E5qiPUr2cjsje HB/zl/SGOwDMcNb4wUEnNJNezPu8EGRGEAmGb2cN0dQFAqmAPXLAujkKEFLsuw97W54q q1Z4px77M6zMr0w7jrHy+UAPGRirtQr8A0ngAaBrMdPG2Wp4nQeveIDR0gQr0RiXfH01 tN9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=x37pcn3M6vPyg8cwbr5kr7DA8TIGTpCINTXyPVgD+Ik=; b=HDhzVTm50w2NxD6SFQvfFaFZlPl9cI9Defb88BHb2Nyj1nmXm4bR4/jLxnbTGOIWGV x8yf2V4yDu+f49XhBNjhDZPHLVQcqxPJCmP3L3C9gwLyIWg6y/G3K/fJyGFNUIKDJKFk xAFVoActgomMc8Af2ks8+/Iid6JPWKuBdq1pLJv3H/9ApKBQ/nEG2EAeZpI+A6cpH21y ppudT9fTSdOthmrKPOaPRJkzN6ol/shdR9iami9QKqbtK5eE478pPbOnFR5mtgCGwI23 3lELe2OSSA6DYpwosbrvF5/jtM0KvdFJlZH7kJWket5HBBIAIK4kJftKBhRVDUIfmtSy 45Sg== X-Gm-Message-State: APjAAAXiChDTH4YyYEN1ZtI1IEZdy07HW/aSf71Il23XG3aPukCU3OxF /x7A8UozrnIdS89PiH510rSnKoyl4OoWeGkeuIwH4Q== X-Received: by 2002:a05:6830:14cd:: with SMTP id t13mr11912335otq.25.1557343091371; Wed, 08 May 2019 12:18:11 -0700 (PDT) MIME-Version: 1.0 References: <20190501230126.229218-1-brendanhiggins@google.com> <54940124-50df-16ec-1a32-ad794ee05da7@gmail.com> <20190507080119.GB28121@kroah.com> <20190507172256.GB5900@mit.edu> In-Reply-To: <20190507172256.GB5900@mit.edu> From: Brendan Higgins Date: Wed, 8 May 2019 12:17:59 -0700 Message-ID: Subject: Re: [PATCH v2 00/17] kunit: introduce KUnit, the Linux kernel unit testing framework To: "Theodore Ts'o" , Greg KH , Frank Rowand , Brendan Higgins , Kees Cook , Kieran Bingham , Luis Chamberlain , Rob Herring , Stephen Boyd , shuah , devicetree , dri-devel , kunit-dev@googlegroups.com, "open list:DOCUMENTATION" , linux-fsdevel@vger.kernel.org, linux-kbuild , Linux Kernel Mailing List , "open list:KERNEL SELFTEST FRAMEWORK" , linux-nvdimm , linux-um@lists.infradead.org, Sasha Levin , "Bird, Timothy" , Amir Goldstein , Dan Carpenter , Dan Williams , Daniel Vetter , Jeff Dike , Joel Stanley , Julia Lawall , Kevin Hilman , Knut Omang , Logan Gunthorpe , Michael Ellerman , Petr Mladek , Richard Weinberger , David Rientjes , Steven Rostedt , wfg@linux.intel.com Content-Type: text/plain; charset="UTF-8" Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org > On Tue, May 07, 2019 at 10:01:19AM +0200, Greg KH wrote: > > > My understanding is that the intent of KUnit is to avoid booting a kernel on > > > real hardware or in a virtual machine. That seems to be a matter of semantics > > > to me because isn't invoking a UML Linux just running the Linux kernel in > > > a different form of virtualization? > > > > > > So I do not understand why KUnit is an improvement over kselftest. > > > > > > It seems to me that KUnit is just another piece of infrastructure that I > > > am going to have to be familiar with as a kernel developer. More overhead, > > > more information to stuff into my tiny little brain. > > > > > > I would guess that some developers will focus on just one of the two test > > > environments (and some will focus on both), splitting the development > > > resources instead of pooling them on a common infrastructure. > > > > > > What am I missing? > > > > kselftest provides no in-kernel framework for testing kernel code > > specifically. That should be what kunit provides, an "easy" way to > > write in-kernel tests for things. > > > > Brendan, did I get it right? > > Yes, that's basically right. You don't *have* to use KUnit. It's > supposed to be a simple way to run a large number of small tests that > for specific small components in a system. > > For example, I currently use xfstests using KVM and GCE to test all of > ext4. These tests require using multiple 5 GB and 20GB virtual disks, > and it works by mounting ext4 file systems and exercising ext4 through > the system call interfaces, using userspace tools such as fsstress, > fsx, fio, etc. It requires time overhead to start the VM, create and > allocate virtual disks, etc. For example, to run a single 3 seconds > xfstest (generic/001), it requires full 10 seconds to run it via > kvm-xfstests. > > KUnit is something else; it's specifically intended to allow you to > create lightweight tests quickly and easily, and by reducing the > effort needed to write and run unit tests, hopefully we'll have a lot > more of them and thus improve kernel quality. > > As an example, I have a volunteer working on developing KUinit tests > for ext4. We're going to start by testing the ext4 extent status > tree. The source code is at fs/ext4/extent_status.c; it's > approximately 1800 LOC. The Kunit tests for the extent status tree > will exercise all of the corner cases for the various extent status > tree functions --- e.g., ext4_es_insert_delayed_block(), > ext4_es_remove_extent(), ext4_es_cache_extent(), etc. And it will do > this in isolation without our needing to create a test file system or > using a test block device. > > Next we'll test the ext4 block allocator, again in isolation. To test > the block allocator we will have to write "mock functions" which > simulate reading allocation bitmaps from disk. Again, this will allow > the test writer to explicitly construct corner cases and validate that > the block allocator works as expected without having to reverese > engineer file system data structures which will force a particular > code path to be executed. > > So this is why it's largely irrelevant to me that KUinit uses UML. In > fact, it's a feature. We're not testing device drivers, or the > scheduler, or anything else architecture-specific. UML is not about > virtualization. What it's about in this context is allowing us to > start running test code as quickly as possible. Booting KVM takes > about 3-4 seconds, and this includes initializing virtio_scsi and > other device drivers. If by using UML we can hold the amount of > unnecessary kernel subsystem initialization down to the absolute > minimum, and if it means that we can communicating to the test > framework via a userspace "printf" from UML/KUnit code, as opposed to > via a virtual serial port to KVM's virtual console, it all makes for > lighter weight testing. > > Why did I go looking for a volunteer to write KUnit tests for ext4? > Well, I have a plan to make some changes in restructing how ext4's > write path works, in order to support things like copy-on-write, a > more efficient delayed allocation system, etc. This will require > making changes to the extent status tree, and by having unit tests for > the extent status tree, we'll be able to detect any bugs that we might > accidentally introduce in the es tree far more quickly than if we > didn't have those tests available. Google has long found that having > these sorts of unit tests is a real win for developer velocity for any > non-trivial code module (or C++ class), even when you take into > account the time it takes to create the unit tests. > > - Ted > > P.S. Many thanks to Brendan for finding such a volunteer for me; the > person in question is a SRE from Switzerland who is interested in > getting involved with kernel testing, and this is going to be their > 20% project. :-) Thanks Ted, I really appreciate it! Since Ted provided such an awesome detailed response, I don't think I really need to go into any detail; nevertheless, I think that Greg and Shuah have the right idea; in particular, Shuah provides a good summary. Thanks everyone!