Received: by 2002:a05:6902:102b:0:0:0:0 with SMTP id x11csp904002ybt; Fri, 26 Jun 2020 14:37:26 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyzgt0nX4uWHroULtYV7V1ZGflerqE1+pvQ0ma2u/2Bvpp1OFhEz4pcJZMAN8SYtyclkiQ9 X-Received: by 2002:aa7:cc19:: with SMTP id q25mr5454579edt.26.1593207446235; Fri, 26 Jun 2020 14:37:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1593207446; cv=none; d=google.com; s=arc-20160816; b=BigHs+MSiO0RZbha5V2C1w1i51SR2qInvbc6tvlTqaz6fK+Q45uISlFckZnDwEcaVK 1W+wtN5Jxoey9/gvulY9nHwA4KuFlYXuNZGiTTYy5yvZugR7BrDWLwuIjne5yhYAYvzx /wY2kxHsuT1a+gVvM5G5Ayg4pldLyab+p9sysLnJhTtkj96+YnbTcD+jhsivNcgMXbqo lBZHSnzsjTuNSWTAaMtAK9P1rA5H1b0LzsiEymVOjsZIclCdpOqhmV6Wzi5ML+gifNZF AEd0yXYLgW242JTvItovbLArAd8QpzMz5TVhh4W54ipLUYexs9eIt8C35b2aMWgRBksS iJtA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:in-reply-to:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :dkim-signature; bh=DqOaaV58EjOIY1VepCG+GQtVeu6B1YTzNiLFMCCxxtU=; b=KDKgrXUlVsAGMXtDDsU4cIJMvkAX3qi78lINtIBdKcjJ9EBVivpZYvye8wJrvSG0eC BrYTsQXUsk3eTrp89Kgl9Nji3Gz/sFTzbSmz5J6BYNYDDj6QUznxGshOIFrokKTDcfqt peEwrtmiiySpF/qk+4+AxIk7/+NOygBK/6WfASsiPM2OVlo/d4l0hXWDQvxXX7h1nzlx wApquRoj4/dUR7+hTDkU6MLRchsLoVBCiQw7cbnvPMinBIvv/tnb93YU8M0r5j3V5MGz znksosdwEb27oyZ9nYGTwy5HVAT+/PeyG+dFIeEBJoW0l/Cf/IXq4Oql1kvdpECZ/kb3 RzCQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@chromium.org header.s=google header.b="YEN4h/JS"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=chromium.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id sb27si1339179ejb.538.2020.06.26.14.37.02; Fri, 26 Jun 2020 14:37:26 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@chromium.org header.s=google header.b="YEN4h/JS"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=chromium.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1725855AbgFZVfq (ORCPT + 99 others); Fri, 26 Jun 2020 17:35:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52882 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725803AbgFZVfp (ORCPT ); Fri, 26 Jun 2020 17:35:45 -0400 Received: from mail-pf1-x441.google.com (mail-pf1-x441.google.com [IPv6:2607:f8b0:4864:20::441]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A66DFC03E97B for ; Fri, 26 Jun 2020 14:35:45 -0700 (PDT) Received: by mail-pf1-x441.google.com with SMTP id 67so1100826pfg.5 for ; Fri, 26 Jun 2020 14:35:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=DqOaaV58EjOIY1VepCG+GQtVeu6B1YTzNiLFMCCxxtU=; b=YEN4h/JS6rsbM2TA8UohIHdNyG9IbIr7SJGjr/bFVHhrG/Xom9uuW4VVJlxpjCalyc VzYDhmbP8IO2Aln4UuRyuqd6Tgy9Qku2VWWKJ3oZdsqEk7RJQDbYUlIcQKhd+fpKbBRk cijP8DnapIIWZQIeC0Y6ZFYLszmtMLfDZQXr0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=DqOaaV58EjOIY1VepCG+GQtVeu6B1YTzNiLFMCCxxtU=; b=Q/ggecLsWm0wpIQYq6ROri3YlB4tnINWJhO7ReimxRny/ZodQkw9L3TS+ZwacbWmxH 9ttA3rq5Ku5nD7XLFx4yQG3nWU1wWIAeP+X5jfc6p3z2CoWA+SfyyO25RIwLGWiqxMZ7 LUMam9W62sjdjugJAwSJMXYiTYSk8sV5zbNoVgyOIbE10aZLM0WWA6p7kmLZDd9dBnTJ H571KheOqEQjy5cdbMQYz9OvxDj383ggzIAeUf9BzInkIl/kYKwX0OGNxu49bNbW2ql5 WDrC2XQd7BQQ55P7UR+NgulR3mrzpZy/qYnEjkQu7DER0pinSXLamvq6oSKp8n5hX3Rl cxag== X-Gm-Message-State: AOAM533YKntLwWZkBELkRhP8GT6LT5xRK3mrv87WlaZjlKSesmjhIn1C 08BdqftASPAjKRdbsBifAwOIoQ== X-Received: by 2002:a63:371c:: with SMTP id e28mr692197pga.114.1593207345125; Fri, 26 Jun 2020 14:35:45 -0700 (PDT) Received: from www.outflux.net (smtp.outflux.net. [198.145.64.163]) by smtp.gmail.com with ESMTPSA id w135sm7739381pfc.106.2020.06.26.14.35.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 26 Jun 2020 14:35:43 -0700 (PDT) Date: Fri, 26 Jun 2020 14:35:43 -0700 From: Kees Cook To: Brendan Higgins Cc: jdike@addtoit.com, richard@nod.at, anton.ivanov@cambridgegreys.com, arnd@arndb.de, skhan@linuxfoundation.org, alan.maguire@oracle.com, yzaikin@google.com, davidgow@google.com, akpm@linux-foundation.org, rppt@linux.ibm.com, frowand.list@gmail.com, catalin.marinas@arm.com, will@kernel.org, monstr@monstr.eu, mpe@ellerman.id.au, benh@kernel.crashing.org, paulus@samba.org, chris@zankel.net, jcmvbkbc@gmail.com, gregkh@linuxfoundation.org, sboyd@kernel.org, logang@deltatee.com, mcgrof@kernel.org, linux-um@lists.infradead.org, linux-arch@vger.kernel.org, linux-kselftest@vger.kernel.org, kunit-dev@googlegroups.com, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-xtensa@linux-xtensa.org Subject: Re: [PATCH v5 09/12] kunit: test: add test plan to KUnit TAP format Message-ID: <202006261434.119AE33DBB@keescook> References: <20200626210917.358969-1-brendanhiggins@google.com> <20200626210917.358969-10-brendanhiggins@google.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20200626210917.358969-10-brendanhiggins@google.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, Jun 26, 2020 at 02:09:14PM -0700, Brendan Higgins wrote: > TAP 14 allows an optional test plan to be emitted before the start of > the start of testing[1]; this is valuable because it makes it possible > for a test harness to detect whether the number of tests run matches the > number of tests expected to be run, ensuring that no tests silently > failed. > > Link[1]: https://github.com/isaacs/testanything.github.io/blob/tap14/tap-version-14-specification.md#the-plan > Signed-off-by: Brendan Higgins > Reviewed-by: Stephen Boyd Look good, except... > diff --git a/tools/testing/kunit/test_data/test_is_test_passed-all_passed.log b/tools/testing/kunit/test_data/test_is_test_passed-all_passed.log > index 62ebc0288355c4b122ccc18ae2505f971efa57bc..bc0dc8fe35b760b1feb74ec419818dbfae1adb5c 100644 > GIT binary patch > delta 28 > jcmbQmGoME|#4$jjEVZaOGe1wk(1goSPtRy09}gP > delta 23 > ecmbQwGmD2W#4$jjEVZaOGe1wk&}5@94;uhhkp{*9 > > diff --git a/tools/testing/kunit/test_data/test_is_test_passed-crash.log b/tools/testing/kunit/test_data/test_is_test_passed-crash.log > index 0b249870c8be417a5865bd40a24c8597bb7f5ab1..4d97f6708c4a5ad5bb2ac879e12afca6e816d83d 100644 > GIT binary patch > delta 15 > WcmX>hepY;fFN>j`p3z318g2k9Uj*m? > > delta 10 > RcmX>renNbL@5Z2NZU7lr1S$Xk > > diff --git a/tools/testing/kunit/test_data/test_is_test_passed-failure.log b/tools/testing/kunit/test_data/test_is_test_passed-failure.log > index 9e89d32d5667a59d137f8adacf3a88fdb7f88baf..7a416497e3bec044eefc1535f7d84ee85703ba97 100644 > GIT binary patch > delta 28 > jcmZ3&yOLKp#4$jjEVZaOGe1wk(1goSPtRy0-!wJ=eKrU$ > > delta 23 > ecmZ3