Received: by 2002:a25:4158:0:0:0:0:0 with SMTP id o85csp416822yba; Sat, 4 May 2019 05:14:29 -0700 (PDT) X-Google-Smtp-Source: APXvYqzPYdCC6R/QPbTzcjnZnJ8A3xwQ7wNlsKzRc6sdhiKkv9nWrC9prvMDiPyGrPGl9/LcOFtM X-Received: by 2002:a17:902:29c9:: with SMTP id h67mr18503481plb.114.1556972069396; Sat, 04 May 2019 05:14:29 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556972069; cv=none; d=google.com; s=arc-20160816; b=B6aAYXVAP9J9HEWJCFge8zuELd0d7+wmT/WQE7s+mwZ9OI5i6BZmwrayChBYPru0WP B22Z/1c+iLRzfWyGW6ugktteBuPoK3NnJEaUL17lF7RkU1ykxSkNngmDSIvBsTuEVRNq cNucAjV2g6FHB5fAUtSa5+D106LTDu+b0uAHM5QvM2g9Mxs1thdN0mzS1URYJhinfujG AfHt5qli2XY1bGL5HTQlP0MgBgo6yfF6TeCTuYKX7VUH9KFI9Rsx2R8MDO79VCtFYccs 0U85f9Es4PmSm6u7XkyudwpZr/hy1ztWxJRGLD7KpEgJGeXg4cFNeu8kjPKbp3pQhnR0 Tv3Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:user-agent:in-reply-to :content-disposition:mime-version:references:message-id:subject:cc :to:from:date:dkim-signature; bh=QXeJP+Xrs5IBYA/CKxIY68tI/1KMckY30b8ucTCrJbQ=; b=RlEvYxyAmTyjt0iQjL5GE+h+Lbi4SPxNlkqQPZIuB3TpQ6+9bQhF57WXvYYqBHY0x0 gsvAxWjVENEA6hI7PdjWQmSn/+eZt+ZNT03Lbw9cOuJw8fAEsxAyLYIxYC5P8jca2iqg /yZcy1RhrPuRplJo70pbYq3+5PUJLbSc+scRv7/+pKi5Uel/ZXfEh3FjvAGhQ1W9fFED aslLx/ul8hn0K6cfHtUO8Z+78YqWmNnfj0SiGX1NlcemA1IV3KC/7Y6TiozY3xFZKc/B Ob5Jf7YSjrB4OT0sKQrvmxfCyCiBX9pnu3QHZgJATlQ/JnX+aC1nuYv0tfnZ7MVcZIxd Snhg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=g2qAYqK9; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id e22si6665817pgv.323.2019.05.04.05.13.49; Sat, 04 May 2019 05:14:29 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=g2qAYqK9; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727333AbfEDKm4 (ORCPT + 99 others); Sat, 4 May 2019 06:42:56 -0400 Received: from mail.kernel.org ([198.145.29.99]:44924 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725981AbfEDKmz (ORCPT ); Sat, 4 May 2019 06:42:55 -0400 Received: from localhost (83-86-89-107.cable.dynamic.v4.ziggo.nl [83.86.89.107]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 83EC9206DF; Sat, 4 May 2019 10:42:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1556966574; bh=2wZMR8kL7s1krXCBVQ2gHWVCoJe2nAbegB0hEA5FL6g=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=g2qAYqK93MmeEO5EtmmbYxkKej3oQNG4Xe6XPJD6vrQPt38T1svHdBKrBvdMiYWgD AZ7qWDqABTJ6olUjiTR2iLAVWnkx5H2eC/KCt6WPfdYa51NsGiNAAeStPcWNLEDQJV lDwxJTqazCCtkTaH5ybwPNJUh0FxQBUrP7ZHlU+A= Date: Sat, 4 May 2019 12:42:51 +0200 From: Greg KH To: Brendan Higgins Cc: Frank Rowand , Kees Cook , Kieran Bingham , Luis Chamberlain , Rob Herring , Stephen Boyd , shuah , devicetree , dri-devel , kunit-dev@googlegroups.com, linux-doc@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kbuild@vger.kernel.org, Linux Kernel Mailing List , linux-kselftest@vger.kernel.org, linux-nvdimm , linux-um@lists.infradead.org, Sasha Levin , "Bird, Timothy" , Amir Goldstein , Dan Carpenter , Dan Williams , Daniel Vetter , Jeff Dike , Joel Stanley , Julia Lawall , Kevin Hilman , Knut Omang , Logan Gunthorpe , Michael Ellerman , Petr Mladek , Richard Weinberger , David Rientjes , Steven Rostedt , wfg@linux.intel.com, Felix Guo Subject: Re: [PATCH v2 12/17] kunit: tool: add Python wrappers for running KUnit tests Message-ID: <20190504104251.GB1478@kroah.com> References: <20190501230126.229218-1-brendanhiggins@google.com> <20190501230126.229218-13-brendanhiggins@google.com> <20190502110220.GD12416@kroah.com> <1a5f3c44-9fa9-d423-66bf-45255a90c468@gmail.com> <052fa196-4ea9-8384-79b7-fe6bacc0ee82@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.11.4 (2019-03-13) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, May 03, 2019 at 04:14:49PM -0700, Brendan Higgins wrote: > In any case, it sounds like you and Greg are in agreement on the core > libraries generating the output in TAP13, so I won't argue that point > further. Great! > ## Analysis of using TAP13 > > One of my earlier concerns was that TAP13 is a bit over constrained > for what I would like to output from the KUnit core. It only allows > data to be output as either: > - test number > - ok/not ok with single line description > - directive > - diagnostics > - YAML block > > The test number must become before a set of ok/not ok lines, and does > not contain any additional information. One annoying thing about this > is it doesn't provide any kind of nesting or grouping. It should handle nesting just fine, I think we do that already today. > There is one ok/not ok line per test and it may have a short > description of the test immediately after 'ok' or 'not ok'; this is > problematic because it wants the first thing you say about a test to > be after you know whether it passes or not. Take a look at the output of our current tests, I think you might find it to be a bit more flexible than you think. Also, this isn't our standard, we picked it because we needed a standard that the tools of today already understand. It might have issues and other problems, but we are not in the business of writing test output parsing tools, and we don't want to force everyone out there to write custom parsers. We want them to be able to use the tools they already have so they can test the kernel, and to do so as easily as possible. thanks, greg k-h