Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752892AbbDBNn4 (ORCPT ); Thu, 2 Apr 2015 09:43:56 -0400 Received: from lists.s-osg.org ([54.187.51.154]:33156 "EHLO lists.s-osg.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751260AbbDBNny (ORCPT ); Thu, 2 Apr 2015 09:43:54 -0400 Message-ID: <551D4792.4010909@osg.samsung.com> Date: Thu, 02 Apr 2015 07:43:46 -0600 From: Shuah Khan Organization: Samsung Open Source Group User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.5.0 MIME-Version: 1.0 To: Prarit Bhargava , Tyler Baker CC: John Stultz , lkml , Thomas Gleixner , Richard Cochran Subject: Re: [PATCH 1/2] kselftests: timers: Make set-timer-lat fail more gracefully for !CAP_WAKE_ALARM References: <1427327073-19011-1-git-send-email-john.stultz@linaro.org> <5513EE03.1020507@redhat.com> <551D177E.80203@redhat.com> In-Reply-To: <551D177E.80203@redhat.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 4280 Lines: 87 On 04/02/2015 04:18 AM, Prarit Bhargava wrote: > > > On 03/26/2015 01:33 PM, Tyler Baker wrote: >> On 26 March 2015 at 09:29, John Stultz wrote: >>> On Thu, Mar 26, 2015 at 4:31 AM, Prarit Bhargava wrote: >>>> On 03/25/2015 07:44 PM, John Stultz wrote: >>>>> + printf("%-22s %s missing CAP_WAKE_ALARM? : [UNSUPPORTED]\n", >>>>> + clockstring(clock_id), >>>>> + flags ? "ABSTIME":"RELTIME"); >>>> >>>> Something to think about: Do you want to write these tests to be more human >>>> readable or machine readable? In theory with awk I guess it doesn't matter too >>>> much, however, it is something that we should think about moving forward. >>> >>> So this came up at ELC in a few discussions. Right now there isn't any >>> established output format, but there's some nice and simple >>> infrastructure for counting pass/fails. >>> >>> However, in talking to Tyler, I know he has started looking at how to >>> integrate the selftests into our automated infrastructure and was >>> interested in how we improve the output parsing for reports. So there >>> is interest in improving this, and I'm open to whatever changes might >>> be needed (adding extra arguments to the test to put them into "easy >>> parse" mode or whatever). >> >> Thanks for looping me in John. My interest in kselftest stems from my >> involvement with kernelci.org, a communityservice focused on upstream >> kernel validation across multiple architectures. In it's current form, >> it is merely build and boot testing boards. However, we are at a point >> where we'd like to start running some tests. The automation framework >> (LAVA) used to execute these tests essentially uses a regular >> expression to parse the test's standard output. This is advantageous >> as a test can be written in any language, as long as it produces sane >> uniform output. >> >> Ideally, we would like to perform the kernel builds as we do today >> along with building all the kseltests present in the tree, and >> inserting them into a 'testing' ramdisk for deployment. Once we >> successfully boot the platform, we execute all the kselftests, parse >> standard out, and report the results. The benefit from this >> implementation is that a developer writing a test does have to do >> anything 'special' to get his/her test to run once it has been applied >> to a upstream tree. I'll explain below some concerns I have about >> accomplishing this. >> >> Currently, we have had to write wrappers[1][2] for some kselftests to >> be able parse the output. If we can choose/agree on a standard output >> format all of this complexity goes away, and then we can dynamically >> run kselftests. Integration of new tests will not be needed, as they >> all produce output in standard way. I've taken a look at the wiki page >> for standardizing output[3] and TAP looks like the good format IMO. >> >> Also, for arch != x86 there are some barriers to overcome to get all >> the kselftests cross compiling, which would be nice to have as well. >> >> I realize this may be a good amount of work, so I'd like to help out. >> Perhaps working John to convert his timer tests to use TAP output >> would be a good starting point? > > John, I could probably do that for you. I'm always willing to give it a shot. > Improving reporting and output is a good idea, as long as the reporting framework doesn't add external dependencies and makes it difficult to build and run tests from the kernel git tree environment. Being able to run tests on a development system is the primary objective for these tests. As long as the TAP doesn't require additional libraries and tools to be installed on the development systems, I will be happy with the improvements to reporting and output. thanks, -- Shuah -- Shuah Khan Sr. Linux Kernel Developer Open Source Innovation Group Samsung Research America (Silicon Valley) shuahkh@osg.samsung.com | (970) 217-8978 -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/