Hi Folks,
This is a call for participation for the Linux Testing microconference
at LPC this year.
For those who were at LPC last year, as the closing panel mentioned,
testing is probably the next big push needed to improve quality. From
getting more selftests in, to regression testing to ensure we don't
break realtime as more of PREEMPT_RT comes in, to more stable distros,
we need more testing around the kernel.
We have talked about different efforts around testing, such as fuzzing
(using syzkaller and trinity), automating fuzzing with syzbot, 0day
testing, test frameworks such as ktests, smatch to find bugs in the
past. We want to push this discussion further this year and are
interested in hearing from you what you want to talk about, and where
kernel testing needs to go next.
Please let us know what topics you believe should be a part of the
micro conference this year.
Thanks!
Sasha and Dhaval
Hi,
On Thursday, April 11, 2019 14:37 -03, Dhaval Giani <[email protected]> wrote:
> Hi Folks,
>
> This is a call for participation for the Linux Testing microconference
> at LPC this year.
>
> For those who were at LPC last year, as the closing panel mentioned,
> testing is probably the next big push needed to improve quality. From
> getting more selftests in, to regression testing to ensure we don't
> break realtime as more of PREEMPT_RT comes in, to more stable distros,
> we need more testing around the kernel.
>
> We have talked about different efforts around testing, such as fuzzing
> (using syzkaller and trinity), automating fuzzing with syzbot, 0day
> testing, test frameworks such as ktests, smatch to find bugs in the
> past. We want to push this discussion further this year and are
> interested in hearing from you what you want to talk about, and where
> kernel testing needs to go next.
>
> Please let us know what topics you believe should be a part of the
> micro conference this year.
Guillaume would like to talk about the his work on kernelCI on automated bisection, functional testing and modular pipelines.
Regards,
Gustavo
On Thu, 18 Apr 2019 11:22:47 +0100
"Gustavo Padovan" <[email protected]> wrote:
> Guillaume would like to talk about the his work on kernelCI on automated bisection, functional testing and modular pipelines.
Can you be more specific about what Guillaume wants to talk about for
those topics? I hope it's not just a "let everyone know what Guillaume
has done" talk. Plumbers is about discussions of on going and future
work. If these are all work-in-progress and Guillaume is looking for
input from other stakeholder developers, then that is exactly what
Plumbers is about. But if this is just to show developers what was done
and how to use the finished work, then save that for something like
Open Source Summit.
-- Steve
Hi Steve,
On 18/04/2019 14:26, Steven Rostedt wrote:
> On Thu, 18 Apr 2019 11:22:47 +0100
> "Gustavo Padovan" <[email protected]> wrote:
>
>> Guillaume would like to talk about the his work on kernelCI on automated bisection, functional testing and modular pipelines.
>
> Can you be more specific about what Guillaume wants to talk about for
> those topics? I hope it's not just a "let everyone know what Guillaume
> has done" talk. Plumbers is about discussions of on going and future
> work. If these are all work-in-progress and Guillaume is looking for
> input from other stakeholder developers, then that is exactly what
> Plumbers is about. But if this is just to show developers what was done
> and how to use the finished work, then save that for something like
> Open Source Summit.
Sure, this is very much about discussing how to grow KernelCI in
a more community-driven way. It should also help defining the
role of KernelCI in the wider kernel community, alongside other
test infrastructures.
As Gustavo mentioned, there is the bisection tool which has
already started to bear fruit. It has a few known limitations
that I've already started to address and several aspects that
would be worth discussing for future development. In particular,
it quickly gets a lot more complex when bisecting long test
suites compared to the simple boot tests we're doing now.
Then there is the topic of adding more test suites to cover more
areas of the kernel. Ultimately it would be good to have a way
to enable anyone to submit a new test suite to KernelCI and not
just the small team of developers working on it now, to scale
proportionally with the size of the task. It's also worth
discussing a strategy as to how to expand testing (which areas to
cover first, how to make best use of the available test capacity
etc.).
One more thing I have in mind is an idea I started to explain in
a document[1] whereby some components of the current KernelCI
system could have alternative instances. For example, if an
organisation is doing some upstream kernel testing and wants to
join KernelCI to contribute the results, rather than requiring
all the tests to be run in LAVA or all the kernels to be built
with Jenkins on kernelci.org, there could be alternative
automated build systems for some binaries to be produced and
alternative test systems to test them. Having a common test
results format also helps, but it's only the last step in the
pipeline so there is more to it if we want to explore the full
potential of collaborative testing. So that seems like a good
subject for discussion too as it's mostly still all up in the
air.
Best wishes,
Guillaume
[1] https://groups.io/g/kernelci/topic/kernelci_modular_pipeline/29692355
Hi,
On Thu, 2019-04-11 at 10:37 -0700, Dhaval Giani wrote:
> Hi Folks,
>
> This is a call for participation for the Linux Testing microconference
> at LPC this year.
>
> For those who were at LPC last year, as the closing panel mentioned,
> testing is probably the next big push needed to improve quality. From
> getting more selftests in, to regression testing to ensure we don't
> break realtime as more of PREEMPT_RT comes in, to more stable distros,
> we need more testing around the kernel.
>
> We have talked about different efforts around testing, such as fuzzing
> (using syzkaller and trinity), automating fuzzing with syzbot, 0day
> testing, test frameworks such as ktests, smatch to find bugs in the
> past. We want to push this discussion further this year and are
> interested in hearing from you what you want to talk about, and where
> kernel testing needs to go next.
>
> Please let us know what topics you believe should be a part of the
> micro conference this year.
I'd like to propose another topic on unit test framework
support in the kernel:
From the initial reactions and interest I have seen wrt. KTF
(http://heim.ifi.uio.no/~knuto/ktf/, https://github.com/oracle/ktf)
and the discussions on LKML around KUnit (https://lkml.org/lkml/2018/11/29/82),
it seems there's a general belief that some form of unit test framework
like these can be a good addition to the tools and infrastructure already available
in the kernel.
It seems however that different people have different notions about what
and how such a framework should ideally look, and what features belong there.
I'd like to see if we can bring that discussion forward by focusing on
some of these items, where people seem to have quite differing views
depending on where they come from. Here is a non extensive list of
some topics that seems to pop up when this gets discussed:
- "Purity" of unit testing - what constitutes a "unit" in the kernel?
- Testing kernel code - user space vs kernel space? (both useful)
- Immediate development/debugging requirements vs longer term needs
- Driver/hardware interaction testing?
- "Neat"-factor
- ease of use
- Network testing (more than 1 kernel involved)
...
I'd like to make a short intro into this, and hopefully we can have some
good exchange based on that.
[While our contribution (KTF) is currently available on Github,
at that point in time I plan for us to have submitted a version
of it to the LKML as well]
Thanks,
Knut
On Thu, Apr 11, 2019 at 10:37:51AM -0700, Dhaval Giani wrote:
> Hi Folks,
>
> This is a call for participation for the Linux Testing microconference
> at LPC this year.
>
> For those who were at LPC last year, as the closing panel mentioned,
> testing is probably the next big push needed to improve quality. From
> getting more selftests in, to regression testing to ensure we don't
> break realtime as more of PREEMPT_RT comes in, to more stable distros,
> we need more testing around the kernel.
>
> We have talked about different efforts around testing, such as fuzzing
> (using syzkaller and trinity), automating fuzzing with syzbot, 0day
> testing, test frameworks such as ktests, smatch to find bugs in the
> past. We want to push this discussion further this year and are
> interested in hearing from you what you want to talk about, and where
> kernel testing needs to go next.
I'd be interested to discuss what we could do with annotations and
compiler instrumentation to make the kernel more amenable to static and
dynamic analysis (and to some extent, documenting implicit
requirements).
One idea that I'd like to explore in the context of RT is to annotate
function signatures with their required IRQ/preempt context, such that
we could dynamically check whether those requirements were violated
(even if it didn't happen to cause a problem at that point in time), and
static analysis would be able to find some obviously broken usage. I had
some rough ideas of how to do the dynamic part atop/within ftrace. Maybe
there are similar problems elsewhere.
I know that some clang folk were interested in similar stuff. IIRC Nick
Desaulniers was interested in whether clang's thread safety analysis
tooling could be applied to the kernel (e.g. based on lockdep
annotations).
Thanks,
Mark.
----- Original Message -----
> From: "Dhaval Giani" <[email protected]>
> To: "Sasha Levin" <[email protected]>, "shuah" <[email protected]>, "Kevin Hilman" <[email protected]>,
> "Tim Bird" <[email protected]>, "LKML" <[email protected]>
> Cc: "Steven Rostedt" <[email protected]>, "Dan Carpenter" <[email protected]>, [email protected], "gustavo
> padovan" <[email protected]>, "Dmitry Vyukov" <[email protected]>, "knut omang"
> <[email protected]>
> Sent: Thursday, April 11, 2019 7:37:51 PM
> Subject: Linux Testing Microconference at LPC
>
> Hi Folks,
>
> This is a call for participation for the Linux Testing microconference
> at LPC this year.
>
> For those who were at LPC last year, as the closing panel mentioned,
> testing is probably the next big push needed to improve quality. From
> getting more selftests in, to regression testing to ensure we don't
> break realtime as more of PREEMPT_RT comes in, to more stable distros,
> we need more testing around the kernel.
>
> We have talked about different efforts around testing, such as fuzzing
> (using syzkaller and trinity), automating fuzzing with syzbot, 0day
> testing, test frameworks such as ktests, smatch to find bugs in the
> past. We want to push this discussion further this year and are
> interested in hearing from you what you want to talk about, and where
> kernel testing needs to go next.
>
> Please let us know what topics you believe should be a part of the
> micro conference this year.
>
Hi,
as CKI team, we would like to join the Plumbers discussions about CI for
kernel. We obviously have our own working pipeline but would like to get
more involved with the other upstream projects working on the same and
collaborate on common solutions.
I already started some of these discussions on automated-testing mailing
list and got in contact with Kevin about the possibility of joining forces
with KernelCI project.
Our team planned to organize a 'hackfest' for upstream CI during the
conference but I heard that the Automated Testing Summit should likely take
place during that time too. If that's the case, we should meet up and
discuss everything there instead of organizing a separate event.
Information and links about CKI Project can be be found at [1] in case
people are curious. Some of the things we'd like to achieve is having a
single source for all upstream kernel test results where anyone doing the
testing can publish them and collaborate on solving common problems (like
interpreting test results, making build times faster, increasing the test
coverage, detecting regressions etc.)
With one of my colleagues, we will submit a microconference proposal that
could serve as the starting point for the followup discussions about these
topics.
We would like to get feedback from kernel developers and maintainers about
their expectations for testing and receiving results, as well as discuss
the collaboration with other testing/CI projects.
Veronika Kabatova
CKI Project
[1] https://cki-project.org/
> Thanks!
> Sasha and Dhaval
>
I'm in the process now of planning Automated Testing Summit 2019,
which is tentatively planned for Lyon, France on October 31. This is
the day after Embedded Linux Conference Europe and Open Source Summit
Europe, in Lyon. I've been working with the
Linux Foundation event staff to set this up.
The focus of that event is test standards, including standards for
test definition, results formats, lab and board management, and APIs
between elements of the Automated Testing and CI stack.
I think that the set of things to discuss is somewhat different
between the Plumbers testing microconference and ATS. But I hope that
I'm not fragmenting the space too much.
With regards to the Testing microconference at Plumbers, I would like
to do a presentation on the current status of test standards and test
framework interoperability. We recently had some good meetings
between the LAVA and Fuego people at Linaro Connect
on this topic.
Let me know what you think.
-- Tim
On Thu, Apr 25, 2019 at 6:37 AM Veronika Kabatova <[email protected]> wrote:
>
>
>
> ----- Original Message -----
> > From: "Dhaval Giani" <[email protected]>
> > To: "Sasha Levin" <[email protected]>, "shuah" <[email protected]>, "Kevin Hilman" <[email protected]>,
> > "Tim Bird" <[email protected]>, "LKML" <[email protected]>
> > Cc: "Steven Rostedt" <[email protected]>, "Dan Carpenter" <[email protected]>, [email protected], "gustavo
> > padovan" <[email protected]>, "Dmitry Vyukov" <[email protected]>, "knut omang"
> > <[email protected]>
> > Sent: Thursday, April 11, 2019 7:37:51 PM
> > Subject: Linux Testing Microconference at LPC
> >
> > Hi Folks,
> >
> > This is a call for participation for the Linux Testing microconference
> > at LPC this year.
> >
> > For those who were at LPC last year, as the closing panel mentioned,
> > testing is probably the next big push needed to improve quality. From
> > getting more selftests in, to regression testing to ensure we don't
> > break realtime as more of PREEMPT_RT comes in, to more stable distros,
> > we need more testing around the kernel.
> >
> > We have talked about different efforts around testing, such as fuzzing
> > (using syzkaller and trinity), automating fuzzing with syzbot, 0day
> > testing, test frameworks such as ktests, smatch to find bugs in the
> > past. We want to push this discussion further this year and are
> > interested in hearing from you what you want to talk about, and where
> > kernel testing needs to go next.
> >
> > Please let us know what topics you believe should be a part of the
> > micro conference this year.
> >
>
> Hi,
>
> as CKI team, we would like to join the Plumbers discussions about CI for
> kernel. We obviously have our own working pipeline but would like to get
> more involved with the other upstream projects working on the same and
> collaborate on common solutions.
>
> I already started some of these discussions on automated-testing mailing
> list and got in contact with Kevin about the possibility of joining forces
> with KernelCI project.
>
> Our team planned to organize a 'hackfest' for upstream CI during the
> conference but I heard that the Automated Testing Summit should likely take
> place during that time too. If that's the case, we should meet up and
> discuss everything there instead of organizing a separate event.
>
>
> Information and links about CKI Project can be be found at [1] in case
> people are curious. Some of the things we'd like to achieve is having a
> single source for all upstream kernel test results where anyone doing the
> testing can publish them and collaborate on solving common problems (like
> interpreting test results, making build times faster, increasing the test
> coverage, detecting regressions etc.)
>
> With one of my colleagues, we will submit a microconference proposal that
> could serve as the starting point for the followup discussions about these
> topics.
>
>
> We would like to get feedback from kernel developers and maintainers about
> their expectations for testing and receiving results, as well as discuss
> the collaboration with other testing/CI projects.
>
>
>
> Veronika Kabatova
> CKI Project
>
>
> [1] https://cki-project.org/
>
>
> > Thanks!
> > Sasha and Dhaval
> >
--
-- Tim Bird
Senior Staff Software Engineer, Sony Corporation
Architecture Group Chair, Core Embedded Linux Project, Linux Foundation
On Tue, Apr 23, 2019 at 11:22:50AM +0100, Mark Rutland wrote:
> On Thu, Apr 11, 2019 at 10:37:51AM -0700, Dhaval Giani wrote:
> > Hi Folks,
> >
> > This is a call for participation for the Linux Testing microconference
> > at LPC this year.
> >
> > For those who were at LPC last year, as the closing panel mentioned,
> > testing is probably the next big push needed to improve quality. From
> > getting more selftests in, to regression testing to ensure we don't
> > break realtime as more of PREEMPT_RT comes in, to more stable distros,
> > we need more testing around the kernel.
> >
> > We have talked about different efforts around testing, such as fuzzing
> > (using syzkaller and trinity), automating fuzzing with syzbot, 0day
> > testing, test frameworks such as ktests, smatch to find bugs in the
> > past. We want to push this discussion further this year and are
> > interested in hearing from you what you want to talk about, and where
> > kernel testing needs to go next.
>
> I'd be interested to discuss what we could do with annotations and
> compiler instrumentation to make the kernel more amenable to static and
> dynamic analysis (and to some extent, documenting implicit
> requirements).
>
> One idea that I'd like to explore in the context of RT is to annotate
> function signatures with their required IRQ/preempt context, such that
> we could dynamically check whether those requirements were violated
> (even if it didn't happen to cause a problem at that point in time), and
> static analysis would be able to find some obviously broken usage. I had
> some rough ideas of how to do the dynamic part atop/within ftrace. Maybe
> there are similar problems elsewhere.
>
> I know that some clang folk were interested in similar stuff. IIRC Nick
> Desaulniers was interested in whether clang's thread safety analysis
> tooling could be applied to the kernel (e.g. based on lockdep
> annotations).
FWIW, I'd also be interested in discussing these developments.
There have been several activities/projects related to such "tooling"
(thread safety analysis) recently: I could point out the (brand new)
Google Summer of Code "Applying Clang Thread Safety Analyser to Linux
Kernel" project [1] and (for the "dynamic analysis" side) the efforts
to revive the Kernel Thread sanitizer [2]. I should also mention the
efforts to add (support for) "unmarked" accesses and to formalize the
notion of "data race" in the memory consistency model [3].
So, again, I'd welcome a discussion on these works/ideas.
Thanks,
Andrea
[1] https://summerofcode.withgoogle.com/projects/#5358212549705728
https://github.com/ClangBuiltLinux/thread-safety-analysis
[2] https://github.com/google/ktsan/commits/ktsan
[3] https://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git/commit/?h=dev&id=c602b9e58cb9c13f260791dd7da6687e06809923
https://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git/commit/?h=dev&id=3b1fe99c68b5673879a8018a46b23f431e4d4b7a
https://lkml.kernel.org/r/[email protected]
Hi Sasha and Dhaval,
On 4/11/19 11:37 AM, Dhaval Giani wrote:
> Hi Folks,
>
> This is a call for participation for the Linux Testing microconference
> at LPC this year.
>
> For those who were at LPC last year, as the closing panel mentioned,
> testing is probably the next big push needed to improve quality. From
> getting more selftests in, to regression testing to ensure we don't
> break realtime as more of PREEMPT_RT comes in, to more stable distros,
> we need more testing around the kernel.
>
> We have talked about different efforts around testing, such as fuzzing
> (using syzkaller and trinity), automating fuzzing with syzbot, 0day
> testing, test frameworks such as ktests, smatch to find bugs in the
> past. We want to push this discussion further this year and are
> interested in hearing from you what you want to talk about, and where
> kernel testing needs to go next.
>
> Please let us know what topics you believe should be a part of the
> micro conference this year.
>
> Thanks!
> Sasha and Dhaval
>
A talk on KUnit from Brendan Higgins will be good addition to this
Micro-conference. I am cc'ing Brendan on this thread.
Please consider adding it.
thanks,
-- Shuah
On Wed, May 15, 2019 at 04:44:19PM -0600, shuah wrote:
> Hi Sasha and Dhaval,
>
> On 4/11/19 11:37 AM, Dhaval Giani wrote:
> > Hi Folks,
> >
> > This is a call for participation for the Linux Testing microconference
> > at LPC this year.
> >
> > For those who were at LPC last year, as the closing panel mentioned,
> > testing is probably the next big push needed to improve quality. From
> > getting more selftests in, to regression testing to ensure we don't
> > break realtime as more of PREEMPT_RT comes in, to more stable distros,
> > we need more testing around the kernel.
> >
> > We have talked about different efforts around testing, such as fuzzing
> > (using syzkaller and trinity), automating fuzzing with syzbot, 0day
> > testing, test frameworks such as ktests, smatch to find bugs in the
> > past. We want to push this discussion further this year and are
> > interested in hearing from you what you want to talk about, and where
> > kernel testing needs to go next.
> >
> > Please let us know what topics you believe should be a part of the
> > micro conference this year.
> >
> > Thanks!
> > Sasha and Dhaval
> >
>
> A talk on KUnit from Brendan Higgins will be good addition to this
> Micro-conference. I am cc'ing Brendan on this thread.
>
> Please consider adding it.
Thanks Shuah!
Presumably I should still submit the talk on the website (however, it
looks like the Testing Microconference isn't available as a track option
yet...)? Or is it okay if I just post the proposal here?
Also, for the framing of the talk (assuming people are indeed
interested). I figure people will want an intro along with some
background context, and a discussion of future work. Nevertheless, would
people like more of a demo talk or more of an audience driven discussion
on where we should go and what we should do? Or something else? Really,
I am open to talk about whatever everyone else wants.
For context on KUnit, you can read the LWN article about it here[1], or
you can see the current version of the patchset here[2].
Thanks!
[1] https://lwn.net/Articles/780985/
[2] https://lkml.org/lkml/2019/5/14/834
On Fri, Apr 26, 2019 at 02:02:53PM -0700, Tim Bird wrote:
>I'm in the process now of planning Automated Testing Summit 2019,
>which is tentatively planned for Lyon, France on October 31. This is
>the day after Embedded Linux Conference Europe and Open Source Summit
>Europe, in Lyon. I've been working with the
>Linux Foundation event staff to set this up.
>
>The focus of that event is test standards, including standards for
>test definition, results formats, lab and board management, and APIs
>between elements of the Automated Testing and CI stack.
>
>I think that the set of things to discuss is somewhat different
>between the Plumbers testing microconference and ATS. But I hope that
>I'm not fragmenting the space too much.
>
>With regards to the Testing microconference at Plumbers, I would like
>to do a presentation on the current status of test standards and test
>framework interoperability. We recently had some good meetings
>between the LAVA and Fuego people at Linaro Connect
>on this topic.
Hi Tim,
Sorry for the delayed response, this mail got marked as read as a result
of fat fingers :(
I'd want to avoid having an 'overview' talk as part of the MC. We have
quite a few discussion topics this year and in the spirit of LPC I'd
prefer to avoid presentations.
Maybe it's more appropriate for the refereed track?
--
Thanks,
Sasha
> -----Original Message-----
> From: Sasha Levin
>
> On Fri, Apr 26, 2019 at 02:02:53PM -0700, Tim Bird wrote:
...
> >
> >With regards to the Testing microconference at Plumbers, I would like
> >to do a presentation on the current status of test standards and test
> >framework interoperability. We recently had some good meetings
> >between the LAVA and Fuego people at Linaro Connect
> >on this topic.
>
> Hi Tim,
>
> Sorry for the delayed response, this mail got marked as read as a result
> of fat fingers :(
>
> I'd want to avoid having an 'overview' talk as part of the MC. We have
> quite a few discussion topics this year and in the spirit of LPC I'd
> prefer to avoid presentations.
OK. Sounds good.
> Maybe it's more appropriate for the refereed track?
I'll consider submitting it there, but there's a certain "fun" aspect
to attending a conference that I don't have to prepare a talk for. :-)
Thanks for getting back to me. I'm already registered for Plumbers,
so I'll see you there.
-- Tim
On Wed, May 15, 2019 at 04:44:19PM -0600, shuah wrote:
>Hi Sasha and Dhaval,
>
>On 4/11/19 11:37 AM, Dhaval Giani wrote:
>>Hi Folks,
>>
>>This is a call for participation for the Linux Testing microconference
>>at LPC this year.
>>
>>For those who were at LPC last year, as the closing panel mentioned,
>>testing is probably the next big push needed to improve quality. From
>>getting more selftests in, to regression testing to ensure we don't
>>break realtime as more of PREEMPT_RT comes in, to more stable distros,
>>we need more testing around the kernel.
>>
>>We have talked about different efforts around testing, such as fuzzing
>>(using syzkaller and trinity), automating fuzzing with syzbot, 0day
>>testing, test frameworks such as ktests, smatch to find bugs in the
>>past. We want to push this discussion further this year and are
>>interested in hearing from you what you want to talk about, and where
>>kernel testing needs to go next.
>>
>>Please let us know what topics you believe should be a part of the
>>micro conference this year.
>>
>>Thanks!
>>Sasha and Dhaval
>>
>
>A talk on KUnit from Brendan Higgins will be good addition to this
>Micro-conference. I am cc'ing Brendan on this thread.
>
>Please consider adding it.
FWIW, the topic of unit tests is already on the schedule. There seems to
be two different sub-topics here (kunit vs KTF) so there's a good
discussion to be had here on many levels.
--
Thanks,
Sasha
On Fri, Apr 26, 2019 at 11:03 PM Tim Bird <[email protected]> wrote:
>
> I'm in the process now of planning Automated Testing Summit 2019,
> which is tentatively planned for Lyon, France on October 31. This is
This is _November_ 1, right?
> the day after Embedded Linux Conference Europe and Open Source Summit
> Europe, in Lyon. I've been working with the
> Linux Foundation event staff to set this up.
>
> The focus of that event is test standards, including standards for
> test definition, results formats, lab and board management, and APIs
> between elements of the Automated Testing and CI stack.
>
> I think that the set of things to discuss is somewhat different
> between the Plumbers testing microconference and ATS. But I hope that
> I'm not fragmenting the space too much.
>
> With regards to the Testing microconference at Plumbers, I would like
> to do a presentation on the current status of test standards and test
> framework interoperability. We recently had some good meetings
> between the LAVA and Fuego people at Linaro Connect
> on this topic.
>
> Let me know what you think.
> -- Tim
>
> On Thu, Apr 25, 2019 at 6:37 AM Veronika Kabatova <[email protected]> wrote:
> >
> >
> >
> > ----- Original Message -----
> > > From: "Dhaval Giani" <[email protected]>
> > > To: "Sasha Levin" <[email protected]>, "shuah" <[email protected]>, "Kevin Hilman" <[email protected]>,
> > > "Tim Bird" <[email protected]>, "LKML" <[email protected]>
> > > Cc: "Steven Rostedt" <[email protected]>, "Dan Carpenter" <[email protected]>, [email protected], "gustavo
> > > padovan" <[email protected]>, "Dmitry Vyukov" <[email protected]>, "knut omang"
> > > <[email protected]>
> > > Sent: Thursday, April 11, 2019 7:37:51 PM
> > > Subject: Linux Testing Microconference at LPC
> > >
> > > Hi Folks,
> > >
> > > This is a call for participation for the Linux Testing microconference
> > > at LPC this year.
> > >
> > > For those who were at LPC last year, as the closing panel mentioned,
> > > testing is probably the next big push needed to improve quality. From
> > > getting more selftests in, to regression testing to ensure we don't
> > > break realtime as more of PREEMPT_RT comes in, to more stable distros,
> > > we need more testing around the kernel.
> > >
> > > We have talked about different efforts around testing, such as fuzzing
> > > (using syzkaller and trinity), automating fuzzing with syzbot, 0day
> > > testing, test frameworks such as ktests, smatch to find bugs in the
> > > past. We want to push this discussion further this year and are
> > > interested in hearing from you what you want to talk about, and where
> > > kernel testing needs to go next.
> > >
> > > Please let us know what topics you believe should be a part of the
> > > micro conference this year.
> > >
> >
> > Hi,
> >
> > as CKI team, we would like to join the Plumbers discussions about CI for
> > kernel. We obviously have our own working pipeline but would like to get
> > more involved with the other upstream projects working on the same and
> > collaborate on common solutions.
> >
> > I already started some of these discussions on automated-testing mailing
> > list and got in contact with Kevin about the possibility of joining forces
> > with KernelCI project.
> >
> > Our team planned to organize a 'hackfest' for upstream CI during the
> > conference but I heard that the Automated Testing Summit should likely take
> > place during that time too. If that's the case, we should meet up and
> > discuss everything there instead of organizing a separate event.
> >
> >
> > Information and links about CKI Project can be be found at [1] in case
> > people are curious. Some of the things we'd like to achieve is having a
> > single source for all upstream kernel test results where anyone doing the
> > testing can publish them and collaborate on solving common problems (like
> > interpreting test results, making build times faster, increasing the test
> > coverage, detecting regressions etc.)
> >
> > With one of my colleagues, we will submit a microconference proposal that
> > could serve as the starting point for the followup discussions about these
> > topics.
> >
> >
> > We would like to get feedback from kernel developers and maintainers about
> > their expectations for testing and receiving results, as well as discuss
> > the collaboration with other testing/CI projects.
> >
> >
> >
> > Veronika Kabatova
> > CKI Project
> >
> >
> > [1] https://cki-project.org/
> >
> >
> > > Thanks!
> > > Sasha and Dhaval
> > >
>
>
>
> --
> -- Tim Bird
> Senior Staff Software Engineer, Sony Corporation
> Architecture Group Chair, Core Embedded Linux Project, Linux Foundation
On Sun, May 12, 2019 at 2:40 AM Andrea Parri
<[email protected]> wrote:
>
> On Tue, Apr 23, 2019 at 11:22:50AM +0100, Mark Rutland wrote:
> > On Thu, Apr 11, 2019 at 10:37:51AM -0700, Dhaval Giani wrote:
> > > Hi Folks,
> > >
> > > This is a call for participation for the Linux Testing microconference
> > > at LPC this year.
> > >
> > > For those who were at LPC last year, as the closing panel mentioned,
> > > testing is probably the next big push needed to improve quality. From
> > > getting more selftests in, to regression testing to ensure we don't
> > > break realtime as more of PREEMPT_RT comes in, to more stable distros,
> > > we need more testing around the kernel.
> > >
> > > We have talked about different efforts around testing, such as fuzzing
> > > (using syzkaller and trinity), automating fuzzing with syzbot, 0day
> > > testing, test frameworks such as ktests, smatch to find bugs in the
> > > past. We want to push this discussion further this year and are
> > > interested in hearing from you what you want to talk about, and where
> > > kernel testing needs to go next.
> >
> > I'd be interested to discuss what we could do with annotations and
> > compiler instrumentation to make the kernel more amenable to static and
> > dynamic analysis (and to some extent, documenting implicit
> > requirements).
> >
> > One idea that I'd like to explore in the context of RT is to annotate
> > function signatures with their required IRQ/preempt context, such that
> > we could dynamically check whether those requirements were violated
> > (even if it didn't happen to cause a problem at that point in time), and
> > static analysis would be able to find some obviously broken usage. I had
> > some rough ideas of how to do the dynamic part atop/within ftrace. Maybe
> > there are similar problems elsewhere.
> >
> > I know that some clang folk were interested in similar stuff. IIRC Nick
> > Desaulniers was interested in whether clang's thread safety analysis
> > tooling could be applied to the kernel (e.g. based on lockdep
> > annotations).
>
> FWIW, I'd also be interested in discussing these developments.
>
> There have been several activities/projects related to such "tooling"
> (thread safety analysis) recently: I could point out the (brand new)
> Google Summer of Code "Applying Clang Thread Safety Analyser to Linux
> Kernel" project [1] and (for the "dynamic analysis" side) the efforts
> to revive the Kernel Thread sanitizer [2]. I should also mention the
> efforts to add (support for) "unmarked" accesses and to formalize the
> notion of "data race" in the memory consistency model [3].
>
> So, again, I'd welcome a discussion on these works/ideas.
>
> Thanks,
> Andrea
I would be interested in discussing all of this too: thread safety
annotations, ktsan, unmarked accesses.
On Thu, May 16, 2019 at 2:51 AM <[email protected]> wrote:
> > -----Original Message-----
> > From: Sasha Levin
> >
> > On Fri, Apr 26, 2019 at 02:02:53PM -0700, Tim Bird wrote:
> ...
> > >
> > >With regards to the Testing microconference at Plumbers, I would like
> > >to do a presentation on the current status of test standards and test
> > >framework interoperability. We recently had some good meetings
> > >between the LAVA and Fuego people at Linaro Connect
> > >on this topic.
> >
> > Hi Tim,
> >
> > Sorry for the delayed response, this mail got marked as read as a result
> > of fat fingers :(
> >
> > I'd want to avoid having an 'overview' talk as part of the MC. We have
> > quite a few discussion topics this year and in the spirit of LPC I'd
> > prefer to avoid presentations.
>
> OK. Sounds good.
>
> > Maybe it's more appropriate for the refereed track?
> I'll consider submitting it there, but there's a certain "fun" aspect
> to attending a conference that I don't have to prepare a talk for. :-)
>
> Thanks for getting back to me. I'm already registered for Plumbers,
> so I'll see you there.
> -- Tim
I would like to give an update on syzkaller/syzbot and discuss:
- testability of kernel components in this context
- test coverage and what's still not tested
- discussion of the process (again): what works, what doesn't work, feedback
I also submitted a refereed track talk called "Reflections on kernel
quality, development process and testing". If it's not accepted, I
would like to do it on Testing MC.
On Wed, May 22, 2019 at 6:04 PM Dmitry Vyukov <[email protected]> wrote:
>
> On Thu, May 16, 2019 at 2:51 AM <[email protected]> wrote:
> > > -----Original Message-----
> > > From: Sasha Levin
> > >
> > > On Fri, Apr 26, 2019 at 02:02:53PM -0700, Tim Bird wrote:
> > ...
> > > >
> > > >With regards to the Testing microconference at Plumbers, I would like
> > > >to do a presentation on the current status of test standards and test
> > > >framework interoperability. We recently had some good meetings
> > > >between the LAVA and Fuego people at Linaro Connect
> > > >on this topic.
> > >
> > > Hi Tim,
> > >
> > > Sorry for the delayed response, this mail got marked as read as a result
> > > of fat fingers :(
> > >
> > > I'd want to avoid having an 'overview' talk as part of the MC. We have
> > > quite a few discussion topics this year and in the spirit of LPC I'd
> > > prefer to avoid presentations.
> >
> > OK. Sounds good.
> >
> > > Maybe it's more appropriate for the refereed track?
> > I'll consider submitting it there, but there's a certain "fun" aspect
> > to attending a conference that I don't have to prepare a talk for. :-)
> >
> > Thanks for getting back to me. I'm already registered for Plumbers,
> > so I'll see you there.
> > -- Tim
>
>
> I would like to give an update on syzkaller/syzbot and discuss:
> - testability of kernel components in this context
> - test coverage and what's still not tested
> - discussion of the process (again): what works, what doesn't work, feedback
>
This sounds good to me.
> I also submitted a refereed track talk called "Reflections on kernel
> quality, development process and testing". If it's not accepted, I
> would like to do it on Testing MC.
I don't think refereed talks fit in the MC
> Please let us know what topics you believe should be a part of the
> micro conference this year.
At OSPM right now, Douglas and Ionela were talking about their
scheduler behavioral testing framework using LISA and rt-app. This is
an interesting topic, and I think has a lot of scope for making
scheduler testing/behaviour more predictable as well as
analyze/validate scheduler behavior. I am hoping they are able to make
it to LPC this year.
Dhaval
On Wed, May 15, 2019 at 08:36:49PM -0400, Sasha Levin wrote:
> On Wed, May 15, 2019 at 04:44:19PM -0600, shuah wrote:
> > Hi Sasha and Dhaval,
> >
> > On 4/11/19 11:37 AM, Dhaval Giani wrote:
> > > Hi Folks,
> > >
> > > This is a call for participation for the Linux Testing microconference
> > > at LPC this year.
> > >
> > > For those who were at LPC last year, as the closing panel mentioned,
> > > testing is probably the next big push needed to improve quality. From
> > > getting more selftests in, to regression testing to ensure we don't
> > > break realtime as more of PREEMPT_RT comes in, to more stable distros,
> > > we need more testing around the kernel.
> > >
> > > We have talked about different efforts around testing, such as fuzzing
> > > (using syzkaller and trinity), automating fuzzing with syzbot, 0day
> > > testing, test frameworks such as ktests, smatch to find bugs in the
> > > past. We want to push this discussion further this year and are
> > > interested in hearing from you what you want to talk about, and where
> > > kernel testing needs to go next.
> > >
> > > Please let us know what topics you believe should be a part of the
> > > micro conference this year.
> > >
> > > Thanks!
> > > Sasha and Dhaval
> > >
> >
> > A talk on KUnit from Brendan Higgins will be good addition to this
> > Micro-conference. I am cc'ing Brendan on this thread.
> >
> > Please consider adding it.
>
> FWIW, the topic of unit tests is already on the schedule. There seems to
> be two different sub-topics here (kunit vs KTF) so there's a good
> discussion to be had here on many levels.
Cool, so do we just want to go with that? Have a single slot for KUnit
and KTF combined?
We can each present our work up to this point; maybe offer some
background and rationale on why we made the decision we have and then we
can have some moderated discussion on, pros, cons, next steps, etc?
Cheers
> -----Original Message-----
> From: Dmitry Vyukov
> On Fri, Apr 26, 2019 at 11:03 PM Tim Bird <[email protected]> wrote:
> >
> > I'm in the process now of planning Automated Testing Summit 2019,
> > which is tentatively planned for Lyon, France on October 31. This is
>
> This is _November_ 1, right?
No. Thursday, October 31, 2019. Is there some conflict on Thursday?
-- Tim
On Wed, 22 May 2019 14:02:31 -0700
Brendan Higgins <[email protected]> wrote:
> Cool, so do we just want to go with that? Have a single slot for KUnit
> and KTF combined?
>
> We can each present our work up to this point; maybe offer some
> background and rationale on why we made the decision we have and then we
> can have some moderated discussion on, pros, cons, next steps, etc?
>
You have till the end of today to submit a Refereed talk if you want to
present. Otherwise, Microconferences should only have 5 to 10 minutes
to present what they want to discuss before a discussion should
proceed. If you need more than 10 minutes to give your point, then you
need to try to get a Refereed talk in, and that will give you a full 40
minutes.
-- Steve
On Wed, 22 May 2019 20:58:47 -0400
Steven Rostedt <[email protected]> wrote:
> You have till the end of today to submit a Refereed talk if you want to
> present. Otherwise, Microconferences should only have 5 to 10 minutes
> to present what they want to discuss before a discussion should
> proceed. If you need more than 10 minutes to give your point, then you
> need to try to get a Refereed talk in, and that will give you a full 40
> minutes.
I just found out that we are extending the deadline by one week. You
still have time, but you had better hurry!
-- Steve
On Wed, 2019-05-22 at 14:02 -0700, Brendan Higgins wrote:
> On Wed, May 15, 2019 at 08:36:49PM -0400, Sasha Levin wrote:
> > On Wed, May 15, 2019 at 04:44:19PM -0600, shuah wrote:
> > > Hi Sasha and Dhaval,
> > >
> > > On 4/11/19 11:37 AM, Dhaval Giani wrote:
> > > > Hi Folks,
> > > >
> > > > This is a call for participation for the Linux Testing microconference
> > > > at LPC this year.
> > > >
> > > > For those who were at LPC last year, as the closing panel mentioned,
> > > > testing is probably the next big push needed to improve quality. From
> > > > getting more selftests in, to regression testing to ensure we don't
> > > > break realtime as more of PREEMPT_RT comes in, to more stable distros,
> > > > we need more testing around the kernel.
> > > >
> > > > We have talked about different efforts around testing, such as fuzzing
> > > > (using syzkaller and trinity), automating fuzzing with syzbot, 0day
> > > > testing, test frameworks such as ktests, smatch to find bugs in the
> > > > past. We want to push this discussion further this year and are
> > > > interested in hearing from you what you want to talk about, and where
> > > > kernel testing needs to go next.
> > > >
> > > > Please let us know what topics you believe should be a part of the
> > > > micro conference this year.
> > > >
> > > > Thanks!
> > > > Sasha and Dhaval
> > > >
> > >
> > > A talk on KUnit from Brendan Higgins will be good addition to this
> > > Micro-conference. I am cc'ing Brendan on this thread.
> > >
> > > Please consider adding it.
> >
> > FWIW, the topic of unit tests is already on the schedule. There seems to
> > be two different sub-topics here (kunit vs KTF) so there's a good
> > discussion to be had here on many levels.
>
> Cool, so do we just want to go with that? Have a single slot for KUnit
> and KTF combined?
>
> We can each present our work up to this point; maybe offer some
> background and rationale on why we made the decision we have and then we
> can have some moderated discussion on, pros, cons, next steps, etc?
I definitely had KTF and KUnit in mind when proposing this topic.
If you recall from the last time we discussed unit testing, each slot is
fairly limited in time. My plan for the intro for discussion is to
itemize some of the distinct goals we try to achieve with our frameworks and have a
discussion based on that. In light of the discussion around your patch sets,
one topic is also the question of whether a common API would be useful/desired,
and whether we can "capture" a short namespace for that.
Thanks,
Knut
> Cheers
On Thu, May 23, 2019 at 2:08 AM <[email protected]> wrote:
> > From: Dmitry Vyukov
> > On Fri, Apr 26, 2019 at 11:03 PM Tim Bird <[email protected]> wrote:
> > >
> > > I'm in the process now of planning Automated Testing Summit 2019,
> > > which is tentatively planned for Lyon, France on October 31. This is
> >
> > This is _November_ 1, right?
> No. Thursday, October 31, 2019. Is there some conflict on Thursday?
Ah, no, sorry. It's just me incapable of reading numbers.
On Wed, May 22, 2019 at 05:52:17PM +0200, Dmitry Vyukov wrote:
> On Sun, May 12, 2019 at 2:40 AM Andrea Parri
> <[email protected]> wrote:
> >
> > On Tue, Apr 23, 2019 at 11:22:50AM +0100, Mark Rutland wrote:
> > > On Thu, Apr 11, 2019 at 10:37:51AM -0700, Dhaval Giani wrote:
> > > > Hi Folks,
> > > >
> > > > This is a call for participation for the Linux Testing microconference
> > > > at LPC this year.
> > > >
> > > > For those who were at LPC last year, as the closing panel mentioned,
> > > > testing is probably the next big push needed to improve quality. From
> > > > getting more selftests in, to regression testing to ensure we don't
> > > > break realtime as more of PREEMPT_RT comes in, to more stable distros,
> > > > we need more testing around the kernel.
> > > >
> > > > We have talked about different efforts around testing, such as fuzzing
> > > > (using syzkaller and trinity), automating fuzzing with syzbot, 0day
> > > > testing, test frameworks such as ktests, smatch to find bugs in the
> > > > past. We want to push this discussion further this year and are
> > > > interested in hearing from you what you want to talk about, and where
> > > > kernel testing needs to go next.
> > >
> > > I'd be interested to discuss what we could do with annotations and
> > > compiler instrumentation to make the kernel more amenable to static and
> > > dynamic analysis (and to some extent, documenting implicit
> > > requirements).
> > >
> > > One idea that I'd like to explore in the context of RT is to annotate
> > > function signatures with their required IRQ/preempt context, such that
> > > we could dynamically check whether those requirements were violated
> > > (even if it didn't happen to cause a problem at that point in time), and
> > > static analysis would be able to find some obviously broken usage. I had
> > > some rough ideas of how to do the dynamic part atop/within ftrace. Maybe
> > > there are similar problems elsewhere.
> > >
> > > I know that some clang folk were interested in similar stuff. IIRC Nick
> > > Desaulniers was interested in whether clang's thread safety analysis
> > > tooling could be applied to the kernel (e.g. based on lockdep
> > > annotations).
> >
> > FWIW, I'd also be interested in discussing these developments.
> >
> > There have been several activities/projects related to such "tooling"
> > (thread safety analysis) recently: I could point out the (brand new)
> > Google Summer of Code "Applying Clang Thread Safety Analyser to Linux
> > Kernel" project [1] and (for the "dynamic analysis" side) the efforts
> > to revive the Kernel Thread sanitizer [2]. I should also mention the
> > efforts to add (support for) "unmarked" accesses and to formalize the
> > notion of "data race" in the memory consistency model [3].
> >
> > So, again, I'd welcome a discussion on these works/ideas.
> >
> > Thanks,
> > Andrea
>
> I would be interested in discussing all of this too: thread safety
> annotations, ktsan, unmarked accesses.
Sounds like a great discussion! Might this fit into Sasha Levin's
and Dhaval Giani's proposed Testing & Fuzzing MC?
Thanx, Paul
On Wed, May 22, 2019 at 9:55 PM Knut Omang <[email protected]> wrote:
Sorry for the delayed reply.
>
> On Wed, 2019-05-22 at 14:02 -0700, Brendan Higgins wrote:
> > On Wed, May 15, 2019 at 08:36:49PM -0400, Sasha Levin wrote:
> > > On Wed, May 15, 2019 at 04:44:19PM -0600, shuah wrote:
> > > > Hi Sasha and Dhaval,
> > > >
> > > > On 4/11/19 11:37 AM, Dhaval Giani wrote:
> > > > > Hi Folks,
> > > > >
> > > > > This is a call for participation for the Linux Testing microconference
> > > > > at LPC this year.
> > > > >
> > > > > For those who were at LPC last year, as the closing panel mentioned,
> > > > > testing is probably the next big push needed to improve quality. From
> > > > > getting more selftests in, to regression testing to ensure we don't
> > > > > break realtime as more of PREEMPT_RT comes in, to more stable distros,
> > > > > we need more testing around the kernel.
> > > > >
> > > > > We have talked about different efforts around testing, such as fuzzing
> > > > > (using syzkaller and trinity), automating fuzzing with syzbot, 0day
> > > > > testing, test frameworks such as ktests, smatch to find bugs in the
> > > > > past. We want to push this discussion further this year and are
> > > > > interested in hearing from you what you want to talk about, and where
> > > > > kernel testing needs to go next.
> > > > >
> > > > > Please let us know what topics you believe should be a part of the
> > > > > micro conference this year.
> > > > >
> > > > > Thanks!
> > > > > Sasha and Dhaval
> > > > >
> > > >
> > > > A talk on KUnit from Brendan Higgins will be good addition to this
> > > > Micro-conference. I am cc'ing Brendan on this thread.
> > > >
> > > > Please consider adding it.
> > >
> > > FWIW, the topic of unit tests is already on the schedule. There seems to
> > > be two different sub-topics here (kunit vs KTF) so there's a good
> > > discussion to be had here on many levels.
> >
> > Cool, so do we just want to go with that? Have a single slot for KUnit
> > and KTF combined?
> >
> > We can each present our work up to this point; maybe offer some
> > background and rationale on why we made the decision we have and then we
> > can have some moderated discussion on, pros, cons, next steps, etc?
>
> I definitely had KTF and KUnit in mind when proposing this topic.
Awesome!
> If you recall from the last time we discussed unit testing, each slot is
> fairly limited in time. My plan for the intro for discussion is to
Yeah, as per Steven's comment, I also submitted a refereed talk for
more detailed stuff.
> itemize some of the distinct goals we try to achieve with our frameworks and have a
> discussion based on that. In light of the discussion around your patch sets,
Sounds good to me. One thing I would like to talk about is maybe
trying to classify different categories of tests (unit vs. integration
vs. end-to-end), where they fit into the Linux kernel, how
prescriptivist should we be in categorization and what a test is for,
etc. I think this has been a point of disagreement/confusion on my
patchsets as well.
> one topic is also the question of whether a common API would be useful/desired,
> and whether we can "capture" a short namespace for that.
I am not opposed. This could potentially tie in to what kind of test
something is as I mentioned above. In anycase, sounds like there is a
lot of room for good discussion.
Thanks!
Hi Dhaval,
On 5/22/19 5:11 PM, Dhaval Giani wrote:
>> Please let us know what topics you believe should be a part of the
>> micro conference this year.
>
> At OSPM right now, Douglas and Ionela were talking about their
> scheduler behavioral testing framework using LISA and rt-app. This is
> an interesting topic, and I think has a lot of scope for making
> scheduler testing/behaviour more predictable as well as
> analyze/validate scheduler behavior. I am hoping they are able to make
> it to LPC this year.
We unfortunately won't be able to attend on that topic this year. We however do have
some documentation describing the way we use statistics in our testing methodology,
although it requires some level of familiarity with the tooling [LISA].
The [slides] from Valentin at OSPM 2019 describes some other aspects regarding noise handling.
All of that should probably be aggregated in some tool-agnostic part of the LISA documentation to make it
easier to grasp by the wider community, especially when it comes to test framework capabilities comparison.
If someone fancies a chat on tooling capabilities, we are also reachable on #arm-lisa channel on
freenode during European working hours.
[LISA] https://lisa-linux-integrated-system-analysis.readthedocs.io/en/master/workflows/automated_testing.html#analyzing-results
[slides] http://retis.sssup.it/ospm-summit/Downloads/01_07-SchedulerBehaviouralTesting_Schneider.pdf
>
> Dhaval
>
Best regards,
Douglas