Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754818AbdFWTDW (ORCPT ); Fri, 23 Jun 2017 15:03:22 -0400 Received: from resqmta-po-08v.sys.comcast.net ([96.114.154.167]:40172 "EHLO resqmta-po-08v.sys.comcast.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754757AbdFWTDT (ORCPT ); Fri, 23 Jun 2017 15:03:19 -0400 Reply-To: shuah@kernel.org Subject: Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures] To: Tom Gall Cc: Kees Cook , Andy Lutomirski , Sumit Semwal , Brian Norris , "Luis R. Rodriguez" , Greg Kroah-Hartman , LKML , "# 3.4.x" , "open list:KERNEL SELFTEST FRAMEWORK" , Shuah Khan , Shuah Khan References: <334e6a92-2d41-c9e1-c807-19e493f1af83@kernel.org> <1ce2d820-3479-2d65-14e1-b60cd93bd055@kernel.org> From: Shuah Khan Message-ID: <2ffa4cd6-8a50-dc9f-32b1-f9a07be40e4e@kernel.org> Date: Fri, 23 Jun 2017 13:03:17 -0600 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.1.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-CMAE-Envelope: MS4wfMGRgCotaX+H+PYaFB+cL1qvfaV5RvChkVXQ0xmzGDzAW6K8IUlpO3cVgy6WwPHOnhdThWYCoOgs4Uphx2kq/Fshfup7Oa8dYz1iI8bebHp/+pbw/1hz 951XcyXb7Jnk1Ex9tTLJq/7/xbHMJFSQfidPbutAPBbpYjYuztTvRWAQU2hfjRRQPjT0Bh37N//4J2g9APTJgL2egMTEUofiGp/12IS5UuD6aWDofm8Mes/w bC1jOjQdVFkRGKbHnls9PctDRW14v7KKBjcmF0Vp0s+5d2sOXYlKe0++tfEUJO7btEdjvtSxKwI7p4eq2gBT9VFbTpq0G3RrOB9VAW3AMOwtb4FAB+KCdGEe Qi2vJVJ2DQRecG3y4jnq19PTJBmcECaIVz0Ol911J4qnQVcChTqnrExLDhU8SSzq5YOXgpEsULpaC1GUpRcE9o/zfsebyuhr43JLXUrEb1nE5imRcrT798LJ f54Tp/6TzTuY2Qor+yj5hk839mo0xupx8IE0FGEF4x3PGTMEqU/JWyONVqpZ8VU5a4NdhBsmcLzXRjU5CTikuWPijVm88/apGEsrfA== Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 6546 Lines: 146 On 06/22/2017 01:48 PM, Tom Gall wrote: > Hi > > On Thu, Jun 22, 2017 at 2:06 PM, Shuah Khan wrote: >> On 06/22/2017 11:50 AM, Kees Cook wrote: >>> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski wrote: >>>> On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan wrote: >>>>> On 06/22/2017 10:53 AM, Kees Cook wrote: >>>>>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal wrote: >>>>>>> Hi Kees, Andy, >>>>>>> >>>>>>> On 15 June 2017 at 23:26, Sumit Semwal wrote: >>>>>>>> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] - >>>>>>>> feature and test together. >>>>>>>> - This one also seems like a security hole being closed, and the >>>>>>>> 'feature' could be a candidate for stable backports, but Arnd tried >>>>>>>> that, and it was quite non-trivial. So perhaps we'll need some help >>>>>>>> from the subsystem developers here. >>>>>>> >>>>>>> Could you please help us sort this out? Our goal is to help Greg with >>>>>>> testing stable kernels, and currently the seccomp tests fail due to >>>>>>> missing feature (seccomp ptrace hole closure) getting tested via >>>>>>> latest kselftest. >>>>>>> >>>>>>> If you feel the feature isn't a stable candidate, then could you >>>>>>> please help make the test degrade gracefully in its absence? >> >> In some cases, it is not easy to degrade and/or check for a feature. >> Probably several security features could fall in this bucket. >> >>>>>> >>>>>> I don't really want to have that change be a backport -- it's quite >>>>>> invasive across multiple architectures. >> >> Agreed. The same test for kernel applies to tests as well. If a kernel >> feature can't be back-ported, the test for that feature will fall in the >> same bucket. It shouldn't be back-ported. >> >>>>>> >>>>>> I would say just add a kernel version check to the test. This is >>>>>> probably not the only selftest that will need such things. :) >>>>> >>>>> Adding release checks to selftests is going to problematic for maintenance. >>>>> Tests should fail gracefully if feature isn't supported in older kernels. >>>>> >>>>> Several tests do that now and please find a way to check for dependencies >>>>> and feature availability and fail the test gracefully. If there is a test >>>>> that can't do that for some reason, we can discuss it, but as a general >>>>> rule, I don't want to see kselftest patches that check release. >>>> >>>> If a future kernel inadvertently loses the new feature and degrades to >>>> the behavior of old kernels, that would be a serious bug and should be >>>> caught. >> >> Agreed. If I understand you correctly, by not testing stable kernels >> with their own selftests, some serious bugs could go undetected. > > Personally I'm a bit skeptical. I think the reasoning is more that the > latest selftests provide more coverage, and therefore should be better > tests, even on older kernels. > >>> >>> Right. I really think stable kernels should be tested with their own >>> selftests. If some test is needed in a stable kernel it should be >>> backported to that stable kernel. >> >> Correct. This is always a safe option. There might be cases that even >> prevent tests being built, especially if a new feature adds new fields >> to an existing structure. >> >> It appears in some cases, users want to run newer tests on older kernels. >> Some tests can clearly detect feature support using module presence and/or >> Kconfig enabled or disabled. These are conditions even on a kernel that >> supports a new module or new config option. The kernel the test is running >> on might not have the feature enabled or module might not be present. In >> these cases, it would be easier to detect and skip the test. >> >> However, some features aren't so easy. For example: >> >> - a new flag is added to a syscall, and new test is added. It might not >> be easy to detect that. >> - We might have some tests that can't detect and skip. >> >> Based on this discussion, it is probably accurate to say: >> >> 1. It is recommended that selftests from the same release be run on the >> kernel. >> 2. Selftests from newer kernels will run on older kernels, user should >> understand the risks such as some tests might fail and might not >> detect feature degradation related bugs. >> 3. Selftests will fail gracefully on older releases if at all possible. > > How about gracefully be skipped instead of fail? > > The later suggests the test case in some situations can detect it's > pointless to run something and say as much instead of emitting a > failure that would be a waste of time to look into. > > As another example take tools/testing/selftests/net/psock_fanout.c > On 4.9 it'll fail to compile (using master's selftests) because > PACKET_FANOUT_FLAG_UNIQUEID isn't defined. Add a simple #ifdef for > that symbol and the psock_fanout test will compile and run just fine. Unfortunately adding PACKET_FANOUT_FLAG_UNIQUEID isn't correct. This is a new feature that went into 4.12. You don't want to define this for older kernel which will break on newer kernels. This is one concern I have that in am attempt to fix mainline selftest to be able to test older kernels will cause problems. This is a good example of a new test that has dependency on a new define in an include file which will be hard to check - compile will fail on older kernels. The rest of the net tests will compile and run. -- Shuah > >> Sumit! >> >> 1. What are the reasons for testing older kernel with selftests from >> newer kernels? What are the benefits you see for doing so? > > I think the presumption is the latest greatest collection of selftests > are the best, most complete. > >> I am looking to understand the need/reasons for this use-case. In our >> previous discussion on this subject, I did say, you should be able to >> do so with some exceptions. >> >> 2. Do you test kernels with the selftests from the same release? > > We have the ability to do either. The new shiny .... it calls. > >> 3. Do you find testing with newer selftests to be useful? > > I think it comes down to coverage and again the current perception > that latest greatest is better. Quantitatively we haven't collected > data to support that position tho it would be interesting to compare > say a 4.4-lts and it's selftests directory to a mainline, see how much > was new and then find out how much of those new selftests actually > work on the older 4.4-lts. > >> thanks, >> -- Shuah > >