Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Cc:     Rafael David Tinoco <rafael.tinoco@linaro.org>,
        "David S. Miller" <davem@davemloft.net>, netdev@vger.kernel.org,
        linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org,
        Willem de Bruijn <willemb@google.com>,
        Dan Rue <dan.rue@linaro.org>,
        Anders Roxell <anders.roxell@linaro.org>
Subject: Re: selftests/net: udpgso: LTS kernels supportability ?
To:     shuah <shuah@kernel.org>
References: <bbabafb5-e16c-4025-31d0-d9e5feb5f39a@linaro.org>
 <1d2c9a10-6caf-49ff-b921-8b93dbe96f78@kernel.org>
From:   Rafael David Tinoco <rafael.tinoco@linaro.org>
Organization: Linaro
Message-ID: <1de7a003-0aac-1f4f-b836-267136e30a0a@linaro.org>
Date:   Tue, 18 Dec 2018 09:37:58 -0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101
 Thunderbird/60.3.1
MIME-Version: 1.0
In-Reply-To: <1d2c9a10-6caf-49ff-b921-8b93dbe96f78@kernel.org>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On 12/17/18 4:42 PM, shuah wrote:
> Hi Rafael,
> 
> On 12/17/18 10:53 AM, Rafael David Tinoco wrote:
>> Shuah,
>>
>> I was recently investigating some errors coming out of our functional
>> tests and we, Dan and I, came up with a discussion that might not be new
>> for you, but, interests us, in defining how to better use kselftests as
>> a regression mechanism/tool in our LKFT (https://lkft.linaro.org).
>>
>> David / Willem,
>>
>> I'm only using udpgso as an example for what I'd like to ask Shuah. Feel
>> free to jump in in the discussion if you think its worth.
>>
>> All,
>>
>> Regarding: udpgso AND https://bugs.linaro.org/show_bug.cgi?id=3980
>>
>> udpgso tests are failing in kernels bellow 4.18 because of 2 main
>> reasons:
>>
>> 1) udp4_ufo_fragment does not seem to demand the GSO SKB to be > than
>> the MTU for older kernels (4th test case in udpgso.c).
>>
>> 2) setsockopt(...UDP_SEGMENT) support is not present for older kernels.
>> (commits "udp: generate gso with UDP_SEGMENT" and its fixes seem to be
>> needed).
> 
> This case is easy right? Based on the test output below , I can see that
> the failure is due to
> 
> ./udpgso: setsockopt udp segment: Protocol not available. setsockopt()
> is returning an error to clearly indicate that this options isn't
> supported. This will be a test change to say test is a skip as opposed
> to fail.

You referred to (2). (1) isn't that straightforward.

> We have a solution for this - test should SKIP as opposed to FAIL.
> 
>> With that explained, finally the question/discussion:
>>
>> Shouldn't we enforce a versioning mechanism for tests that are testing
>> recently added features ? I mean, some of the tests inside udpgso
>> selftest are good enough for older kernels...
> 
> Right - we do have generic way to handle that by detecting if feature is
> supported and skip instead of using Kernel version which is going to be
> hard to maintain.

You can't distinguish case (1) failures between real failures OR older
kernel behaving differently then testcase expects.

>>
>> But, because we have no control over "kernel features" and "supported
>> test cases", we, Linaro, have to end up blacklisting all selftests that
>> have new feature oriented tests, because one or two test cases only.
>>
>> This has already been solved in other functional tests projects:
>> allowing to check the running kernel version and deciding which test
>> cases to run.
>>
> 
> I would like to see effort going into fixing tests to skip when a
> feature isn't supported. I think that is the solution that will be
> maintainable in the long run.
> 
>> Would that be something we should pursue ? (We could try to make patches
>> here and there, like this case, whenever we face this). Or... should we
>> stick with mainline/next only when talking about kselftest and forget
>> about LTS kernels ?
>>
> 
> There is a middle of the road solution to run Kselftest from the same
> kernel release on LTS kernels and report the results as it is turning
> out be adding overhead in interpreting results when mainline/next
> Kselftest are run on LTS.
> 
> Kselftest mainline/next tends to be in a state where there could be bugs
> in tests like the one you are finding in the example you used to
> describe the problem. As we find them we fix them. That is just the
> nature of mainline/next
> 
> Maybe for LTS kernels it is better for you to stay with Kselftest from
> the same release or close to it. For example, running 4.20 Kselftest on
> 4.4 is going to result in more skips/(false fails) than running 4.4
> Kselftest on 4.4 even though it might provide better coverage. It is a
> judgment call on the overhead vs. advantage running newer Kselftest from
> mainline/next on LTS.
> 
> I don't think versioning (skip or release based) can fully address the
> problem you are seeing considering the fluid nature of mainline/next.

Alright. I needed a statement in that direction to decide how to better
address our "regressions" for LTS kernels.

> 
> thanks,
> -- Shuah

Thanks a lot.

-- 
Rafael D. Tinoco
Linaro - Kernel Validation