Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) client-ip=23.128.96.36;
Date:   Sat, 7 Oct 2023 15:23:30 +0800
From:   Oliver Sang <oliver.sang@intel.com>
To:     Ido Schimmel <idosch@idosch.org>
CC:     Sriram Yagnaraman <sriram.yagnaraman@est.tech>,
        "oe-lkp@lists.linux.dev" <oe-lkp@lists.linux.dev>,
        "lkp@intel.com" <lkp@intel.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "David S. Miller" <davem@davemloft.net>,
        "netdev@vger.kernel.org" <netdev@vger.kernel.org>,
        <oliver.sang@intel.com>
Subject: Re: [linus:master] [selftests]  8ae9efb859:
 kernel-selftests.net.fib_tests.sh.fail
Message-ID: <ZSEHcoakN1FeL6ZM@xsang-OptiPlex-9020>
References: <202309191658.c00d8b8-oliver.sang@intel.com>
 <DBBP189MB1433CECC6CBECFD95352EA3595FCA@DBBP189MB1433.EURP189.PROD.OUTLOOK.COM>
 <ZRmHLPbyTCBvoHWP@shredder>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <ZRmHLPbyTCBvoHWP@shredder>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?9XqDZIOym5Ir9wV5CR7C0gBlo6HBABhwiP92NQToM6Yq3B1mEGNA4PrxJ9re?=
 =?us-ascii?Q?nEyFb1aKfqpmeIfT02dArXlmTNXssmrNiE4/JGR9avI/Iks6MKnu8rqAPjlf?=
 =?us-ascii?Q?MY2tUfKKF93dnMz9nqZDL/UogZQh4N89JEE62kFsdh9TD2sR8I22X0MmETM3?=
 =?us-ascii?Q?KX7nr2Lz3/wV0Z2LCBy7SNCzCWx1v4HgtOZu4Os9lqcR4ufMLTjDaSMYqvsb?=
 =?us-ascii?Q?ZDFjeaFskRCxHfdiOyApdjAACneXb3wgDpf9wb7l3sPrENvkW9Xi8mT3m3yc?=
 =?us-ascii?Q?RPFKzE34xpgH27uLADlc8wE3Ozv5QPm4s2v5m0sb/zPMRLZA/2KJfc+3o1IC?=
 =?us-ascii?Q?m2bn1gh3uXfa40miuobzrxbm9gov8PBRV07sBriap2yotSPqixLKk0qzazAn?=
 =?us-ascii?Q?0uebqv8NyJ1H0ARBgjVVVZIp3qAvFYwtLCESjlC/NIBkPFTGSRq94+1l+w6a?=
 =?us-ascii?Q?BoelsrEZYtd3MrHaofITYugVffyO+YRDso1CXmEOUM8i20Gi+6z33Qv0kJ0l?=
 =?us-ascii?Q?OKT4ie2NvkPjmp5PMLtOncuIEWKAAtlbmNcbLpMDqgMRjOm9OegG3H23qHDx?=
 =?us-ascii?Q?9k6FAXz9L/toZbCY2q+NK0qPZRm5KQPwOUWJh+1x4fMa0PA0aA01HF13zpLZ?=
 =?us-ascii?Q?rR6/PHyQbA1PMtDvhD++nqOcyGMx7fg2D5t50RseBv5QOnz9q7Mo1GPmGIhB?=
 =?us-ascii?Q?Kfnm7PUY6yP15wxRZ4OW9Cd6dygMoe2/R971y1Qcj6NcYSn/cVrq4K0hf4IC?=
 =?us-ascii?Q?MkKD+BPXN49gOAxkwy2F564FlMxq1g0MUqQYz5NJ2Gxn8cPXwgJSktn/wZdU?=
 =?us-ascii?Q?U0jzEPJnJmAYnqJRb+FwW7YbfABhtzz4N4aBl1wDW2f3p1x4DyOyBRavFNb5?=
 =?us-ascii?Q?e0qwyyiCuzq5NKO9UkLEtP4Tpw8MS7r8oymNeTPcoEGWy/ampWiLo6o3HAzY?=
 =?us-ascii?Q?n9odL3wDg/E8PHDhLMFUKZ/0kCICwrero5ykp8dX0RwPZnQqsv5yaTkoJ858?=
 =?us-ascii?Q?p5fu2+QMZdG2bBysqURT6WLz8Jz3UNAH96Go9Dzkf40nvH0Ply4fSFUn+jj2?=
 =?us-ascii?Q?iT31EUcZBI5ygkwCGT1WKXWs+6AdivIJlHb6MoOUdhSBMQ1sBjMUhJW5WiNl?=
 =?us-ascii?Q?udGMkYDXz+yIEp03ShBF0VFdzpZYBk2yZAAtuqyz6DQaHBg5u4HEs/5jPMk6?=
 =?us-ascii?Q?p7rsvIJYKX302cOPECVTtwAmqwd2B5Cjp7BVglsCTD7f0qTuAPTcHD4tfK9s?=
 =?us-ascii?Q?y5LAbxBmACYcSQKo8ivmbviWdZxxY57FfUbp5d+oyrDMVUvuWzq9zT9zELT/?=
 =?us-ascii?Q?QE4PEC/AfKyX0nJQPqNR9RMkgK89TwWYsrqCgxTL613vaWHvi3Rnwr2hnXYs?=
 =?us-ascii?Q?5tpgVSuEFm8ksWqUGcmv2phNt9XLMKh/ap9AQJUsEXFy8ZXZQ++LLdK60Y+A?=
 =?us-ascii?Q?E4Y3ixyZL7SGMTBGZeHw4Nvj5mIqeUwxkr481Id533y4D1EID4T6HmcmRL6R?=
 =?us-ascii?Q?GEOSEN1QwcjQROFAJPhlsxDhi9xjNYCC4NU89PhxZIoS6zEQp8mZP9e6P1se?=
 =?us-ascii?Q?RYi0p3B5XrfQLCa5BtlSyQNOaA3WjkMQmgpXbfwOBYHVPm/ZlCB0Onx/Uyum?=
 =?us-ascii?Q?Xg=3D=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 02f643ba-aa93-4e08-7f45-08dbc706520f
X-MS-Exchange-CrossTenant-AuthSource: PH8PR11MB6779.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Oct 2023 07:23:36.6271
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: NDbuT1zqhbiv6qyJmxcBg6g3ha0EnjrlXqT0/tVB8qQp6ADHznk3AhReNy0Vsu3QV8eo4nQB9ahaX4SMbR0Aww==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR11MB8548
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=2.7 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,RCVD_IN_SBL_CSS,SPF_HELO_NONE,SPF_PASS autolearn=no
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Sat, 07 Oct 2023 00:23:55 -0700 (PDT)
X-Spam-Level: **

hi, Ido Schimmel,

On Sun, Oct 01, 2023 at 05:50:20PM +0300, Ido Schimmel wrote:
> On Mon, Sep 25, 2023 at 06:18:34PM +0000, Sriram Yagnaraman wrote:
> > CC: Ido, who helped a lot with writing these tests.
> > 
> > > -----Original Message-----
> > > From: kernel test robot <oliver.sang@intel.com>
> > > Sent: Tuesday, 19 September 2023 10:32
> > > To: Sriram Yagnaraman <sriram.yagnaraman@est.tech>
> > > Cc: oe-lkp@lists.linux.dev; lkp@intel.com; linux-kernel@vger.kernel.org; David
> > > S. Miller <davem@davemloft.net>; netdev@vger.kernel.org;
> > > oliver.sang@intel.com
> > > Subject: [linus:master] [selftests] 8ae9efb859: kernel-
> > > selftests.net.fib_tests.sh.fail
> > > 
> > > 
> > > hi, Sriram Yagnaraman,
> > > 
> > > we noticed two new added tests failed in our test environment.
> > > want to consult with you what's the dependency and requirement to run
> > > them?
> > > Thanks a lot!
> > 
> > Sorry for the delayed response. I will look at this and get back.
> > I am not an expert with lkp-tests but will try to set it up on my local environment and reproduce the problem.
> > 
> > > 
> > > Hello,
> > > 
> > > kernel test robot noticed "kernel-selftests.net.fib_tests.sh.fail" on:
> > > 
> > > commit: 8ae9efb859c05a54ac92b3336c6ca0597c9c8cdb ("selftests: fib_tests:
> > > Add multipath list receive tests")
> > > https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
> > > 
> > > in testcase: kernel-selftests
> > > version: kernel-selftests-x86_64-60acb023-1_20230329
> > > with following parameters:
> > > 
> > > 	group: net
> > > 
> > > 
> > > 
> > > compiler: gcc-12
> > > test machine: 36 threads 1 sockets Intel(R) Core(TM) i9-10980XE CPU @
> > > 3.00GHz (Cascade Lake) with 32G memory
> > > 
> > > (please refer to attached dmesg/kmsg for entire log/backtrace)
> > > 
> > > 
> > > 
> > > 
> > > If you fix the issue in a separate patch/commit (i.e. not just a new version of the
> > > same patch/commit), kindly add following tags
> > > | Reported-by: kernel test robot <oliver.sang@intel.com>
> > > | Closes:
> > > | https://lore.kernel.org/oe-lkp/202309191658.c00d8b8-oliver.sang@intel.
> > > | com
> > > 
> > > 
> > > 
> > > # timeout set to 1500
> > > # selftests: net: fib_tests.sh
> > > #
> > > # Single path route test
> > > #     Start point
> > > #     TEST: IPv4 fibmatch                                                 [ OK ]
> > > #     TEST: IPv6 fibmatch                                                 [ OK ]
> > > #     Nexthop device deleted
> > > #     TEST: IPv4 fibmatch - no route                                      [ OK ]
> > > #     TEST: IPv6 fibmatch - no route                                      [ OK ]
> > > 
> > > ...
> > > 
> > > #
> > > # Fib6 garbage collection test
> > > #     TEST: ipv6 route garbage collection                                 [ OK ]
> > > #
> > > # IPv4 multipath list receive tests
> > > #     TEST: Multipath route hit ratio (.06)                               [FAIL]
> > > #
> > > # IPv6 multipath list receive tests
> > > #     TEST: Multipath route hit ratio (.10)                               [FAIL]
> 
> I found two possible problems. The first is that in the IPv4 case we
> might get more trace point hits than packets (ratio higher than 1)
> because of the additional FIB lookups for source validation. Fixed by
> disabling source validation:
> 
> diff --git a/tools/testing/selftests/net/fib_tests.sh b/tools/testing/selftests/net/fib_tests.sh
> index e7d2a530618a..66d0db7a2614 100755
> --- a/tools/testing/selftests/net/fib_tests.sh
> +++ b/tools/testing/selftests/net/fib_tests.sh
> @@ -2437,6 +2437,9 @@ ipv4_mpath_list_test()
>         run_cmd "ip -n ns2 route add 203.0.113.0/24
>                 nexthop via 172.16.201.2 nexthop via 172.16.202.2"
>         run_cmd "ip netns exec ns2 sysctl -qw net.ipv4.fib_multipath_hash_policy=1"
> +       run_cmd "ip netns exec ns2 sysctl -qw net.ipv4.conf.veth2.rp_filter=0"
> +       run_cmd "ip netns exec ns2 sysctl -qw net.ipv4.conf.all.rp_filter=0"
> +       run_cmd "ip netns exec ns2 sysctl -qw net.ipv4.conf.default.rp_filter=0"
>         set +e
>  
>         local dmac=$(ip -n ns2 -j link show dev veth2 | jq -r '.[]["address"]')
> 
> The second problem (which I believe is the one you encountered) is that
> we might miss certain trace point hits if they happen from the ksoftirqd
> task instead of the mausezahn task. Fixed by:
> 
> @@ -2449,7 +2452,7 @@ ipv4_mpath_list_test()
>         # words, the FIB lookup tracepoint needs to be triggered for every
>         # packet.
>         local t0_rx_pkts=$(link_stats_get ns2 veth2 rx packets)
> -       run_cmd "perf stat -e fib:fib_table_lookup --filter 'err == 0' -j -o $tmp_file -- $cmd"
> +       run_cmd "perf stat -a -e fib:fib_table_lookup --filter 'err == 0' -j -o $tmp_file -- $cmd"
>         local t1_rx_pkts=$(link_stats_get ns2 veth2 rx packets)
>         local diff=$(echo $t1_rx_pkts - $t0_rx_pkts | bc -l)
>         list_rcv_eval $tmp_file $diff
> @@ -2494,7 +2497,7 @@ ipv6_mpath_list_test()
>         # words, the FIB lookup tracepoint needs to be triggered for every
>         # packet.
>         local t0_rx_pkts=$(link_stats_get ns2 veth2 rx packets)
> -       run_cmd "perf stat -e fib6:fib6_table_lookup --filter 'err == 0' -j -o $tmp_file -- $cmd"
> +       run_cmd "perf stat -a -e fib6:fib6_table_lookup --filter 'err == 0' -j -o $tmp_file -- $cmd"
>         local t1_rx_pkts=$(link_stats_get ns2 veth2 rx packets)
>         local diff=$(echo $t1_rx_pkts - $t0_rx_pkts | bc -l)
>         list_rcv_eval $tmp_file $diff
> 
> Ran both tests in a loop:
> 
> # for i in $(seq 1 20); do ./fib_tests.sh -t ipv4_mpath_list; done
> # for i in $(seq 1 20); do ./fib_tests.sh -t ipv6_mpath_list; done
> 
> And verified that the results are stable. Also verified that the tests
> reliably fail when reverting both fixes:
> 
> 8423be8926aa ipv6: ignore dst hint for multipath routes
> 6ac66cb03ae3 ipv4: ignore dst hint for multipath routes
> 
> Can you please test with the proposed modifications?

we applied above patches upon 8ae9efb859, and two tests passed now:

# IPv4 multipath list receive tests
#     TEST: Multipath route hit ratio (.99)                               [ OK ]
#
# IPv6 multipath list receive tests
#     TEST: Multipath route hit ratio (1.00)                              [ OK ]
#
# Tests passed: 225
# Tests failed:   0
ok 17 selftests: net: fib_tests.sh


Tested-by: kernel test robot <oliver.sang@intel.com>


> 
> Thanks
>