by Guillaume Nault

[permalink] [raw]

Subject: Re: POSSIBLE BUG: selftests/net/fcnal-test.sh: [FAIL] in vrf "bind - ns-B IPv6 LLA" test

On Tue, Jun 06, 2023 at 09:17:24PM +0200, Mirsad Goran Todorovac wrote:
> On 6/6/23 20:50, Guillaume Nault wrote:
> > On Tue, Jun 06, 2023 at 04:28:02PM +0200, Mirsad Todorovac wrote:
> > > On 6/6/23 16:11, Guillaume Nault wrote:
> > > > On Tue, Jun 06, 2023 at 03:57:35PM +0200, Mirsad Todorovac wrote:
> > > > > + if (oif) {
> > > > > + rcu_read_lock();
> > > > > + dev = dev_get_by_index_rcu(net, oif);
> > > > > + rcu_read_unlock();
> > > >
> > > > You can't assume '*dev' is still valid after rcu_read_unlock() unless
> > > > you hold a reference on it.
> > > >
> > > > > + rtnl_lock();
> > > > > + mdev = netdev_master_upper_dev_get(dev);
> > > > > + rtnl_unlock();
> > > >
> > > > Because of that, 'dev' might have already disappeared at the time
> > > > netdev_master_upper_dev_get() is called. So it may dereference an
> > > > invalid pointer here.
> > >
> > > Good point, thanks. I didn't expect those to change.
> > >
> > > This can be fixed, provided that RCU and RTNL locks can be nested:
> >
> > Well, yes and no. You can call rcu_read_{lock,unlock}() while under the
> > rtnl protection, but not the other way around.
> >
> > > rcu_read_lock();
> > > if (oif) {
> > > dev = dev_get_by_index_rcu(net, oif);
> > > rtnl_lock();
> > > mdev = netdev_master_upper_dev_get(dev);
> > > rtnl_unlock();
> > > }
> >
> > This is invalid: rtnl_lock() uses a mutex, so it can sleep and that's
> > forbidden inside an RCU critical section.
>
> Obviously, that's bad. Mea culpa.
>
> > > if (sk->sk_bound_dev_if) {
> > > bdev = dev_get_by_index_rcu(net, sk->sk_bound_dev_if);
> > > }
> > >
> > > addr_type = ipv6_addr_type(daddr);
> > > if ((__ipv6_addr_needs_scope_id(addr_type) && !oif) ||
> > > (addr_type & IPV6_ADDR_MAPPED) ||
> > > (oif && sk->sk_bound_dev_if && oif != sk->sk_bound_dev_if &&
> > > !(mdev && sk->sk_bound_dev_if && bdev && mdev == bdev))) {
> > > rcu_read_unlock();
> > > return -EINVAL;
> > > }
> > > rcu_read_unlock();
> > >
> > > But again this is still probably not race-free (bdev might also disappear before
> > > the mdev == bdev test), even if it passed fcnal-test.sh, there is much duplication
> > > of code, so your one-line solution is obviously by far better. :-)
> >
> > The real problem is choosing the right function for getting the master
> > device. In particular netdev_master_upper_dev_get() was a bad choice.
> > It forces you to take the rtnl, which is unnatural here and obliges you
> > to add extra code, while all this shouldn't be necessary in the first
> > place.
>
> Thank you for the additional insight. I had poor luck with Googling on
> these.
>
> I made a blunder after blunder. But it was insightful and brainstorming.
> Good exercise for my little grey cells.
>
> However, learning without making any errors appears to be simply a lot
> of blunt memorising. :-/
>
> It's good to be in an environment when one can learn from errors.
>
> :-)

I'm happy you found this useful.

> Regards,
> Mirsad
>

2023-06-06 22:36:31

On 6/14/23 10:47, Guillaume Nault wrote:
> On Sat, Jun 10, 2023 at 08:04:02PM +0200, Mirsad Goran Todorovac wrote:
>> This also works on the Lenovo IdeaPad 3 Ubuntu 22.10 laptop, but on the AlmaLinux 8.8
>> Lenovo desktop I have a problem:
>>
>> [root@pc-mtodorov net]# grep FAIL ../fcnal-test-4.log
>> TEST: ping local, VRF bind - ns-A IP [FAIL]
>> TEST: ping local, VRF bind - VRF IP [FAIL]
>> TEST: ping local, device bind - ns-A IP [FAIL]
>> TEST: ping local, VRF bind - ns-A IP [FAIL]
>> TEST: ping local, VRF bind - VRF IP [FAIL]
>> TEST: ping local, device bind - ns-A IP [FAIL]
>> [root@pc-mtodorov net]#
>>
>> Kernel is the recent one:
>>
>> [root@pc-mtodorov net]# uname -rms
>> Linux 6.4.0-rc5-testnet-00003-g5b23878f7ed9 x86_64
>> [root@pc-mtodorov net]#
>
> Maybe a problem with the ping version used by the distribution.
> You can try "./fcnal-test.sh -t ipv4_ping -p -v" to view the commands
> run and make the script stop when there's a test failure (so that you
> can see the ping output and try your own commands in the testing
> environment).

Thank you for taking the time for the reply. And thanks for the hint.
But I am sort of on ebb tide on this.

It would be good to have the test run on both versions of Linux to test
the actual kernel faults. Maybe pack a version of ping command w the test?
But I cannot deploy too much time in this.

I hope then the upgrade AlmaLinux 8.8 -> 9.x (or CentOS clones in general)
would solve the issue, but it is not guaranteed, and I would lose bisect
to the old kernels. Which is why I do not upgrade to the latest releases
in the first place. :-/

If it is just the AlmaLinux ping, then it is just an exotic distro, but it
is a CentOS clone, so the issue might exist in the more popular Rocky, too.

I am not sure what is the right way to do in this case or I would already
have done it. Presumptuous maybe, but true.

>>>> However, I have a question:
>>>>
>>>> In the ping + "With VRF" section, the tests with net.ipv4.raw_l3mdev_accept=1
>>>> are repeated twice, while "No VRF" section has the versions:
>>>>
>>>> SYSCTL: net.ipv4.raw_l3mdev_accept=0
>>>>
>>>> and
>>>>
>>>> SYSCTL: net.ipv4.raw_l3mdev_accept=1
>>>>
>>>> The same happens with the IPv6 ping tests.
>>>>
>>>> In that case, it could be that we have only 2 actual FAIL cases,
>>>> because the error is reported twice.
>>>>
>>>> Is this intentional?
>>>
>>> I don't know why the non-VRF tests are run once with raw_l3mdev_accept=0
>>> and once with raw_l3mdev_accept=1. Unless I'm missing something, this
>>> option shouldn't affect non-VRF users. Maybe the objective is to make
>>> sure that it really doesn't affect them. David certainly knows better.
>>
>> The problem appears to be that non-VRF tests are being ran with
>> raw_l3mdev_accept={0|1}, while VRF tests w raw_l3mdev_accept={1|1} ...
>
> The reason the VRF tests run twice is to test both raw and ping sockets
> (using the "net.ipv4.ping_group_range" sysctl). It doesn't seem anyone
> ever intended to run the VRF tests with raw_l3mdev_accept=0.
>
> Only the non-VRF tests were intended to be tested with
> raw_l3mdev_accept=0 (see commit c032dd8cc7e2 ("selftests: Add ipv4 ping
> tests to fcnal-test")). But I have no idea why.

Well, you are not to blame if it is not documented.

This thing doesn't come out of the testsuite save by prayer and fasting,
I'm afraid ;-)

Best regards,
Mirsad