Create VLAN devices and add default gateways with following commands:
# ip link add link eth2 dev eth2.71 type vlan id 71
# ip link add link eth2 dev eth2.72 type vlan id 72
# ip addr add 192.168.71.41/24 dev eth2.71
# ip addr add 192.168.72.41/24 dev eth2.72
# ip link set eth2.71 up
# ip link set eth2.72 up
# route add -net default gw 192.168.71.1 dev eth2.71
# route add -net default gw 192.168.72.1 dev eth2.72
Add a nameserver configuration in the following file:
# cat /etc/resolv.conf
nameserver 8.8.8.8
Use the following command trigger DNS packet:
# ping http://www.baidu.com
Assume the above test machine is client.
Of course, we should also create VLAN devices in peer server as following:
# ip link add link eth2 dev eth2.71 type vlan id 71
# ip link add link eth2 dev eth2.72 type vlan id 72
# ip addr add 192.168.71.1/24 dev eth2.71
# ip addr add 192.168.72.1/24 dev eth2.72
# ip link set eth2.71 up
# ip link set eth2.72 up
We capture packets with tcpdump in client machine when ping:
# tcpdump -i eth2 -vne
...
20:30:22.996044 52:54:00:20:23:a9 > 52:54:00:d2:4f:e3, ethertype 802.1Q (0x8100), length 77: vlan 71, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 25407, offset 0, flags [DF], proto UDP (17), length 59)
192.168.72.41.42666 > 8.8.8.8.domain: 58562+ A? http://www.baidu.com. (31)
20:30:22.996125 52:54:00:20:23:a9 > 52:54:00:d2:4f:e3, ethertype 802.1Q (0x8100), length 77: vlan 71, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 25408, offset 0, flags [DF], proto UDP (17), length 59)
192.168.72.41.42666 > 8.8.8.8.domain: 25803+ AAAA? http://www.baidu.com. (31)
...
We can find that IPv4 saddr "192.168.72.41" do not match with selected VLAN device "eth2.71".
I tracked the related processes, and found that user space program uses connect() firstly, then sends UDP packet.
The problem happens in the connect() process. Analysis as following with codes:
static inline struct rtable *ip_route_connect(struct flowi4 *fl4,
__be32 dst, __be32 src, u32 tos,
int oif, u8 protocol,
__be16 sport, __be16 dport,
struct sock *sk)
{
struct net *net = sock_net(sk);
struct rtable *rt;
ip_route_connect_init(fl4, dst, src, tos, oif, protocol,
sport, dport, sk);
if (!dst || !src) {
/* rtable and fl4 are matched after the first __ip_route_output_key().
* rtable->dst.dev->name == "eth2.72" && rtable->rt_gw4 == 0x148a8c0
* fl4->saddr == 0x2948a8c0
*/
rt = __ip_route_output_key(net, fl4);
if (IS_ERR(rt))
return rt;
ip_rt_put(rt);
flowi4_update_output(fl4, oif, tos, fl4->daddr, fl4->saddr);
}
security_sk_classify_flow(sk, flowi4_to_flowi_common(fl4));
/* rtable and fl4 do not match after the second __ip_route_output_key().
* rtable->dst.dev->name == "eth2.71" && rtable->rt_gw4 == 0x147a8c0
* fl4->saddr == 0x2948a8c0
*/
return ip_route_output_flow(net, fl4, sk);
}
Deep tracking, it because fa->fa_default has changed in fib_select_default() after first __ip_route_output_key() process,
and a new fib_nh is selected in fib_select_default() within the second __ip_route_output_key() process but not update flowi4.
So the phenomenon described at the beginning happens.
Does it a kernel bug or a user problem? If it is a kernel bug, is there any good solution?
> On 3/7/22 11:41 PM, Ziyang Xuan (William) wrote:
>> Create VLAN devices and add default gateways with following commands:
>>
>> # ip link add link eth2 dev eth2.71 type vlan id 71
>> # ip link add link eth2 dev eth2.72 type vlan id 72
>> # ip addr add 192.168.71.41/24 dev eth2.71
>> # ip addr add 192.168.72.41/24 dev eth2.72
>> # ip link set eth2.71 up
>> # ip link set eth2.72 up
>> # route add -net default gw 192.168.71.1 dev eth2.71
>> # route add -net default gw 192.168.72.1 dev eth2.72
>>
>
> ...
>
>> We can find that IPv4 saddr "192.168.72.41" do not match with selected VLAN device "eth2.71".
>>
>> I tracked the related processes, and found that user space program uses connect() firstly, then sends UDP packet.
>>
>
> ...
>
>> Deep tracking, it because fa->fa_default has changed in fib_select_default() after first __ip_route_output_key() process,
>> and a new fib_nh is selected in fib_select_default() within the second __ip_route_output_key() process but not update flowi4.
>> So the phenomenon described at the beginning happens.
>>
>> Does it a kernel bug or a user problem? If it is a kernel bug, is there any good solution?
>
> That is a known problem with multipath routes.
> .
>
Does the community have a plan to address it?
> Create VLAN devices and add default gateways with following commands:
>
> # ip link add link eth2 dev eth2.71 type vlan id 71
> # ip link add link eth2 dev eth2.72 type vlan id 72
> # ip addr add 192.168.71.41/24 dev eth2.71
> # ip addr add 192.168.72.41/24 dev eth2.72
> # ip link set eth2.71 up
> # ip link set eth2.72 up
> # route add -net default gw 192.168.71.1 dev eth2.71
> # route add -net default gw 192.168.72.1 dev eth2.72
>
> Add a nameserver configuration in the following file:
> # cat /etc/resolv.conf
> nameserver 8.8.8.8
>
> Use the following command trigger DNS packet:
> # ping http://www.baidu.com
>
> Assume the above test machine is client.
>
> Of course, we should also create VLAN devices in peer server as following:
>
> # ip link add link eth2 dev eth2.71 type vlan id 71
> # ip link add link eth2 dev eth2.72 type vlan id 72
> # ip addr add 192.168.71.1/24 dev eth2.71
> # ip addr add 192.168.72.1/24 dev eth2.72
> # ip link set eth2.71 up
> # ip link set eth2.72 up
>
> We capture packets with tcpdump in client machine when ping:
> # tcpdump -i eth2 -vne
> ...
> 20:30:22.996044 52:54:00:20:23:a9 > 52:54:00:d2:4f:e3, ethertype 802.1Q (0x8100), length 77: vlan 71, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 25407, offset 0, flags [DF], proto UDP (17), length 59)
> 192.168.72.41.42666 > 8.8.8.8.domain: 58562+ A? http://www.baidu.com. (31)
> 20:30:22.996125 52:54:00:20:23:a9 > 52:54:00:d2:4f:e3, ethertype 802.1Q (0x8100), length 77: vlan 71, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 25408, offset 0, flags [DF], proto UDP (17), length 59)
> 192.168.72.41.42666 > 8.8.8.8.domain: 25803+ AAAA? http://www.baidu.com. (31)
> ...
>
> We can find that IPv4 saddr "192.168.72.41" do not match with selected VLAN device "eth2.71".
Is there anyone familiar with route/fib realization? And thank you for your warm-hearted help!
>
> I tracked the related processes, and found that user space program uses connect() firstly, then sends UDP packet.
>
> The problem happens in the connect() process. Analysis as following with codes:
>
> static inline struct rtable *ip_route_connect(struct flowi4 *fl4,
> __be32 dst, __be32 src, u32 tos,
> int oif, u8 protocol,
> __be16 sport, __be16 dport,
> struct sock *sk)
> {
> struct net *net = sock_net(sk);
> struct rtable *rt;
>
> ip_route_connect_init(fl4, dst, src, tos, oif, protocol,
> sport, dport, sk);
>
> if (!dst || !src) {
>
> /* rtable and fl4 are matched after the first __ip_route_output_key().
> * rtable->dst.dev->name == "eth2.72" && rtable->rt_gw4 == 0x148a8c0
> * fl4->saddr == 0x2948a8c0
> */
> rt = __ip_route_output_key(net, fl4);
> if (IS_ERR(rt))
> return rt;
> ip_rt_put(rt);
> flowi4_update_output(fl4, oif, tos, fl4->daddr, fl4->saddr);
> }
> security_sk_classify_flow(sk, flowi4_to_flowi_common(fl4));
>
> /* rtable and fl4 do not match after the second __ip_route_output_key().
> * rtable->dst.dev->name == "eth2.71" && rtable->rt_gw4 == 0x147a8c0
> * fl4->saddr == 0x2948a8c0
> */
> return ip_route_output_flow(net, fl4, sk);
> }
>
> Deep tracking, it because fa->fa_default has changed in fib_select_default() after first __ip_route_output_key() process,
> and a new fib_nh is selected in fib_select_default() within the second __ip_route_output_key() process but not update flowi4.
> So the phenomenon described at the beginning happens.
>
> Does it a kernel bug or a user problem? If it is a kernel bug, is there any good solution?
>
On 3/7/22 11:41 PM, Ziyang Xuan (William) wrote:
> Create VLAN devices and add default gateways with following commands:
>
> # ip link add link eth2 dev eth2.71 type vlan id 71
> # ip link add link eth2 dev eth2.72 type vlan id 72
> # ip addr add 192.168.71.41/24 dev eth2.71
> # ip addr add 192.168.72.41/24 dev eth2.72
> # ip link set eth2.71 up
> # ip link set eth2.72 up
> # route add -net default gw 192.168.71.1 dev eth2.71
> # route add -net default gw 192.168.72.1 dev eth2.72
>
...
> We can find that IPv4 saddr "192.168.72.41" do not match with selected VLAN device "eth2.71".
>
> I tracked the related processes, and found that user space program uses connect() firstly, then sends UDP packet.
>
...
> Deep tracking, it because fa->fa_default has changed in fib_select_default() after first __ip_route_output_key() process,
> and a new fib_nh is selected in fib_select_default() within the second __ip_route_output_key() process but not update flowi4.
> So the phenomenon described at the beginning happens.
>
> Does it a kernel bug or a user problem? If it is a kernel bug, is there any good solution?
That is a known problem with multipath routes.
On 3/9/22 8:20 PM, Ziyang Xuan (William) wrote:
> Does the community have a plan to address it?
i do not. someone with the knowledge and time is welcome to create a patch.