Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <20221218234801.579114-1-jmaxwell37@gmail.com> <9f145202ca6a59b48d4430ed26a7ab0fe4c5dfaf.camel@redhat.com>
 <bf56c3aa-85df-734d-f419-835a35e66e03@kernel.org> <CAGHK07BehyHXoS+27=cfZoKz4XNTcJjyB5us33sNS7P+_fudHQ@mail.gmail.com>
 <CAGHK07D2Dy4zFGHqwdyg+nsRC_iL4ArWTPk7L2ndA2PaLfOMYQ@mail.gmail.com>
In-Reply-To: <CAGHK07D2Dy4zFGHqwdyg+nsRC_iL4ArWTPk7L2ndA2PaLfOMYQ@mail.gmail.com>
From:   Jonathan Maxwell <jmaxwell37@gmail.com>
Date:   Thu, 22 Dec 2022 16:39:21 +1100
Message-ID: <CAGHK07DU15NhFvGuLB6WHUF0fffT3MefL3E3JWHmtWR6Wzm0bA@mail.gmail.com>
Subject: Re: [net-next] ipv6: fix routing cache overflow for raw sockets
To:     David Ahern <dsahern@kernel.org>
Cc:     Paolo Abeni <pabeni@redhat.com>, davem@davemloft.net,
        edumazet@google.com, kuba@kernel.org, yoshfuji@linux-ipv6.org,
        netdev@vger.kernel.org, linux-kernel@vger.kernel.org
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

On Wed, Dec 21, 2022 at 3:31 PM Jonathan Maxwell <jmaxwell37@gmail.com> wrote:
>
> On Wed, Dec 21, 2022 at 8:55 AM Jonathan Maxwell <jmaxwell37@gmail.com> wrote:
> >
> > On Wed, Dec 21, 2022 at 2:10 AM David Ahern <dsahern@kernel.org> wrote:
> > >
> > > On 12/20/22 5:35 AM, Paolo Abeni wrote:
> > > > On Mon, 2022-12-19 at 10:48 +1100, Jon Maxwell wrote:
> > > >> Sending Ipv6 packets in a loop via a raw socket triggers an issue where a
> > > >> route is cloned by ip6_rt_cache_alloc() for each packet sent. This quickly
> > > >> consumes the Ipv6 max_size threshold which defaults to 4096 resulting in
> > > >> these warnings:
> > > >>
> > > >> [1]   99.187805] dst_alloc: 7728 callbacks suppressed
> > > >> [2] Route cache is full: consider increasing sysctl net.ipv6.route.max_size.
> > > >> .
> > > >> .
> > > >> [300] Route cache is full: consider increasing sysctl net.ipv6.route.max_size.
> > > >
> > > > If I read correctly, the maximum number of dst that the raw socket can
> > > > use this way is limited by the number of packets it allows via the
> > > > sndbuf limit, right?
> > > >
> > > > Are other FLOWI_FLAG_KNOWN_NH users affected, too? e.g. nf_dup_ipv6,
> > > > ipvs, seg6?
> > > >
> > > > @DavidA: why do we need to create RTF_CACHE clones for KNOWN_NH flows?
> > > >
> > > > Thanks,
> > > >
> > > > Paolo
> > > >
> > >
> > > If I recall the details correctly: that sysctl limit was added back when
> > > ipv6 routes were managed as dst_entries and there was a desire to allow
> > > an admin to limit the memory consumed. At this point in time, IPv6 is
> > > more inline with IPv4 - a separate struct for fib entries from dst
> > > entries. That "Route cache is full" message is now out of date since
> > > this is dst_entries which have a gc mechanism.
> > >
> > > IPv4 does not limit the number of dst_entries that can be allocated
> > > (ip_rt_max_size is the sysctl variable behind the ipv4 version of
> > > max_size and it is a no-op). IPv6 can probably do the same here?
> > >
> >
> > diff --git a/net/ipv6/route.c b/net/ipv6/route.c
> > index dbc224023977..701aba7feaf5 100644
> > --- a/net/ipv6/route.c
> > +++ b/net/ipv6/route.c
> > @@ -6470,7 +6470,7 @@ static int __net_init ip6_route_net_init(struct net *net)
> >  #endif
> >
> >         net->ipv6.sysctl.flush_delay = 0;
> > -       net->ipv6.sysctl.ip6_rt_max_size = 4096;
> > +       net->ipv6.sysctl.ip6_rt_max_size = INT_MAX;
> >         net->ipv6.sysctl.ip6_rt_gc_min_interval = HZ / 2;
> >         net->ipv6.sysctl.ip6_rt_gc_timeout = 60*HZ;
> >         net->ipv6.sysctl.ip6_rt_gc_interval = 30*HZ;
> >
> > The above patch resolved it for the Ipv6 reproducer.
> >
> > Would that be sufficient?
> >
>
> Otherwise if you prefer to make Ipv6 behaviour similar to IPv4.
> Rather than upping max_size.
>
> Here is prototype patch that removes the max_size check for Ipv6:
>

There are some mistakes in this prototype patch. Let me come up with a
better one. I'll submit a new patch in the new year for review. Thanks for
the suggestion DavidA.

Regards

Jon

> diff --git a/include/net/dst_ops.h b/include/net/dst_ops.h
> index 88ff7bb2bb9b..632086b2f644 100644
> --- a/include/net/dst_ops.h
> +++ b/include/net/dst_ops.h
> @@ -16,7 +16,7 @@ struct dst_ops {
>         unsigned short          family;
>         unsigned int            gc_thresh;
>
> -       int                     (*gc)(struct dst_ops *ops);
> +       void                    (*gc)(struct dst_ops *ops);
>         struct dst_entry *      (*check)(struct dst_entry *, __u32 cookie);
>         unsigned int            (*default_advmss)(const struct dst_entry *);
>         unsigned int            (*mtu)(const struct dst_entry *);
> diff --git a/net/core/dst.c b/net/core/dst.c
> index 497ef9b3fc6a..dcb85267bc4c 100644
> --- a/net/core/dst.c
> +++ b/net/core/dst.c
> @@ -82,12 +82,8 @@ void *dst_alloc(struct dst_ops *ops, struct net_device *dev,
>
>         if (ops->gc &&
>             !(flags & DST_NOCOUNT) &&
> -           dst_entries_get_fast(ops) > ops->gc_thresh) {
> -               if (ops->gc(ops)) {
> -                       pr_notice_ratelimited("Route cache is full:
> consider increasing sysctl net.ipv6.route.max_size.\n");
> -                       return NULL;
> -               }
> -       }
> +           dst_entries_get_fast(ops) > ops->gc_thresh)
> +               ops->gc(ops);
>
>         dst = kmem_cache_alloc(ops->kmem_cachep, GFP_ATOMIC);
>         if (!dst)
> diff --git a/net/ipv6/route.c b/net/ipv6/route.c
> index dbc224023977..8db7c5436da4 100644
> --- a/net/ipv6/route.c
> +++ b/net/ipv6/route.c
> @@ -91,7 +91,7 @@ static struct dst_entry *ip6_negative_advice(struct
> dst_entry *);
>  static void            ip6_dst_destroy(struct dst_entry *);
>  static void            ip6_dst_ifdown(struct dst_entry *,
>                                        struct net_device *dev, int how);
> -static int              ip6_dst_gc(struct dst_ops *ops);
> +static void             ip6_dst_gc(struct dst_ops *ops);
>
>  static int             ip6_pkt_discard(struct sk_buff *skb);
>  static int             ip6_pkt_discard_out(struct net *net, struct
> sock *sk, struct sk_buff *skb);
> @@ -3295,32 +3295,21 @@ struct dst_entry *icmp6_dst_alloc(struct
> net_device *dev,
>         return dst;
>  }
>
> -static int ip6_dst_gc(struct dst_ops *ops)
> +static void ip6_dst_gc(struct dst_ops *ops)
>  {
>         struct net *net = container_of(ops, struct net, ipv6.ip6_dst_ops);
> -       int rt_min_interval = net->ipv6.sysctl.ip6_rt_gc_min_interval;
> -       int rt_max_size = net->ipv6.sysctl.ip6_rt_max_size;
>         int rt_elasticity = net->ipv6.sysctl.ip6_rt_gc_elasticity;
>         int rt_gc_timeout = net->ipv6.sysctl.ip6_rt_gc_timeout;
> -       unsigned long rt_last_gc = net->ipv6.ip6_rt_last_gc;
>         int entries;
>
>         entries = dst_entries_get_fast(ops);
> -       if (entries > rt_max_size)
> -               entries = dst_entries_get_slow(ops);
> -
> -       if (time_after(rt_last_gc + rt_min_interval, jiffies) &&
> -           entries <= rt_max_size)
> -               goto out;
>
>         net->ipv6.ip6_rt_gc_expire++;
>         fib6_run_gc(net->ipv6.ip6_rt_gc_expire, net, true);
>         entries = dst_entries_get_slow(ops);
>         if (entries < ops->gc_thresh)
>                 net->ipv6.ip6_rt_gc_expire = rt_gc_timeout>>1;
> -out:
>         net->ipv6.ip6_rt_gc_expire -= net->ipv6.ip6_rt_gc_expire>>rt_elasticity;
> -       return entries > rt_max_size;
>  }
>
>  static int ip6_nh_lookup_table(struct net *net, struct fib6_config *cfg,
>
> > > I do not believe the suggested flag is the right change.
> >
> > Regards
> >
> > Jon