Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Message-ID: <5fffbce3-2722-97b9-c025-1ce3da5e5467@bluematt.me>
Date:   Mon, 19 Apr 2021 13:20:39 -0400
MIME-Version: 1.0
Subject: Re: PROBLEM: DoS Attack on Fragment Cache
Content-Language: en-US
To:     Eric Dumazet <edumazet@google.com>
Cc:     Willy Tarreau <w@1wt.eu>, Keyu Man <kman001@ucr.edu>,
        David Ahern <dsahern@gmail.com>,
        Florian Westphal <fw@strlen.de>,
        David Miller <davem@davemloft.net>,
        Hideaki YOSHIFUJI <yoshfuji@linux-ipv6.org>,
        David Ahern <dsahern@kernel.org>,
        Jakub Kicinski <kuba@kernel.org>,
        netdev <netdev@vger.kernel.org>,
        LKML <linux-kernel@vger.kernel.org>,
        Zhiyun Qian <zhiyunq@cs.ucr.edu>
References: <02917697-4CE2-4BBE-BF47-31F58BC89025@hxcore.ol>
 <52098fa9-2feb-08ae-c24f-1e696076c3b9@gmail.com>
 <CANn89iL_V0WbeA-Zr29cLSp9pCsthkX9ze4W46gx=8-UeK2qMg@mail.gmail.com>
 <20210417072744.GB14109@1wt.eu>
 <CAMqUL6bkp2Dy3AMFZeNLjE1f-sAwnuBWpXH_FSYTSh8=Ac3RKg@mail.gmail.com>
 <20210417075030.GA14265@1wt.eu>
 <c6467c1c-54f5-8681-6e7d-aa1d9fc2ff32@bluematt.me>
 <CAMqUL6bAVE9p=XEnH4HdBmBfThaY3FDosqyr8yrQo6N_9+Jf3w@mail.gmail.com>
 <78d776a9-4299-ff4e-8ca2-096ec5c02d05@bluematt.me>
 <20210418043933.GB18896@1wt.eu>
 <9e2966be-d210-edf9-4f3c-5681f0d07c5f@bluematt.me>
 <CANn89iKXYutm20oi-rCwch0eL1Oo9rq1W=ex6+NzvPitq_jX0Q@mail.gmail.com>
From:   Matt Corallo <netdev-list@mattcorallo.com>
In-Reply-To: <CANn89iKXYutm20oi-rCwch0eL1Oo9rq1W=ex6+NzvPitq_jX0Q@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk

Note that there are two completely separate sysctls here - the timeout on fragments, and the amount of memory available 
for fragment reassembly. You have to multiply them together to reach the "Mbps of lost or deliberately-lost fragments 
before we start dropping all future fragments". See the calculation in the description of the patch I mentioned above 
for exact details, but turning the time down to 1s already gives you 32Mbps, and you can tune the memory usage 
separately (eg 128MB, really 256 between v4 and v6, would give you 1Gbps of "lost" fragments).

Its true, an attacker can use a lot of memory in that case, but 128MiB isn't actually something that rises to the level 
of "trivial for an attacker to use all the memory you allowed" or "cause OOM".

I only chimed in on this thread to note that this isn't just a theoretical attack concern, however - this is a 
real-world non-attack-scenario issue that's pretty trivial to hit. Just losing 1Mbps of traffic on a modern residential 
internet connection is pretty doable, make that flow mostly frags and suddenly your VPN drops out for 30 seconds at a 
time just because.

I agree with others here that actually solving the DoS issue isn't trivial, but making it less absurdly trivial to have 
30 second dropouts of your VPN connection would also be a nice change.

Matt

On 4/19/21 05:43, Eric Dumazet wrote:
> On Sun, Apr 18, 2021 at 4:31 PM Matt Corallo
> <netdev-list@mattcorallo.com> wrote:
>>
>> Should the default, though, be so low? If someone is still using a old modem they can crank up the sysctl, it does seem
>> like such things are pretty rare these days :). Its rather trivial to, without any kind of attack, hit 1Mbps of lost
>> fragments in today's networks, at which point all fragments are dropped. After all, I submitted the patch to "scratch my
>> own itch" :).
> 
> Again, even if you increase the values by 1000x, it is trivial for an
> attacker to use all the memory you allowed.
> 
> And allowing a significant portion of memory to be eaten like that
> might cause OOM on hosts where jobs are consuming all physical memory.
> 
> It is a sysctl, I changed things so that one could really reserve/use
> 16GB of memory if she/he is desperate about frags.
> 
>>
>> Matt
>>
>> On 4/18/21 00:39, Willy Tarreau wrote:
>>> I do agree that we shouldn't keep them that long nowadays, we can't go
>>> too low without risking to break some slow transmission stacks (SLIP/PPP
>>> over modems for example).