Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <737440541.1127428.1656698294694.JavaMail.zimbra@desy.de> <05f3b4e144ec5d12ab87d861222128181e805321.camel@hammerspace.com>
In-Reply-To: <05f3b4e144ec5d12ab87d861222128181e805321.camel@hammerspace.com>
From:   Daire Byrne <daire@dneg.com>
Date:   Fri, 1 Jul 2022 22:51:51 +0100
Message-ID: <CAPt2mGO3HsM6ixecvNioZ=jNCNBZ-DuPWmq+LzEnzJdR3McC9A@mail.gmail.com>
Subject: Re: Per user rate limiter
To:     Trond Myklebust <trondmy@hammerspace.com>
Cc:     "tigran.mkrtchyan@desy.de" <tigran.mkrtchyan@desy.de>,
        "linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

On Fri, 1 Jul 2022 at 19:23, Trond Myklebust <trondmy@hammerspace.com> wrote:
> 2) Define QoS policies for the connections using the kernel Traffic

If it helps, we use HTB qdisc/classes on our Linux NFS servers to
optionally limit the total egress and ingress (ifb) bandwidth to/from
our renderfarm.

User workstations are exempt from these limits so always get full speed.

We can do this fairly easily because our network is well defined and
split into subnet ranges so filtering by these allows us to
differentiate between host classes (farm/workstations etc).

Strictly speaking, it's a bit more complicated in that we only apply
limits and change them dynamically based on the "load" of the server
and how well it is keeping up with demand. This is just a bash script
running in a loop looking at the state, scaling the HTB limits and
applying filters.

Our goal is to always ensure that taff have a good experience on their
interactive desktops and we'll happily slow batch farm jobs to keep it
that way.

It is basically a low-pass filter that limits server load spikes.

To do something similar by user or process, you could run your jobs in
a cgroup and have it mark the packets that the server could then use
to filter. But I think this only works for the client writes to the
server as you have no way to mark and act on the egress packets out of
the server?

Daire