On Fri, 22 Apr 2022 01:40:15 -0400
Kent Overstreet <[email protected]> wrote:
> So I'm honestly not super eager to start modifying tricky arch code that I can't
> test, and digging into what looked like non trivial interactions between the way
> the traceing code using seq_buf (naturally, given that's where it originates).
Yes, seq_buf came from the tracing system but was to be used in a more
broader way. I had originally pushed trace_seq into the lib directory, but
Andrew Morton said it was too specific to tracing. Thus, I gutted the
generic parts out of it and created seq_buf, which looks to be something
that you could use. I had patches to convert seq_file to it, but ran out of
time. I probably can pull them out of the closet and start that again.
>
> Now yes, I _could_ do a wholesale conversion of seq_buf to printbuf and delete
> that code, but doing that job right, to be confident that I'm not introducing
> bugs, is going to take more time than I really want to invest right now. I
> really don't like to play fast and loose with that stuff.
I would be happy to work with you to convert to seq_buf. If there's
something missing from it, I can help you change it so that it doesn't
cause any regressions with the tracing subsystem.
This is how open source programming is suppose to work ;-)
-- Steve
Hi Steve!
On Fri, Apr 22, 2022 at 11:37:36AM -0400, Steven Rostedt wrote:
> On Fri, 22 Apr 2022 01:40:15 -0400
> Kent Overstreet <[email protected]> wrote:
>
> > So I'm honestly not super eager to start modifying tricky arch code that I can't
> > test, and digging into what looked like non trivial interactions between the way
> > the traceing code using seq_buf (naturally, given that's where it originates).
>
> Yes, seq_buf came from the tracing system but was to be used in a more
> broader way. I had originally pushed trace_seq into the lib directory, but
> Andrew Morton said it was too specific to tracing. Thus, I gutted the
> generic parts out of it and created seq_buf, which looks to be something
> that you could use. I had patches to convert seq_file to it, but ran out of
> time. I probably can pull them out of the closet and start that again.
>
> >
> > Now yes, I _could_ do a wholesale conversion of seq_buf to printbuf and delete
> > that code, but doing that job right, to be confident that I'm not introducing
> > bugs, is going to take more time than I really want to invest right now. I
> > really don't like to play fast and loose with that stuff.
>
> I would be happy to work with you to convert to seq_buf. If there's
> something missing from it, I can help you change it so that it doesn't
> cause any regressions with the tracing subsystem.
>
> This is how open source programming is suppose to work ;-)
Is it though? :)
One of the things I've been meaning to talk more about, that
came out of a recent Rust discussion, is that we in the kernel community could
really do a better job with how we interact with the outside world, particularly
with regards to the sharing of code.
The point was made to me when another long standing kernel dev was complaining
about Facebook being a large, insular, difficult to work with organization, that
likes to pretend it is the center of the universe and not bend to the outside
world, while doing the exact same thing with respect to new concerns brought by
the Rust community. The irony was illuminating :)
The reason I bring that up is that in this case, printbuf is the more evolved,
more widely used implementation, and you're asking me to discard it so the
kernel can stick with its more primitive, less widely used implementation.
$ git grep -w seq_buf|wc -l
86
$ git grep -w printbuf|wc -l
366
So, going to have to push back on that one :)
Printbufs aren't new code; everything in them is there because I've found it
valuable, which is why I decided to try promoting them to the kernel proper (and
more importantly, the idea of a standard way to pretty-print anything).
I'm happy to discuss the merits of the code more, and try to convince you why
you'll like them :)
On Fri, 2022-04-22 at 15:30 -0400, Kent Overstreet wrote:
> Hi Steve!
>
> On Fri, Apr 22, 2022 at 11:37:36AM -0400, Steven Rostedt wrote:
> > On Fri, 22 Apr 2022 01:40:15 -0400
> > Kent Overstreet <[email protected]> wrote:
[...]
> > > Now yes, I _could_ do a wholesale conversion of seq_buf to
> > > printbuf and delete that code, but doing that job right, to be
> > > confident that I'm not introducing bugs, is going to take more
> > > time than I really want to invest right now. I really don't like
> > > to play fast and loose with that stuff.
> >
> > I would be happy to work with you to convert to seq_buf. If there's
> > something missing from it, I can help you change it so that it
> > doesn't cause any regressions with the tracing subsystem.
> >
> > This is how open source programming is suppose to work ;-)
>
> Is it though? :)
>
> One of the things I've been meaning to talk more about, that
> came out of a recent Rust discussion, is that we in the kernel
> community could really do a better job with how we interact with the
> outside world, particularly with regards to the sharing of code.
>
> The point was made to me when another long standing kernel dev was
> complaining about Facebook being a large, insular, difficult to work
> with organization, that likes to pretend it is the center of the
> universe and not bend to the outside world, while doing the exact
> same thing with respect to new concerns brought by the Rust
> community. The irony was illuminating :)
Hey, I didn't say that at all. I said vendoring the facebook reference
implementation wouldn't work (it being 74k lines and us using 300) but
that facebook was doing the right thing for us with zstd because they
were maintaining the core code we needed, even if we couldn't vendor it
from their code base:
https://lore.kernel.org/rust-for-linux/ea85b3bce5f172dc73e2be8eb4dbd21fae826fa1.camel@HansenPartnership.com/
You were the one who said all that about facebook, while incorrectly
implying I said it first (which is an interesting variation on the
strawman fallacy):
https://lore.kernel.org/rust-for-linux/[email protected]/
James
On Fri, Apr 22, 2022 at 04:03:12PM -0400, James Bottomley wrote:
> Hey, I didn't say that at all. I said vendoring the facebook reference
> implementation wouldn't work (it being 74k lines and us using 300) but
> that facebook was doing the right thing for us with zstd because they
> were maintaining the core code we needed, even if we couldn't vendor it
> from their code base:
>
> https://lore.kernel.org/rust-for-linux/ea85b3bce5f172dc73e2be8eb4dbd21fae826fa1.camel@HansenPartnership.com/
>
> You were the one who said all that about facebook, while incorrectly
> implying I said it first (which is an interesting variation on the
> strawman fallacy):
Hey, sorry for picking on you James - I didn't mean to single you out.
Let me explain where I'm coming from - actually, this email I received puts it
better than I could:
From: "John Ericson" <[email protected]>
To: "Kent Overstreet" <[email protected]>
Subject: Nice email on cargo crates in Linux
Hi. I saw your email linked from https://lwn.net/Articles/889924/. Nicely put!
I was involved in some of the allocating try_* function work to avoid nasty
panics, and thus make sure Rust in Linux didn't have to reinvent the wheel
redoing the alloc library. I very much share your goals but suspect this will
happen in a sort of boil-the-frog way over time. The basic portability method of
"incentivize users to write code that's *more* portable than what they currently
need" is a culture shock for user space and kernel space alike. But if we get
all our ducks in a row on the Rust side, eventually it should just be "too
tempting", and the code sharing will happen for economic reasons (rather than
ideological ones about trying to smash down the arbitrary dichotomies :) going
it alone).
The larger issue is major projects closed off unto themselves such as Linux,
PostgreSQL, or even the Glasgow Haskell Compiler (something I am currently
writing a plan to untangle) have fallen too deep in Conway's law's embrace, and
thus are full of bad habits like
https://johno.com/composition-over-configuration is arguing against. Over time,
I hope using external libraries will not only result in less duplicated work,
but also "pry open" their architecture a bit, tamping down on crazy
configurability and replacing it with more principled composition.
I fear this is all way too spicy to bring up on the LKML in 2022, at least
coming from someone like me without any landed kernel patches under his belt,
but I wanted to let you know other people have similar thoughts.
Cheers,
John
-------
Re: Rust in Linux, I get frustrated when I see senior people tell the new Rust
people "don't do that" to things that are standard practice in the outside
world.
I think Linus said recently that Rust in the kernel is something that could
fail, and he's right - but if it fails, it won't just be the failure of the Rust
people to do the required work, it'll be _our_ failure too, a failure to work
with them.
And I think the kernel needs Rust, in the long term. Rust is the biggest advance
towards programming languages with embedded correctness proofs in the systems
space since ever, and that's huge. It's only a first step, it doesn't solve
everything - the next step will probably be dependent types (as in Idris); after
that it's hard to say. But maybe 50, 100 years out, systems programming is going
to look very different.
And that excites me! What I love about programming is being able to sink into
something new, difficult and unsolved, and let my mind wander and explore and
play with ideas until I come across a simple and elegant way of solving it.
What I hate about programming is starting to do that, and then an hour in
getting interrupted by a bug report in an area of the code that I should have
been done with. And then again. And then again.
But that's just the nature of the beast in C, with the current state of the art
of the tools we have. Proving C programs correct can be done (I understand Paul
McKenney is on the bleeding edge of that), but it's too expensive and time
consuming to be practical for most things. We need better languages.
And we've got people trying to work with us, so I hope we work with them and
make it happen. That's going to require us to update our thinking in some areas
- like with cargo.
C people are used to projects being monoliths, but in the modern world things
that in prior days would have required hacking the compiler are now done as
libraries - and we're going to want that code! We don't need to be
reimplementing rust bitvecs, or string parsing, or - I could make up a long,
long list. That code is all going to just work in kernel land, and if we adopt
it it'll lead to a lot less duplication of effort.
Also, what John said in that email about "large projects closed off unto
themselves" - he's spot on about that. When I was at Google, their was this deep
unspoken assumption that anything done at Google was better than anything in the
outside world, and it _rankled_. It was a major reason I departed.
The kernel community has a lot of that going on here. Again, sorry to pick on
you James, but I wanted to make the argument that - maybe the kernel _should_ be
adopting a more structured way of using code from outside repositories, like
cargo, or git submodules (except I've never had a positive experience with git
submodules, so ignore that suggestion, unless I've just been using them wrong,
in which case someone please teach me). To read you and Greg saying "nah, just
copy code from other repos, it's fine" - it felt like being back in the old days
when we were still trying to get people to use source control, and having that
one older colleague who _insisted_ on not using source control of any kind, and
that's a bit disheartening.
I just want to make both the kernel and the wider world of open source software
a better place, and the thing I'm itching to see get better is that currently
it's a massive hassle to share code between kernel space and user space. This is
understandable, because before git we definitely didn't have the tooling, and in
C this kind of code sharing has historically been questionable.
But:
- We've got a new language coming that makes this sort of thing a lot saner,
both from a source code management perspective (cargo) and with much stronger
language guarantees that code won't do something surprising on you
- Even in C, we're gaining a better appreciation for why someone would want to
do such a thing.
Another random case in point: liblwipv6, a userspace TCP/IP library, lets you
do really cool things with building/simulating networks entirely in userspace
without root access, just over unix domain sockets. But it's a completely new
implementation, and thus buggy and incomplete. Imagine if, in some far away
future, we could someday just reuse the whole Linux network stack in
userspace, for purposes undreamed of by the likes of us...
And also, personally I find that code gets better when you have to spend the
effort to clean it up, make it more modular, and separate it from the wider
project - that's when you find out what it's really about. There's a ton of
_really effing cool code_ in the kernel that could be seeing wider use in
userspace, too (workqueues!). And, like with pretty printers, once you start
down a good path it often begets more of itself - modularity will lead to more
modularity. If we start making thinks like workqueues more modular and also
usable as a userspace library, that's going to make it easier to port more
kernel code, and more... which will make writing C in userspace a nicer
experience, and maybe even lead to more code sharing in the reverse direction..
Anyways, just one man's dream :)
[change of subject for an easy thread kill, since I'm sure most people
on cc don't want to follow this]
On Fri, 2022-04-22 at 17:13 -0400, Kent Overstreet wrote:
> On Fri, Apr 22, 2022 at 04:03:12PM -0400, James Bottomley wrote:
> > Hey, I didn't say that at all. I said vendoring the facebook
> > reference implementation wouldn't work (it being 74k lines and us
> > using 300) but that facebook was doing the right thing for us with
> > zstd because they were maintaining the core code we needed, even if
> > we couldn't vendor it from their code base:
> >
> > https://lore.kernel.org/rust-for-linux/ea85b3bce5f172dc73e2be8eb4dbd21fae826fa1.camel@HansenPartnership.com/
> >
> > You were the one who said all that about facebook, while
> > incorrectly implying I said it first (which is an interesting
> > variation on the strawman fallacy):
>
> Hey, sorry for picking on you James - I didn't mean to single you
> out.
>
> Let me explain where I'm coming from - actually, this email I
> received puts it better than I could:
>
> From: "John Ericson" <[email protected]>
> To: "Kent Overstreet" <[email protected]>
> Subject: Nice email on cargo crates in Linux
>
> Hi. I saw your email linked from https://lwn.net/Articles/889924/.
> Nicely put!
>
> I was involved in some of the allocating try_* function work to avoid
> nasty panics, and thus make sure Rust in Linux didn't have to
> reinvent the wheel redoing the alloc library. I very much share your
> goals but suspect this will happen in a sort of boil-the-frog way
> over time. The basic portability method of "incentivize users to
> write code that's *more* portable than what they currently
> need" is a culture shock for user space and kernel space alike. But
> if we get all our ducks in a row on the Rust side, eventually it
> should just be "too tempting", and the code sharing will happen for
> economic reasons (rather than ideological ones about trying to smash
> down the arbitrary dichotomies :) going it alone).
>
> The larger issue is major projects closed off unto themselves such as
> Linux, PostgreSQL, or even the Glasgow Haskell Compiler (something I
> am currently writing a plan to untangle) have fallen too deep in
> Conway's law's embrace, and thus are full of bad habits like
> https://johno.com/composition-over-configuration is arguing against.
> Over time, I hope using external libraries will not only result in
> less duplicated work, but also "pry open" their architecture a bit,
> tamping down on crazy configurability and replacing it with more
> principled composition.
>
> I fear this is all way too spicy to bring up on the LKML in 2022, at
> least coming from someone like me without any landed kernel patches
> under his belt, but I wanted to let you know other people have
> similar thoughts.
>
> Cheers,
>
> John
>
> -------
>
> Re: Rust in Linux, I get frustrated when I see senior people tell the
> new Rust people "don't do that" to things that are standard practice
> in the outside world.
You stripped the nuance of that. I said many no_std crates could be
used in the kernel. I also said that the async crate couldn't because
the rust compiler itself would have to support the kernel threading
model.
> I think Linus said recently that Rust in the kernel is something that
> could fail, and he's right - but if it fails, it won't just be the
> failure of the Rust people to do the required work, it'll be _our_
> failure too, a failure to work with them.
The big risk is that rust needs to adapt to the kernel environment.
This isn't rust specific, llvm had similar issues as an alternative C
compiler. I think rust in the kernel would fail if it were only the
rust kernel people asking. Fortunately the pressure to support rust in
embedded leading to the rise in no_std crates is a force which can also
get rust in the kernel over the finish line because of the focus it
puts on getting the language and crates to adapt to non standard
environments.
[...]
> The kernel community has a lot of that going on here. Again, sorry to
> pick on you James, but I wanted to make the argument that - maybe the
> kernel _should_ be adopting a more structured way of using code from
> outside repositories, like cargo, or git submodules (except I've
> never had a positive experience with git submodules, so ignore that
> suggestion, unless I've just been using them wrong, in which case
> someone please teach me). To read you and Greg saying "nah, just
> copy code from other repos, it's fine" - it felt like being back in
> the old days when we were still trying to get people to use source
> control, and having that one older colleague who _insisted_ on not
> using source control of any kind, and that's a bit disheartening.
Even in C terms, the kernel is a nostdlib environment. If a C project
has too much libc dependency it's not going to work directly in the
kernel, nor should it. Let's look at zstd (which is pretty much a
nostdlib project) as a great example: the facebook people didn't
actually port the top of their tree (1.5) to the kernel, they
backported bug fixes to the 1.4 branch and made a special release
(1.4.10) just for us. Why did they do this? It was because the 1.5
version vastly increased stack use to the extent it would run off the
end of the limited kernel stack so couldn't be ported directly into the
kernel. A lot of C libraries that are nostdlib have problems like this
as well (you can use recursion, but not in the kernel). There's no
easy way of shimming environmental constraints like this.
The lesson: it is possible to make the core of a project mobile, but
only if you're aware of all the environmental constraints it will run
into as it gets ported. The list of possible environments is huge:
kernel, embedded, industrial control ..., so naturally not every (or
more accurately hardly any) project wants to do this.
James
On Sat, Apr 23, 2022 at 10:16:37AM -0400, James Bottomley wrote:
> You stripped the nuance of that. I said many no_std crates could be
> used in the kernel. I also said that the async crate couldn't because
> the rust compiler itself would have to support the kernel threading
> model.
I just scanned through that thread and that's not what you said. What you said
was:
> The above is also the rust crate problem in miniature: the crates grow
> API features the kernel will never care about and importing them
> wholesale is going to take forever because of the internal kernel
> support issue. In the end, to take rust async as an example, it will
> be much better to do for rust what we've done for zlib: take the core
> that can support the kernel threading model and reimplement that in the
> kernel crate. The act of doing that will a) prove people care enough
> about the functionality and b) allow us to refine it nicely.
>
> I also don't think rust would really want to import crates wholesale.
> The reason for no_std is that rust is trying to adapt to embedded
> environments, which the somewhat harsh constraints of the kernel is
> very similar to.
But maybe your position has changed somewhat? It sounds like you've been
arguing against just directly depending on foreign reposotories and for the
staus quo of just ad-hoc copying of code.
I'll help by stating my own position: I think we should be coming up with a
process for how dependencies on other git repositories are going to work,
something better than just cut and paste. Whether or not we vendorize code isn't
really that important, but I'd say that if we are vendorizing code and we're not
including entire sub-repositories (like cargo vendor does) we ought to still
make this a scripted process that takes as an input a list of files we're
pulling and a remote repository we're pulling from, and the file list and the
remote repo (and commit ID we're pulling from) should all be checked in.
I think using cargo would be _great_ because it would handle this part for us
(perhaps minus pulling of individual files? haven't checked) instead of
home-growing our own. However, I'd like something for C repositories too, and I
don't think we want to start depending on cargo for non Rust development... :)
But much more important that the technical details of how we import code is just
having good answers for people who aren't embedded in Linux kernel development
culture, so we aren't just telling them "no" by default because we haven't
thought about this stuff and having them walk away frustrated.
> > I think Linus said recently that Rust in the kernel is something that
> > could fail, and he's right - but if it fails, it won't just be the
> > failure of the Rust people to do the required work, it'll be _our_
> > failure too, a failure to work with them.
>
> The big risk is that rust needs to adapt to the kernel environment.
> This isn't rust specific, llvm had similar issues as an alternative C
> compiler. I think rust in the kernel would fail if it were only the
> rust kernel people asking. Fortunately the pressure to support rust in
> embedded leading to the rise in no_std crates is a force which can also
> get rust in the kernel over the finish line because of the focus it
> puts on getting the language and crates to adapt to non standard
> environments.
It's both! It's on all of us to make this work.
> > The kernel community has a lot of that going on here. Again, sorry to
> > pick on you James, but I wanted to make the argument that - maybe the
> > kernel _should_ be adopting a more structured way of using code from
> > outside repositories, like cargo, or git submodules (except I've
> > never had a positive experience with git submodules, so ignore that
> > suggestion, unless I've just been using them wrong, in which case
> > someone please teach me). To read you and Greg saying "nah, just
> > copy code from other repos, it's fine" - it felt like being back in
> > the old days when we were still trying to get people to use source
> > control, and having that one older colleague who _insisted_ on not
> > using source control of any kind, and that's a bit disheartening.
>
> Even in C terms, the kernel is a nostdlib environment. If a C project
> has too much libc dependency it's not going to work directly in the
> kernel, nor should it. Let's look at zstd (which is pretty much a
> nostdlib project) as a great example: the facebook people didn't
> actually port the top of their tree (1.5) to the kernel, they
> backported bug fixes to the 1.4 branch and made a special release
> (1.4.10) just for us. Why did they do this? It was because the 1.5
> version vastly increased stack use to the extent it would run off the
> end of the limited kernel stack so couldn't be ported directly into the
> kernel. A lot of C libraries that are nostdlib have problems like this
> as well (you can use recursion, but not in the kernel). There's no
> easy way of shimming environmental constraints like this.
I wonder if we might have come up with a better solution if there'd been more
cross-project communication and less siloing. Small stacks aren't particular to
the kernel - it's definitely not unheard of to write userspace code where you
want to have a lot of small stacks (especially if you're doing some kind of
coroutine style threading; I've done stuff like this in the past) - and to me,
as someone who's been incrementing on and maintaining a codebase in active use
for 10 years, having multiple older versions in active use that need bugfixes
gives me cold shivers.
I wouldn't be surprised if at some point the zstd people walk back some of their
changes or make it configurable at some point :)
On Sun, 2022-04-24 at 16:36 -0400, Kent Overstreet wrote:
> On Sat, Apr 23, 2022 at 10:16:37AM -0400, James Bottomley wrote:
> > You stripped the nuance of that. I said many no_std crates could
> > be used in the kernel. I also said that the async crate couldn't
> > because the rust compiler itself would have to support the kernel
> > threading model.
>
> I just scanned through that thread and that's not what you said. What
> you said was:
>
> > The above is also the rust crate problem in miniature: the crates
> > grow API features the kernel will never care about and importing
> > them wholesale is going to take forever because of the internal
> > kernel support issue. In the end, to take rust async as an
> > example, it will be much better to do for rust what we've done for
> > zlib: take the core that can support the kernel threading model and
> > reimplement that in the kernel crate. The act of doing that will
> > a) prove people care enough about the functionality and b) allow us
> > to refine it nicely.
> >
> > I also don't think rust would really want to import crates
> > wholesale. The reason for no_std is that rust is trying to adapt
> > to embedded environments, which the somewhat harsh constraints of
> > the kernel is very similar to.
>
> But maybe your position has changed somewhat?
I read those two as saying the same thing just with differing levels of
detail.
> It sounds like you've been arguing against just directly depending
> on foreign reposotories and for the staus quo of just ad-hoc copying
> of code.
I don't think I've said anything generalised about that either way.
However, I have noted that most of the external repositories I've
looked at can't (or shouldn't) be imported directly. Perhaps if we
could find one that could be pulled in directly, we could use that to
drive a discussion of how.
> I'll help by stating my own position: I think we should be coming up
> with a process for how dependencies on other git repositories are
> going to work, something better than just cut and paste. Whether or
> not we vendorize code isn't really that important, but I'd say that
> if we are vendorizing code and we're notincluding entire sub-
> repositories (like cargo vendor does) we ought to still make this a
> scripted process that takes as an input a list of files we're
> pulling and a remote repository we're pulling from, and the file list
> and the remote repo (and commit ID we're pulling from) should all be
> checked in.
Do we have an example of an external piece of code we could do this to?
[...]
> > > The kernel community has a lot of that going on here. Again,
> > > sorry to pick on you James, but I wanted to make the argument
> > > that - maybe the kernel _should_ be adopting a more structured
> > > way of using code from outside repositories, like cargo, or git
> > > submodules (except I've never had a positive experience with git
> > > submodules, so ignore that suggestion, unless I've just been
> > > using them wrong, in which case someone please teach me). To read
> > > you and Greg saying "nah, just copy code from other repos, it's
> > > fine" - it felt like being back in the old days when we were
> > > still trying to get people to use source control, and having that
> > > one older colleague who _insisted_ on not using source control of
> > > any kind, and that's a bit disheartening.
> >
> > Even in C terms, the kernel is a nostdlib environment. If a C
> > project has too much libc dependency it's not going to work
> > directly in the kernel, nor should it. Let's look at zstd (which
> > is pretty much a nostdlib project) as a great example: the facebook
> > people didn't actually port the top of their tree (1.5) to the
> > kernel, they backported bug fixes to the 1.4 branch and made a
> > special release (1.4.10) just for us. Why did they do this? It
> > was because the 1.5 version vastly increased stack use to the
> > extent it would run off the end of the limited kernel stack so
> > couldn't be ported directly into the kernel. A lot of C libraries
> > that are nostdlib have problems like this as well (you can use
> > recursion, but not in the kernel). There's no easy way of shimming
> > environmental constraints like this.
>
> I wonder if we might have come up with a better solution if there'd
> been more cross-project communication and less siloing. Small stacks
> aren't particular to the kernel - it's definitely not unheard of to
> write userspace code where you want to have a lot of small stacks
> (especially if you're doing some kind of coroutine style threading;
> I've done stuff like this in the past)
But would you say that every piece of userspace code should reject
recursion and write for small stacks just in case it needs to be reused
in a more extreme environment?
I don't; I think it's fine there are loads of implementations that
would never work in our environment, because mostly there's no reason
to use them in the kernel (or another embedded environment). I also
understand why people build for the standard userspace environment
first ... it reduces complexity and makes the construction way easier.
> - and to me, as someone who's been incrementing on and maintaining a
> codebase in active use for 10 years, having multiple older versions
> in active use that need bugfixes gives me cold shivers.
>
> I wouldn't be surprised if at some point the zstd people walk back
> some of their changes or make it configurable at some point :)
Many things can happen in the future, but right at the moment I still
think zstd is working fine for both parties.
James