We've embraced 9p on kdevops [0] for support to let a guest use a host's
path to install Linux. This works well except I notice that if the host
makes new changes the guest does not see it. This just means the guest
has to reboot every time before running 'make modules_install install -j 8'
and it is unexpected and not user friendly.
I wanted to see if someone was working on fixing this yet. I've tested
with both cache=loose and cache=fscache. We use msize 131072, no
security and virtio-9p-pci.
[0] https://github.com/linux-kdevops/kdevops
Luis
I was looking at kdevops the other day - cool stuff. Was trying to
figure out how we could do v9fs CI with it.
Both cache=loose and cache=fscache currently don't validate via host.
I have a patch series which adds some new cache modes (readahead and
writeback) that give closer to the loose performance but with tighter
consistency (open-to-close) - these patches are in my ericvh/for-next
tree. I've also started on some patches to add cache=tight which will
do dir caching like loose but with validation, and potentially also
will add a new flag that will indicate how often loose should
revalidate. There is also some discussion of decoupling fscache from
the revalidation policy.
So the stuff in for-next will make it into the next kernel, there is a
writeback fix patch that still has some bugs when cache=fscache that
I'm working through, but the base patches add the new cache modes.
The new dir cache along with tight policy will hopefully get done in
the next month and so should be ready for next merge window as well.
-eric
On Sat, Mar 11, 2023 at 4:47 PM Luis Chamberlain <[email protected]> wrote:
>
> We've embraced 9p on kdevops [0] for support to let a guest use a host's
> path to install Linux. This works well except I notice that if the host
> makes new changes the guest does not see it. This just means the guest
> has to reboot every time before running 'make modules_install install -j 8'
> and it is unexpected and not user friendly.
>
> I wanted to see if someone was working on fixing this yet. I've tested
> with both cache=loose and cache=fscache. We use msize 131072, no
> security and virtio-9p-pci.
>
> [0] https://github.com/linux-kdevops/kdevops
>
> Luis
On Sun, Mar 12, 2023 at 01:22:34PM -0500, Eric Van Hensbergen wrote:
> I was looking at kdevops the other day - cool stuff. Was trying to
> figure out how we could do v9fs CI with it.
Happy to help any questions you may have about it!
> Both cache=loose and cache=fscache currently don't validate via host.
What does this mean exactly?
Right now a host with debian 6.0.0-6-amd64 certainly does not seem to push
out changes to 9p clients on VMs but Josef informs me that with 6.2-rc8
he did see the changes propagate.
So I'm about to upgrade the host forward now in hopes that this is
somehow resolved on a newer debian testing kernel.
1) The simple test is boot up the guest, read a file mounted with 9p.
2) On the host append a file and sha1sum it.
3) Client side sha1sums after a few seconds
Today on 6.0.0 I see absolutely no updates, I have to reboot the guest.
Josef seems to indicate he does not see this issue on 6.2-rc8. So
curious if this was a bug fixed.
> I have a patch series which adds some new cache modes (readahead and
> writeback) that give closer to the loose performance but with tighter
> consistency (open-to-close) - these patches are in my ericvh/for-next
> tree.
Good to know!
For the typical workflow for kdevops we do kernel compilations on the
host over and over and over again. Take your typical kernel day, that's
how often we do that. So it is conventiant to just have the guest run
'make modules_install install' instead of:'reboot; make modules_instal install',
Do none of the existing 9p cache modes not support open-to-close policies
at all?
Right now the cache mode used is cache=loose as that's the default,
what do you recommend for a typical kernel development environemnt?
> I've also started on some patches to add cache=tight which will
> do dir caching like loose but with validation, and potentially also
> will add a new flag that will indicate how often loose should
> revalidate. There is also some discussion of decoupling fscache from
> the revalidation policy.
I see !
> So the stuff in for-next will make it into the next kernel, there is a
> writeback fix patch that still has some bugs when cache=fscache that
> I'm working through, but the base patches add the new cache modes.
> The new dir cache along with tight policy will hopefully get done in
> the next month and so should be ready for next merge window as well.
Nice!
Luis
Sorry, took a bit to unstack from day job, but while going through the
patch queue I remembered I still had some questions to answer here.
On Fri, Mar 17, 2023 at 12:01 PM Luis Chamberlain <[email protected]> wrote:
>
> On Sun, Mar 12, 2023 at 01:22:34PM -0500, Eric Van Hensbergen wrote:
> > I was looking at kdevops the other day - cool stuff. Was trying to
> > figure out how we could do v9fs CI with it.
>
> Happy to help any questions you may have about it!
>
> > Both cache=loose and cache=fscache currently don't validate via host.
>
> What does this mean exactly?
>
That's a good question - I guess the answer is "by design" they don't
do anything special to check that the cache is up to date with the
host. That being said, there are several places in the code where the
cache will be invalidated (in some cases even if its up to date with
the host, yes, those are likely bugs). Many of the invalidations that
are present are overly conservative. The key missing one is
v9fs_lookup_revalidate which currently fills missing inode data but
should potentially also check to see if anything in the file changed
and react accordingly. The other thing is that we aren't always using
the caches when we should, if you look at the traffic even when
everything should be perfectly cached in fscache we are getting data
and metadata from the wire -- those are mostly bugs in the cache
implementation that I'm trying to go through and fix now.
From the 9p perspective, we should be looking at qid.version values
returned from the host and matching them to our internal notion of
version. If the versions don't match we should be invalidating
caches. qid.versions get returned on open, lookup, and most meta-data
operations so there's lots of opportunities there. There are still
some issues with this approach, namely that not all servers populate
qid.version and right now the ones that do use a hashed timestamp
(since qid.version is only 32-bits). This probably covers most bases,
but its not ideal -- so currrently thinking through whether we do a
further extension of the protocol or use some other mechanism.
There's a possibility of using a full getattr every so often to
back-up qid.version validation, but if we want truly tight coherence
(not just open to close) then we need to open up some sort of back
channel for invalidates from the server -- but I think the qid based
invalidation probably gets us most of what we need so going to start
with that.
> Right now a host with debian 6.0.0-6-amd64 certainly does not seem to push
> out changes to 9p clients on VMs but Josef informs me that with 6.2-rc8
> he did see the changes propagate.
I did tighten up some of the invalidation in the last round of
patches, however these are likely more on the overly conservative side
versus doing the right thing -- however, its really not at the point
where you can rely on it. If consistency is something you care about,
I'd suggest cache=none until you can get cache=readahead.
>
> Do none of the existing 9p cache modes not support open-to-close policies
> at all?
>
not specifically open-to-close, loose supports file and dir caching
but without consistency, it might be tempting to try cache=mmap to see
if it gets you closer, but my frame of reference is more the current
patches versus the old code so not sure it would buy you anything.
>
> Right now the cache mode used is cache=loose as that's the default,
> what do you recommend for a typical kernel development environemnt?
>
As I said, if you are interactively changing things I think you'd want
to go for cache=none for now (as painful as it is). I have fixed what
I hope to be my last bug with the new patch series so it should be
going into linux-next today.
-eric
On Mon, Mar 27, 2023 at 08:05:21AM -0500, Eric Van Hensbergen wrote:
> Sorry, took a bit to unstack from day job, but while going through the
> patch queue I remembered I still had some questions to answer here.
>
> On Fri, Mar 17, 2023 at 12:01 PM Luis Chamberlain <[email protected]> wrote:
> >
> > On Sun, Mar 12, 2023 at 01:22:34PM -0500, Eric Van Hensbergen wrote:
> > > I was looking at kdevops the other day - cool stuff. Was trying to
> > > figure out how we could do v9fs CI with it.
> >
> > Happy to help any questions you may have about it!
> >
> > > Both cache=loose and cache=fscache currently don't validate via host.
> >
> > What does this mean exactly?
> >
>
> That's a good question - I guess the answer is "by design" they don't
> do anything special to check that the cache is up to date with the
> host.
From a user perspective none of this is clear from any of the
documentation I read.
> That being said, there are several places in the code where the
> cache will be invalidated
<-- elaboration of current invalidation schemes when cache is enabled -->
>
> > Right now a host with debian 6.0.0-6-amd64 certainly does not seem to push
> > out changes to 9p clients on VMs but Josef informs me that with 6.2-rc8
> > he did see the changes propagate.
>
> I did tighten up some of the invalidation in the last round of
> patches, however these are likely more on the overly conservative side
> versus doing the right thing -- however, its really not at the point
> where you can rely on it. If consistency is something you care about,
> I'd suggest cache=none until you can get cache=readahead.
Indeed that fixes things and makes things work *exactly* as one would
expect. The simple case of just editing a file and immediately comparing
sha1sum on the host and guest yields the exact same result.
In comparison using the default cache of loose... things are not what
you'd expect. The cache is completley useless even if you do a full
kernel compilation on the host. The guest will not get any updates at
all.
> > Do none of the existing 9p cache modes not support open-to-close policies
> > at all?
> >
>
> not specifically open-to-close, loose supports file and dir caching
> but without consistency, it might be tempting to try cache=mmap to see
> if it gets you closer, but my frame of reference is more the current
> patches versus the old code so not sure it would buy you anything.
OK thanks, it seems we should probably help test 9p with fstests and get
a baseline, and then check to see how many tests actually do test cache
modes. If none exists, we should add some to test for each invalidation
strategy possible.
We can certainly use kdevops to help test this but I suspect that
since it relies on a client / host model it may make sense to share
some code for automation on bringup with what NFS folks are doing
with kdevops.
> > Right now the cache mode used is cache=loose as that's the default,
> > what do you recommend for a typical kernel development environemnt?
>
> As I said, if you are interactively changing things I think you'd want
> to go for cache=none for now (as painful as it is).
The documentation is not so clear what the pain is in terms of
performance.
> I have fixed what
> I hope to be my last bug with the new patch series so it should be
> going into linux-next today.
Nice, thanks, since kdevops relies on a host kernel though and we strive
to have stability for that, I personally like to recommend distro
kernels and so they're a few kernel releases out of date. So debian-testing
is on 6.1 as of today for example.
So I'll go with the following for now on kdevops for now. This seems to
fix all the woes I had for now. Once a cache mode with proper testign
proves to pass most tests in fstests we can switch to it on kdevops.
diff --git a/playbooks/roles/bootlinux/tasks/main.yml b/playbooks/roles/bootlinux/tasks/main.yml
index 297a42794a09..477cacdee887 100644
--- a/playbooks/roles/bootlinux/tasks/main.yml
+++ b/playbooks/roles/bootlinux/tasks/main.yml
@@ -111,7 +111,7 @@
name: "{{ target_linux_dir_path }}"
src: "{{ bootlinux_9p_mount_tag }}"
fstype: "9p"
- opts: "ro,trans=virtio,version=9p2000.L,posixacl,cache=loose"
+ opts: "ro,trans=virtio,version=9p2000.L,posixacl,cache=none"
state: "mounted"
tags: [ 'data_partition' ]
when:
BTW the qemu wiki seems to suggest cache=loose and its why I used it on
kdevops as a default. What about the following so to avoid folks running
into similar issues? I can go and update the wiki too.
diff --git a/Documentation/filesystems/9p.rst b/Documentation/filesystems/9p.rst
index 0e800b8f73cc..476944912312 100644
--- a/Documentation/filesystems/9p.rst
+++ b/Documentation/filesystems/9p.rst
@@ -56,6 +56,12 @@ seen by reading /sys/bus/virtio/drivers/9pnet_virtio/virtio<n>/mount_tag files.
Options
=======
+Note that a none of the currently supported cache modes validate against data
+againgst the host, they don't do anything special to check that the cache is up
+to date with the host. If you really need some synchronization the cache mode
+you should use is default "none", that will ensure data is kept up to date
+synchronously right away despite some performance overhead.
+
============= ===============================================================
trans=name select an alternative transport. Valid options are
currently:
Luis Chamberlain wrote on Mon, Mar 27, 2023 at 10:39:54AM -0700:
> > I have fixed what
> > I hope to be my last bug with the new patch series so it should be
> > going into linux-next today.
>
> Nice, thanks, since kdevops relies on a host kernel though and we strive
> to have stability for that, I personally like to recommend distro
> kernels and so they're a few kernel releases out of date. So debian-testing
> is on 6.1 as of today for example.
> [...]
> - opts: "ro,trans=virtio,version=9p2000.L,posixacl,cache=loose"
> + opts: "ro,trans=virtio,version=9p2000.L,posixacl,cache=none"
Yes, if you want something mostly coherent with the host, cache=none (or
cache=mmap if you need mmap, iirc linux build does for linking? if you
want to do that on guest...) is what you'll want to use on current
kernels.
> BTW the qemu wiki seems to suggest cache=loose and its why I used it on
> kdevops as a default. What about the following so to avoid folks running
> into similar issues? I can go and update the wiki too.
I've added Christian in Cc for this point, he's more active on the qemu
side
(thread started here:
https://lkml.kernel.org/r/[email protected]
)
I have no opinion on the current wording, the default is there for a
reason and it's a safe default (none), and cache=loose is clearly
described with "no attempts are made at consistency, intended for
exclusive, read-only mounts" which I think ought to be clear enough
(exclusive means not shared with the host), but if you think it's not
clear enough it probably isn't.
A word on the qemu wiki "if you want to share with host..." would
probably be good though.
--
Dominique
On Tue, Mar 28, 2023 at 11:31:50AM +0900, Dominique Martinet wrote:
> Luis Chamberlain wrote on Mon, Mar 27, 2023 at 10:39:54AM -0700:
> > > I have fixed what
> > > I hope to be my last bug with the new patch series so it should be
> > > going into linux-next today.
> >
> > Nice, thanks, since kdevops relies on a host kernel though and we strive
> > to have stability for that, I personally like to recommend distro
> > kernels and so they're a few kernel releases out of date. So debian-testing
> > is on 6.1 as of today for example.
> > [...]
> > - opts: "ro,trans=virtio,version=9p2000.L,posixacl,cache=loose"
> > + opts: "ro,trans=virtio,version=9p2000.L,posixacl,cache=none"
>
> Yes, if you want something mostly coherent with the host, cache=none (or
> cache=mmap if you need mmap, iirc linux build does for linking? if you
> want to do that on guest...) is what you'll want to use on current
> kernels.
OK cool, we use 9p to build on the host and so all the guest has to do
is just 'make modules_install install'.
> > BTW the qemu wiki seems to suggest cache=loose and its why I used it on
> > kdevops as a default. What about the following so to avoid folks running
> > into similar issues? I can go and update the wiki too.
>
> I've added Christian in Cc for this point, he's more active on the qemu
> side
> (thread started here:
> https://lkml.kernel.org/r/[email protected]
> )
>
> I have no opinion on the current wording, the default is there for a
> reason and it's a safe default (none), and cache=loose is clearly
> described with "no attempts are made at consistency, intended for
> exclusive, read-only mounts" which I think ought to be clear enough
> (exclusive means not shared with the host), but if you think it's not
> clear enough it probably isn't.
It certainly was not.
LUis
On Tuesday, March 28, 2023 4:31:50 AM CEST Dominique Martinet wrote:
> Luis Chamberlain wrote on Mon, Mar 27, 2023 at 10:39:54AM -0700:
> > > I have fixed what
> > > I hope to be my last bug with the new patch series so it should be
> > > going into linux-next today.
> >
> > Nice, thanks, since kdevops relies on a host kernel though and we strive
> > to have stability for that, I personally like to recommend distro
> > kernels and so they're a few kernel releases out of date. So debian-testing
> > is on 6.1 as of today for example.
> > [...]
> > - opts: "ro,trans=virtio,version=9p2000.L,posixacl,cache=loose"
> > + opts: "ro,trans=virtio,version=9p2000.L,posixacl,cache=none"
>
> Yes, if you want something mostly coherent with the host, cache=none (or
> cache=mmap if you need mmap, iirc linux build does for linking? if you
> want to do that on guest...) is what you'll want to use on current
> kernels.
>
> > BTW the qemu wiki seems to suggest cache=loose and its why I used it on
> > kdevops as a default. What about the following so to avoid folks running
> > into similar issues? I can go and update the wiki too.
>
> I've added Christian in Cc for this point, he's more active on the qemu
> side
> (thread started here:
> https://lkml.kernel.org/r/[email protected]
> )
>
> I have no opinion on the current wording, the default is there for a
> reason and it's a safe default (none), and cache=loose is clearly
> described with "no attempts are made at consistency, intended for
> exclusive, read-only mounts" which I think ought to be clear enough
> (exclusive means not shared with the host), but if you think it's not
> clear enough it probably isn't.
>
> A word on the qemu wiki "if you want to share with host..." would
> probably be good though.
Hi Luis,
not sure which QEMU wiki page you are referring to. AFAIK we currently have 3
QEMU wiki pages concerning 9p:
1. 9p documentation for users:
https://wiki.qemu.org/Documentation/9psetup
2. 9p documentation for developers only:
https://wiki.qemu.org/Documentation/9p
3. How to setup an entire guest on top of a 9p root filesystem:
https://wiki.qemu.org/Documentation/9p_root_fs
Only the latter wiki page mentions cache=loose at all:
"To speedup things you can also consider to use e.g. cache=loose instead.
That will deploy a filesystem cache on guest side and reduces the amount
of 9p requests to hosts. As a consequence however guest might not
immediately see file changes performed on host side. So choose wisely upon
intended use case scenario. You can change between cache=mmap or e.g.
cache=loose at any time."
Which I now changed to:
"To speedup things you can also consider to use e.g. cache=loose instead.
That will deploy a filesystem cache on guest side and reduces the amount of
9p requests to hosts. As a consequence however guest might not see file
changes performed on host side *at* *all* (as Linux kernel's 9p client
currently does not revalidate for fs changes on host side at all, which is
planned to be changed on Linux kernel side soon though). So choose wisely
upon intended use case scenario. You can change between cache=mmap or e.g.
cache=loose at any time."
On the user page it was already clearly mentioned though:
"Mount the shared folder on guest using
mount -t 9p -o trans=virtio test_mount /tmp/shared/ -oversion=9p2000.L,posixacl,msize=104857600,cache=none
In the above example the folder /home/guest/9p_setup/ shared of the host
is shared with the folder /tmp/shared on the guest. We use no cache because
current caching mechanisms need more work and the results are not what you
would expect."
Best regards,
Christian Schoenebeck
On Tue, Mar 28, 2023 at 01:53:49PM +0200, Christian Schoenebeck wrote:
> Hi Luis,
>
> not sure which QEMU wiki page you are referring to. AFAIK we currently have 3
> QEMU wiki pages concerning 9p:
>
> 1. 9p documentation for users:
> https://wiki.qemu.org/Documentation/9psetup
It was this one. I hadn't looked at the other ones.
> 2. 9p documentation for developers only:
> https://wiki.qemu.org/Documentation/9p
>
> 3. How to setup an entire guest on top of a 9p root filesystem:
> https://wiki.qemu.org/Documentation/9p_root_fs
>
> Only the latter wiki page mentions cache=loose at all:
>
> "To speedup things you can also consider to use e.g. cache=loose instead.
> That will deploy a filesystem cache on guest side and reduces the amount
> of 9p requests to hosts. As a consequence however guest might not
> immediately see file changes performed on host side. So choose wisely upon
> intended use case scenario. You can change between cache=mmap or e.g.
> cache=loose at any time."
>
> Which I now changed to:
>
> "To speedup things you can also consider to use e.g. cache=loose instead.
My experience is that cache=loose is totally useless.
> That will deploy a filesystem cache on guest side and reduces the amount of
> 9p requests to hosts. As a consequence however guest might not see file
> changes performed on host side *at* *all*
I think that makes it pretty useless, aren't most setups on the guest read-only?
It is not about "may not see", just won't. For example I modified the
Makefile and compiled a full kernel and even with those series of
changes, the guest *minutes later* never saw any updates.
> (as Linux kernel's 9p client
> currently does not revalidate for fs changes on host side at all, which is
> planned to be changed on Linux kernel side soon though). So choose wisely
> upon intended use case scenario. You can change between cache=mmap or e.g.
> cache=loose at any time."
>
> On the user page it was already clearly mentioned though:
>
> "Mount the shared folder on guest using
>
> mount -t 9p -o trans=virtio test_mount /tmp/shared/ -oversion=9p2000.L,posixacl,msize=104857600,cache=none
>
> In the above example the folder /home/guest/9p_setup/ shared of the host
> is shared with the folder /tmp/shared on the guest. We use no cache because
> current caching mechanisms need more work and the results are not what you
> would expect."
I got a wiki account now and I was the one who had clarified this.
Luis
Luis Chamberlain wrote on Tue, Mar 28, 2023 at 10:41:02AM -0700:
> > "To speedup things you can also consider to use e.g. cache=loose instead.
>
> My experience is that cache=loose is totally useless.
If the fs you mount isn't accessed by the host while the VM is up, and
isn't shared with another guest (e.g. "exclusive share"), you'll get
what you expect.
I have no idea what people use qemu's virtfs for but this is apparently
common enough that it was recommended before without anyone complaining
since that started being recommended in 2011[1] until now?
[1] https://wiki.qemu.org/index.php?title=Documentation/9psetup&diff=2178&oldid=2177
(now I'm not arguing it should be recommended, my stance as a 9p
maintainer is that the default should be used unless you know what
you're doing, so the new code should just remove the 'cache=none'
altogether as that's the default.
With the new cache models Eric is preparing comes, we'll get a new safe
default that will likely be better than cache=none, there is no reason
to explicitly recommend the historic safe model as the default has
always been on the safe side and we have no plan of changing that.)
> > That will deploy a filesystem cache on guest side and reduces the amount of
> > 9p requests to hosts. As a consequence however guest might not see file
> > changes performed on host side *at* *all*
>
> I think that makes it pretty useless, aren't most setups on the guest read-only?
>
> It is not about "may not see", just won't. For example I modified the
> Makefile and compiled a full kernel and even with those series of
> changes, the guest *minutes later* never saw any updates.
read-only on the guest has nothing to do with it, nor has time.
If the directory is never accessed on the guest before the kernel has
been built, you'll be able to make install on the guest -- once, even if
the build was done after the VM booted and fs mounted.
After it's been read once, it'll stay in cache until memory pressure (or
an admin action like umount/mount or sysctl vm.drop_caches=3) clears it.
I believe that's why it appeared to work until you noticed the issue and
had to change the mount option -- I'd expect in most case you'll run
make install once and reboot/kexec into the new kernel.
It's not safe for your usecase and cache=none definitely sounds better
to me, but people should use defaults make their own informed decision.
>> In the above example the folder /home/guest/9p_setup/ shared of the
>> host is shared with the folder /tmp/shared on the guest. We use no
>> cache because current caching mechanisms need more work and the
>> results are not what you would expect."
>
> I got a wiki account now and I was the one who had clarified this.
Thanks for helping making this clearer.
--
Dominique
On Wednesday, March 29, 2023 12:08:26 AM CEST Dominique Martinet wrote:
> Luis Chamberlain wrote on Tue, Mar 28, 2023 at 10:41:02AM -0700:
> > > "To speedup things you can also consider to use e.g. cache=loose instead.
> >
> > My experience is that cache=loose is totally useless.
>
> If the fs you mount isn't accessed by the host while the VM is up, and
> isn't shared with another guest (e.g. "exclusive share"), you'll get
> what you expect.
>
> I have no idea what people use qemu's virtfs for but this is apparently
> common enough that it was recommended before without anyone complaining
> since that started being recommended in 2011[1] until now?
>
> [1] https://wiki.qemu.org/index.php?title=Documentation/9psetup&diff=2178&oldid=2177
>
> (now I'm not arguing it should be recommended, my stance as a 9p
> maintainer is that the default should be used unless you know what
> you're doing, so the new code should just remove the 'cache=none'
> altogether as that's the default.
> With the new cache models Eric is preparing comes, we'll get a new safe
> default that will likely be better than cache=none, there is no reason
> to explicitly recommend the historic safe model as the default has
> always been on the safe side and we have no plan of changing that.)
It's not that I receive a lot of feedback for what people use 9p for, nor am I
QEMU's 9p maintainer for a long time, but so far contributors cared more about
performance and other issues than propagating changes host -> guest without
reboot/remount/drop_caches. At least they did not care enough to work on
patches.
Personally I also use cache=loose and only need to push changes host->guest
once in a while.
> > > That will deploy a filesystem cache on guest side and reduces the amount of
> > > 9p requests to hosts. As a consequence however guest might not see file
> > > changes performed on host side *at* *all*
> >
> > I think that makes it pretty useless, aren't most setups on the guest read-only?
> >
> > It is not about "may not see", just won't. For example I modified the
> > Makefile and compiled a full kernel and even with those series of
> > changes, the guest *minutes later* never saw any updates.
>
> read-only on the guest has nothing to do with it, nor has time.
>
> If the directory is never accessed on the guest before the kernel has
> been built, you'll be able to make install on the guest -- once, even if
> the build was done after the VM booted and fs mounted.
>
> After it's been read once, it'll stay in cache until memory pressure (or
> an admin action like umount/mount or sysctl vm.drop_caches=3) clears it.
>
> I believe that's why it appeared to work until you noticed the issue and
> had to change the mount option -- I'd expect in most case you'll run
> make install once and reboot/kexec into the new kernel.
>
> It's not safe for your usecase and cache=none definitely sounds better
> to me, but people should use defaults make their own informed decision.
It appears to me that read-only seems not to be the average use case for 9p,
at least from the command lines I received. It is often used in combination
with overlayfs though.
I (think) the reason why cache=loose was recommended as default option on the
QEMU wiki page ages ago, was because of its really poor performance at that
point. I would personally not go that far and discourage people from using
cache=loose in general, as long as they get informed about the consequences.
You still get a great deal of performance boost, the rest is for each
individual to decide.
Considering that Eric already has patches for revalidating the cache in the
works, I think the changes I made on the other QEMU wiki page are appropriate,
including the word "might" as it's soon only a matter of kernel version.
> >> In the above example the folder /home/guest/9p_setup/ shared of the
> >> host is shared with the folder /tmp/shared on the guest. We use no
> >> cache because current caching mechanisms need more work and the
> >> results are not what you would expect."
> >
> > I got a wiki account now and I was the one who had clarified this.
>
> Thanks for helping making this clearer.
Yep, and thanks for making a wiki account and improving the content there
directly. Always appreciated!
On Wed, 2023-03-29 at 13:19 +0200, Christian Schoenebeck wrote:
> On Wednesday, March 29, 2023 12:08:26 AM CEST Dominique Martinet wrote:
> > Luis Chamberlain wrote on Tue, Mar 28, 2023 at 10:41:02AM -0700:
> > > > "To speedup things you can also consider to use e.g. cache=loose instead.
> > >
> > > My experience is that cache=loose is totally useless.
> >
> > If the fs you mount isn't accessed by the host while the VM is up, and
> > isn't shared with another guest (e.g. "exclusive share"), you'll get
> > what you expect.
> >
> > I have no idea what people use qemu's virtfs for but this is apparently
> > common enough that it was recommended before without anyone complaining
> > since that started being recommended in 2011[1] until now?
> >
> > [1] https://wiki.qemu.org/index.php?title=Documentation/9psetup&diff=2178&oldid=2177
> >
> > (now I'm not arguing it should be recommended, my stance as a 9p
> > maintainer is that the default should be used unless you know what
> > you're doing, so the new code should just remove the 'cache=none'
> > altogether as that's the default.
> > With the new cache models Eric is preparing comes, we'll get a new safe
> > default that will likely be better than cache=none, there is no reason
> > to explicitly recommend the historic safe model as the default has
> > always been on the safe side and we have no plan of changing that.)
>
> It's not that I receive a lot of feedback for what people use 9p for, nor am I
> QEMU's 9p maintainer for a long time, but so far contributors cared more about
> performance and other issues than propagating changes host -> guest without
> reboot/remount/drop_caches. At least they did not care enough to work on
> patches.
>
> Personally I also use cache=loose and only need to push changes host->guest
> once in a while.
>
> > > > That will deploy a filesystem cache on guest side and reduces the amount of
> > > > 9p requests to hosts. As a consequence however guest might not see file
> > > > changes performed on host side *at* *all*
> > >
> > > I think that makes it pretty useless, aren't most setups on the guest read-only?
> > >
> > > It is not about "may not see", just won't. For example I modified the
> > > Makefile and compiled a full kernel and even with those series of
> > > changes, the guest *minutes later* never saw any updates.
> >
> > read-only on the guest has nothing to do with it, nor has time.
> >
> > If the directory is never accessed on the guest before the kernel has
> > been built, you'll be able to make install on the guest -- once, even if
> > the build was done after the VM booted and fs mounted.
> >
> > After it's been read once, it'll stay in cache until memory pressure (or
> > an admin action like umount/mount or sysctl vm.drop_caches=3) clears it.
> >
> > I believe that's why it appeared to work until you noticed the issue and
> > had to change the mount option -- I'd expect in most case you'll run
> > make install once and reboot/kexec into the new kernel.
> >
> > It's not safe for your usecase and cache=none definitely sounds better
> > to me, but people should use defaults make their own informed decision.
>
> It appears to me that read-only seems not to be the average use case for 9p,
> at least from the command lines I received. It is often used in combination
> with overlayfs though.
>
> I (think) the reason why cache=loose was recommended as default option on the
> QEMU wiki page ages ago, was because of its really poor performance at that
> point. I would personally not go that far and discourage people from using
> cache=loose in general, as long as they get informed about the consequences.
> You still get a great deal of performance boost, the rest is for each
> individual to decide.
>
> Considering that Eric already has patches for revalidating the cache in the
> works, I think the changes I made on the other QEMU wiki page are appropriate,
> including the word "might" as it's soon only a matter of kernel version.
>
> > > > In the above example the folder /home/guest/9p_setup/ shared of the
> > > > host is shared with the folder /tmp/shared on the guest. We use no
> > > > cache because current caching mechanisms need more work and the
> > > > results are not what you would expect."
> > >
> > > I got a wiki account now and I was the one who had clarified this.
> >
> > Thanks for helping making this clearer.
>
> Yep, and thanks for making a wiki account and improving the content there
> directly. Always appreciated!
>
Catching up on this thread.
Getting cache coherency right on a network filesystem is quite
difficult. It's always a balance between correctness and performance.
Some protocols (e.g. CIFS and Ceph) take a very heavy-handed approach to
try ensure that the caches are always coherent. Basically, these clients
are only allowed to cache when the server grants permission for it. That
can have a negative effect on performance, of course.
NFS as a protocol is more "loose", but we've generally beat its cache
coherency mechanisms into shape over the years, so you don't see these
sorts of problems there as much. FWIW, NFS uses a sliding time window to
revalidate the cache, such that it'll revalidate frequently when an
inodes is changing frequently, but less so when it's more stable.
9P I haven't worked with as much, but it sounds like it doesn't try to
keep caches coherent (at least not with cache=loose).
Probably the simplest solution here is to simply unmount/mount before
you have the clients call "make modules_install && make install". That
should ensure that the client doesn't have any stale data in the cache
when the time comes to do the reads. A full reboot shouldn't be
required.
--
Jeff Layton <[email protected]>
I like the sliding window concept - I wasn't aware NFS was doing that,
I'll have a look as part of my rework.
The unmount/mount should indeed flush any cache (unless using
fscache), so that might be a good workaround if it can be automated in
the workflow.
-eric
On Wed, Mar 29, 2023 at 6:32 AM Jeff Layton <[email protected]> wrote:
>
> On Wed, 2023-03-29 at 13:19 +0200, Christian Schoenebeck wrote:
> > On Wednesday, March 29, 2023 12:08:26 AM CEST Dominique Martinet wrote:
> > > Luis Chamberlain wrote on Tue, Mar 28, 2023 at 10:41:02AM -0700:
> > > > > "To speedup things you can also consider to use e.g. cache=loose instead.
> > > >
> > > > My experience is that cache=loose is totally useless.
> > >
> > > If the fs you mount isn't accessed by the host while the VM is up, and
> > > isn't shared with another guest (e.g. "exclusive share"), you'll get
> > > what you expect.
> > >
> > > I have no idea what people use qemu's virtfs for but this is apparently
> > > common enough that it was recommended before without anyone complaining
> > > since that started being recommended in 2011[1] until now?
> > >
> > > [1] https://wiki.qemu.org/index.php?title=Documentation/9psetup&diff=2178&oldid=2177
> > >
> > > (now I'm not arguing it should be recommended, my stance as a 9p
> > > maintainer is that the default should be used unless you know what
> > > you're doing, so the new code should just remove the 'cache=none'
> > > altogether as that's the default.
> > > With the new cache models Eric is preparing comes, we'll get a new safe
> > > default that will likely be better than cache=none, there is no reason
> > > to explicitly recommend the historic safe model as the default has
> > > always been on the safe side and we have no plan of changing that.)
> >
> > It's not that I receive a lot of feedback for what people use 9p for, nor am I
> > QEMU's 9p maintainer for a long time, but so far contributors cared more about
> > performance and other issues than propagating changes host -> guest without
> > reboot/remount/drop_caches. At least they did not care enough to work on
> > patches.
> >
> > Personally I also use cache=loose and only need to push changes host->guest
> > once in a while.
> >
> > > > > That will deploy a filesystem cache on guest side and reduces the amount of
> > > > > 9p requests to hosts. As a consequence however guest might not see file
> > > > > changes performed on host side *at* *all*
> > > >
> > > > I think that makes it pretty useless, aren't most setups on the guest read-only?
> > > >
> > > > It is not about "may not see", just won't. For example I modified the
> > > > Makefile and compiled a full kernel and even with those series of
> > > > changes, the guest *minutes later* never saw any updates.
> > >
> > > read-only on the guest has nothing to do with it, nor has time.
> > >
> > > If the directory is never accessed on the guest before the kernel has
> > > been built, you'll be able to make install on the guest -- once, even if
> > > the build was done after the VM booted and fs mounted.
> > >
> > > After it's been read once, it'll stay in cache until memory pressure (or
> > > an admin action like umount/mount or sysctl vm.drop_caches=3) clears it.
> > >
> > > I believe that's why it appeared to work until you noticed the issue and
> > > had to change the mount option -- I'd expect in most case you'll run
> > > make install once and reboot/kexec into the new kernel.
> > >
> > > It's not safe for your usecase and cache=none definitely sounds better
> > > to me, but people should use defaults make their own informed decision.
> >
> > It appears to me that read-only seems not to be the average use case for 9p,
> > at least from the command lines I received. It is often used in combination
> > with overlayfs though.
> >
> > I (think) the reason why cache=loose was recommended as default option on the
> > QEMU wiki page ages ago, was because of its really poor performance at that
> > point. I would personally not go that far and discourage people from using
> > cache=loose in general, as long as they get informed about the consequences.
> > You still get a great deal of performance boost, the rest is for each
> > individual to decide.
> >
> > Considering that Eric already has patches for revalidating the cache in the
> > works, I think the changes I made on the other QEMU wiki page are appropriate,
> > including the word "might" as it's soon only a matter of kernel version.
> >
> > > > > In the above example the folder /home/guest/9p_setup/ shared of the
> > > > > host is shared with the folder /tmp/shared on the guest. We use no
> > > > > cache because current caching mechanisms need more work and the
> > > > > results are not what you would expect."
> > > >
> > > > I got a wiki account now and I was the one who had clarified this.
> > >
> > > Thanks for helping making this clearer.
> >
> > Yep, and thanks for making a wiki account and improving the content there
> > directly. Always appreciated!
> >
>
> Catching up on this thread.
>
> Getting cache coherency right on a network filesystem is quite
> difficult. It's always a balance between correctness and performance.
>
> Some protocols (e.g. CIFS and Ceph) take a very heavy-handed approach to
> try ensure that the caches are always coherent. Basically, these clients
> are only allowed to cache when the server grants permission for it. That
> can have a negative effect on performance, of course.
>
> NFS as a protocol is more "loose", but we've generally beat its cache
> coherency mechanisms into shape over the years, so you don't see these
> sorts of problems there as much. FWIW, NFS uses a sliding time window to
> revalidate the cache, such that it'll revalidate frequently when an
> inodes is changing frequently, but less so when it's more stable.
>
> 9P I haven't worked with as much, but it sounds like it doesn't try to
> keep caches coherent (at least not with cache=loose).
>
> Probably the simplest solution here is to simply unmount/mount before
> you have the clients call "make modules_install && make install". That
> should ensure that the client doesn't have any stale data in the cache
> when the time comes to do the reads. A full reboot shouldn't be
> required.
>
> --
> Jeff Layton <[email protected]>