Before people get too excited this isn't a proposal to kill DAX. The
topic proposal is a discussion to resolve lingering open questions
that currently motivate ext4 and xfs to scream "EXPERIMENTAL" when the
current DAX facilities are enabled. The are 2 primary concerns to
resolve. Enumerate the remaining features/fixes, and identify a path
to implement it all without regressing any existing application use
cases.
An enumeration of remaining projects follows, please expand this list
if I missed something:
* "DAX" has no specific meaning by itself, users have 2 use cases for
"DAX" capabilities: userspace cache management via MAP_SYNC, and page
cache avoidance where the latter aspect of DAX has no current api to
discover / use it. The project is to supplement MAP_SYNC with a
MAP_DIRECT facility and MADV_SYNC / MADV_DIRECT to indicate the same
dynamically via madvise. Similar to O_DIRECT, MAP_DIRECT would be an
application hint to avoid / minimiize page cache usage, but no strict
guarantee like what MAP_SYNC provides.
* Resolve all "if (dax) goto fail;" patterns in the kernel. Outside of
longterm-GUP (a topic in its own right) the projects here are
XFS-reflink and XFS-realtime-device support. DAX+reflink effectively
requires a given physical page to be mapped into two different inodes
at different (page->index) offsets. The challenge is to support
DAX-reflink without violating any existing application visible
semantics, the operating assumption / strawman to debate is that
experimental status is not blanket permission to go change existing
semantics in backwards incompatible ways.
* Deprecate, but not remove, the DAX mount option. Too many flows
depend on the option so it will never go away, but the facility is too
coarse. Provide an option to enable MAP_SYNC and
more-likely-to-do-something-useful-MAP_DIRECT on a per-directory
basis. The current proposal is to allow this property to only be
toggled while the directory is empty to avoid the complications of
racing page invalidation with new DAX mappings.
Secondary projects, i.e. important but I would submit are not in the
critical path to removing the "experimental" designation:
* Filesystem-integrated badblock management. Hook up the media error
notifications from libnvdimm to the filesystem to allow for operations
like "list files with media errors" and "enumerate bad file offsets on
a granulatiy smaller than a page". Another consideration along these
lines is to integrate machine-check-handling and dynamic error
notification into a filesystem interface. I've heard complaints that
the sigaction() based mechanism to receive BUS_MCEERR_* information,
while sufficient for the "System RAM" use case, is not precise enough
for the "Persistent Memory / DAX" use case where errors are repairable
and sub-page error information is useful.
* Userfaultfd for file-backed mappings and DAX
Ideally all the usual DAX, persistent memory, and GUP suspects could
be in the room to discuss this:
* Jan Kara
* Dave Chinner
* Christoph Hellwig
* Jeff Moyer
* Johannes Thumshirn
* Matthew Wilcox
* John Hubbard
* Jérôme Glisse
* MM folks for the reflink vs 'struct page' vs Xarray considerations
On Wed, 2019-02-06 at 13:12 -0800, Dan Williams wrote:
> Before people get too excited this isn't a proposal to kill DAX. The
> topic proposal is a discussion to resolve lingering open questions
> that currently motivate ext4 and xfs to scream "EXPERIMENTAL" when the
> current DAX facilities are enabled. The are 2 primary concerns to
> resolve. Enumerate the remaining features/fixes, and identify a path
> to implement it all without regressing any existing application use
> cases.
>
> An enumeration of remaining projects follows, please expand this list
> if I missed something:
>
> * "DAX" has no specific meaning by itself, users have 2 use cases for
> "DAX" capabilities: userspace cache management via MAP_SYNC, and page
> cache avoidance where the latter aspect of DAX has no current api to
> discover / use it. The project is to supplement MAP_SYNC with a
> MAP_DIRECT facility and MADV_SYNC / MADV_DIRECT to indicate the same
> dynamically via madvise. Similar to O_DIRECT, MAP_DIRECT would be an
> application hint to avoid / minimiize page cache usage, but no strict
> guarantee like what MAP_SYNC provides.
Agreed that basic dax programming model needs to be settled.
> * Resolve all "if (dax) goto fail;" patterns in the kernel. Outside of
> longterm-GUP (a topic in its own right) the projects here are
> XFS-reflink and XFS-realtime-device support. DAX+reflink effectively
> requires a given physical page to be mapped into two different inodes
> at different (page->index) offsets. The challenge is to support
> DAX-reflink without violating any existing application visible
> semantics, the operating assumption / strawman to debate is that
> experimental status is not blanket permission to go change existing
> semantics in backwards incompatible ways.
I do not think "if (dax) goto fail;" is actually bad. It helps users to
immediately realize that dax may not be enabled and they have a setup
issue. I also think that FS-specific enhancements like DAX-reflink=1
support should be managed separately and should be possible after
EXPERIMENTAL is removed from 'mount -o dax'. That is, removing
EXPERIMENTAL for DAX-reflink=0 should not require support of DAX-
reflink=1.
I've also had multiple users complained about 'mount -o dax' succeeded
by falling back to non-dax despite of their intent. Such users wasted
many hours without knowing their setup error / current restrictions.
The next think they always ask is how to check if dax is enabled in such
case, and they are not happy about limited interfaces (ex. look for
mount entry in /proc/mounts), either.
> * Deprecate, but not remove, the DAX mount option. Too many flows
> depend on the option so it will never go away, but the facility is too
> coarse. Provide an option to enable MAP_SYNC and
> more-likely-to-do-something-useful-MAP_DIRECT on a per-directory
> basis. The current proposal is to allow this property to only be
> toggled while the directory is empty to avoid the complications of
> racing page invalidation with new DAX mappings.
Same as above. Such enhancement should be possible after EXPERIMENTAL
is removed from 'mount -o dax'. IOW, a separate EXPERIMENTAL message
can be shown when user requests per-directory dax.
Thanks,
-Toshi
On 07/02/2019 22:51, Kani, Toshi wrote:
> I've also had multiple users complained about 'mount -o dax' succeeded
> by falling back to non-dax despite of their intent. Such users wasted
> many hours without knowing their setup error / current restrictions.
> The next think they always ask is how to check if dax is enabled in such
> case, and they are not happy about limited interfaces (ex. look for
> mount entry in /proc/mounts), either.
Exactly, there's a need for admins and user-space applications to check if they
can use DAX or not. See also [1] for reference.
> Same as above. Such enhancement should be possible after EXPERIMENTAL
> is removed from 'mount -o dax'. IOW, a separate EXPERIMENTAL message
> can be shown when user requests per-directory dax.
Just as a data point here, in our Kernels we've removed the EXPERIMENTAL
warning for DAX on XFS (but not EXT4) and we are supporting it for some limited
use-cases.
[1] https://lore.kernel.org/linux-fsdevel/[email protected]/
Byte,
Johannes
--
Johannes Thumshirn SUSE Labs Filesystems
[email protected] +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850
On Wed 06-02-19 13:12:59, Dan Williams wrote:
[...]
> * Userfaultfd for file-backed mappings and DAX
I assume that other topics are meant to be FS track but this one is MM,
right?
--
Michal Hocko
SUSE Labs
On Thu, Feb 14, 2019 at 5:46 AM Michal Hocko <[email protected]> wrote:
>
> On Wed 06-02-19 13:12:59, Dan Williams wrote:
> [...]
> > * Userfaultfd for file-backed mappings and DAX
>
> I assume that other topics are meant to be FS track but this one is MM,
> right?
Yes, but I think it is the lowest priority of all the noted sub-topics
in this proposal. The DAX-reflink discussion, where a given
physical-page may need to be mapped into multiple inodes at different
offsets, might be more fruitful to have as a joint discussion with MM.
On Thu, Feb 14, 2019 at 10:25:07AM -0800, Dan Williams wrote:
> On Thu, Feb 14, 2019 at 5:46 AM Michal Hocko <[email protected]> wrote:
> >
> > On Wed 06-02-19 13:12:59, Dan Williams wrote:
> > [...]
> > > * Userfaultfd for file-backed mappings and DAX
> >
> > I assume that other topics are meant to be FS track but this one is MM,
> > right?
>
> Yes, but I think it is the lowest priority of all the noted sub-topics
> in this proposal. The DAX-reflink discussion, where a given
> physical-page may need to be mapped into multiple inodes at different
> offsets, might be more fruitful to have as a joint discussion with MM.
Note that my generic page write protection work can be use for that ie
having a single page correspond to multiple different mapping with also
different offset within each mapping. While in my patchset i only solve
the mapping aliasing issue, the index can be solve in much the same way
because same thinking apply. Namely that when you work on a file you
know the mapping and file offset and thus the index and when you work on
the vma you know the mapping and offset within the vma which translate
to offset within the file. They are only few places that do not have the
informations available and those do not care about it.
I am just again working on my struct page mapping patchset as well as
the generic page write protection that sits on top. I hope to be able
to post the v2 in couple weeks. You can always look at my posting last
year to see more details.
Cheers,
J?r?me
On Thu, Feb 14, 2019 at 11:10 AM Jerome Glisse <[email protected]> wrote:
>
> On Thu, Feb 14, 2019 at 10:25:07AM -0800, Dan Williams wrote:
> > On Thu, Feb 14, 2019 at 5:46 AM Michal Hocko <[email protected]> wrote:
> > >
> > > On Wed 06-02-19 13:12:59, Dan Williams wrote:
> > > [...]
> > > > * Userfaultfd for file-backed mappings and DAX
> > >
> > > I assume that other topics are meant to be FS track but this one is MM,
> > > right?
> >
> > Yes, but I think it is the lowest priority of all the noted sub-topics
> > in this proposal. The DAX-reflink discussion, where a given
> > physical-page may need to be mapped into multiple inodes at different
> > offsets, might be more fruitful to have as a joint discussion with MM.
>
> Note that my generic page write protection work can be use for that ie
> having a single page correspond to multiple different mapping with also
> different offset within each mapping. While in my patchset i only solve
> the mapping aliasing issue, the index can be solve in much the same way
> because same thinking apply. Namely that when you work on a file you
> know the mapping and file offset and thus the index and when you work on
> the vma you know the mapping and offset within the vma which translate
> to offset within the file. They are only few places that do not have the
> informations available and those do not care about it.
>
> I am just again working on my struct page mapping patchset as well as
> the generic page write protection that sits on top. I hope to be able
> to post the v2 in couple weeks. You can always look at my posting last
> year to see more details.
Yes, I have that in mind as one of the contenders. However, it's not
clear to me that its a suitable fit for filesystem-reflink. Others
have floated the 'page proxy' idea, so it would be good to discuss the
merits of the general approaches.
On Thu, Feb 14, 2019 at 11:31:24AM -0800, Dan Williams wrote:
> On Thu, Feb 14, 2019 at 11:10 AM Jerome Glisse <[email protected]> wrote:
> > I am just again working on my struct page mapping patchset as well as
> > the generic page write protection that sits on top. I hope to be able
> > to post the v2 in couple weeks. You can always look at my posting last
> > year to see more details.
>
> Yes, I have that in mind as one of the contenders. However, it's not
> clear to me that its a suitable fit for filesystem-reflink. Others
> have floated the 'page proxy' idea, so it would be good to discuss the
> merits of the general approaches.
... and my preferred option of putting pfn entries in the page cache.
Or is that what you meant by "page proxy"?
On Thu, Feb 14, 2019 at 12:09 PM Matthew Wilcox <[email protected]> wrote:
>
> On Thu, Feb 14, 2019 at 11:31:24AM -0800, Dan Williams wrote:
> > On Thu, Feb 14, 2019 at 11:10 AM Jerome Glisse <[email protected]> wrote:
> > > I am just again working on my struct page mapping patchset as well as
> > > the generic page write protection that sits on top. I hope to be able
> > > to post the v2 in couple weeks. You can always look at my posting last
> > > year to see more details.
> >
> > Yes, I have that in mind as one of the contenders. However, it's not
> > clear to me that its a suitable fit for filesystem-reflink. Others
> > have floated the 'page proxy' idea, so it would be good to discuss the
> > merits of the general approaches.
>
> ... and my preferred option of putting pfn entries in the page cache.
Another option to include the discussion.
> Or is that what you meant by "page proxy"?
Page proxy would be an object that a filesystem could allocate to
point back to a single physical 'struct page *'. The proxy would
contain an override for page->index.
On Thu, Feb 14, 2019 at 12:20:11PM -0800, Dan Williams wrote:
> On Thu, Feb 14, 2019 at 12:09 PM Matthew Wilcox <[email protected]> wrote:
> >
> > On Thu, Feb 14, 2019 at 11:31:24AM -0800, Dan Williams wrote:
> > > On Thu, Feb 14, 2019 at 11:10 AM Jerome Glisse <[email protected]> wrote:
> > > > I am just again working on my struct page mapping patchset as well as
> > > > the generic page write protection that sits on top. I hope to be able
> > > > to post the v2 in couple weeks. You can always look at my posting last
> > > > year to see more details.
> > >
> > > Yes, I have that in mind as one of the contenders. However, it's not
> > > clear to me that its a suitable fit for filesystem-reflink. Others
> > > have floated the 'page proxy' idea, so it would be good to discuss the
> > > merits of the general approaches.
> >
> > ... and my preferred option of putting pfn entries in the page cache.
>
> Another option to include the discussion.
>
> > Or is that what you meant by "page proxy"?
>
> Page proxy would be an object that a filesystem could allocate to
> point back to a single physical 'struct page *'. The proxy would
> contain an override for page->index.
Note that generic page write protection has such object, kind of like
stable_node in KSM. You overwritte page->mapping to point to this
generic struct which has a pointer to set of callback so that whatever
is protecting the page can offer API to break protection (break sharing
here).
So instead of having struct proxy_page -> struct page you would have the
reverse: struct page -> struct proxy and so you do not have to change
much in all the file system beside removing the reliance on page->mapping
which is what most of my patches are about.
Cheers,
J?r?me
On Thu, Feb 14, 2019 at 12:20:11PM -0800, Dan Williams wrote:
> On Thu, Feb 14, 2019 at 12:09 PM Matthew Wilcox <[email protected]> wrote:
> >
> > On Thu, Feb 14, 2019 at 11:31:24AM -0800, Dan Williams wrote:
> > > On Thu, Feb 14, 2019 at 11:10 AM Jerome Glisse <[email protected]> wrote:
> > > > I am just again working on my struct page mapping patchset as well as
> > > > the generic page write protection that sits on top. I hope to be able
> > > > to post the v2 in couple weeks. You can always look at my posting last
> > > > year to see more details.
> > >
> > > Yes, I have that in mind as one of the contenders. However, it's not
> > > clear to me that its a suitable fit for filesystem-reflink. Others
> > > have floated the 'page proxy' idea, so it would be good to discuss the
> > > merits of the general approaches.
> >
> > ... and my preferred option of putting pfn entries in the page cache.
>
> Another option to include the discussion.
>
> > Or is that what you meant by "page proxy"?
>
> Page proxy would be an object that a filesystem could allocate to
> point back to a single physical 'struct page *'. The proxy would
> contain an override for page->index.
Needs to override page->mapping, potentially the page flags, what
happens when someone takes a gup reference to the proxy structure,
etc?
Besides, didn't we discuss all this last year? how about we start
with a summary of all the options considered last year, the pros and
cons, etc, and then go from there? Regurgitating everything we've
already talked about last year doesn't seem particularly useful to
me...
Cheers,
Dave.
--
Dave Chinner
[email protected]
On 2/6/19 4:12 PM, Dan Williams wrote:
> Before people get too excited this isn't a proposal to kill DAX. The
> topic proposal is a discussion to resolve lingering open questions
> that currently motivate ext4 and xfs to scream "EXPERIMENTAL" when the
> current DAX facilities are enabled. The are 2 primary concerns to
> resolve. Enumerate the remaining features/fixes, and identify a path
> to implement it all without regressing any existing application use
> cases.
>
> An enumeration of remaining projects follows, please expand this list
> if I missed something:
>
> * "DAX" has no specific meaning by itself, users have 2 use cases for
> "DAX" capabilities: userspace cache management via MAP_SYNC, and page
> cache avoidance where the latter aspect of DAX has no current api to
> discover / use it. The project is to supplement MAP_SYNC with a
> MAP_DIRECT facility and MADV_SYNC / MADV_DIRECT to indicate the same
> dynamically via madvise. Similar to O_DIRECT, MAP_DIRECT would be an
> application hint to avoid / minimiize page cache usage, but no strict
> guarantee like what MAP_SYNC provides.
Sounds like a great topic to me. Having just gone through a new round of USENIX
paper reviews, it is interesting to see how many academic systems are being
pitched in this space (and most of them don't mention the kernel based xfs/ext4
with dax).
Regards,
Ric
>
> * Resolve all "if (dax) goto fail;" patterns in the kernel. Outside of
> longterm-GUP (a topic in its own right) the projects here are
> XFS-reflink and XFS-realtime-device support. DAX+reflink effectively
> requires a given physical page to be mapped into two different inodes
> at different (page->index) offsets. The challenge is to support
> DAX-reflink without violating any existing application visible
> semantics, the operating assumption / strawman to debate is that
> experimental status is not blanket permission to go change existing
> semantics in backwards incompatible ways.
>
> * Deprecate, but not remove, the DAX mount option. Too many flows
> depend on the option so it will never go away, but the facility is too
> coarse. Provide an option to enable MAP_SYNC and
> more-likely-to-do-something-useful-MAP_DIRECT on a per-directory
> basis. The current proposal is to allow this property to only be
> toggled while the directory is empty to avoid the complications of
> racing page invalidation with new DAX mappings.
>
>
> Secondary projects, i.e. important but I would submit are not in the
> critical path to removing the "experimental" designation:
> * Filesystem-integrated badblock management. Hook up the media error
> notifications from libnvdimm to the filesystem to allow for operations
> like "list files with media errors" and "enumerate bad file offsets on
> a granulatiy smaller than a page". Another consideration along these
> lines is to integrate machine-check-handling and dynamic error
> notification into a filesystem interface. I've heard complaints that
> the sigaction() based mechanism to receive BUS_MCEERR_* information,
> while sufficient for the "System RAM" use case, is not precise enough
> for the "Persistent Memory / DAX" use case where errors are repairable
> and sub-page error information is useful.
>
> * Userfaultfd for file-backed mappings and DAX
>
>
> Ideally all the usual DAX, persistent memory, and GUP suspects could
> be in the room to discuss this:
> * Jan Kara
> * Dave Chinner
> * Christoph Hellwig
> * Jeff Moyer
> * Johannes Thumshirn
> * Matthew Wilcox
> * John Hubbard
> * Jérôme Glisse
> * MM folks for the reflink vs 'struct page' vs Xarray considerations
On Sun, Feb 17, 2019 at 12:29 PM Ric Wheeler <[email protected]> wrote:
>
> On 2/6/19 4:12 PM, Dan Williams wrote:
> > Before people get too excited this isn't a proposal to kill DAX. The
> > topic proposal is a discussion to resolve lingering open questions
> > that currently motivate ext4 and xfs to scream "EXPERIMENTAL" when the
> > current DAX facilities are enabled. The are 2 primary concerns to
> > resolve. Enumerate the remaining features/fixes, and identify a path
> > to implement it all without regressing any existing application use
> > cases.
> >
> > An enumeration of remaining projects follows, please expand this list
> > if I missed something:
> >
> > * "DAX" has no specific meaning by itself, users have 2 use cases for
> > "DAX" capabilities: userspace cache management via MAP_SYNC, and page
> > cache avoidance where the latter aspect of DAX has no current api to
> > discover / use it. The project is to supplement MAP_SYNC with a
> > MAP_DIRECT facility and MADV_SYNC / MADV_DIRECT to indicate the same
> > dynamically via madvise. Similar to O_DIRECT, MAP_DIRECT would be an
> > application hint to avoid / minimiize page cache usage, but no strict
> > guarantee like what MAP_SYNC provides.
>
>
> Sounds like a great topic to me. Having just gone through a new round of USENIX
> paper reviews, it is interesting to see how many academic systems are being
> pitched in this space (and most of them don't mention the kernel based xfs/ext4
> with dax).
Makes sense, the current fsdax facility is chasing feature
deficiencies relative to device-dax (longterm pin support) and nova
(dax techniques for fs-metadata).