Revised patches based on 2.6.21-rc4 kernel and nfs-utils-1.1.0-rc1 that
address issues discussed in:
https://www.redhat.com/archives/cluster-devel/2006-September/msg00034.html
Quick How-to:
1) Failover server exports filesystem with "fsid" option as:
/etc/exports entry> /mnt/shared/exports *(fsid=1234,sync,rw)
2) Failover server dispatch rpc.statd with "-H" option.
3) Failover server drops locks based on fsid by:
shell> echo 1234 > /proc/fs/nfsd/nlm_unlock
4) Takeover server enters per fsid grace period by:
shell> echo 1234 > /proc/fs/nfsd/nlm_set_igrace
5) Takeover server notifies clients for lock reclaim by:
shell> /usr/sbin/sm-notify -f -v floating_ip_address -P an_sm_directory
Patch Summary:
4-1: implement /proc/fs/nfsd/nlm_unlock
4-2: implement /proc/fs/nfsd/nlm_set_igrace
4-3: correctly record and pass incoming server ip interface into rpc.statd.
4-4: nfs-utils statd changes
4-1 includes an existing lockd bug fix as discussed in:
http://sourceforge.net/mailarchive/forum.php?thread_name=4603506D.5040807%40redhat.com&forum_name=nfs
(subject: [NFS] Question about f_count in struct nlm_file)
4-4 includes an existing nfs-utils statd bug fix as discussed in:
http://sourceforge.net/mailarchive/message.php?msg_name=46142B4F.1030507%40redhat.com
(subject: Re: [NFS] lockd and statd)
Misc:
o No IPV6 support due to testing efforts
o NFS V3 only - will compare notes with CITI folks (NFS V4 issues)
o Still need some error-inject tests.
-- Wendy
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Thu, Apr 05, 2007 at 05:50:55PM -0400, Wendy Cheng wrote:
> 1) Failover server exports filesystem with "fsid" option as:
> /etc/exports entry> /mnt/shared/exports *(fsid=1234,sync,rw)
> 2) Failover server dispatch rpc.statd with "-H" option.
> 3) Failover server drops locks based on fsid by:
> shell> echo 1234 > /proc/fs/nfsd/nlm_unlock
> 4) Takeover server enters per fsid grace period by:
> shell> echo 1234 > /proc/fs/nfsd/nlm_set_igrace
> 5) Takeover server notifies clients for lock reclaim by:
> shell> /usr/sbin/sm-notify -f -v floating_ip_address -P an_sm_directory
I don't understand statd and lockd as well as I should. Where exactly
does the takeover server stop serving requests, and the failover server
start? If this isn't done carefully, you can leave a window between
steps 3 and 4 where a client could acquire a lock before its rightful
owner reclaims it, right?
--b.
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
J. Bruce Fields wrote:
> On Thu, Apr 05, 2007 at 05:50:55PM -0400, Wendy Cheng wrote:
>
>> 1) Failover server exports filesystem with "fsid" option as:
>> /etc/exports entry> /mnt/shared/exports *(fsid=1234,sync,rw)
>> 2) Failover server dispatch rpc.statd with "-H" option.
>> 3) Failover server drops locks based on fsid by:
>> shell> echo 1234 > /proc/fs/nfsd/nlm_unlock
>> 4) Takeover server enters per fsid grace period by:
>> shell> echo 1234 > /proc/fs/nfsd/nlm_set_igrace
>> 5) Takeover server notifies clients for lock reclaim by:
>> shell> /usr/sbin/sm-notify -f -v floating_ip_address -P an_sm_directory
>>
>
> I don't understand statd and lockd as well as I should. Where exactly
> does the takeover server stop serving requests, and the failover server
> start? If this isn't done carefully, you can leave a window between
> steps 3 and 4 where a client could acquire a lock before its rightful
> owner reclaims it, right?
>
>
The detailed overall steps were described in the first email we sent
*long* time (> 6 months, I think) ago. The first step of the whole
process is tearing down the floating IP from the failover server. The IP
is not accessible until filesystem is safely fail-over and SM_NOTIFY
ready to be sent.
Last round of discussion gave me an impression that as long as I rebased
the code into akpm's mm tree, these patches would get accepted. So I
have been quite careless in this submission and just realized people
have a very short memory :) .. Will do the write-up and put it somewhere
so we don't need to go thru this again.
-- Wendy
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
[email protected] wrote on 04/25/2007 07:10:31 AM:
> J. Bruce Fields wrote:
> > On Thu, Apr 05, 2007 at 05:50:55PM -0400, Wendy Cheng wrote:
> >
> >> 1) Failover server exports filesystem with "fsid" option as:
> >> /etc/exports entry> /mnt/shared/exports *(fsid=1234,sync,rw)
> >> 2) Failover server dispatch rpc.statd with "-H" option.
> >> 3) Failover server drops locks based on fsid by:
> >> shell> echo 1234 > /proc/fs/nfsd/nlm_unlock
> >> 4) Takeover server enters per fsid grace period by:
> >> shell> echo 1234 > /proc/fs/nfsd/nlm_set_igrace
> >> 5) Takeover server notifies clients for lock reclaim by:
> >> shell> /usr/sbin/sm-notify -f -v floating_ip_address -P
an_sm_directory
> >>
> >
> > I don't understand statd and lockd as well as I should. Where exactly
> > does the takeover server stop serving requests, and the failover
server
> > start? If this isn't done carefully, you can leave a window between
> > steps 3 and 4 where a client could acquire a lock before its rightful
> > owner reclaims it, right?
> >
> >
> The detailed overall steps were described in the first email we sent
> *long* time (> 6 months, I think) ago. The first step of the whole
> process is tearing down the floating IP from the failover server. The IP
> is not accessible until filesystem is safely fail-over and SM_NOTIFY
> ready to be sent.
I thought this is a solution for an active active server where a cluster
file system can export the same file system from multiple NFS servers.
Marc.
>
> Last round of discussion gave me an impression that as long as I rebased
> the code into akpm's mm tree, these patches would get accepted. So I
> have been quite careless in this submission and just realized people
> have a very short memory :) .. Will do the write-up and put it somewhere
> so we don't need to go thru this again.
>
> -- Wendy
>
>
>
-------------------------------------------------------------------------
> This SF.net email is sponsored by DB2 Express
> Download DB2 Express C - the FREE version of DB2 express and take
> control of your XML. No limits. Just data. Click to get it now.
> http://sourceforge.net/powerbar/db2/
> _______________________________________________
> NFS maillist - [email protected]
> https://lists.sourceforge.net/lists/listinfo/nfs
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Marc Eshel wrote:
>> The detailed overall steps were described in the first email we sent
>> *long* time (> 6 months, I think) ago. The first step of the whole
>> process is tearing down the floating IP from the failover server. The IP
>> is not accessible until filesystem is safely fail-over and SM_NOTIFY
>> ready to be sent.
>>
>
> I thought this is a solution for an active active server where a cluster
> file system can export the same file system from multiple NFS servers.
> Marc.
>
>
Yes ... but remember we should have two cases here:
1) Local filesystems such as ext3 - both IP and filesystem are not
accessible during the transition.
2) Cluster filesystem such as GFS or GPFS - filesystem may still be
accessible (depending on the configuration, say you have advertised two
exported IP addresses, each serving different subdirectories for the
very same cluster filesystem). The failover IP should be suspended
during the transition until SM_NOTIFY is ready to go out (but the other
IP should be up and services the requests as it should).
I assume people understand that the affected export entries should have
been un-exported (as part of the over-all process).
-- Wendy
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Wed, Apr 25, 2007 at 10:10:31AM -0400, Wendy Cheng wrote:
> The detailed overall steps were described in the first email we sent
> *long* time (> 6 months, I think) ago. The first step of the whole
> process is tearing down the floating IP from the failover server. The IP
> is not accessible until filesystem is safely fail-over and SM_NOTIFY
> ready to be sent.
I understand, thanks.
> Last round of discussion gave me an impression that as long as I rebased
> the code into akpm's mm tree, these patches would get accepted. So I
> have been quite careless in this submission and just realized people
> have a very short memory :) .. Will do the write-up and put it somewhere
> so we don't need to go thru this again.
Yeah, apologies for the short memory. I'll try to follow more closely
from now on!
If practical, it would be helpful to have any such documentation in the
final version of the patches, though, either as patch comments or (maybe
better in this case) as comments in the code or in Documentation/. When
someone needs to go back and find out how this was all meant to work,
it'll be easier to find in the source tree or the git history than in
the mail archives.
--b.
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Wendy Cheng wrote:
> Marc Eshel wrote:
>>> The detailed overall steps were described in the first email we sent
>>> *long* time (> 6 months, I think) ago. The first step of the whole
>>> process is tearing down the floating IP from the failover server.
>>> The IP is not accessible until filesystem is safely fail-over and
>>> SM_NOTIFY ready to be sent.
>>
>> I thought this is a solution for an active active server where a
>> cluster file system can export the same file system from multiple NFS
>> servers.
>> Marc.
>>
>
> Yes ... but remember we should have two cases here:
>
> 1) Local filesystems such as ext3 - both IP and filesystem are not
> accessible during the transition.
> 2) Cluster filesystem such as GFS or GPFS - filesystem may still be
> accessible (depending on the configuration, say you have advertised
> two exported IP addresses, each serving different subdirectories for
> the very same cluster filesystem). The failover IP should be suspended
> during the transition until SM_NOTIFY is ready to go out (but the
> other IP should be up and services the requests as it should).
>
> I assume people understand that the affected export entries should
> have been un-exported (as part of the over-all process).
To remind people, we had described the overall steps in the first round
of kernel interface discussions submitted to nfs mailing list (before
implemented the code). These steps included un-export the entries,
tearing down the IP, filesystem migration, SM_NOTIFY, grace period, etc.
I'm in the middle of redoing the write-up. It will be uploaded into
somewhere soon.
-- Wendy
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
J. Bruce Fields wrote:
> If practical, it would be helpful to have any such documentation in the
> final version of the patches, though, either as patch comments or (maybe
> better in this case) as comments in the code or in Documentation/. When
> someone needs to go back and find out how this was all meant to work,
> it'll be easier to find in the source tree or the git history than in
> the mail archives.
>
>
I'm correcting the oversight at this moment .... had been working in
cluster-only development group(s) for too long and forgot NFS mailing
list is a community consisting of people from different background(s).
-- Wendy
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Few new thoughts from the latest round of review are really good and
worth doing....
However, since this particular NLM patch set is only part of the overall
scaffolding code to allow NFS V3 server fail over before NFS V4 is
widely adopted and stabilized, I'm wondering whether we should drag
ourselves too far for something that will be replaced soon. Lon and I
had been discussing the possibility of proposing new design changes into
the existing state monitoring protocol itself - but I'm leaning toward
*not* doing client SM_NOTIFY eventually (by passing the lock states
directly from fail-over server to take-over server if all possible).
This would consolidate few next work items such as NFSD V3 request reply
cache entires (or at least non-idempotent operation entries) or NFS V4
states that need to get moved around between the fail over servers.
In general, NFS cluster failover has been error prone and has timing
constraints (e.g. failover must finish within a sensible time interval).
Would it make more sense to have a workable solution with restricted
application first ? We can always merge various pieces together later as
we learn more from our users. For this reasoning, simple and plain
patches like this set would work best for now.
In any case, the following collect the review comments so far:
o 1-1 [from hch]
"Dropping locks should also support uuid or dev_t based exports."
A valid request. The easiest solution might be simply taking Neil's idea
by using export path name. So this issue is combined into 1-3 (see below
for details).
o 1-2 [from hch]
"It would be nice to have a more general push api for changes to
filesystem state, that works on a similar basis as getting information
from /etc/exports."
Could hch (or anyone) elaborate more on this ? Should I interpret it as
implementing a configuration file (that describes the failover options
that has a format similar to /etc/exports (including filesystem
identifiers, the length of grace period, etc) and a command (maybe two -
one on failover server and one on take-over server) to kick off the
failover based on the pre-defined configuration file ?
o 1-3 [from neilb]
"It would seem to make more sense to use the filesystem name (i.e. a
path) by writing a directory name to /proc/fs/nfsd/nlm_unlock and maybe
also to /proc/fs/nlm_restart_grace_for_fs" and have 'my_name' in the
SM_MON request be the path name of the export point rather the network
address."
It was my mistake to mention that we could use "fsid" in the "my_name"
field in previous post. As Lon pointed out, SM_MON requires server
address so we do not blindly notify clients that could result with
unspecified behaviors. On the other hand, the "path name" idea does
solve various problems if we want to support different types of existing
filesystem identifiers for failover purpose. Combining the configuration
file mentioned in 1-2, this could be a nice long term solution. Few
concerns (about using path name alone) :
*String comparison can be error-prone and slow
* It loses the abstraction provided by the "fsid" approach, particularly
for a cluster filesystem load balancing purpose. With "fsid" approach,
we could simply export the same directory using two different fsid(s)
(associated with two different IP addresses) for various purposes on the
same node.
* Will have to repeatedly educate users that "dev_t" is not unique
across reboots or nodes; uuid is restricted to one single disk
partition; and both of them require extra steps to obtain the values
somewhere else that are not easily read by human eyes. My support
experiences taught me that by the time users really understand the
difference, they'll switch to fsid anyway.
1-4 [from bfields]
"Unrelated bug fix should break out from the feature patches".
Will do
2-1 [from cluster coherent NFS conf. call]
"Hooks to allow cluster filesystem does its own "start" and "stop" of
grace period."
This could be solved by using a configuration file as described in 1-2.
3-1 [from okir]
"There's not enough room in the SM_MON request to accommodate additional
network addresses (e.g. IPv6)".
SM_MON is sent and received *within* the very same server. Is it really
matter whether we follow the protocol standard in this particular RPC
call ? My guess is not. Current patch writes server IP into "my_name"
field as a variable length character array. I see no reason this can't
be a larger character array (say 128 bytes for IPV6) to accommodate all
the existing network addressing we know of.
3-2 [from okir]
"Should we think about replacing SM_MON with some new design altogether
(think netlink) ?"
Yes. But before we spend the efforts, I would suggest we focus on
1. Offering a tentative workable NFS V3 solution for our users first.
2. Check out the requirement from NFS V4 implementation so we don't end
up revising the new changes again when V4 failover arrives.
In short, my vote is taking this (NLM) patch set and let people try it
out while we switch our gear to look into other NFS V3 failover issues
(nfsd in particular). Neil ?
-- Wendy
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Arjan, I need an objective opinion and am wondering whether you could
give me some advices ...
I'm quite upset about this Christoph guy and really want to "talk back".
Will my response be too strong that ends up harm my later on patches ?
-- Wendy
Christoph Hellwig wrote:
> On Tue, Apr 17, 2007 at 10:11:13PM -0400, Wendy Cheng wrote:
> > However, since this particular NLM patch set is only part of the
> overall
> > scaffolding code to allow NFS V3 server fail over before NFS V4 is
> > widely adopted and stabilized, I'm wondering whether we should drag
> > ourselves too far for something that will be replaced soon.
>
> I don't think that's a valid argument. We hack this up because it's
> going to be obsolete mid-term never was a really good argument. And in
> this case it's a particularly bad one. People won't rush to NFSv4 just
> because someone declares it stable now. And if they did we couldn't
> simply rip out existing functionality.
>
The "hack" and "bad" are very subjective words in this context.
Comparing to many other code currently living inside Linux kernel tree,
this patch set, gone thru 3 rounds of extensive review and discussions,
deserves at least "average" standing in terms of solution, quality and
testing efforts.
On the other hand, I certainly welcome further constructive suggestions
and ideas though.
-- Wendy
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
oops :( ...
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Wed, Apr 18, 2007 at 02:56:18PM -0400, Wendy Cheng wrote:
> Arjan, I need an objective opinion and am wondering whether you could
> give me some advices ...
>
> I'm quite upset about this Christoph guy and really want to "talk back".
> Will my response be too strong that ends up harm my later on patches ?
I don't mind strong answers, but I'd generally prefer clueful strong
answers :)
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Christoph Hellwig wrote:
> On Wed, Apr 18, 2007 at 02:56:18PM -0400, Wendy Cheng wrote:
>
>> Arjan, I need an objective opinion and am wondering whether you could
>> give me some advices ...
>>
>> I'm quite upset about this Christoph guy and really want to "talk back".
>> Will my response be too strong that ends up harm my later on patches ?
>>
>
> I don't mind strong answers, but I'd generally prefer clueful strong
> answers :)
>
>
Well, I have been seriously considering your previous advice about
getting another job. Maybe working on a new email system that would
allow people to cancel or stop mistakenly sent out emails would be a
very good new project ? :)
Glad to know you don't feel offended.
-- Wendy
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Thu, Apr 05, 2007 at 05:50:55PM -0400, Wendy Cheng wrote:
> Revised patches based on 2.6.21-rc4 kernel and nfs-utils-1.1.0-rc1 that
> address issues discussed in:
> https://www.redhat.com/archives/cluster-devel/2006-September/msg00034.html
>
> Quick How-to:
> 1) Failover server exports filesystem with "fsid" option as:
> /etc/exports entry> /mnt/shared/exports *(fsid=1234,sync,rw)
> 2) Failover server dispatch rpc.statd with "-H" option.
> 3) Failover server drops locks based on fsid by:
> shell> echo 1234 > /proc/fs/nfsd/nlm_unlock
> 4) Takeover server enters per fsid grace period by:
> shell> echo 1234 > /proc/fs/nfsd/nlm_set_igrace
> 5) Takeover server notifies clients for lock reclaim by:
> shell> /usr/sbin/sm-notify -f -v floating_ip_address -P an_sm_directory
>
> Patch Summary:
> 4-1: implement /proc/fs/nfsd/nlm_unlock
> 4-2: implement /proc/fs/nfsd/nlm_set_igrace
> 4-3: correctly record and pass incoming server ip interface into rpc.statd.
> 4-4: nfs-utils statd changes
> 4-1 includes an existing lockd bug fix as discussed in:
> http://sourceforge.net/mailarchive/forum.php?thread_name=4603506D.5040807%40redhat.com&forum_name=nfs
> (subject: [NFS] Question about f_count in struct nlm_file)
That's the one separate chunk in nlm_traverse_files()? Could you keep
that split out as a separate patch? I see that it got some discussion
before but I'm not clear what the resolution was....
--b.
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs