The uploaded patches implement NLM lock failover as discussed in:
http://www.redhat.com/archives/linux-cluster/2006-June/msg00050.html
[PATCH 1/3]
Add a new admin interface into current nfsd procfs filesystem to trigger
NLM lock releasing logic. The command is invoked by echoing the server
virtual IP address into /proc/fs/nfsd/nlm_unlock file as:
shell> cd /prod/fs/nfsd
shell> echo 10.10.1.1 > nlm_unlock
It is currently restricted to IPV4 addressing and the command should be
invoked from the taken-over NFS server. To do list (not yet implemented)
include:
1. IPv6 addessing enablement
2. Add "client:server" ip pair to allow NFS V4 lock failover as proposed
by Andy Adamson (CITI).
[PATCH 2/3]
Add take-over server counter-part command into nfsd procfs interface to
allow selectively setting of per (virtual) ip (lockd) grace period. It
is also invoked by echoing the virtual IP into
/proc/fs/nfsd/nlm_set_ip_grace file as:
shell> cd /proc/fs/nfsd
shell> echo 10.10.1.1 > nlm_set_ip_grace
It is also restricted to IPV4 addressing and the command is expected to
be invoked from the take-over NFS server.
[PATCH 3/3]
This kernel patch has *not* been unit tested out yet and it needs to be
paired with user mode nfs-utils changes (not ready in time for this
RFC). It puts the taken-over IPv4 address in standard dot notation into
the 3rd parameter of ha_callout program (see man rpc.statd for details)
for "add-client" event. For "del-client", we would assume the monitored
host should be removed from machine-wide lists, regardless server's
interface.
Upon agree-ed on, we will integrate the changes into our cluster suite
to start a full function verification test. Please comment.
-- Wendy
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Fri, 2006-08-04 at 10:56 -0400, Wendy Cheng wrote:
> Anyway, better be conservative than sorry - I think we want to switch to
> "fsid" approach to avoid messing with these networking issues, including
> IPV6 modification. That is, we use fsid as the key to drop the lock and
> set per-fsid NLM grace period. The ha-callout will have a 4th argument
> (fsid) when invoked.
What is the point of doing that? As far as the client is concerned, a
server has either rebooted or it hasn't. It doesn't know about single
filesystems rebooting.
Cheers,
Trond
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Fri, 2006-08-04 at 11:51 -0400, Trond Myklebust wrote:
> On Fri, 2006-08-04 at 10:56 -0400, Wendy Cheng wrote:
> > Anyway, better be conservative than sorry - I think we want to switch to
> > "fsid" approach to avoid messing with these networking issues, including
> > IPV6 modification. That is, we use fsid as the key to drop the lock and
> > set per-fsid NLM grace period. The ha-callout will have a 4th argument
> > (fsid) when invoked.
>
> What is the point of doing that? As far as the client is concerned, a
> server has either rebooted or it hasn't. It doesn't know about single
> filesystems rebooting.
>
For active-active failover, the submitted patches allow:
1: Drop the locks tied with one particular floating ip (in old server).
2: Notify relevant clients that the floating ip has been rebooted.
3: Set per-ip nlm grace period.
4: The (notified) nfs clients reclaim locks into the new server.
While the above 4 steps are being executed, both servers keep alive with
other nfs services un-interrupted. (1) and (3) are accomplished by Patch
3-1 and Patch 3-2. (4) is nfs client's task that follows its existing
logic without changes.
For (2), the basics are built upon the existing rpc.statd's HA features,
specifically the -H and -nNP option. It, however, needs Patch 3-3 to
pass the correct floating ip address into rpc.statd user mode daemon as
the following:
For system not involved in HA failover, nothing has change. All new
functions are optional with added-on feature. For cluster failover,
1. The rpc.statd is dispatched as "rpc.statd -H ha-callout"
2. Upon each monitor RPC calls (SM_MON or SM_UNMON), rpc.statd
received the following from kernel:
2-a: event (mon or unmon)
2-b: server interface
2-c: client interface.
3. The rpc.statd does its normal chores by writing or deleting
the client interface to/from the default sm directory. Server
interface is not used here.
(btw, this is the existing logic without changes).
4. Then, the rpc.statd invokes ha-callout with the following three
arguments:
4-a: event (add-client or del-client)
4-b: server interface
4-c: client interface
The ha-callout (in our case, it will be part of RHCS cluster suite)
builds multiple sm directories based on 4-b, say
/shared_storage/sm_x, where x is server's ip interface.
5. Upon failover, the cluster suite invokes
"rpc.statd -n x -N -P /shared_storage/sm_x" to notify affected
clients. The new short-life rpc.statd will send the notification to
relevant (nlm) clients and subsequently exits. The old rpc.statd
(from step 1) is not aware of the failover event.
Note that before patch 3-3, the kernel always sets 2-b to
system_utsname.nodename. For rpc.statd, if RESTRICTED_STATD flag is on,
the rpc.statd always set 4-b to 127.0.0.1. Without RESTRICTED_STATD on,
it sets 4-b with whatever was passed by kernel (via 2-b). What (kernel)
patch 3-3 does is setting 2-b to the floating ip so rpc.statd could get
the correct ip and pass it into 4-b.
Greg said (I havn't figured out how) without setting 4-b to 127.0.0.1,
we "may" open a security hole. So the thinking here is, then, let's not
change anything but add an fsid as 4th argument for ha-callout as:
4-d: fsid.
where "fsid" can be viewed as an unique identifier for an NFS export
specified in exports file (check "man exports"); e.g.
/failover_dir *(rw, fsid=1234)
With the added fsid info from ha-callout program, the cluster suite (or
human administrator) should be able to associated which (nlm) client has
been affected by one particular failover.
>From implementation point of view, since fsid, if specified, has already
been part of the filehandle that is part of the nlm_file structure, we
should be able to replace the floating ip in the submitted patches with
fsid and still accomplish the very same thing. In short, the failover
sequence with the new interface would look like:
taken-over server:
A-1. tear down floating ip, say 10.10.1.1
A-2. unexport subject filesystem
A-3. "echo 1234 > /proc/fs/nfsd/nlm_unlock" //fsid=1234
A-4. umount filesystem.
take-over server:
B-1. mount the subject filesystem
B-2. "echo 1234 > /proc/fs/nfsd/nlm_set_ip_grace"
B-3. "rpc.statd -n 10.10.1.1 -N -P /shared_storage/sm_10.10.1.1"
B-4. bring up 10.10.1.1
B-5. re-export the filesystem
A-3 and B-2 could be issued multiple times if the floating ip is
associated with multiple fsid(s).
Make sense ?
This fsid can also resolve Neil's concern (about nlm client using wrong
server interface to access filesystem) that I'll follow up sometime next
week.
-- Wendy
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Fri, 2006-08-04 at 23:27, Wendy Cheng wrote:
> On Fri, 2006-08-04 at 19:27 +1000, Greg Banks wrote:
> > On Tue, 2006-08-01 at 11:55, Wendy Cheng wrote:
> > > o The nfs-utils config flag RESTRICTED_STATD must be off for NLM
> > > failover to be functional correctly.
> >
> > That would reopen this ancient security hole:
> >
> > http://www.cert.org/advisories/CA-99-05-statd-automountd.html
> >
> > which might not be the best of ideas.
> >
>
> ok, thanks ! I'll look into this. But I believe nfs-utils-1.0.8-rc4 has
> this off by default ?
I really hope distros have --enable-secure-statd in their .specs.
I know SLES9+ doesn't need it, because SLES has Olaf's in-kernel
rpc.statd which (IIRC) has the equivalent of RESTRICTED_STATD
hardcoded.
Greg.
--
Greg Banks, R&D Software Engineer, SGI Australian Software Group.
I don't speak for SGI.
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Sat, 2006-08-05 at 15:44, Wendy Cheng wrote:
> On Fri, 2006-08-04 at 11:51 -0400, Trond Myklebust wrote:
> > On Fri, 2006-08-04 at 10:56 -0400, Wendy Cheng wrote:
> Note that before patch 3-3, the kernel always sets 2-b to
> system_utsname.nodename. For rpc.statd, if RESTRICTED_STATD flag is on,
> the rpc.statd always set 4-b to 127.0.0.1. Without RESTRICTED_STATD on,
> it sets 4-b with whatever was passed by kernel (via 2-b). What (kernel)
> patch 3-3 does is setting 2-b to the floating ip so rpc.statd could get
> the correct ip and pass it into 4-b.
>
> Greg said (I havn't figured out how) without setting 4-b to 127.0.0.1,
> we "may" open a security hole.
Aha, I see what you needed. You could have changed the logic
in the RESTRICTED_STATD case of sm_mon_1_svc() not to ignore the
passed my_addr.s_addr if svc_getcaller(rqstp->rq_xprt) is a
privileged port on localhost. This would probably give you your
logic without reopening the security hole.
> take-over server:
> B-1. mount the subject filesystem
> B-2. "echo 1234 > /proc/fs/nfsd/nlm_set_ip_grace"
> B-3. "rpc.statd -n 10.10.1.1 -N -P /shared_storage/sm_10.10.1.1"
> B-4. bring up 10.10.1.1
> B-5. re-export the filesystem
Umm, don't you want to do B-3 after B-4 and B-5 ? Otherwise
clients might racily fail on the first try.
Also, just curious here, when do you purge the clients' ARP caches?
Greg.
--
Greg Banks, R&D Software Engineer, SGI Australian Software Group.
I don't speak for SGI.
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
For background info, please check out:
o http://www.redhat.com/archives/linux-cluster/2006-June/msg00050.html
for interface discussion.
o https://www.redhat.com/archives/cluster-devel/2006-June/msg00231.html
for first drafted code review.
Note and Restrictions:
o With nfs-utils-1.0.8-rc4 and nfs-utils-lib-1.0.8, the tests went
surprisingly well, particularly the ha-callout feature. *No* change
is made into these two user mode utility packages.
o The nfs-utils config flag RESTRICTED_STATD must be off for NLM
failover to be functional correctly.
o The third parameter passed to rpc.statd ha-callout program is no
longer be the system_utsname.nodename (set by sethostname()). It
is, instead, the specific IP interface where the server receives
the client's request.
o The patches are for NFS v2/v3 only. However, we do leave room for
future NFS V4 expansion. For example, echoing client_ip@server_ip
into /proc/fs/nfsd/nlm_unlock could be used to drop the V4 locks.
o IP V6 modification is not included in this patch set. If required,
it will be submitted as another patch set.
PATCH 1/3
---------
Add a new admin interface into current nfsd procfs filesystem to trigger
NLM lock releasing logic. The command is invoked by echoing the server
IP V4 address (in standard dot notation) into /proc/fs/nfsd/nlm_unlock
file as:
shell> cd /prod/fs/nfsd
shell> echo 10.10.1.1 > nlm_unlock
PATCH 2/3
---------
Add take-over server counter-part command into nfsd procfs interface to
allow selectively setting of per (virtual) ip (lockd) grace period. The
grace period setting follows current system-wide grace period rule and
default. It is also invoked by echoing the server IP V4 address (in dot
notation) into /proc/fs/nfsd/nlm_set_ip_grace file:
shell> cd /proc/fs/nfsd
shell> echo 10.10.1.1 > nlm_set_ip_grace
PATCH 3/3
---------
This kernel patch adds a new field into struct nlm_host that holds the
server IP address. Upon SM_MON and SM_UNMON procedure calls, the IP (in
standard V4 dot notation) is placed as the "my_name" string and passed
to local statd daemon. This enables ha-callout program ("man rpc.statd"
for details) to receive the IP address of the server that has received
the client's request. Before this change, my_name string is universally
set to system_utsname.nodename.
The user mode HA implementation is expected to:
1. Specify a user mode ha-calloupt program (rpc.statd -H) for receiving
client monitored states.
2. Based on info passed by ha-callout, individual state-directory should
be created and can be read from take-over server.
3. Upon failover, on take-over server, send out notification to nfs
client via (rpc.statd -n server_ip -P individual_state_directory -N)
command.
-- Wendy
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Thanks for these.
First note: it helps a lot if the Subject line for each patch
contains a distinctive short description of what the patch does.
Rather than just "NLM lock failover" three times, maybe
Add nlm_lock file to nfsd fs to allow selective unlocked based on server IP
Add nlm_set_ip_grace file to allow selective setting of grace time
Use IP address rather than hostname in SM_{UN,}MON calls to statd
Or something like that.
>
> PATCH 1/3
> ---------
> Add a new admin interface into current nfsd procfs filesystem to trigger
> NLM lock releasing logic. The command is invoked by echoing the server
> IP V4 address (in standard dot notation) into /proc/fs/nfsd/nlm_unlock
> file as:
>
> shell> cd /prod/fs/nfsd
> shell> echo 10.10.1.1 > nlm_unlock
>
This patch makes an assumption that any given filehandle will only arrive at
one particular interface - never more. This is implicit in the fact
that f_iaddr is stored in 'struct nlm_file' which is indexed by
filehandle.
In the case where you are intending to support active-active failover
this is should be the case, but obviously configuration errors are
possible.
I think what I would like is that if requests arrive at two (or more)
different interfaces for the one file, then f_iaddr is cleared and some
flag is set.
When an IP is written to nlm_unlock, if the flag is set, then a
warning message is printer
Note: some files access via multiple interfaces will not be
unlocked.
A consequence of this is that you cannot have a virtual server with
two (or more interfaces). Is this likely to be a problem?
e.g. if you have 4 physical interfaces on your server, might you want
to bind a different IP to each for each virtual server?
If you did, then my change above would mean that you couldn't do
failover, and we might need to look at other options...
Possibly (and maybe this is more work than is justified), lockd can
monitor interface usage and deduce interface pools based on seeing the
same filehandle on multiple interfaces. Then when an unlock request
arrives on nlm_unlock, lockd would require all interfaces that touched
a file to be 'unlocked' before actually dropping the locks on the
file.
As you can probably tell I was "thinking out loud" there and it may
not be particularly coherent or cohesive.
Do you have any thoughts on this issues?
> PATCH 2/3
> ---------
> Add take-over server counter-part command into nfsd procfs interface to
> allow selectively setting of per (virtual) ip (lockd) grace period. The
> grace period setting follows current system-wide grace period rule and
> default. It is also invoked by echoing the server IP V4 address (in dot
> notation) into /proc/fs/nfsd/nlm_set_ip_grace file:
>
> shell> cd /proc/fs/nfsd
> shell> echo 10.10.1.1 > nlm_set_ip_grace
>
I think nlm_ip_mutex should be a spinlock, and I don't think you
should need to hold the lock in __nlm_servs_gc, as you have already
disconnected these entries from the global list.
+extern unsigned long set_grace_period(void); /* see fs/lockd/svc.c */
That should go in a header file.
+ switch (passthru_check) {
+ case NLMSVC_FO_PASSTHRU:
+ break;
+ case NLMSVC_FO_RECLAIM:
+ if (argp->reclaim) break;
+ default:
+ return nlm_lck_denied_grace_period;
+ }
I'd rather you spelt out
case NLMSVC_FO_BLOCK_ANY:
rather than used 'default:' - it makes the code more readable.
and surely you should check for NLMSVC_FO_PASSTHRU before calling
nlmsvc_fo_check ???
It seems to me that it would be clearer not to put the nlmsvc_fo_check
call inside nlm*svc_retrieve_args, but rather to define e.g.
nlmsvc_is_grace_period(rqstp)
which checks nlmsvc_grace_period and the list of IPs, and then replace
every
if (nlmsvc_grace_period) {
resp->status = nlm_lck_denied_grace_period;
return rpc_success;
}
with
if (nlmsvc_is_grace_period(rqstp)) {
resp->status = nlm_lck_denied_grace_period;
return rpc_success;
}
and every
if (nlmsvc_grace_period && !argp->reclaim) {
resp->status = nlm_lck_denied_grace_period;
return rpc_success;
}
with
if (!argp->reclaim && nlmsvc_is_grace_period(rqstp)) {
resp->status = nlm_lck_denied_grace_period;
return rpc_success;
}
Does that seem reasonable to you?
> PATCH 3/3
> ---------
> This kernel patch adds a new field into struct nlm_host that holds the
> server IP address. Upon SM_MON and SM_UNMON procedure calls, the IP (in
> standard V4 dot notation) is placed as the "my_name" string and passed
> to local statd daemon. This enables ha-callout program ("man rpc.statd"
> for details) to receive the IP address of the server that has received
> the client's request. Before this change, my_name string is universally
> set to system_utsname.nodename.
>
> The user mode HA implementation is expected to:
> 1. Specify a user mode ha-calloupt program (rpc.statd -H) for receiving
> client monitored states.
> 2. Based on info passed by ha-callout, individual state-directory should
> be created and can be read from take-over server.
> 3. Upon failover, on take-over server, send out notification to nfs
> client via (rpc.statd -n server_ip -P individual_state_directory -N)
> command.
Was it necessary to rename "s_server" to "s_where"? Could you not
have just introduced "s_server_ip"???
And if you really want s_where, then I would like some #defines and
make
+ if (server)
+ host->h_where = 1;
+ else
+ host->h_where = 0;
something like
host->where = server ? ON_SERVER : ON_CLIENT
(reading some more) In fact, you don't need h_where at all. Just
change h_server to be the IP address, and then use e.g.
- args.proto= (host->h_proto<<1) | host->h_server;
+ args.serv = host->h_server;
+ args.proto= (host->h_proto<<1) | (host->h_server?1:0)
(or host->server!=0 or !!host->server - whatever takes your fancy).
@@ -142,7 +142,7 @@ out_err:
static u32 *
xdr_encode_common(struct rpc_rqst *rqstp, u32 *p, struct nsm_args *argp)
{
- char buffer[20];
+ char buffer[100];
This looks like it should really be a separate patch, and should
probably be __NEW_UTS_LEN+1 rather than 100.
But I worry about other people who might be using ha-callout already
and expecting a host name there. We are making a non-optional
user-visible change here. Is it really a safe thing to do?
Thanks,
NeilBrown
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Tue, 2006-08-01 at 11:55, Wendy Cheng wrote:
> o The nfs-utils config flag RESTRICTED_STATD must be off for NLM
> failover to be functional correctly.
That would reopen this ancient security hole:
http://www.cert.org/advisories/CA-99-05-statd-automountd.html
which might not be the best of ideas.
Greg.
--
Greg Banks, R&D Software Engineer, SGI Australian Software Group.
I don't speak for SGI.
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Fri, 2006-08-04 at 19:27 +1000, Greg Banks wrote:
> On Tue, 2006-08-01 at 11:55, Wendy Cheng wrote:
> > o The nfs-utils config flag RESTRICTED_STATD must be off for NLM
> > failover to be functional correctly.
>
> That would reopen this ancient security hole:
>
> http://www.cert.org/advisories/CA-99-05-statd-automountd.html
>
> which might not be the best of ideas.
>
ok, thanks ! I'll look into this. But I believe nfs-utils-1.0.8-rc4 has
this off by default ?
-- Wendy
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Fri, 2006-08-04 at 09:27 -0400, Wendy Cheng wrote:
> On Fri, 2006-08-04 at 19:27 +1000, Greg Banks wrote:
> > On Tue, 2006-08-01 at 11:55, Wendy Cheng wrote:
> > > o The nfs-utils config flag RESTRICTED_STATD must be off for NLM
> > > failover to be functional correctly.
> >
> > That would reopen this ancient security hole:
> >
> > http://www.cert.org/advisories/CA-99-05-statd-automountd.html
> >
> > which might not be the best of ideas.
> >
>
> ok, thanks ! I'll look into this. But I believe nfs-utils-1.0.8-rc4 has
> this off by default ?
>
Anyway, better be conservative than sorry - I think we want to switch to
"fsid" approach to avoid messing with these networking issues, including
IPV6 modification. That is, we use fsid as the key to drop the lock and
set per-fsid NLM grace period. The ha-callout will have a 4th argument
(fsid) when invoked.
-- Wendy
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs