From: Bryan Schumaker <[email protected]>
While working on p2p-nfs, I discovered that I sometimes need to clear
state for a specific client to test all possible error recovery conditions.
The current fault injection code deletes state as it find it, so it can
be difficult to guess which state will be forgotten. In addition, I
currently print out the amount of state forgotten but I don't give details
like "Forgot 3 locks from client w.x.y.z". These patches set out to
improve that.
The first 6 patches clean up the current code and prepare it for specific
client state removal. Patch 7 adds printing information to the server's logs
when a fault injection file is read (such as "Client w.x.y.z has 3 open
files"). Patch 8 adds in a custom file operations structure so users can
write strings to fault injection files in addition to u64s. Finally, patch
9 allows users to remove state by writing a client's IP address to one of
the files.
Changes in v4:
- Access lock state from open owners rather than requiring a network namespace
pointer.
- Don't produce a null pointer expection if fault injection is attempted after
loading the nfsd module but before running any nfsd threads.
- Bryan
Bryan Schumaker (9):
NFSD: Lock state before calling fault injection function
NFSD: Clean up forgetting clients
NFSD: Clean up forgetting locks
NFSD: Clean up forgetting openowners
NFSD: Clean up forgetting and recalling delegations
NFSD: Fault injection operations take a per-client forget function
NFSD: Reading a fault injection file prints a state count
NFSD: Add a custom file operations structure for fault injection
NFSD: Forget state for a specific client
fs/nfsd/fault_inject.c | 112 ++++++++++++++++++++++++----
fs/nfsd/netns.h | 3 +
fs/nfsd/nfs4state.c | 197 ++++++++++++++++++++++++++++++-------------------
fs/nfsd/state.h | 18 +++--
4 files changed, 237 insertions(+), 93 deletions(-)
--
1.8.0.1
From: Bryan Schumaker <[email protected]>
Each function touches state in some way, so getting the lock earlier
can help simplify code.
Signed-off-by: Bryan Schumaker <[email protected]>
---
fs/nfsd/fault_inject.c | 2 ++
fs/nfsd/nfs4state.c | 18 ++----------------
2 files changed, 4 insertions(+), 16 deletions(-)
diff --git a/fs/nfsd/fault_inject.c b/fs/nfsd/fault_inject.c
index 0278112..4b385a1 100644
--- a/fs/nfsd/fault_inject.c
+++ b/fs/nfsd/fault_inject.c
@@ -51,7 +51,9 @@ static int nfsd_inject_set(void *op_ptr, u64 val)
else
printk(KERN_INFO "NFSD Fault Injection: %s (n = %llu)", op->file, val);
+ nfs4_lock_state();
op->func(val);
+ nfs4_unlock_state();
return 0;
}
diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index b1aa577..8e19c69 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -4611,13 +4611,11 @@ void nfsd_forget_clients(u64 num)
int count = 0;
struct nfsd_net *nn = net_generic(current->nsproxy->net_ns, nfsd_net_id);
- nfs4_lock_state();
list_for_each_entry_safe(clp, next, &nn->client_lru, cl_lru) {
expire_client(clp);
if (++count == num)
break;
}
- nfs4_unlock_state();
printk(KERN_INFO "NFSD: Forgot %d clients", count);
}
@@ -4653,25 +4651,15 @@ static int nfsd_release_n_owners(u64 num, bool is_open_owner,
void nfsd_forget_locks(u64 num)
{
- int count;
struct nfsd_net *nn = net_generic(&init_net, nfsd_net_id);
-
- nfs4_lock_state();
- count = nfsd_release_n_owners(num, false, release_lockowner_sop, nn);
- nfs4_unlock_state();
-
+ int count = nfsd_release_n_owners(num, false, release_lockowner_sop, nn);
printk(KERN_INFO "NFSD: Forgot %d locks", count);
}
void nfsd_forget_openowners(u64 num)
{
- int count;
struct nfsd_net *nn = net_generic(&init_net, nfsd_net_id);
-
- nfs4_lock_state();
- count = nfsd_release_n_owners(num, true, release_openowner_sop, nn);
- nfs4_unlock_state();
-
+ int count = nfsd_release_n_owners(num, true, release_openowner_sop, nn);
printk(KERN_INFO "NFSD: Forgot %d open owners", count);
}
@@ -4704,10 +4692,8 @@ void nfsd_forget_delegations(u64 num)
count = nfsd_process_n_delegations(num, &victims);
spin_unlock(&recall_lock);
- nfs4_lock_state();
list_for_each_entry_safe(dp, dnext, &victims, dl_recall_lru)
unhash_delegation(dp);
- nfs4_unlock_state();
printk(KERN_INFO "NFSD: Forgot %d delegations", count);
}
--
1.8.0.1
From: Bryan Schumaker <[email protected]>
The eventual goal is to forget state based on ip address, so it makes
sense to call this function in a for-each-client loop until the correct
amount of state is forgotten. I also use this patch as an opportunity
to rename the forget function from "func()" to "forget()".
Signed-off-by: Bryan Schumaker <[email protected]>
---
fs/nfsd/fault_inject.c | 16 +++++++++-------
fs/nfsd/nfs4state.c | 30 ------------------------------
fs/nfsd/state.h | 12 +++++++-----
3 files changed, 16 insertions(+), 42 deletions(-)
diff --git a/fs/nfsd/fault_inject.c b/fs/nfsd/fault_inject.c
index 4b385a1..bf6161a 100644
--- a/fs/nfsd/fault_inject.c
+++ b/fs/nfsd/fault_inject.c
@@ -13,29 +13,29 @@
struct nfsd_fault_inject_op {
char *file;
- void (*func)(u64);
+ u64 (*forget)(struct nfs4_client *, u64);
};
static struct nfsd_fault_inject_op inject_ops[] = {
{
.file = "forget_clients",
- .func = nfsd_forget_clients,
+ .forget = nfsd_forget_client,
},
{
.file = "forget_locks",
- .func = nfsd_forget_locks,
+ .forget = nfsd_forget_client_locks,
},
{
.file = "forget_openowners",
- .func = nfsd_forget_openowners,
+ .forget = nfsd_forget_client_openowners,
},
{
.file = "forget_delegations",
- .func = nfsd_forget_delegations,
+ .forget = nfsd_forget_client_delegations,
},
{
.file = "recall_delegations",
- .func = nfsd_recall_delegations,
+ .forget = nfsd_recall_client_delegations,
},
};
@@ -44,6 +44,7 @@ static struct dentry *debug_dir;
static int nfsd_inject_set(void *op_ptr, u64 val)
{
+ u64 count = 0;
struct nfsd_fault_inject_op *op = op_ptr;
if (val == 0)
@@ -52,8 +53,9 @@ static int nfsd_inject_set(void *op_ptr, u64 val)
printk(KERN_INFO "NFSD Fault Injection: %s (n = %llu)", op->file, val);
nfs4_lock_state();
- op->func(val);
+ count = nfsd_for_n_state(val, op->forget);
nfs4_unlock_state();
+ printk(KERN_INFO "NFSD: %s: found %llu", op->file, count);
return 0;
}
diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index dc7c22f..ab45cdd 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -4721,36 +4721,6 @@ u64 nfsd_for_n_state(u64 max, u64 (*func)(struct nfs4_client *, u64))
return count;
}
-void nfsd_forget_clients(u64 num)
-{
- u64 count = nfsd_for_n_state(num, nfsd_forget_client);
- printk(KERN_INFO "NFSD: Forgot %llu clients", count);
-}
-
-void nfsd_forget_locks(u64 num)
-{
- u64 count = nfsd_for_n_state(num, nfsd_forget_client_locks);
- printk(KERN_INFO "NFSD: Forgot %llu locks", count);
-}
-
-void nfsd_forget_openowners(u64 num)
-{
- u64 count = nfsd_for_n_state(num, nfsd_forget_client_openowners);
- printk(KERN_INFO "NFSD: Forgot %llu open owners", count);
-}
-
-void nfsd_forget_delegations(u64 num)
-{
- u64 count = nfsd_for_n_state(num, nfsd_forget_client_delegations);
- printk(KERN_INFO "NFSD: Forgot %llu delegations", count);
-}
-
-void nfsd_recall_delegations(u64 num)
-{
- u64 count = nfsd_for_n_state(num, nfsd_recall_client_delegations);
- printk(KERN_INFO "NFSD: Recalled %llu delegations", count);
-}
-
#endif /* CONFIG_NFSD_FAULT_INJECTION */
/* initialization to perform at module load time: */
diff --git a/fs/nfsd/state.h b/fs/nfsd/state.h
index b542bf2..423ac64 100644
--- a/fs/nfsd/state.h
+++ b/fs/nfsd/state.h
@@ -501,11 +501,13 @@ extern void nfsd4_record_grace_done(struct nfsd_net *nn, time_t boot_time);
#ifdef CONFIG_NFSD_FAULT_INJECTION
int nfsd_fault_inject_init(void);
void nfsd_fault_inject_cleanup(void);
-void nfsd_forget_clients(u64);
-void nfsd_forget_locks(u64);
-void nfsd_forget_openowners(u64);
-void nfsd_forget_delegations(u64);
-void nfsd_recall_delegations(u64);
+u64 nfsd_for_n_state(u64, u64 (*)(struct nfs4_client *, u64));
+
+u64 nfsd_forget_client(struct nfs4_client *, u64);
+u64 nfsd_forget_client_locks(struct nfs4_client*, u64);
+u64 nfsd_forget_client_openowners(struct nfs4_client *, u64);
+u64 nfsd_forget_client_delegations(struct nfs4_client *, u64);
+u64 nfsd_recall_client_delegations(struct nfs4_client *, u64);
#else /* CONFIG_NFSD_FAULT_INJECTION */
static inline int nfsd_fault_inject_init(void) { return 0; }
static inline void nfsd_fault_inject_cleanup(void) {}
--
1.8.0.1
From: Bryan Schumaker <[email protected]>
Controlling the read and write functions allows me to add in "forget
client w.x.y.z", since we won't be limited to reading and writing only
u64 values.
Signed-off-by: Bryan Schumaker <[email protected]>
---
fs/nfsd/fault_inject.c | 56 +++++++++++++++++++++++++++++++++++++++++++-------
1 file changed, 49 insertions(+), 7 deletions(-)
diff --git a/fs/nfsd/fault_inject.c b/fs/nfsd/fault_inject.c
index 545f8e4..19f9094 100644
--- a/fs/nfsd/fault_inject.c
+++ b/fs/nfsd/fault_inject.c
@@ -8,6 +8,7 @@
#include <linux/fs.h>
#include <linux/debugfs.h>
#include <linux/module.h>
+#include <asm/uaccess.h>
#include "state.h"
@@ -48,10 +49,9 @@ static struct nfsd_fault_inject_op inject_ops[] = {
static long int NUM_INJECT_OPS = sizeof(inject_ops) / sizeof(struct nfsd_fault_inject_op);
static struct dentry *debug_dir;
-static int nfsd_inject_set(void *op_ptr, u64 val)
+static void nfsd_inject_set(struct nfsd_fault_inject_op *op, u64 val)
{
u64 count = 0;
- struct nfsd_fault_inject_op *op = op_ptr;
if (val == 0)
printk(KERN_INFO "NFSD Fault Injection: %s (all)", op->file);
@@ -62,19 +62,61 @@ static int nfsd_inject_set(void *op_ptr, u64 val)
count = nfsd_for_n_state(val, op->forget);
nfs4_unlock_state();
printk(KERN_INFO "NFSD: %s: found %llu", op->file, count);
- return 0;
}
-static int nfsd_inject_get(void *op_ptr, u64 *val)
+static void nfsd_inject_get(struct nfsd_fault_inject_op *op, u64 *val)
{
- struct nfsd_fault_inject_op *op = op_ptr;
nfs4_lock_state();
*val = nfsd_for_n_state(0, op->print);
nfs4_unlock_state();
- return 0;
}
-DEFINE_SIMPLE_ATTRIBUTE(fops_nfsd, nfsd_inject_get, nfsd_inject_set, "%llu\n");
+static ssize_t fault_inject_read(struct file *file, char __user *buf,
+ size_t len, loff_t *ppos)
+{
+ static u64 val;
+ char read_buf[25];
+ size_t size, ret;
+ loff_t pos = *ppos;
+
+ if (!pos)
+ nfsd_inject_get(file->f_dentry->d_inode->i_private, &val);
+ size = scnprintf(read_buf, sizeof(read_buf), "%llu\n", val);
+
+ if (pos < 0)
+ return -EINVAL;
+ if (pos >= size || !len)
+ return 0;
+ if (len > size - pos)
+ len = size - pos;
+ ret = copy_to_user(buf, read_buf + pos, len);
+ if (ret == len)
+ return -EFAULT;
+ len -= ret;
+ *ppos = pos + len;
+ return len;
+}
+
+static ssize_t fault_inject_write(struct file *file, const char __user *buf,
+ size_t len, loff_t *ppos)
+{
+ char write_buf[24];
+ size_t size = min(sizeof(write_buf), len) - 1;
+ u64 val;
+
+ if (copy_from_user(write_buf, buf, size))
+ return -EFAULT;
+
+ val = simple_strtoll(write_buf, NULL, 0);
+ nfsd_inject_set(file->f_dentry->d_inode->i_private, val);
+ return len; /* on success, claim we got the whole input */
+}
+
+static const struct file_operations fops_nfsd = {
+ .owner = THIS_MODULE,
+ .read = fault_inject_read,
+ .write = fault_inject_write,
+};
void nfsd_fault_inject_cleanup(void)
{
--
1.8.0.1
From: Bryan Schumaker <[email protected]>
Using "forget_n_state()" forces me to implement the code needed to
forget a specific client's openowners.
Signed-off-by: Bryan Schumaker <[email protected]>
---
fs/nfsd/nfs4state.c | 49 ++++++++++++++++++++++---------------------------
1 file changed, 22 insertions(+), 27 deletions(-)
diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index 46bece4..00d4398 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -4637,6 +4637,26 @@ u64 nfsd_forget_client_locks(struct nfs4_client *clp, u64 max)
return nfsd_foreach_client_lock(clp, max, release_lockowner);
}
+static u64 nfsd_foreach_client_open(struct nfs4_client *clp, u64 max, void (*func)(struct nfs4_openowner *))
+{
+ struct nfs4_openowner *oop, *next;
+ u64 count = 0;
+
+ list_for_each_entry_safe(oop, next, &clp->cl_openowners, oo_perclient) {
+ if (func)
+ func(oop);
+ if (++count == max)
+ break;
+ }
+
+ return count;
+}
+
+u64 nfsd_forget_client_openowners(struct nfs4_client *clp, u64 max)
+{
+ return nfsd_foreach_client_open(clp, max, release_openowner);
+}
+
u64 nfsd_for_n_state(u64 max, u64 (*func)(struct nfs4_client *, u64))
{
struct nfs4_client *clp, *next;
@@ -4661,30 +4681,6 @@ void nfsd_forget_clients(u64 num)
printk(KERN_INFO "NFSD: Forgot %llu clients", count);
}
-static void release_openowner_sop(struct nfs4_stateowner *sop)
-{
- release_openowner(openowner(sop));
-}
-
-static int nfsd_release_n_owners(u64 num, bool is_open_owner,
- void (*release_sop)(struct nfs4_stateowner *),
- struct nfsd_net *nn)
-{
- int i, count = 0;
- struct nfs4_stateowner *sop, *next;
-
- for (i = 0; i < OWNER_HASH_SIZE; i++) {
- list_for_each_entry_safe(sop, next, &nn->ownerstr_hashtbl[i], so_strhash) {
- if (sop->so_is_open_owner != is_open_owner)
- continue;
- release_sop(sop);
- if (++count == num)
- return count;
- }
- }
- return count;
-}
-
void nfsd_forget_locks(u64 num)
{
u64 count = nfsd_for_n_state(num, nfsd_forget_client_locks);
@@ -4693,9 +4689,8 @@ void nfsd_forget_locks(u64 num)
void nfsd_forget_openowners(u64 num)
{
- struct nfsd_net *nn = net_generic(&init_net, nfsd_net_id);
- int count = nfsd_release_n_owners(num, true, release_openowner_sop, nn);
- printk(KERN_INFO "NFSD: Forgot %d open owners", count);
+ u64 count = nfsd_for_n_state(num, nfsd_forget_client_openowners);
+ printk(KERN_INFO "NFSD: Forgot %llu open owners", count);
}
static int nfsd_process_n_delegations(u64 num, struct list_head *list)
--
1.8.0.1
From: Bryan Schumaker <[email protected]>
I also log basic information that I can figure out about the type of
state (such as number of locks for each client IP address). This can be
useful for checking that state was actually dropped and later for
checking if the client was able to recover.
Signed-off-by: Bryan Schumaker <[email protected]>
---
fs/nfsd/fault_inject.c | 13 +++++++++++--
fs/nfsd/nfs4state.c | 42 ++++++++++++++++++++++++++++++++++++++++++
fs/nfsd/state.h | 5 +++++
3 files changed, 58 insertions(+), 2 deletions(-)
diff --git a/fs/nfsd/fault_inject.c b/fs/nfsd/fault_inject.c
index bf6161a..545f8e4 100644
--- a/fs/nfsd/fault_inject.c
+++ b/fs/nfsd/fault_inject.c
@@ -14,28 +14,34 @@
struct nfsd_fault_inject_op {
char *file;
u64 (*forget)(struct nfs4_client *, u64);
+ u64 (*print)(struct nfs4_client *, u64);
};
static struct nfsd_fault_inject_op inject_ops[] = {
{
.file = "forget_clients",
.forget = nfsd_forget_client,
+ .print = nfsd_print_client,
},
{
.file = "forget_locks",
.forget = nfsd_forget_client_locks,
+ .print = nfsd_print_client_locks,
},
{
.file = "forget_openowners",
.forget = nfsd_forget_client_openowners,
+ .print = nfsd_print_client_openowners,
},
{
.file = "forget_delegations",
.forget = nfsd_forget_client_delegations,
+ .print = nfsd_print_client_delegations,
},
{
.file = "recall_delegations",
.forget = nfsd_recall_client_delegations,
+ .print = nfsd_print_client_delegations,
},
};
@@ -59,9 +65,12 @@ static int nfsd_inject_set(void *op_ptr, u64 val)
return 0;
}
-static int nfsd_inject_get(void *data, u64 *val)
+static int nfsd_inject_get(void *op_ptr, u64 *val)
{
- *val = 0;
+ struct nfsd_fault_inject_op *op = op_ptr;
+ nfs4_lock_state();
+ *val = nfsd_for_n_state(0, op->print);
+ nfs4_unlock_state();
return 0;
}
diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index ab45cdd..9fb8e52 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -4611,6 +4611,22 @@ u64 nfsd_forget_client(struct nfs4_client *clp, u64 max)
return 1;
}
+u64 nfsd_print_client(struct nfs4_client *clp, u64 num)
+{
+ char buf[INET6_ADDRSTRLEN];
+ rpc_ntop((struct sockaddr *)&clp->cl_addr, buf, 129);
+ printk(KERN_INFO "NFS Client: %s\n", buf);
+ return 1;
+}
+
+static void nfsd_print_count(struct nfs4_client *clp, unsigned int count,
+ const char *type)
+{
+ char buf[INET6_ADDRSTRLEN];
+ rpc_ntop((struct sockaddr *)&clp->cl_addr, buf, 129);
+ printk(KERN_INFO "NFS Client: %s has %u %s\n", buf, count, type);
+}
+
static u64 nfsd_foreach_client_lock(struct nfs4_client *clp, u64 max, void (*func)(struct nfs4_lockowner *))
{
struct nfs4_openowner *oop;
@@ -4637,6 +4653,13 @@ u64 nfsd_forget_client_locks(struct nfs4_client *clp, u64 max)
return nfsd_foreach_client_lock(clp, max, release_lockowner);
}
+u64 nfsd_print_client_locks(struct nfs4_client *clp, u64 max)
+{
+ u64 count = nfsd_foreach_client_lock(clp, max, NULL);
+ nfsd_print_count(clp, count, "locked files");
+ return count;
+}
+
static u64 nfsd_foreach_client_open(struct nfs4_client *clp, u64 max, void (*func)(struct nfs4_openowner *))
{
struct nfs4_openowner *oop, *next;
@@ -4657,6 +4680,13 @@ u64 nfsd_forget_client_openowners(struct nfs4_client *clp, u64 max)
return nfsd_foreach_client_open(clp, max, release_openowner);
}
+u64 nfsd_print_client_openowners(struct nfs4_client *clp, u64 max)
+{
+ u64 count = nfsd_foreach_client_open(clp, max, NULL);
+ nfsd_print_count(clp, count, "open files");
+ return count;
+}
+
static u64 nfsd_find_all_delegations(struct nfs4_client *clp, u64 max,
struct list_head *victims)
{
@@ -4703,6 +4733,18 @@ u64 nfsd_recall_client_delegations(struct nfs4_client *clp, u64 max)
return count;
}
+u64 nfsd_print_client_delegations(struct nfs4_client *clp, u64 max)
+{
+ u64 count = 0;
+
+ spin_lock(&recall_lock);
+ count = nfsd_find_all_delegations(clp, max, NULL);
+ spin_unlock(&recall_lock);
+
+ nfsd_print_count(clp, count, "delegations");
+ return count;
+}
+
u64 nfsd_for_n_state(u64 max, u64 (*func)(struct nfs4_client *, u64))
{
struct nfs4_client *clp, *next;
diff --git a/fs/nfsd/state.h b/fs/nfsd/state.h
index 423ac64..4017f35 100644
--- a/fs/nfsd/state.h
+++ b/fs/nfsd/state.h
@@ -508,6 +508,11 @@ u64 nfsd_forget_client_locks(struct nfs4_client*, u64);
u64 nfsd_forget_client_openowners(struct nfs4_client *, u64);
u64 nfsd_forget_client_delegations(struct nfs4_client *, u64);
u64 nfsd_recall_client_delegations(struct nfs4_client *, u64);
+
+u64 nfsd_print_client(struct nfs4_client *, u64);
+u64 nfsd_print_client_locks(struct nfs4_client *, u64);
+u64 nfsd_print_client_openowners(struct nfs4_client *, u64);
+u64 nfsd_print_client_delegations(struct nfs4_client *, u64);
#else /* CONFIG_NFSD_FAULT_INJECTION */
static inline int nfsd_fault_inject_init(void) { return 0; }
static inline void nfsd_fault_inject_cleanup(void) {}
--
1.8.0.1
From: Bryan Schumaker <[email protected]>
Once I have a client, I can easily use its delegation list rather than
searching the file hash table for delegations to remove.
Signed-off-by: Bryan Schumaker <[email protected]>
---
fs/nfsd/nfs4state.c | 94 ++++++++++++++++++++++++++++-------------------------
1 file changed, 50 insertions(+), 44 deletions(-)
diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index 00d4398..dc7c22f 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -4657,6 +4657,52 @@ u64 nfsd_forget_client_openowners(struct nfs4_client *clp, u64 max)
return nfsd_foreach_client_open(clp, max, release_openowner);
}
+static u64 nfsd_find_all_delegations(struct nfs4_client *clp, u64 max,
+ struct list_head *victims)
+{
+ struct nfs4_delegation *dp, *next;
+ u64 count = 0;
+
+ list_for_each_entry_safe(dp, next, &clp->cl_delegations, dl_perclnt) {
+ if (victims)
+ list_move(&dp->dl_recall_lru, victims);
+ if (++count == max)
+ break;
+ }
+ return count;
+}
+
+u64 nfsd_forget_client_delegations(struct nfs4_client *clp, u64 max)
+{
+ struct nfs4_delegation *dp, *next;
+ LIST_HEAD(victims);
+ u64 count;
+
+ spin_lock(&recall_lock);
+ count = nfsd_find_all_delegations(clp, max, &victims);
+ spin_unlock(&recall_lock);
+
+ list_for_each_entry_safe(dp, next, &victims, dl_recall_lru)
+ unhash_delegation(dp);
+
+ return count;
+}
+
+u64 nfsd_recall_client_delegations(struct nfs4_client *clp, u64 max)
+{
+ struct nfs4_delegation *dp, *next;
+ LIST_HEAD(victims);
+ u64 count;
+
+ spin_lock(&recall_lock);
+ count = nfsd_find_all_delegations(clp, max, &victims);
+ list_for_each_entry_safe(dp, next, &victims, dl_recall_lru)
+ nfsd_break_one_deleg(dp);
+ spin_unlock(&recall_lock);
+
+ return count;
+}
+
u64 nfsd_for_n_state(u64 max, u64 (*func)(struct nfs4_client *, u64))
{
struct nfs4_client *clp, *next;
@@ -4693,56 +4739,16 @@ void nfsd_forget_openowners(u64 num)
printk(KERN_INFO "NFSD: Forgot %llu open owners", count);
}
-static int nfsd_process_n_delegations(u64 num, struct list_head *list)
-{
- int i, count = 0;
- struct nfs4_file *fp, *fnext;
- struct nfs4_delegation *dp, *dnext;
-
- for (i = 0; i < FILE_HASH_SIZE; i++) {
- list_for_each_entry_safe(fp, fnext, &file_hashtbl[i], fi_hash) {
- list_for_each_entry_safe(dp, dnext, &fp->fi_delegations, dl_perfile) {
- list_move(&dp->dl_recall_lru, list);
- if (++count == num)
- return count;
- }
- }
- }
-
- return count;
-}
-
void nfsd_forget_delegations(u64 num)
{
- unsigned int count;
- LIST_HEAD(victims);
- struct nfs4_delegation *dp, *dnext;
-
- spin_lock(&recall_lock);
- count = nfsd_process_n_delegations(num, &victims);
- spin_unlock(&recall_lock);
-
- list_for_each_entry_safe(dp, dnext, &victims, dl_recall_lru)
- unhash_delegation(dp);
-
- printk(KERN_INFO "NFSD: Forgot %d delegations", count);
+ u64 count = nfsd_for_n_state(num, nfsd_forget_client_delegations);
+ printk(KERN_INFO "NFSD: Forgot %llu delegations", count);
}
void nfsd_recall_delegations(u64 num)
{
- unsigned int count;
- LIST_HEAD(victims);
- struct nfs4_delegation *dp, *dnext;
-
- spin_lock(&recall_lock);
- count = nfsd_process_n_delegations(num, &victims);
- list_for_each_entry_safe(dp, dnext, &victims, dl_recall_lru) {
- list_del(&dp->dl_recall_lru);
- nfsd_break_one_deleg(dp);
- }
- spin_unlock(&recall_lock);
-
- printk(KERN_INFO "NFSD: Recalled %d delegations", count);
+ u64 count = nfsd_for_n_state(num, nfsd_recall_client_delegations);
+ printk(KERN_INFO "NFSD: Recalled %llu delegations", count);
}
#endif /* CONFIG_NFSD_FAULT_INJECTION */
--
1.8.0.1
From: Bryan Schumaker <[email protected]>
I added in a generic for-each loop that takes a pass over the client_lru
list for the current net namespace and calls some function. The next few
patches will update other operations to use this function as well. A value
of 0 still means "forget everything that is found".
Signed-off-by: Bryan Schumaker <[email protected]>
---
fs/nfsd/netns.h | 3 +++
fs/nfsd/nfs4state.c | 25 ++++++++++++++++++++-----
2 files changed, 23 insertions(+), 5 deletions(-)
diff --git a/fs/nfsd/netns.h b/fs/nfsd/netns.h
index 2c4b2e2..964b554 100644
--- a/fs/nfsd/netns.h
+++ b/fs/nfsd/netns.h
@@ -92,5 +92,8 @@ struct nfsd_net {
time_t nfsd4_grace;
};
+/* Simple check to find out if a given net was properly initialized */
+#define nfsd_netns_ready(nn) ((nn)->sessionid_hashtbl)
+
extern int nfsd_net_id;
#endif /* __NFSD_NETNS_H__ */
diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index 8e19c69..2478c89 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -4605,19 +4605,34 @@ nfs4_check_open_reclaim(clientid_t *clid, bool sessions, struct nfsd_net *nn)
#ifdef CONFIG_NFSD_FAULT_INJECTION
-void nfsd_forget_clients(u64 num)
+u64 nfsd_forget_client(struct nfs4_client *clp, u64 max)
+{
+ expire_client(clp);
+ return 1;
+}
+
+u64 nfsd_for_n_state(u64 max, u64 (*func)(struct nfs4_client *, u64))
{
struct nfs4_client *clp, *next;
- int count = 0;
+ u64 count = 0;
struct nfsd_net *nn = net_generic(current->nsproxy->net_ns, nfsd_net_id);
+ if (!nfsd_netns_ready(nn))
+ return 0;
+
list_for_each_entry_safe(clp, next, &nn->client_lru, cl_lru) {
- expire_client(clp);
- if (++count == num)
+ count += func(clp, max - count);
+ if ((max != 0) && (count >= max))
break;
}
- printk(KERN_INFO "NFSD: Forgot %d clients", count);
+ return count;
+}
+
+void nfsd_forget_clients(u64 num)
+{
+ u64 count = nfsd_for_n_state(num, nfsd_forget_client);
+ printk(KERN_INFO "NFSD: Forgot %llu clients", count);
}
static void release_lockowner_sop(struct nfs4_stateowner *sop)
--
1.8.0.1
From: Bryan Schumaker <[email protected]>
I use the new "forget_n_state()" function to iterate through each client
first when searching for locks. This may slow down forgetting locks a
little bit, but it implements most of the code needed to forget a
specified client's locks.
Signed-off-by: Bryan Schumaker <[email protected]>
---
fs/nfsd/nfs4state.c | 36 ++++++++++++++++++++++++++++--------
1 file changed, 28 insertions(+), 8 deletions(-)
diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index 2478c89..46bece4 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -4611,6 +4611,32 @@ u64 nfsd_forget_client(struct nfs4_client *clp, u64 max)
return 1;
}
+static u64 nfsd_foreach_client_lock(struct nfs4_client *clp, u64 max, void (*func)(struct nfs4_lockowner *))
+{
+ struct nfs4_openowner *oop;
+ struct nfs4_lockowner *lop, *lo_next;
+ struct nfs4_ol_stateid *stp, *st_next;
+ u64 count = 0;
+
+ list_for_each_entry(oop, &clp->cl_openowners, oo_perclient) {
+ list_for_each_entry_safe(stp, st_next, &oop->oo_owner.so_stateids, st_perstateowner) {
+ list_for_each_entry_safe(lop, lo_next, &stp->st_lockowners, lo_perstateid) {
+ if (func)
+ func(lop);
+ if (++count == max)
+ return count;
+ }
+ }
+ }
+
+ return count;
+}
+
+u64 nfsd_forget_client_locks(struct nfs4_client *clp, u64 max)
+{
+ return nfsd_foreach_client_lock(clp, max, release_lockowner);
+}
+
u64 nfsd_for_n_state(u64 max, u64 (*func)(struct nfs4_client *, u64))
{
struct nfs4_client *clp, *next;
@@ -4635,11 +4661,6 @@ void nfsd_forget_clients(u64 num)
printk(KERN_INFO "NFSD: Forgot %llu clients", count);
}
-static void release_lockowner_sop(struct nfs4_stateowner *sop)
-{
- release_lockowner(lockowner(sop));
-}
-
static void release_openowner_sop(struct nfs4_stateowner *sop)
{
release_openowner(openowner(sop));
@@ -4666,9 +4687,8 @@ static int nfsd_release_n_owners(u64 num, bool is_open_owner,
void nfsd_forget_locks(u64 num)
{
- struct nfsd_net *nn = net_generic(&init_net, nfsd_net_id);
- int count = nfsd_release_n_owners(num, false, release_lockowner_sop, nn);
- printk(KERN_INFO "NFSD: Forgot %d locks", count);
+ u64 count = nfsd_for_n_state(num, nfsd_forget_client_locks);
+ printk(KERN_INFO "NFSD: Forgot %llu locks", count);
}
void nfsd_forget_openowners(u64 num)
--
1.8.0.1
From: Bryan Schumaker <[email protected]>
Write the client's ip address to any state file and all appropriate
state for that client will be forgotten.
Signed-off-by: Bryan Schumaker <[email protected]>
---
fs/nfsd/fault_inject.c | 37 +++++++++++++++++++++++++++++++++----
fs/nfsd/nfs4state.c | 15 +++++++++++++++
fs/nfsd/state.h | 1 +
3 files changed, 49 insertions(+), 4 deletions(-)
diff --git a/fs/nfsd/fault_inject.c b/fs/nfsd/fault_inject.c
index 19f9094..96ffdf5 100644
--- a/fs/nfsd/fault_inject.c
+++ b/fs/nfsd/fault_inject.c
@@ -8,9 +8,12 @@
#include <linux/fs.h>
#include <linux/debugfs.h>
#include <linux/module.h>
+#include <linux/nsproxy.h>
+#include <linux/sunrpc/clnt.h>
#include <asm/uaccess.h>
#include "state.h"
+#include "netns.h"
struct nfsd_fault_inject_op {
char *file;
@@ -64,6 +67,24 @@ static void nfsd_inject_set(struct nfsd_fault_inject_op *op, u64 val)
printk(KERN_INFO "NFSD: %s: found %llu", op->file, count);
}
+static void nfsd_inject_set_client(struct nfsd_fault_inject_op *op,
+ struct sockaddr_storage *addr,
+ size_t addr_size)
+{
+ char buf[INET6_ADDRSTRLEN];
+ struct nfs4_client *clp;
+ u64 count;
+
+ nfs4_lock_state();
+ clp = nfsd_find_client(addr, addr_size);
+ if (clp) {
+ count = op->forget(clp, 0);
+ rpc_ntop((struct sockaddr *)&clp->cl_addr, buf, 129);
+ printk(KERN_INFO "NFSD [%s]: Client %s had %llu state object(s)\n", op->file, buf, count);
+ }
+ nfs4_unlock_state();
+}
+
static void nfsd_inject_get(struct nfsd_fault_inject_op *op, u64 *val)
{
nfs4_lock_state();
@@ -100,15 +121,23 @@ static ssize_t fault_inject_read(struct file *file, char __user *buf,
static ssize_t fault_inject_write(struct file *file, const char __user *buf,
size_t len, loff_t *ppos)
{
- char write_buf[24];
+ char write_buf[INET6_ADDRSTRLEN];
size_t size = min(sizeof(write_buf), len) - 1;
+ struct net *net = current->nsproxy->net_ns;
+ struct sockaddr_storage sa;
u64 val;
if (copy_from_user(write_buf, buf, size))
return -EFAULT;
-
- val = simple_strtoll(write_buf, NULL, 0);
- nfsd_inject_set(file->f_dentry->d_inode->i_private, val);
+ write_buf[size] = '\0';
+
+ size = rpc_pton(net, write_buf, size, (struct sockaddr *)&sa, sizeof(sa));
+ if (size > 0)
+ nfsd_inject_set_client(file->f_dentry->d_inode->i_private, &sa, size);
+ else {
+ val = simple_strtoll(write_buf, NULL, 0);
+ nfsd_inject_set(file->f_dentry->d_inode->i_private, val);
+ }
return len; /* on success, claim we got the whole input */
}
diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index 9fb8e52..eff7340 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -4763,6 +4763,21 @@ u64 nfsd_for_n_state(u64 max, u64 (*func)(struct nfs4_client *, u64))
return count;
}
+struct nfs4_client *nfsd_find_client(struct sockaddr_storage *addr, size_t addr_size)
+{
+ struct nfs4_client *clp;
+ struct nfsd_net *nn = net_generic(current->nsproxy->net_ns, nfsd_net_id);
+
+ if (!nfsd_netns_ready(nn))
+ return NULL;
+
+ list_for_each_entry(clp, &nn->client_lru, cl_lru) {
+ if (memcmp(&clp->cl_addr, addr, addr_size) == 0)
+ return clp;
+ }
+ return NULL;
+}
+
#endif /* CONFIG_NFSD_FAULT_INJECTION */
/* initialization to perform at module load time: */
diff --git a/fs/nfsd/state.h b/fs/nfsd/state.h
index 4017f35..d1c229f 100644
--- a/fs/nfsd/state.h
+++ b/fs/nfsd/state.h
@@ -502,6 +502,7 @@ extern void nfsd4_record_grace_done(struct nfsd_net *nn, time_t boot_time);
int nfsd_fault_inject_init(void);
void nfsd_fault_inject_cleanup(void);
u64 nfsd_for_n_state(u64, u64 (*)(struct nfs4_client *, u64));
+struct nfs4_client *nfsd_find_client(struct sockaddr_storage *, size_t);
u64 nfsd_forget_client(struct nfs4_client *, u64);
u64 nfsd_forget_client_locks(struct nfs4_client*, u64);
--
1.8.0.1
On 12/03/2012 09:53 AM, J. Bruce Fields wrote:
> On Thu, Nov 29, 2012 at 11:40:37AM -0500, [email protected] wrote:
>> From: Bryan Schumaker <[email protected]>
>>
>> While working on p2p-nfs, I discovered that I sometimes need to clear
>> state for a specific client to test all possible error recovery conditions.
>> The current fault injection code deletes state as it find it, so it can
>> be difficult to guess which state will be forgotten. In addition, I
>> currently print out the amount of state forgotten but I don't give details
>> like "Forgot 3 locks from client w.x.y.z". These patches set out to
>> improve that.
>>
>> The first 6 patches clean up the current code and prepare it for specific
>> client state removal. Patch 7 adds printing information to the server's logs
>> when a fault injection file is read (such as "Client w.x.y.z has 3 open
>> files"). Patch 8 adds in a custom file operations structure so users can
>> write strings to fault injection files in addition to u64s. Finally, patch
>> 9 allows users to remove state by writing a client's IP address to one of
>> the files.
>>
>> Changes in v4:
>> - Access lock state from open owners rather than requiring a network namespace
>> pointer.
>> - Don't produce a null pointer expection if fault injection is attempted after
>> loading the nfsd module but before running any nfsd threads.
>
> OK, thanks for your patience, applying this version (pending some basic
> testing).
Awesome, thanks! :)
- Bryan
>
> --b.
>
>>
>> - Bryan
>>
>> Bryan Schumaker (9):
>> NFSD: Lock state before calling fault injection function
>> NFSD: Clean up forgetting clients
>> NFSD: Clean up forgetting locks
>> NFSD: Clean up forgetting openowners
>> NFSD: Clean up forgetting and recalling delegations
>> NFSD: Fault injection operations take a per-client forget function
>> NFSD: Reading a fault injection file prints a state count
>> NFSD: Add a custom file operations structure for fault injection
>> NFSD: Forget state for a specific client
>>
>> fs/nfsd/fault_inject.c | 112 ++++++++++++++++++++++++----
>> fs/nfsd/netns.h | 3 +
>> fs/nfsd/nfs4state.c | 197 ++++++++++++++++++++++++++++++-------------------
>> fs/nfsd/state.h | 18 +++--
>> 4 files changed, 237 insertions(+), 93 deletions(-)
>>
>> --
>> 1.8.0.1
>>
On Thu, Nov 29, 2012 at 11:40:37AM -0500, [email protected] wrote:
> From: Bryan Schumaker <[email protected]>
>
> While working on p2p-nfs, I discovered that I sometimes need to clear
> state for a specific client to test all possible error recovery conditions.
> The current fault injection code deletes state as it find it, so it can
> be difficult to guess which state will be forgotten. In addition, I
> currently print out the amount of state forgotten but I don't give details
> like "Forgot 3 locks from client w.x.y.z". These patches set out to
> improve that.
>
> The first 6 patches clean up the current code and prepare it for specific
> client state removal. Patch 7 adds printing information to the server's logs
> when a fault injection file is read (such as "Client w.x.y.z has 3 open
> files"). Patch 8 adds in a custom file operations structure so users can
> write strings to fault injection files in addition to u64s. Finally, patch
> 9 allows users to remove state by writing a client's IP address to one of
> the files.
>
> Changes in v4:
> - Access lock state from open owners rather than requiring a network namespace
> pointer.
> - Don't produce a null pointer expection if fault injection is attempted after
> loading the nfsd module but before running any nfsd threads.
OK, thanks for your patience, applying this version (pending some basic
testing).
--b.
>
> - Bryan
>
> Bryan Schumaker (9):
> NFSD: Lock state before calling fault injection function
> NFSD: Clean up forgetting clients
> NFSD: Clean up forgetting locks
> NFSD: Clean up forgetting openowners
> NFSD: Clean up forgetting and recalling delegations
> NFSD: Fault injection operations take a per-client forget function
> NFSD: Reading a fault injection file prints a state count
> NFSD: Add a custom file operations structure for fault injection
> NFSD: Forget state for a specific client
>
> fs/nfsd/fault_inject.c | 112 ++++++++++++++++++++++++----
> fs/nfsd/netns.h | 3 +
> fs/nfsd/nfs4state.c | 197 ++++++++++++++++++++++++++++++-------------------
> fs/nfsd/state.h | 18 +++--
> 4 files changed, 237 insertions(+), 93 deletions(-)
>
> --
> 1.8.0.1
>