This patch series allows setting all zswap params at runtime, instead
of only being settable at boot-time.
The changes to zswap are rather large, due to the creation of zswap pools,
which contain both a compressor function as well as a zpool. When either
the compressor or zpool param is changed at runtime, a new zswap pool is
created with the new compressor and zpool, and used for all new compressed
pages. Any old zswap pools that still contain pages are retained only to
load pages from, and destroyed once they become empty.
One notable change required for this to work is to split the currently
global kernel param mutex into a global mutex only for built-in params,
and a per-module mutex for loadable module params. The reason this change
is required is because zswap's compressor and zpool param handler callback
functions attempt to load, via crypto_has_comp() and the new zpool_has_pool()
functions, any required compressor or zpool modules. The problem there is
that the zswap param callback functions run while the global param mutex is
locked, but when they attempt to load another module, if the loading module
has any params set e.g. via /etc/modprobe.d/*.conf, modprobe will also try
to take the global param mutex, and a deadlock will result, with the mutex
held by the zswap param callback which is waiting for modprobe, but modprobe
waiting for the mutex to change the loading module's param. Using a
per-module mutex for all loadable modules prevents this, since each module
will take its own mutex and never conflict with another module's param
changes.
Dan Streetman (5):
zpool: add zpool_has_pool()
module: add per-module params lock
zswap: runtime enable/disable
zswap: dynamic pool creation
zswap: change zpool/compressor at runtime
arch/um/drivers/hostaudio_kern.c | 20 +-
drivers/net/ethernet/myricom/myri10ge/myri10ge.c | 6 +-
drivers/net/wireless/libertas_tf/if_usb.c | 6 +-
drivers/usb/atm/ueagle-atm.c | 4 +-
drivers/video/fbdev/vt8623fb.c | 4 +-
include/linux/module.h | 1 +
include/linux/moduleparam.h | 67 +--
include/linux/zpool.h | 2 +
kernel/module.c | 1 +
kernel/params.c | 45 +-
mm/zpool.c | 25 +
mm/zswap.c | 696 +++++++++++++++++------
net/mac80211/rate.c | 4 +-
13 files changed, 640 insertions(+), 241 deletions(-)
--
2.1.0
Add zpool_has_pool() function, indicating if the specified type of zpool
is available (i.e. zsmalloc or zbud). This allows checking if a pool is
available, without actually trying to allocate it, similar to
crypto_has_alg().
This is used by a following patch to zswap that enables the dynamic
runtime creation of zswap zpools.
Signed-off-by: Dan Streetman <[email protected]>
---
include/linux/zpool.h | 2 ++
mm/zpool.c | 25 +++++++++++++++++++++++++
2 files changed, 27 insertions(+)
diff --git a/include/linux/zpool.h b/include/linux/zpool.h
index 56529b3..31b79e2 100644
--- a/include/linux/zpool.h
+++ b/include/linux/zpool.h
@@ -36,6 +36,8 @@ enum zpool_mapmode {
ZPOOL_MM_DEFAULT = ZPOOL_MM_RW
};
+bool zpool_has_pool(char *type);
+
struct zpool *zpool_create_pool(char *type, char *name,
gfp_t gfp, struct zpool_ops *ops);
diff --git a/mm/zpool.c b/mm/zpool.c
index 884659d..bff40ce 100644
--- a/mm/zpool.c
+++ b/mm/zpool.c
@@ -127,6 +127,31 @@ static void zpool_put_driver(struct zpool_driver *driver)
}
/**
+ * zpool_has_pool() - Check if the pool driver is available
+ * @type The type of the zpool to check (e.g. zbud, zsmalloc)
+ *
+ * This checks if the @type pool driver is available.
+ *
+ * Returns: true if @type pool is available, false if not
+ */
+bool zpool_has_pool(char *type)
+{
+ struct zpool_driver *driver = zpool_get_driver(type);
+
+ if (!driver) {
+ request_module("zpool-%s", type);
+ driver = zpool_get_driver(type);
+ }
+
+ if (!driver)
+ return false;
+
+ zpool_put_driver(driver);
+ return true;
+}
+EXPORT_SYMBOL(zpool_has_pool);
+
+/**
* zpool_create_pool() - Create a new zpool
* @type The type of the zpool to create (e.g. zbud, zsmalloc)
* @name The name of the zpool (e.g. zram0, zswap)
--
2.1.0
Add a "param_lock" mutex to each module, and add a pointer to the owning
module to struct kernel_param. Then change the param sysfs modification
code to only use the global param_lock for built-ins, and use the
per-module param_lock for all modules. Also simplify the
kernel_[un]block_sysfs_[read|write]() functions, to simply
kernel_[un]block_sysfs().
The kernel param code currently uses a single global mutex to protect
modification of any and all kernel params. While this generally works,
there is one specific problem with it; a module callback function
cannot safely load another module, i.e. with request_module() or even
with indirect calls such as crypto_has_alg(). If the module to be
loaded has any of its params configured (e.g. with a /etc/modprobe.d/*
config file), then the attempt will result in a deadlock between the
first module param callback waiting for modprobe, and modprobe trying to
lock the global kernel param mutex to set the new module's param.
This fixes that by using per-module mutexes, so that each individual module
is protected against concurrent changes in its own kernel params, but is
not blocked by changes to other module params. All built-in modules
continue to use the global mutex, since they will always be loaded at
runtime and references (e.g. request_module(), crypto_has_alg()) to them
will never cause load-time param changing.
This also simplifies the interface used by modules to block sysfs access
to their params; while there are currently functions to block and unblock
sysfs param access which are split up by read and write and expect a single
kernel param to be passed, their actual operation is identical and applies
to all params, not just the one passed to them; they simply lock and unlock
the global param mutex. They are thus replaced with simplified functions
that clearly indicate that all params for a module are being blocked, both
for read and write access.
Signed-off-by: Dan Streetman <[email protected]>
---
arch/um/drivers/hostaudio_kern.c | 20 +++----
drivers/net/ethernet/myricom/myri10ge/myri10ge.c | 6 +--
drivers/net/wireless/libertas_tf/if_usb.c | 6 +--
drivers/usb/atm/ueagle-atm.c | 4 +-
drivers/video/fbdev/vt8623fb.c | 4 +-
include/linux/module.h | 1 +
include/linux/moduleparam.h | 67 +++++++++---------------
kernel/module.c | 1 +
kernel/params.c | 45 ++++++++++------
net/mac80211/rate.c | 4 +-
10 files changed, 77 insertions(+), 81 deletions(-)
diff --git a/arch/um/drivers/hostaudio_kern.c b/arch/um/drivers/hostaudio_kern.c
index 9b90fdc..1bcb798 100644
--- a/arch/um/drivers/hostaudio_kern.c
+++ b/arch/um/drivers/hostaudio_kern.c
@@ -185,9 +185,9 @@ static int hostaudio_open(struct inode *inode, struct file *file)
int ret;
#ifdef DEBUG
- kparam_block_sysfs_write(dsp);
+ kparam_block_sysfs();
printk(KERN_DEBUG "hostaudio: open called (host: %s)\n", dsp);
- kparam_unblock_sysfs_write(dsp);
+ kparam_unblock_sysfs();
#endif
state = kmalloc(sizeof(struct hostaudio_state), GFP_KERNEL);
@@ -199,11 +199,11 @@ static int hostaudio_open(struct inode *inode, struct file *file)
if (file->f_mode & FMODE_WRITE)
w = 1;
- kparam_block_sysfs_write(dsp);
+ kparam_block_sysfs();
mutex_lock(&hostaudio_mutex);
ret = os_open_file(dsp, of_set_rw(OPENFLAGS(), r, w), 0);
mutex_unlock(&hostaudio_mutex);
- kparam_unblock_sysfs_write(dsp);
+ kparam_unblock_sysfs();
if (ret < 0) {
kfree(state);
@@ -260,17 +260,17 @@ static int hostmixer_open_mixdev(struct inode *inode, struct file *file)
if (file->f_mode & FMODE_WRITE)
w = 1;
- kparam_block_sysfs_write(mixer);
+ kparam_block_sysfs();
mutex_lock(&hostaudio_mutex);
ret = os_open_file(mixer, of_set_rw(OPENFLAGS(), r, w), 0);
mutex_unlock(&hostaudio_mutex);
- kparam_unblock_sysfs_write(mixer);
+ kparam_unblock_sysfs();
if (ret < 0) {
- kparam_block_sysfs_write(dsp);
+ kparam_block_sysfs();
printk(KERN_ERR "hostaudio_open_mixdev failed to open '%s', "
"err = %d\n", dsp, -ret);
- kparam_unblock_sysfs_write(dsp);
+ kparam_unblock_sysfs();
kfree(state);
return ret;
}
@@ -326,10 +326,10 @@ MODULE_LICENSE("GPL");
static int __init hostaudio_init_module(void)
{
- __kernel_param_lock();
+ kparam_block_sysfs();
printk(KERN_INFO "UML Audio Relay (host dsp = %s, host mixer = %s)\n",
dsp, mixer);
- __kernel_param_unlock();
+ kparam_unblock_sysfs();
module_data.dev_audio = register_sound_dsp(&hostaudio_fops, -1);
if (module_data.dev_audio < 0) {
diff --git a/drivers/net/ethernet/myricom/myri10ge/myri10ge.c b/drivers/net/ethernet/myricom/myri10ge/myri10ge.c
index 2bae502..6d7ac3f 100644
--- a/drivers/net/ethernet/myricom/myri10ge/myri10ge.c
+++ b/drivers/net/ethernet/myricom/myri10ge/myri10ge.c
@@ -279,7 +279,7 @@ MODULE_FIRMWARE("myri10ge_eth_z8e.dat");
MODULE_FIRMWARE("myri10ge_rss_ethp_z8e.dat");
MODULE_FIRMWARE("myri10ge_rss_eth_z8e.dat");
-/* Careful: must be accessed under kparam_block_sysfs_write */
+/* Careful: must be accessed under kparam_block_sysfs */
static char *myri10ge_fw_name = NULL;
module_param(myri10ge_fw_name, charp, S_IRUGO | S_IWUSR);
MODULE_PARM_DESC(myri10ge_fw_name, "Firmware image name");
@@ -3427,7 +3427,7 @@ static void myri10ge_select_firmware(struct myri10ge_priv *mgp)
}
}
- kparam_block_sysfs_write(myri10ge_fw_name);
+ kparam_block_sysfs();
if (myri10ge_fw_name != NULL) {
char *fw_name = kstrdup(myri10ge_fw_name, GFP_KERNEL);
if (fw_name) {
@@ -3435,7 +3435,7 @@ static void myri10ge_select_firmware(struct myri10ge_priv *mgp)
set_fw_name(mgp, fw_name, true);
}
}
- kparam_unblock_sysfs_write(myri10ge_fw_name);
+ kparam_unblock_sysfs();
if (mgp->board_number < MYRI10GE_MAX_BOARDS &&
myri10ge_fw_names[mgp->board_number] != NULL &&
diff --git a/drivers/net/wireless/libertas_tf/if_usb.c b/drivers/net/wireless/libertas_tf/if_usb.c
index 1a20cee..7cdbbce 100644
--- a/drivers/net/wireless/libertas_tf/if_usb.c
+++ b/drivers/net/wireless/libertas_tf/if_usb.c
@@ -821,15 +821,15 @@ static int if_usb_prog_firmware(struct if_usb_card *cardp)
lbtf_deb_enter(LBTF_DEB_USB);
- kparam_block_sysfs_write(fw_name);
+ kparam_block_sysfs();
ret = request_firmware(&cardp->fw, lbtf_fw_name, &cardp->udev->dev);
if (ret < 0) {
pr_err("request_firmware() failed with %#x\n", ret);
pr_err("firmware %s not found\n", lbtf_fw_name);
- kparam_unblock_sysfs_write(fw_name);
+ kparam_unblock_sysfs();
goto done;
}
- kparam_unblock_sysfs_write(fw_name);
+ kparam_unblock_sysfs();
if (check_fwfile_format(cardp->fw->data, cardp->fw->size))
goto release_fw;
diff --git a/drivers/usb/atm/ueagle-atm.c b/drivers/usb/atm/ueagle-atm.c
index 888998a..8262989 100644
--- a/drivers/usb/atm/ueagle-atm.c
+++ b/drivers/usb/atm/ueagle-atm.c
@@ -1599,7 +1599,7 @@ static void cmvs_file_name(struct uea_softc *sc, char *const cmv_name, int ver)
char file_arr[] = "CMVxy.bin";
char *file;
- kparam_block_sysfs_write(cmv_file);
+ kparam_block_sysfs();
/* set proper name corresponding modem version and line type */
if (cmv_file[sc->modem_index] == NULL) {
if (UEA_CHIP_VERSION(sc) == ADI930)
@@ -1618,7 +1618,7 @@ static void cmvs_file_name(struct uea_softc *sc, char *const cmv_name, int ver)
strlcat(cmv_name, file, UEA_FW_NAME_MAX);
if (ver == 2)
strlcat(cmv_name, ".v2", UEA_FW_NAME_MAX);
- kparam_unblock_sysfs_write(cmv_file);
+ kparam_unblock_sysfs();
}
static int request_cmvs_old(struct uea_softc *sc,
diff --git a/drivers/video/fbdev/vt8623fb.c b/drivers/video/fbdev/vt8623fb.c
index ea7f056..a940626 100644
--- a/drivers/video/fbdev/vt8623fb.c
+++ b/drivers/video/fbdev/vt8623fb.c
@@ -754,9 +754,9 @@ static int vt8623_pci_probe(struct pci_dev *dev, const struct pci_device_id *id)
/* Prepare startup mode */
- kparam_block_sysfs_write(mode_option);
+ kparam_block_sysfs();
rc = fb_find_mode(&(info->var), info, mode_option, NULL, 0, NULL, 8);
- kparam_unblock_sysfs_write(mode_option);
+ kparam_unblock_sysfs();
if (! ((rc == 1) || (rc == 2))) {
rc = -EINVAL;
dev_err(info->device, "mode %s not found\n", mode_option);
diff --git a/include/linux/module.h b/include/linux/module.h
index c883b86..e8c24d8 100644
--- a/include/linux/module.h
+++ b/include/linux/module.h
@@ -232,6 +232,7 @@ struct module {
unsigned int num_syms;
/* Kernel parameters. */
+ struct mutex param_lock;
struct kernel_param *kp;
unsigned int num_kp;
diff --git a/include/linux/moduleparam.h b/include/linux/moduleparam.h
index 1c9effa..d9f689f 100644
--- a/include/linux/moduleparam.h
+++ b/include/linux/moduleparam.h
@@ -67,6 +67,7 @@ enum {
struct kernel_param {
const char *name;
+ struct module *mod;
const struct kernel_param_ops *ops;
u16 perm;
s8 level;
@@ -108,7 +109,7 @@ struct kparam_array
*
* @perm is 0 if the the variable is not to appear in sysfs, or 0444
* for world-readable, 0644 for root-writable, etc. Note that if it
- * is writable, you may need to use kparam_block_sysfs_write() around
+ * is writable, you may need to use kparam_block_sysfs() around
* accesses (esp. charp, which can be kfreed when it changes).
*
* The @type is simply pasted to refer to a param_ops_##type and a
@@ -220,8 +221,8 @@ struct kparam_array
static struct kernel_param __moduleparam_const __param_##name \
__used \
__attribute__ ((unused,__section__ ("__param"),aligned(sizeof(void *)))) \
- = { __param_str_##name, ops, VERIFY_OCTAL_PERMISSIONS(perm), \
- level, flags, { arg } }
+ = { __param_str_##name, THIS_MODULE, ops, \
+ VERIFY_OCTAL_PERMISSIONS(perm), level, flags, { arg } }
/* Obsolete - use module_param_cb() */
#define module_param_call(name, set, get, arg, perm) \
@@ -239,57 +240,37 @@ __check_old_set_param(int (*oldset)(const char *, struct kernel_param *))
}
/**
- * kparam_block_sysfs_write - make sure a parameter isn't written via sysfs.
- * @name: the name of the parameter
+ * kparam_block_sysfs - lock THIS_MODULE's param(s) from being r/w
*
- * There's no point blocking write on a paramter that isn't writable via sysfs!
- */
-#define kparam_block_sysfs_write(name) \
- do { \
- BUG_ON(!(__param_##name.perm & 0222)); \
- __kernel_param_lock(); \
- } while (0)
-
-/**
- * kparam_unblock_sysfs_write - allows sysfs to write to a parameter again.
- * @name: the name of the parameter
- */
-#define kparam_unblock_sysfs_write(name) \
- do { \
- BUG_ON(!(__param_##name.perm & 0222)); \
- __kernel_param_unlock(); \
- } while (0)
-
-/**
- * kparam_block_sysfs_read - make sure a parameter isn't read via sysfs.
- * @name: the name of the parameter
+ * Each module has its own mutex that protects all of its sysfs params
+ * during read and write from sysfs of any of its params. All built-ins
+ * share a single mutex that similarly protects all built-in sysfs params.
+ * This locks the appropriate mutex; either the built-in mutex if called
+ * from a built-in, or the specific module's mutex if called from a module.
*
- * This also blocks sysfs writes.
+ * As with any mutex, this call will block if any of the corresponding
+ * param(s) are currently being read or written. After calling this, the
+ * @kparam_unblock_sysfs() function must be called to unlock the mutex and
+ * allow the corresponding param(s) to be read and written.
*/
-#define kparam_block_sysfs_read(name) \
- do { \
- BUG_ON(!(__param_##name.perm & 0444)); \
- __kernel_param_lock(); \
- } while (0)
+#define kparam_block_sysfs() __kernel_param_lock(THIS_MODULE)
/**
- * kparam_unblock_sysfs_read - allows sysfs to read a parameter again.
- * @name: the name of the parameter
+ * kparam_unblock_sysfs - unlock THIS_MODULE's param(s) from being r/w
+ *
+ * This unlocks the corresponding mutex; see @kparam_block_sysfs() for details.
*/
-#define kparam_unblock_sysfs_read(name) \
- do { \
- BUG_ON(!(__param_##name.perm & 0444)); \
- __kernel_param_unlock(); \
- } while (0)
+#define kparam_unblock_sysfs() __kernel_param_unlock(THIS_MODULE)
#ifdef CONFIG_SYSFS
-extern void __kernel_param_lock(void);
-extern void __kernel_param_unlock(void);
+/* don't use these directly, use block/unblock_sysfs above */
+extern void __kernel_param_lock(struct module *);
+extern void __kernel_param_unlock(struct module *);
#else
-static inline void __kernel_param_lock(void)
+static inline void __kernel_param_lock(struct module *)
{
}
-static inline void __kernel_param_unlock(void)
+static inline void __kernel_param_unlock(struct module *)
{
}
#endif
diff --git a/kernel/module.c b/kernel/module.c
index 42a1d2a..d3632e7 100644
--- a/kernel/module.c
+++ b/kernel/module.c
@@ -3341,6 +3341,7 @@ static int load_module(struct load_info *info, const char __user *uargs,
goto ddebug_cleanup;
/* Module is ready to execute: parsing args may do that. */
+ mutex_init(&mod->param_lock);
after_dashes = parse_args(mod->name, mod->args, mod->kp, mod->num_kp,
-32768, 32767, unknown_module_param_cb);
if (IS_ERR(after_dashes)) {
diff --git a/kernel/params.c b/kernel/params.c
index a22d6a7..2d1849b 100644
--- a/kernel/params.c
+++ b/kernel/params.c
@@ -25,8 +25,9 @@
#include <linux/slab.h>
#include <linux/ctype.h>
-/* Protects all parameters, and incidentally kmalloced_param list. */
+/* Protects all built-in parameters; modules use their own param_lock */
static DEFINE_MUTEX(param_lock);
+static bool __kernel_param_is_locked(struct module *mod);
/* This just allows us to keep track of which parameters are kmalloced. */
struct kmalloced_param {
@@ -34,6 +35,7 @@ struct kmalloced_param {
char val[];
};
static LIST_HEAD(kmalloced_params);
+static DEFINE_SPINLOCK(kmalloced_params_lock);
static void *kmalloc_parameter(unsigned int size)
{
@@ -43,7 +45,10 @@ static void *kmalloc_parameter(unsigned int size)
if (!p)
return NULL;
+ spin_lock(&kmalloced_params_lock);
list_add(&p->list, &kmalloced_params);
+ spin_unlock(&kmalloced_params_lock);
+
return p->val;
}
@@ -52,6 +57,7 @@ static void maybe_kfree_parameter(void *param)
{
struct kmalloced_param *p;
+ spin_lock(&kmalloced_params_lock);
list_for_each_entry(p, &kmalloced_params, list) {
if (p->val == param) {
list_del(&p->list);
@@ -59,6 +65,7 @@ static void maybe_kfree_parameter(void *param)
break;
}
}
+ spin_unlock(&kmalloced_params_lock);
}
static char dash2underscore(char c)
@@ -118,10 +125,10 @@ static int parse_one(char *param,
return -EINVAL;
pr_debug("handling %s with %p\n", param,
params[i].ops->set);
- mutex_lock(¶m_lock);
+ __kernel_param_lock(params[i].mod);
param_check_unsafe(¶ms[i]);
err = params[i].ops->set(val, ¶ms[i]);
- mutex_unlock(¶m_lock);
+ __kernel_param_unlock(params[i].mod);
return err;
}
}
@@ -387,7 +394,8 @@ struct kernel_param_ops param_ops_bint = {
EXPORT_SYMBOL(param_ops_bint);
/* We break the rule and mangle the string. */
-static int param_array(const char *name,
+static int param_array(struct module *mod,
+ const char *name,
const char *val,
unsigned int min, unsigned int max,
void *elem, int elemsize,
@@ -418,7 +426,7 @@ static int param_array(const char *name,
/* nul-terminate and parse */
save = val[len];
((char *)val)[len] = '\0';
- BUG_ON(!mutex_is_locked(¶m_lock));
+ BUG_ON(!__kernel_param_is_locked(mod));
ret = set(val, &kp);
if (ret != 0)
@@ -440,7 +448,7 @@ static int param_array_set(const char *val, const struct kernel_param *kp)
const struct kparam_array *arr = kp->arr;
unsigned int temp_num;
- return param_array(kp->name, val, 1, arr->max, arr->elem,
+ return param_array(kp->mod, kp->name, val, 1, arr->max, arr->elem,
arr->elemsize, arr->ops->set, kp->level,
arr->num ?: &temp_num);
}
@@ -456,7 +464,7 @@ static int param_array_get(char *buffer, const struct kernel_param *kp)
if (i)
buffer[off++] = ',';
p.arg = arr->elem + arr->elemsize * i;
- BUG_ON(!mutex_is_locked(¶m_lock));
+ BUG_ON(!__kernel_param_is_locked(kp->mod));
ret = arr->ops->get(buffer + off, &p);
if (ret < 0)
return ret;
@@ -539,9 +547,9 @@ static ssize_t param_attr_show(struct module_attribute *mattr,
if (!attribute->param->ops->get)
return -EPERM;
- mutex_lock(¶m_lock);
+ __kernel_param_lock(mk->mod);
count = attribute->param->ops->get(buf, attribute->param);
- mutex_unlock(¶m_lock);
+ __kernel_param_unlock(mk->mod);
if (count > 0) {
strcat(buf, "\n");
++count;
@@ -551,7 +559,7 @@ static ssize_t param_attr_show(struct module_attribute *mattr,
/* sysfs always hands a nul-terminated string in buf. We rely on that. */
static ssize_t param_attr_store(struct module_attribute *mattr,
- struct module_kobject *km,
+ struct module_kobject *mk,
const char *buf, size_t len)
{
int err;
@@ -560,10 +568,10 @@ static ssize_t param_attr_store(struct module_attribute *mattr,
if (!attribute->param->ops->set)
return -EPERM;
- mutex_lock(¶m_lock);
+ __kernel_param_lock(mk->mod);
param_check_unsafe(attribute->param);
err = attribute->param->ops->set(buf, attribute->param);
- mutex_unlock(¶m_lock);
+ __kernel_param_unlock(mk->mod);
if (!err)
return len;
return err;
@@ -576,16 +584,21 @@ static ssize_t param_attr_store(struct module_attribute *mattr,
#define __modinit __init
#endif
+static bool __kernel_param_is_locked(struct module *mod)
+{
+ return mutex_is_locked(mod ? &mod->param_lock : ¶m_lock);
+}
+
#ifdef CONFIG_SYSFS
-void __kernel_param_lock(void)
+void __kernel_param_lock(struct module *mod)
{
- mutex_lock(¶m_lock);
+ mutex_lock(mod ? &mod->param_lock : ¶m_lock);
}
EXPORT_SYMBOL(__kernel_param_lock);
-void __kernel_param_unlock(void)
+void __kernel_param_unlock(struct module *mod)
{
- mutex_unlock(¶m_lock);
+ mutex_unlock(mod ? &mod->param_lock : ¶m_lock);
}
EXPORT_SYMBOL(__kernel_param_unlock);
diff --git a/net/mac80211/rate.c b/net/mac80211/rate.c
index d53355b..ca82d5e 100644
--- a/net/mac80211/rate.c
+++ b/net/mac80211/rate.c
@@ -103,7 +103,7 @@ ieee80211_rate_control_ops_get(const char *name)
const struct rate_control_ops *ops;
const char *alg_name;
- kparam_block_sysfs_write(ieee80211_default_rc_algo);
+ kparam_block_sysfs();
if (!name)
alg_name = ieee80211_default_rc_algo;
else
@@ -117,7 +117,7 @@ ieee80211_rate_control_ops_get(const char *name)
/* try built-in one if specific alg requested but not found */
if (!ops && strlen(CONFIG_MAC80211_RC_DEFAULT))
ops = ieee80211_try_rate_control_ops_get(CONFIG_MAC80211_RC_DEFAULT);
- kparam_unblock_sysfs_write(ieee80211_default_rc_algo);
+ kparam_unblock_sysfs();
return ops;
}
--
2.1.0
Change the "enabled" parameter to be configurable at runtime. Remove
the enabled check from init(), and move it to the frontswap store()
function; when enabled, pages will be stored, and when disabled, pages
won't be stored.
Signed-off-by: Dan Streetman <[email protected]>
---
mm/zswap.c | 13 +++++++------
1 file changed, 7 insertions(+), 6 deletions(-)
diff --git a/mm/zswap.c b/mm/zswap.c
index 4249e82..e070b10 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -75,9 +75,10 @@ static u64 zswap_duplicate_entry;
/*********************************
* tunables
**********************************/
-/* Enable/disable zswap (disabled by default, fixed at boot for now) */
-static bool zswap_enabled __read_mostly;
-module_param_named(enabled, zswap_enabled, bool, 0444);
+
+/* Enable/disable zswap (disabled by default) */
+static bool zswap_enabled;
+module_param_named(enabled, zswap_enabled, bool, 0644);
/* Compressor to be used by zswap (fixed at boot for now) */
#define ZSWAP_COMPRESSOR_DEFAULT "lzo"
@@ -648,6 +649,9 @@ static int zswap_frontswap_store(unsigned type, pgoff_t offset,
u8 *src, *dst;
struct zswap_header *zhdr;
+ if (!zswap_enabled)
+ return -EPERM;
+
if (!tree) {
ret = -ENODEV;
goto reject;
@@ -901,9 +905,6 @@ static int __init init_zswap(void)
{
gfp_t gfp = __GFP_NORETRY | __GFP_NOWARN;
- if (!zswap_enabled)
- return 0;
-
pr_info("loading zswap\n");
zswap_pool = zpool_create_pool(zswap_zpool_type, "zswap", gfp,
--
2.1.0
Add dynamic creation of pools. Move the static crypto compression
per-cpu transforms into each pool. Add a pointer to zswap_entry to
the pool it's in.
This is required by the following patch which enables changing the
zswap zpool and compressor params at runtime.
Signed-off-by: Dan Streetman <[email protected]>
---
mm/zswap.c | 550 +++++++++++++++++++++++++++++++++++++++++++++----------------
1 file changed, 408 insertions(+), 142 deletions(-)
diff --git a/mm/zswap.c b/mm/zswap.c
index e070b10..6eb0d93 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -99,66 +99,19 @@ module_param_named(zpool, zswap_zpool_type, charp, 0444);
static struct zpool *zswap_pool;
/*********************************
-* compression functions
+* data structures
**********************************/
-/* per-cpu compression transforms */
-static struct crypto_comp * __percpu *zswap_comp_pcpu_tfms;
-enum comp_op {
- ZSWAP_COMPOP_COMPRESS,
- ZSWAP_COMPOP_DECOMPRESS
+struct zswap_pool {
+ struct zpool *zpool;
+ struct kref kref;
+ struct list_head list;
+ struct rcu_head rcu_head;
+ struct notifier_block notifier;
+ char tfm_name[CRYPTO_MAX_ALG_NAME];
+ struct crypto_comp * __percpu *tfm;
};
-static int zswap_comp_op(enum comp_op op, const u8 *src, unsigned int slen,
- u8 *dst, unsigned int *dlen)
-{
- struct crypto_comp *tfm;
- int ret;
-
- tfm = *per_cpu_ptr(zswap_comp_pcpu_tfms, get_cpu());
- switch (op) {
- case ZSWAP_COMPOP_COMPRESS:
- ret = crypto_comp_compress(tfm, src, slen, dst, dlen);
- break;
- case ZSWAP_COMPOP_DECOMPRESS:
- ret = crypto_comp_decompress(tfm, src, slen, dst, dlen);
- break;
- default:
- ret = -EINVAL;
- }
-
- put_cpu();
- return ret;
-}
-
-static int __init zswap_comp_init(void)
-{
- if (!crypto_has_comp(zswap_compressor, 0, 0)) {
- pr_info("%s compressor not available\n", zswap_compressor);
- /* fall back to default compressor */
- zswap_compressor = ZSWAP_COMPRESSOR_DEFAULT;
- if (!crypto_has_comp(zswap_compressor, 0, 0))
- /* can't even load the default compressor */
- return -ENODEV;
- }
- pr_info("using %s compressor\n", zswap_compressor);
-
- /* alloc percpu transforms */
- zswap_comp_pcpu_tfms = alloc_percpu(struct crypto_comp *);
- if (!zswap_comp_pcpu_tfms)
- return -ENOMEM;
- return 0;
-}
-
-static void __init zswap_comp_exit(void)
-{
- /* free percpu transforms */
- free_percpu(zswap_comp_pcpu_tfms);
-}
-
-/*********************************
-* data structures
-**********************************/
/*
* struct zswap_entry
*
@@ -166,22 +119,24 @@ static void __init zswap_comp_exit(void)
* page within zswap.
*
* rbnode - links the entry into red-black tree for the appropriate swap type
+ * offset - the swap offset for the entry. Index into the red-black tree.
* refcount - the number of outstanding reference to the entry. This is needed
* to protect against premature freeing of the entry by code
* concurrent calls to load, invalidate, and writeback. The lock
* for the zswap_tree structure that contains the entry must
* be held while changing the refcount. Since the lock must
* be held, there is no reason to also make refcount atomic.
- * offset - the swap offset for the entry. Index into the red-black tree.
- * handle - zpool allocation handle that stores the compressed page data
* length - the length in bytes of the compressed page data. Needed during
* decompression
+ * pool - the zswap_pool the entry's data is in
+ * handle - zpool allocation handle that stores the compressed page data
*/
struct zswap_entry {
struct rb_node rbnode;
pgoff_t offset;
int refcount;
unsigned int length;
+ struct zswap_pool *pool;
unsigned long handle;
};
@@ -201,6 +156,44 @@ struct zswap_tree {
static struct zswap_tree *zswap_trees[MAX_SWAPFILES];
+/* RCU-protected iteration */
+static LIST_HEAD(zswap_pools);
+/* protects zswap_pools list modification */
+static DEFINE_SPINLOCK(zswap_pools_lock);
+
+/*********************************
+* helpers and fwd declarations
+**********************************/
+
+#define zswap_pool_debug(msg, p) \
+ pr_debug("%s pool %s/%s\n", msg, (p)->tfm_name, \
+ zpool_get_type((p)->zpool))
+
+static int zswap_writeback_entry(struct zpool *pool, unsigned long handle);
+static int zswap_pool_get(struct zswap_pool *pool);
+static void zswap_pool_put(struct zswap_pool *pool);
+
+static bool zswap_is_full(void)
+{
+ return totalram_pages * zswap_max_pool_percent / 100 <
+ DIV_ROUND_UP(zswap_pool_total_size, PAGE_SIZE);
+}
+
+static void zswap_update_total_size(void)
+{
+ struct zswap_pool *pool;
+ u64 total = 0;
+
+ rcu_read_lock();
+
+ list_for_each_entry_rcu(pool, &zswap_pools, list)
+ total += zpool_get_total_size(pool->zpool);
+
+ rcu_read_unlock();
+
+ zswap_pool_total_size = total;
+}
+
/*********************************
* zswap entry functions
**********************************/
@@ -294,10 +287,11 @@ static void zswap_rb_erase(struct rb_root *root, struct zswap_entry *entry)
*/
static void zswap_free_entry(struct zswap_entry *entry)
{
- zpool_free(zswap_pool, entry->handle);
+ zpool_free(entry->pool->zpool, entry->handle);
+ zswap_pool_put(entry->pool);
zswap_entry_cache_free(entry);
atomic_dec(&zswap_stored_pages);
- zswap_pool_total_size = zpool_get_total_size(zswap_pool);
+ zswap_update_total_size();
}
/* caller must hold the tree lock */
@@ -339,35 +333,21 @@ static struct zswap_entry *zswap_entry_find_get(struct rb_root *root,
**********************************/
static DEFINE_PER_CPU(u8 *, zswap_dstmem);
-static int __zswap_cpu_notifier(unsigned long action, unsigned long cpu)
+static int __zswap_cpu_dstmem_notifier(unsigned long action, unsigned long cpu)
{
- struct crypto_comp *tfm;
u8 *dst;
switch (action) {
case CPU_UP_PREPARE:
- tfm = crypto_alloc_comp(zswap_compressor, 0, 0);
- if (IS_ERR(tfm)) {
- pr_err("can't allocate compressor transform\n");
- return NOTIFY_BAD;
- }
- *per_cpu_ptr(zswap_comp_pcpu_tfms, cpu) = tfm;
dst = kmalloc_node(PAGE_SIZE * 2, GFP_KERNEL, cpu_to_node(cpu));
if (!dst) {
pr_err("can't allocate compressor buffer\n");
- crypto_free_comp(tfm);
- *per_cpu_ptr(zswap_comp_pcpu_tfms, cpu) = NULL;
return NOTIFY_BAD;
}
per_cpu(zswap_dstmem, cpu) = dst;
break;
case CPU_DEAD:
case CPU_UP_CANCELED:
- tfm = *per_cpu_ptr(zswap_comp_pcpu_tfms, cpu);
- if (tfm) {
- crypto_free_comp(tfm);
- *per_cpu_ptr(zswap_comp_pcpu_tfms, cpu) = NULL;
- }
dst = per_cpu(zswap_dstmem, cpu);
kfree(dst);
per_cpu(zswap_dstmem, cpu) = NULL;
@@ -378,43 +358,309 @@ static int __zswap_cpu_notifier(unsigned long action, unsigned long cpu)
return NOTIFY_OK;
}
-static int zswap_cpu_notifier(struct notifier_block *nb,
- unsigned long action, void *pcpu)
+static int zswap_cpu_dstmem_notifier(struct notifier_block *nb,
+ unsigned long action, void *pcpu)
{
- unsigned long cpu = (unsigned long)pcpu;
- return __zswap_cpu_notifier(action, cpu);
+ return __zswap_cpu_dstmem_notifier(action, (unsigned long)pcpu);
}
-static struct notifier_block zswap_cpu_notifier_block = {
- .notifier_call = zswap_cpu_notifier
+static struct notifier_block zswap_dstmem_notifier = {
+ .notifier_call = zswap_cpu_dstmem_notifier,
};
-static int __init zswap_cpu_init(void)
+static int __init zswap_cpu_dstmem_init(void)
{
unsigned long cpu;
cpu_notifier_register_begin();
for_each_online_cpu(cpu)
- if (__zswap_cpu_notifier(CPU_UP_PREPARE, cpu) != NOTIFY_OK)
+ if (__zswap_cpu_dstmem_notifier(CPU_UP_PREPARE, cpu) ==
+ NOTIFY_BAD)
goto cleanup;
- __register_cpu_notifier(&zswap_cpu_notifier_block);
+ __register_cpu_notifier(&zswap_dstmem_notifier);
cpu_notifier_register_done();
return 0;
cleanup:
for_each_online_cpu(cpu)
- __zswap_cpu_notifier(CPU_UP_CANCELED, cpu);
+ __zswap_cpu_dstmem_notifier(CPU_UP_CANCELED, cpu);
cpu_notifier_register_done();
return -ENOMEM;
}
+static void zswap_cpu_dstmem_destroy(void)
+{
+ unsigned long cpu;
+
+ cpu_notifier_register_begin();
+ for_each_online_cpu(cpu)
+ __zswap_cpu_dstmem_notifier(CPU_UP_CANCELED, cpu);
+ __unregister_cpu_notifier(&zswap_dstmem_notifier);
+ cpu_notifier_register_done();
+}
+
+static int __zswap_cpu_comp_notifier(struct zswap_pool *pool,
+ unsigned long action, unsigned long cpu)
+{
+ struct crypto_comp *tfm;
+
+ switch (action) {
+ case CPU_UP_PREPARE:
+ if (WARN_ON(*per_cpu_ptr(pool->tfm, cpu)))
+ break;
+ tfm = crypto_alloc_comp(pool->tfm_name, 0, 0);
+ if (IS_ERR_OR_NULL(tfm)) {
+ pr_err("could not alloc crypto comp %s : %ld\n",
+ pool->tfm_name, PTR_ERR(tfm));
+ return NOTIFY_BAD;
+ }
+ *per_cpu_ptr(pool->tfm, cpu) = tfm;
+ break;
+ case CPU_DEAD:
+ case CPU_UP_CANCELED:
+ tfm = *per_cpu_ptr(pool->tfm, cpu);
+ if (!IS_ERR_OR_NULL(tfm))
+ crypto_free_comp(tfm);
+ *per_cpu_ptr(pool->tfm, cpu) = NULL;
+ break;
+ default:
+ break;
+ }
+ return NOTIFY_OK;
+}
+
+static int zswap_cpu_comp_notifier(struct notifier_block *nb,
+ unsigned long action, void *pcpu)
+{
+ unsigned long cpu = (unsigned long)pcpu;
+ struct zswap_pool *pool = container_of(nb, typeof(*pool), notifier);
+
+ return __zswap_cpu_comp_notifier(pool, action, cpu);
+}
+
+static int zswap_cpu_comp_init(struct zswap_pool *pool)
+{
+ unsigned long cpu;
+
+ memset(&pool->notifier, 0, sizeof(pool->notifier));
+ pool->notifier.notifier_call = zswap_cpu_comp_notifier;
+
+ cpu_notifier_register_begin();
+ for_each_online_cpu(cpu)
+ if (__zswap_cpu_comp_notifier(pool, CPU_UP_PREPARE, cpu) ==
+ NOTIFY_BAD)
+ goto cleanup;
+ __register_cpu_notifier(&pool->notifier);
+ cpu_notifier_register_done();
+ return 0;
+
+cleanup:
+ for_each_online_cpu(cpu)
+ __zswap_cpu_comp_notifier(pool, CPU_UP_CANCELED, cpu);
+ cpu_notifier_register_done();
+ return -ENOMEM;
+}
+
+static void zswap_cpu_comp_destroy(struct zswap_pool *pool)
+{
+ unsigned long cpu;
+
+ cpu_notifier_register_begin();
+ for_each_online_cpu(cpu)
+ __zswap_cpu_comp_notifier(pool, CPU_UP_CANCELED, cpu);
+ __unregister_cpu_notifier(&pool->notifier);
+ cpu_notifier_register_done();
+}
+
/*********************************
-* helpers
+* pool functions
**********************************/
-static bool zswap_is_full(void)
+
+static struct zswap_pool *__zswap_pool_current(void)
{
- return totalram_pages * zswap_max_pool_percent / 100 <
- DIV_ROUND_UP(zswap_pool_total_size, PAGE_SIZE);
+ struct zswap_pool *pool;
+
+ pool = list_first_or_null_rcu(&zswap_pools, typeof(*pool), list);
+ WARN_ON(!pool);
+
+ return pool;
+}
+
+static struct zswap_pool *zswap_pool_current(void)
+{
+ assert_spin_locked(&zswap_pools_lock);
+
+ return __zswap_pool_current();
+}
+
+static struct zswap_pool *zswap_pool_current_get(void)
+{
+ struct zswap_pool *pool;
+
+ rcu_read_lock();
+
+ pool = __zswap_pool_current();
+ if (!pool || !zswap_pool_get(pool))
+ pool = NULL;
+
+ rcu_read_unlock();
+
+ return pool;
+}
+
+static struct zswap_pool *zswap_pool_last_get(void)
+{
+ struct zswap_pool *pool, *last = NULL;
+
+ rcu_read_lock();
+
+ list_for_each_entry_rcu(pool, &zswap_pools, list)
+ last = pool;
+ if (!WARN_ON(!last) && !zswap_pool_get(last))
+ last = NULL;
+
+ rcu_read_unlock();
+
+ return last;
+}
+
+static struct zpool_ops zswap_zpool_ops = {
+ .evict = zswap_writeback_entry
+};
+
+static struct zswap_pool *zswap_pool_find_get(char *type, char *compressor)
+{
+ struct zswap_pool *pool;
+
+ assert_spin_locked(&zswap_pools_lock);
+
+ list_for_each_entry_rcu(pool, &zswap_pools, list) {
+ if (strncmp(pool->tfm_name, compressor, sizeof(pool->tfm_name)))
+ continue;
+ if (strncmp(zpool_get_type(pool->zpool), type,
+ sizeof(zswap_zpool_type)))
+ continue;
+ /* if we can't get it, it's about to be destroyed */
+ if (!zswap_pool_get(pool))
+ continue;
+ return pool;
+ }
+
+ return NULL;
+}
+
+static struct zswap_pool *zswap_pool_create(char *type, char *compressor)
+{
+ struct zswap_pool *pool;
+ gfp_t gfp = __GFP_NORETRY | __GFP_NOWARN;
+
+ pool = kzalloc(sizeof(*pool), GFP_KERNEL);
+ if (!pool) {
+ pr_err("pool alloc failed\n");
+ return NULL;
+ }
+
+ pool->zpool = zpool_create_pool(type, "zswap", gfp, &zswap_zpool_ops);
+ if (!pool->zpool) {
+ pr_err("%s zpool not available\n", type);
+ goto error;
+ }
+ pr_debug("using %s zpool\n", zpool_get_type(pool->zpool));
+
+ strlcpy(pool->tfm_name, compressor, sizeof(pool->tfm_name));
+ pool->tfm = alloc_percpu(struct crypto_comp *);
+ if (!pool->tfm) {
+ pr_err("percpu alloc failed\n");
+ goto error;
+ }
+
+ if (zswap_cpu_comp_init(pool))
+ goto error;
+ pr_debug("using %s compressor\n", pool->tfm_name);
+
+ /* being the current pool takes 1 ref; this func expects the
+ * caller to always add the new pool as the current pool
+ */
+ kref_init(&pool->kref);
+ INIT_LIST_HEAD(&pool->list);
+
+ zswap_pool_debug("created", pool);
+
+ return pool;
+
+error:
+ free_percpu(pool->tfm);
+ if (pool->zpool)
+ zpool_destroy_pool(pool->zpool);
+ kfree(pool);
+ return NULL;
+}
+
+static struct zswap_pool *__zswap_pool_create_fallback(void)
+{
+ if (!crypto_has_comp(zswap_compressor, 0, 0)) {
+ pr_err("compressor %s not available, using default %s\n",
+ zswap_compressor, ZSWAP_COMPRESSOR_DEFAULT);
+ strncpy(zswap_compressor, ZSWAP_COMPRESSOR_DEFAULT,
+ sizeof(zswap_compressor));
+ }
+ if (!zpool_has_pool(zswap_zpool_type)) {
+ pr_err("zpool %s not available, using default %s\n",
+ zswap_zpool_type, ZSWAP_ZPOOL_DEFAULT);
+ strncpy(zswap_zpool_type, ZSWAP_ZPOOL_DEFAULT,
+ sizeof(zswap_zpool_type));
+ }
+
+ return zswap_pool_create(zswap_zpool_type, zswap_compressor);
+}
+
+static void zswap_pool_destroy(struct zswap_pool *pool)
+{
+ zswap_pool_debug("destroying", pool);
+
+ zswap_cpu_comp_destroy(pool);
+ free_percpu(pool->tfm);
+ zpool_destroy_pool(pool->zpool);
+ kfree(pool);
+}
+
+static int __must_check zswap_pool_get(struct zswap_pool *pool)
+{
+ return kref_get_unless_zero(&pool->kref);
+}
+
+static void __zswap_pool_release(struct rcu_head *head)
+{
+ struct zswap_pool *pool = container_of(head, typeof(*pool), rcu_head);
+
+ /* nobody should have been able to get a kref... */
+ WARN_ON(kref_get_unless_zero(&pool->kref));
+
+ /* pool is now off zswap_pools list and has no references. */
+ zswap_pool_destroy(pool);
+}
+
+static void __zswap_pool_empty(struct kref *kref)
+{
+ struct zswap_pool *pool;
+
+ pool = container_of(kref, typeof(*pool), kref);
+
+ spin_lock(&zswap_pools_lock);
+
+ WARN_ON(pool == zswap_pool_current());
+
+ list_del_rcu(&pool->list);
+ call_rcu(&pool->rcu_head, __zswap_pool_release);
+
+ spin_unlock(&zswap_pools_lock);
+}
+
+static void zswap_pool_put(struct zswap_pool *pool)
+{
+ kref_put(&pool->kref, __zswap_pool_empty);
+}
+
}
/*********************************
@@ -538,6 +784,7 @@ static int zswap_writeback_entry(struct zpool *pool, unsigned long handle)
pgoff_t offset;
struct zswap_entry *entry;
struct page *page;
+ struct crypto_comp *tfm;
u8 *src, *dst;
unsigned int dlen;
int ret;
@@ -578,13 +825,15 @@ static int zswap_writeback_entry(struct zpool *pool, unsigned long handle)
case ZSWAP_SWAPCACHE_NEW: /* page is locked */
/* decompress */
dlen = PAGE_SIZE;
- src = (u8 *)zpool_map_handle(zswap_pool, entry->handle,
+ src = (u8 *)zpool_map_handle(entry->pool->zpool, entry->handle,
ZPOOL_MM_RO) + sizeof(struct zswap_header);
dst = kmap_atomic(page);
- ret = zswap_comp_op(ZSWAP_COMPOP_DECOMPRESS, src,
- entry->length, dst, &dlen);
+ tfm = *get_cpu_ptr(entry->pool->tfm);
+ ret = crypto_comp_decompress(tfm, src, entry->length,
+ dst, &dlen);
+ put_cpu_ptr(entry->pool->tfm);
kunmap_atomic(dst);
- zpool_unmap_handle(zswap_pool, entry->handle);
+ zpool_unmap_handle(entry->pool->zpool, entry->handle);
BUG_ON(ret);
BUG_ON(dlen != PAGE_SIZE);
@@ -633,6 +882,22 @@ end:
return ret;
}
+static int zswap_shrink(void)
+{
+ struct zswap_pool *pool;
+ int ret;
+
+ pool = zswap_pool_last_get();
+ if (!pool)
+ return -ENOENT;
+
+ ret = zpool_shrink(pool->zpool, 1, NULL);
+
+ zswap_pool_put(pool);
+
+ return ret;
+}
+
/*********************************
* frontswap hooks
**********************************/
@@ -642,6 +907,7 @@ static int zswap_frontswap_store(unsigned type, pgoff_t offset,
{
struct zswap_tree *tree = zswap_trees[type];
struct zswap_entry *entry, *dupentry;
+ struct crypto_comp *tfm;
int ret;
unsigned int dlen = PAGE_SIZE, len;
unsigned long handle;
@@ -660,7 +926,7 @@ static int zswap_frontswap_store(unsigned type, pgoff_t offset,
/* reclaim space if needed */
if (zswap_is_full()) {
zswap_pool_limit_hit++;
- if (zpool_shrink(zswap_pool, 1, NULL)) {
+ if (zswap_shrink()) {
zswap_reject_reclaim_fail++;
ret = -ENOMEM;
goto reject;
@@ -675,33 +941,42 @@ static int zswap_frontswap_store(unsigned type, pgoff_t offset,
goto reject;
}
+ /* if entry is successfully added, it keeps the reference */
+ entry->pool = zswap_pool_current_get();
+ if (!entry->pool) {
+ ret = -EINVAL;
+ goto freepage;
+ }
+
/* compress */
dst = get_cpu_var(zswap_dstmem);
+ tfm = *get_cpu_ptr(entry->pool->tfm);
src = kmap_atomic(page);
- ret = zswap_comp_op(ZSWAP_COMPOP_COMPRESS, src, PAGE_SIZE, dst, &dlen);
+ ret = crypto_comp_compress(tfm, src, PAGE_SIZE, dst, &dlen);
kunmap_atomic(src);
+ put_cpu_ptr(entry->pool->tfm);
if (ret) {
ret = -EINVAL;
- goto freepage;
+ goto put_dstmem;
}
/* store */
len = dlen + sizeof(struct zswap_header);
- ret = zpool_malloc(zswap_pool, len, __GFP_NORETRY | __GFP_NOWARN,
- &handle);
+ ret = zpool_malloc(entry->pool->zpool, len,
+ __GFP_NORETRY | __GFP_NOWARN, &handle);
if (ret == -ENOSPC) {
zswap_reject_compress_poor++;
- goto freepage;
+ goto put_dstmem;
}
if (ret) {
zswap_reject_alloc_fail++;
- goto freepage;
+ goto put_dstmem;
}
- zhdr = zpool_map_handle(zswap_pool, handle, ZPOOL_MM_RW);
+ zhdr = zpool_map_handle(entry->pool->zpool, handle, ZPOOL_MM_RW);
zhdr->swpentry = swp_entry(type, offset);
buf = (u8 *)(zhdr + 1);
memcpy(buf, dst, dlen);
- zpool_unmap_handle(zswap_pool, handle);
+ zpool_unmap_handle(entry->pool->zpool, handle);
put_cpu_var(zswap_dstmem);
/* populate entry */
@@ -724,12 +999,14 @@ static int zswap_frontswap_store(unsigned type, pgoff_t offset,
/* update stats */
atomic_inc(&zswap_stored_pages);
- zswap_pool_total_size = zpool_get_total_size(zswap_pool);
+ zswap_update_total_size();
return 0;
-freepage:
+put_dstmem:
put_cpu_var(zswap_dstmem);
+ zswap_pool_put(entry->pool);
+freepage:
zswap_entry_cache_free(entry);
reject:
return ret;
@@ -744,6 +1021,7 @@ static int zswap_frontswap_load(unsigned type, pgoff_t offset,
{
struct zswap_tree *tree = zswap_trees[type];
struct zswap_entry *entry;
+ struct crypto_comp *tfm;
u8 *src, *dst;
unsigned int dlen;
int ret;
@@ -760,13 +1038,14 @@ static int zswap_frontswap_load(unsigned type, pgoff_t offset,
/* decompress */
dlen = PAGE_SIZE;
- src = (u8 *)zpool_map_handle(zswap_pool, entry->handle,
+ src = (u8 *)zpool_map_handle(entry->pool->zpool, entry->handle,
ZPOOL_MM_RO) + sizeof(struct zswap_header);
dst = kmap_atomic(page);
- ret = zswap_comp_op(ZSWAP_COMPOP_DECOMPRESS, src, entry->length,
- dst, &dlen);
+ tfm = *get_cpu_ptr(entry->pool->tfm);
+ ret = crypto_comp_decompress(tfm, src, entry->length, dst, &dlen);
+ put_cpu_ptr(entry->pool->tfm);
kunmap_atomic(dst);
- zpool_unmap_handle(zswap_pool, entry->handle);
+ zpool_unmap_handle(entry->pool->zpool, entry->handle);
BUG_ON(ret);
spin_lock(&tree->lock);
@@ -819,10 +1098,6 @@ static void zswap_frontswap_invalidate_area(unsigned type)
zswap_trees[type] = NULL;
}
-static struct zpool_ops zswap_zpool_ops = {
- .evict = zswap_writeback_entry
-};
-
static void zswap_frontswap_init(unsigned type)
{
struct zswap_tree *tree;
@@ -903,49 +1178,40 @@ static void __exit zswap_debugfs_exit(void) { }
**********************************/
static int __init init_zswap(void)
{
- gfp_t gfp = __GFP_NORETRY | __GFP_NOWARN;
+ struct zswap_pool *pool;
- pr_info("loading zswap\n");
-
- zswap_pool = zpool_create_pool(zswap_zpool_type, "zswap", gfp,
- &zswap_zpool_ops);
- if (!zswap_pool && strcmp(zswap_zpool_type, ZSWAP_ZPOOL_DEFAULT)) {
- pr_info("%s zpool not available\n", zswap_zpool_type);
- zswap_zpool_type = ZSWAP_ZPOOL_DEFAULT;
- zswap_pool = zpool_create_pool(zswap_zpool_type, "zswap", gfp,
- &zswap_zpool_ops);
- }
- if (!zswap_pool) {
- pr_err("%s zpool not available\n", zswap_zpool_type);
- pr_err("zpool creation failed\n");
- goto error;
- }
- pr_info("using %s pool\n", zswap_zpool_type);
+ pr_info("loading\n");
if (zswap_entry_cache_create()) {
pr_err("entry cache creation failed\n");
- goto cachefail;
+ goto cache_fail;
}
- if (zswap_comp_init()) {
- pr_err("compressor initialization failed\n");
- goto compfail;
+
+ if (zswap_cpu_dstmem_init()) {
+ pr_err("dstmem alloc failed\n");
+ goto dstmem_fail;
}
- if (zswap_cpu_init()) {
- pr_err("per-cpu initialization failed\n");
- goto pcpufail;
+
+ pool = __zswap_pool_create_fallback();
+ if (!pool) {
+ pr_err("pool creation failed\n");
+ goto pool_fail;
}
+ pr_info("loaded using pool %s/%s\n", pool->tfm_name,
+ zpool_get_type(pool->zpool));
+
+ list_add(&pool->list, &zswap_pools);
frontswap_register_ops(&zswap_frontswap_ops);
if (zswap_debugfs_init())
pr_warn("debugfs initialization failed\n");
return 0;
-pcpufail:
- zswap_comp_exit();
-compfail:
+
+pool_fail:
+ zswap_cpu_dstmem_destroy();
+dstmem_fail:
zswap_entry_cache_destroy();
-cachefail:
- zpool_destroy_pool(zswap_pool);
-error:
+cache_fail:
return -ENOMEM;
}
/* must be late so crypto has time to come up */
--
2.1.0
Update the zpool and compressor parameters to be changeable at runtime.
When changed, a new pool is created with the requested zpool/compressor,
and added as the current pool at the front of the pool list. Previous
pools remain in the list only to remove existing compressed pages from.
The old pool(s) are removed once they become empty.
Signed-off-by: Dan Streetman <[email protected]>
---
mm/zswap.c | 135 +++++++++++++++++++++++++++++++++++++++++++++++++++++++------
1 file changed, 122 insertions(+), 13 deletions(-)
diff --git a/mm/zswap.c b/mm/zswap.c
index 6eb0d93..af74bd2 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -80,23 +80,39 @@ static u64 zswap_duplicate_entry;
static bool zswap_enabled;
module_param_named(enabled, zswap_enabled, bool, 0644);
-/* Compressor to be used by zswap (fixed at boot for now) */
+/* Crypto compressor to use */
#define ZSWAP_COMPRESSOR_DEFAULT "lzo"
-static char *zswap_compressor = ZSWAP_COMPRESSOR_DEFAULT;
-module_param_named(compressor, zswap_compressor, charp, 0444);
-
-/* The maximum percentage of memory that the compressed pool can occupy */
-static unsigned int zswap_max_pool_percent = 20;
-module_param_named(max_pool_percent,
- zswap_max_pool_percent, uint, 0644);
+static char zswap_compressor[CRYPTO_MAX_ALG_NAME] = ZSWAP_COMPRESSOR_DEFAULT;
+static struct kparam_string zswap_compressor_kparam = {
+ .string = zswap_compressor,
+ .maxlen = sizeof(zswap_compressor),
+};
+static int zswap_compressor_param_set(const char *,
+ const struct kernel_param *);
+static struct kernel_param_ops zswap_compressor_param_ops = {
+ .set = zswap_compressor_param_set,
+ .get = param_get_string,
+};
+module_param_cb(compressor, &zswap_compressor_param_ops,
+ &zswap_compressor_kparam, 0644);
-/* Compressed storage to use */
+/* Compressed storage zpool to use */
#define ZSWAP_ZPOOL_DEFAULT "zbud"
-static char *zswap_zpool_type = ZSWAP_ZPOOL_DEFAULT;
-module_param_named(zpool, zswap_zpool_type, charp, 0444);
+static char zswap_zpool_type[32 /* arbitrary */] = ZSWAP_ZPOOL_DEFAULT;
+static struct kparam_string zswap_zpool_kparam = {
+ .string = zswap_zpool_type,
+ .maxlen = sizeof(zswap_zpool_type),
+};
+static int zswap_zpool_param_set(const char *, const struct kernel_param *);
+static struct kernel_param_ops zswap_zpool_param_ops = {
+ .set = zswap_zpool_param_set,
+ .get = param_get_string,
+};
+module_param_cb(zpool, &zswap_zpool_param_ops, &zswap_zpool_kparam, 0644);
-/* zpool is shared by all of zswap backend */
-static struct zpool *zswap_pool;
+/* The maximum percentage of memory that the compressed pool can occupy */
+static unsigned int zswap_max_pool_percent = 20;
+module_param_named(max_pool_percent, zswap_max_pool_percent, uint, 0644);
/*********************************
* data structures
@@ -161,6 +177,9 @@ static LIST_HEAD(zswap_pools);
/* protects zswap_pools list modification */
static DEFINE_SPINLOCK(zswap_pools_lock);
+/* used by param callback function */
+static bool zswap_init_started;
+
/*********************************
* helpers and fwd declarations
**********************************/
@@ -661,6 +680,94 @@ static void zswap_pool_put(struct zswap_pool *pool)
kref_put(&pool->kref, __zswap_pool_empty);
}
+/*********************************
+* param callbacks
+**********************************/
+
+static int __zswap_param_set(const char *val, const struct kernel_param *kp,
+ char *type, char *compressor)
+{
+ struct zswap_pool *pool, *put_pool = NULL;
+ char str[kp->str->maxlen], *s;
+ int ret;
+
+ strlcpy(str, val, kp->str->maxlen);
+ s = strim(str);
+
+ /* if this is load-time (pre-init) param setting,
+ * don't create a pool; that's done during init.
+ */
+ if (!zswap_init_started)
+ return param_set_copystring(s, kp);
+
+ /* no change required */
+ if (!strncmp(kp->str->string, s, kp->str->maxlen))
+ return 0;
+
+ if (!type) {
+ type = s;
+ if (!zpool_has_pool(type)) {
+ pr_err("zpool %s not available\n", type);
+ return -ENOENT;
+ }
+ } else if (!compressor) {
+ compressor = s;
+ if (!crypto_has_comp(compressor, 0, 0)) {
+ pr_err("compressor %s not available\n", compressor);
+ return -ENOENT;
+ }
+ }
+
+ spin_lock(&zswap_pools_lock);
+
+ pool = zswap_pool_find_get(type, compressor);
+ if (pool) {
+ zswap_pool_debug("using existing", pool);
+ list_del_rcu(&pool->list);
+ } else {
+ spin_unlock(&zswap_pools_lock);
+ pool = zswap_pool_create(type, compressor);
+ spin_lock(&zswap_pools_lock);
+ }
+
+ if (pool)
+ ret = param_set_copystring(s, kp);
+ else
+ ret = -EINVAL;
+
+ if (!ret) {
+ put_pool = zswap_pool_current();
+ list_add_rcu(&pool->list, &zswap_pools);
+ } else if (pool) {
+ /* add the possibly pre-existing pool to the end of the pools
+ * list; if it's new (and empty) then it'll be removed and
+ * destroyed by the put after we drop the lock
+ */
+ list_add_tail_rcu(&pool->list, &zswap_pools);
+ put_pool = pool;
+ }
+
+ spin_unlock(&zswap_pools_lock);
+
+ /* drop the ref from either the old current pool,
+ * or the new pool we failed to add
+ */
+ if (put_pool)
+ zswap_pool_put(put_pool);
+
+ return ret;
+}
+
+static int zswap_compressor_param_set(const char *val,
+ const struct kernel_param *kp)
+{
+ return __zswap_param_set(val, kp, zswap_zpool_type, NULL);
+}
+
+static int zswap_zpool_param_set(const char *val,
+ const struct kernel_param *kp)
+{
+ return __zswap_param_set(val, kp, NULL, zswap_compressor);
}
/*********************************
@@ -1182,6 +1289,8 @@ static int __init init_zswap(void)
pr_info("loading\n");
+ zswap_init_started = true;
+
if (zswap_entry_cache_create()) {
pr_err("entry cache creation failed\n");
goto cache_fail;
--
2.1.0
On Tue, Jun 02, 2015 at 11:11:55AM -0400, Dan Streetman wrote:
> Change the "enabled" parameter to be configurable at runtime. Remove
> the enabled check from init(), and move it to the frontswap store()
> function; when enabled, pages will be stored, and when disabled, pages
> won't be stored.
I like this one. So much so I wrote it about 2 years ago :)
http://lkml.iu.edu/hypermail/linux/kernel/1307.2/04289.html
It didn't go in though and I forgot about it.
We need to update the documentation too (see my patch).
Thanks,
Seth
>
> Signed-off-by: Dan Streetman <[email protected]>
> ---
> mm/zswap.c | 13 +++++++------
> 1 file changed, 7 insertions(+), 6 deletions(-)
>
> diff --git a/mm/zswap.c b/mm/zswap.c
> index 4249e82..e070b10 100644
> --- a/mm/zswap.c
> +++ b/mm/zswap.c
> @@ -75,9 +75,10 @@ static u64 zswap_duplicate_entry;
> /*********************************
> * tunables
> **********************************/
> -/* Enable/disable zswap (disabled by default, fixed at boot for now) */
> -static bool zswap_enabled __read_mostly;
> -module_param_named(enabled, zswap_enabled, bool, 0444);
> +
> +/* Enable/disable zswap (disabled by default) */
> +static bool zswap_enabled;
> +module_param_named(enabled, zswap_enabled, bool, 0644);
>
> /* Compressor to be used by zswap (fixed at boot for now) */
> #define ZSWAP_COMPRESSOR_DEFAULT "lzo"
> @@ -648,6 +649,9 @@ static int zswap_frontswap_store(unsigned type, pgoff_t offset,
> u8 *src, *dst;
> struct zswap_header *zhdr;
>
> + if (!zswap_enabled)
> + return -EPERM;
> +
> if (!tree) {
> ret = -ENODEV;
> goto reject;
> @@ -901,9 +905,6 @@ static int __init init_zswap(void)
> {
> gfp_t gfp = __GFP_NORETRY | __GFP_NOWARN;
>
> - if (!zswap_enabled)
> - return 0;
> -
> pr_info("loading zswap\n");
>
> zswap_pool = zpool_create_pool(zswap_zpool_type, "zswap", gfp,
> --
> 2.1.0
>
On Tue, Jun 02, 2015 at 11:11:52AM -0400, Dan Streetman wrote:
> This patch series allows setting all zswap params at runtime, instead
> of only being settable at boot-time.
>
> The changes to zswap are rather large, due to the creation of zswap pools,
> which contain both a compressor function as well as a zpool. When either
> the compressor or zpool param is changed at runtime, a new zswap pool is
> created with the new compressor and zpool, and used for all new compressed
> pages. Any old zswap pools that still contain pages are retained only to
> load pages from, and destroyed once they become empty.
>
> One notable change required for this to work is to split the currently
> global kernel param mutex into a global mutex only for built-in params,
> and a per-module mutex for loadable module params. The reason this change
> is required is because zswap's compressor and zpool param handler callback
> functions attempt to load, via crypto_has_comp() and the new zpool_has_pool()
> functions, any required compressor or zpool modules. The problem there is
> that the zswap param callback functions run while the global param mutex is
> locked, but when they attempt to load another module, if the loading module
> has any params set e.g. via /etc/modprobe.d/*.conf, modprobe will also try
> to take the global param mutex, and a deadlock will result, with the mutex
> held by the zswap param callback which is waiting for modprobe, but modprobe
> waiting for the mutex to change the loading module's param. Using a
> per-module mutex for all loadable modules prevents this, since each module
> will take its own mutex and never conflict with another module's param
> changes.
Nice work Dan :)
I'm trying to look at this as three different efforts. In order of
increasing difficulty:
- Enabling/disabling zswap at runtime
- Changing the compressor at runtime, which doesn't involve the zpool layer
- Changing the allocator (type) at runtime which does involve the zpool layer.
In other words, we can store entries that use a different compressor in
the same zpool, but not entries stored in different allocators.
Enabling zswap at runtime is very straightforward, especially if you
aren't going to attempt to flush out all the pages on a disable; only
prevent new stores. I like that.
Changing the compressor at runtime is the next easiest one, since you
have to allocate new compressor transforms, but not a new zpool. You
just store which compressor was used on a per-entry basis.
Changing the allocator (type) is the hardest since it involves a new
zpool, and all the code for managing multiple zpools in zswap.
This is a lot of change all at once. Maybe we could just do the runtime
enable/disable of zswap and the runtime change of compressors first? I
think those two alone would be a lot less invasive. Then we can look at
runtime change of the allocator as a separate thing.
Thanks,
Seth
>
>
> Dan Streetman (5):
> zpool: add zpool_has_pool()
> module: add per-module params lock
> zswap: runtime enable/disable
> zswap: dynamic pool creation
> zswap: change zpool/compressor at runtime
>
> arch/um/drivers/hostaudio_kern.c | 20 +-
> drivers/net/ethernet/myricom/myri10ge/myri10ge.c | 6 +-
> drivers/net/wireless/libertas_tf/if_usb.c | 6 +-
> drivers/usb/atm/ueagle-atm.c | 4 +-
> drivers/video/fbdev/vt8623fb.c | 4 +-
> include/linux/module.h | 1 +
> include/linux/moduleparam.h | 67 +--
> include/linux/zpool.h | 2 +
> kernel/module.c | 1 +
> kernel/params.c | 45 +-
> mm/zpool.c | 25 +
> mm/zswap.c | 696 +++++++++++++++++------
> net/mac80211/rate.c | 4 +-
> 13 files changed, 640 insertions(+), 241 deletions(-)
>
> --
> 2.1.0
>
On Tue, Jun 2, 2015 at 4:11 PM, Seth Jennings <[email protected]> wrote:
> On Tue, Jun 02, 2015 at 11:11:55AM -0400, Dan Streetman wrote:
>> Change the "enabled" parameter to be configurable at runtime. Remove
>> the enabled check from init(), and move it to the frontswap store()
>> function; when enabled, pages will be stored, and when disabled, pages
>> won't be stored.
>
> I like this one. So much so I wrote it about 2 years ago :)
>
> http://lkml.iu.edu/hypermail/linux/kernel/1307.2/04289.html
>
> It didn't go in though and I forgot about it.
>
> We need to update the documentation too (see my patch).
ok sure, forgot about the docs. I'll resend just this patch, with doc
updates, so we can split things up as you suggested.
>
> Thanks,
> Seth
>
>>
>> Signed-off-by: Dan Streetman <[email protected]>
>> ---
>> mm/zswap.c | 13 +++++++------
>> 1 file changed, 7 insertions(+), 6 deletions(-)
>>
>> diff --git a/mm/zswap.c b/mm/zswap.c
>> index 4249e82..e070b10 100644
>> --- a/mm/zswap.c
>> +++ b/mm/zswap.c
>> @@ -75,9 +75,10 @@ static u64 zswap_duplicate_entry;
>> /*********************************
>> * tunables
>> **********************************/
>> -/* Enable/disable zswap (disabled by default, fixed at boot for now) */
>> -static bool zswap_enabled __read_mostly;
>> -module_param_named(enabled, zswap_enabled, bool, 0444);
>> +
>> +/* Enable/disable zswap (disabled by default) */
>> +static bool zswap_enabled;
>> +module_param_named(enabled, zswap_enabled, bool, 0644);
>>
>> /* Compressor to be used by zswap (fixed at boot for now) */
>> #define ZSWAP_COMPRESSOR_DEFAULT "lzo"
>> @@ -648,6 +649,9 @@ static int zswap_frontswap_store(unsigned type, pgoff_t offset,
>> u8 *src, *dst;
>> struct zswap_header *zhdr;
>>
>> + if (!zswap_enabled)
>> + return -EPERM;
>> +
>> if (!tree) {
>> ret = -ENODEV;
>> goto reject;
>> @@ -901,9 +905,6 @@ static int __init init_zswap(void)
>> {
>> gfp_t gfp = __GFP_NORETRY | __GFP_NOWARN;
>>
>> - if (!zswap_enabled)
>> - return 0;
>> -
>> pr_info("loading zswap\n");
>>
>> zswap_pool = zpool_create_pool(zswap_zpool_type, "zswap", gfp,
>> --
>> 2.1.0
>>
On Tue, Jun 2, 2015 at 4:26 PM, Seth Jennings <[email protected]> wrote:
> On Tue, Jun 02, 2015 at 11:11:52AM -0400, Dan Streetman wrote:
>> This patch series allows setting all zswap params at runtime, instead
>> of only being settable at boot-time.
>>
>> The changes to zswap are rather large, due to the creation of zswap pools,
>> which contain both a compressor function as well as a zpool. When either
>> the compressor or zpool param is changed at runtime, a new zswap pool is
>> created with the new compressor and zpool, and used for all new compressed
>> pages. Any old zswap pools that still contain pages are retained only to
>> load pages from, and destroyed once they become empty.
>>
>> One notable change required for this to work is to split the currently
>> global kernel param mutex into a global mutex only for built-in params,
>> and a per-module mutex for loadable module params. The reason this change
>> is required is because zswap's compressor and zpool param handler callback
>> functions attempt to load, via crypto_has_comp() and the new zpool_has_pool()
>> functions, any required compressor or zpool modules. The problem there is
>> that the zswap param callback functions run while the global param mutex is
>> locked, but when they attempt to load another module, if the loading module
>> has any params set e.g. via /etc/modprobe.d/*.conf, modprobe will also try
>> to take the global param mutex, and a deadlock will result, with the mutex
>> held by the zswap param callback which is waiting for modprobe, but modprobe
>> waiting for the mutex to change the loading module's param. Using a
>> per-module mutex for all loadable modules prevents this, since each module
>> will take its own mutex and never conflict with another module's param
>> changes.
>
> Nice work Dan :)
>
> I'm trying to look at this as three different efforts. In order of
> increasing difficulty:
> - Enabling/disabling zswap at runtime
> - Changing the compressor at runtime, which doesn't involve the zpool layer
> - Changing the allocator (type) at runtime which does involve the zpool layer.
>
> In other words, we can store entries that use a different compressor in
> the same zpool, but not entries stored in different allocators.
>
> Enabling zswap at runtime is very straightforward, especially if you
> aren't going to attempt to flush out all the pages on a disable; only
> prevent new stores. I like that.
>
> Changing the compressor at runtime is the next easiest one, since you
> have to allocate new compressor transforms, but not a new zpool. You
> just store which compressor was used on a per-entry basis.
>
> Changing the allocator (type) is the hardest since it involves a new
> zpool, and all the code for managing multiple zpools in zswap.
>
> This is a lot of change all at once. Maybe we could just do the runtime
> enable/disable of zswap and the runtime change of compressors first? I
> think those two alone would be a lot less invasive. Then we can look at
> runtime change of the allocator as a separate thing.
Sure I'll send the enable/disable individually first, with doc updates.
I'll send the other patches as well, to consider separately.
>
> Thanks,
> Seth
>
>>
>>
>> Dan Streetman (5):
>> zpool: add zpool_has_pool()
>> module: add per-module params lock
>> zswap: runtime enable/disable
>> zswap: dynamic pool creation
>> zswap: change zpool/compressor at runtime
>>
>> arch/um/drivers/hostaudio_kern.c | 20 +-
>> drivers/net/ethernet/myricom/myri10ge/myri10ge.c | 6 +-
>> drivers/net/wireless/libertas_tf/if_usb.c | 6 +-
>> drivers/usb/atm/ueagle-atm.c | 4 +-
>> drivers/video/fbdev/vt8623fb.c | 4 +-
>> include/linux/module.h | 1 +
>> include/linux/moduleparam.h | 67 +--
>> include/linux/zpool.h | 2 +
>> kernel/module.c | 1 +
>> kernel/params.c | 45 +-
>> mm/zpool.c | 25 +
>> mm/zswap.c | 696 +++++++++++++++++------
>> net/mac80211/rate.c | 4 +-
>> 13 files changed, 640 insertions(+), 241 deletions(-)
>>
>> --
>> 2.1.0
>>