These patches implement support for Open-Channel SSDs.
Applies against axboe's linux-block/for-4.3/drivers.
Any feedback is greatly appreciated.
Changes since v4:
- Remove gendisk->nvm dependency
- Remove device driver rq private field dependency.
- Update submission and completion. The flow is now
Target -> Block Manager -> Device Driver, replacing callbacks in
device driver.
- Abstracted out the block manager into its own module. Other block
managers can now be implemented. For example to support fully
host-based SSDs.
- No longer exposes the device driver gendisk to user-space.
- Management is moved into /sys/modules/lnvm/parameters/configure_debug
Changes since v3:
- Remove dependency on REQ_NVM_GC
- Refactor nvme integration to use nvme_submit_sync_cmd for
internal commands.
- Fix race condition bug on multiple threads on RRPC target.
- Rename sysfs entry under the block device from nvm to lightnvm.
The configuration is found in /sys/block/*/lightnvm/
Changes since v2:
Feedback from Paul Bolle:
- Fix license to GPLv2, documentation, compilation.
Feedback from Keith Busch:
- nvme: Move lightnvm out and into nvme-lightnvm.c.
- nvme: Set controller css on lightnvm command set.
- nvme: Remove OACS.
Feedback from Christoph Hellwig:
- lightnvm: Move out of block layer into /drivers/lightnvm/core.c
- lightnvm: refactor request->phys_sector into device drivers.
- lightnvm: refactor prep/unprep into device drivers.
- lightnvm: move nvm_dev from request_queue to gendisk.
New
- Bad block table support (From Javier).
- Update maintainers file.
Changes since v1:
- Splitted LightNVM into two parts. A get/put interface for flash
blocks and the respective targets that implement flash translation
layer logic.
- Updated the patches according to the LightNVM specification changes.
- Added interface to add/remove targets for a block device.
Thanks to Jens Axboe, Christoph Hellwig, Keith Busch, Javier Gonzalez
and Jesper Madsen for discussions and contributions.
Matias Bjørling (5):
lightnvm: Support for Open-Channel SSDs
rrpc: RRPC target
bm_hb: Hybrid Open-Channel SSD block manager
null_blk: LightNVM support
nvme: LightNVM support
Documentation/block/null_blk.txt | 8 +
MAINTAINERS | 8 +
drivers/Kconfig | 2 +
drivers/Makefile | 5 +
drivers/block/Makefile | 2 +-
drivers/block/null_blk.c | 138 ++++-
drivers/block/nvme-core.c | 22 +-
drivers/block/nvme-lightnvm.c | 504 ++++++++++++++++
drivers/lightnvm/Kconfig | 33 ++
drivers/lightnvm/Makefile | 7 +
drivers/lightnvm/bm_hb.c | 379 ++++++++++++
drivers/lightnvm/bm_hb.h | 31 +
drivers/lightnvm/core.c | 598 +++++++++++++++++++
drivers/lightnvm/rrpc.c | 1190 ++++++++++++++++++++++++++++++++++++++
drivers/lightnvm/rrpc.h | 220 +++++++
include/linux/lightnvm.h | 393 +++++++++++++
include/linux/nvme.h | 6 +
include/uapi/linux/nvme.h | 3 +
18 files changed, 3539 insertions(+), 10 deletions(-)
create mode 100644 drivers/block/nvme-lightnvm.c
create mode 100644 drivers/lightnvm/Kconfig
create mode 100644 drivers/lightnvm/Makefile
create mode 100644 drivers/lightnvm/bm_hb.c
create mode 100644 drivers/lightnvm/bm_hb.h
create mode 100644 drivers/lightnvm/core.c
create mode 100644 drivers/lightnvm/rrpc.c
create mode 100644 drivers/lightnvm/rrpc.h
create mode 100644 include/linux/lightnvm.h
--
2.1.4
Open-channel SSDs are devices that share responsibilities with the host
in order to implement and maintain features that typical SSDs keep
strictly in firmware. These include (i) the Flash Translation Layer
(FTL), (ii) bad block management, and (iii) hardware units such as the
flash controller, the interface controller, and large amounts of flash
chips. In this way, Open-channels SSDs exposes direct access to their
physical flash storage, while keeping a subset of the internal features
of SSDs.
LightNVM is a specification that gives support to Open-channel SSDs
LightNVM allows the host to manage data placement, garbage collection,
and parallelism. Device specific responsibilities such as bad block
management, FTL extensions to support atomic IOs, or metadata
persistence are still handled by the device.
The implementation of LightNVM consists of two parts: core and
(multiple) targets. The core implements functionality shared across
targets. This is initialization, teardown and statistics. The targets
implement the interface that exposes physical flash to user-space
applications. Examples of such targets include key-value store,
object-store, as well as traditional block devices, which can be
application-specific.
Contributions in this patch from:
Javier Gonzalez <[email protected]>
Jesper Madsen <[email protected]>
Signed-off-by: Matias Bjørling <[email protected]>
---
MAINTAINERS | 8 +
drivers/Kconfig | 2 +
drivers/Makefile | 5 +
drivers/lightnvm/Kconfig | 16 ++
drivers/lightnvm/Makefile | 5 +
drivers/lightnvm/core.c | 598 ++++++++++++++++++++++++++++++++++++++++++++++
include/linux/lightnvm.h | 393 ++++++++++++++++++++++++++++++
7 files changed, 1027 insertions(+)
create mode 100644 drivers/lightnvm/Kconfig
create mode 100644 drivers/lightnvm/Makefile
create mode 100644 drivers/lightnvm/core.c
create mode 100644 include/linux/lightnvm.h
diff --git a/MAINTAINERS b/MAINTAINERS
index 2d3d55c..d149104 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -6162,6 +6162,14 @@ S: Supported
F: drivers/nvdimm/pmem.c
F: include/linux/pmem.h
+LIGHTNVM PLATFORM SUPPORT
+M: Matias Bjorling <[email protected]>
+W: http://github/OpenChannelSSD
+S: Maintained
+F: drivers/lightnvm/
+F: include/linux/lightnvm.h
+F: include/uapi/linux/lightnvm.h
+
LINUX FOR IBM pSERIES (RS/6000)
M: Paul Mackerras <[email protected]>
W: http://www.ibm.com/linux/ltc/projects/ppc
diff --git a/drivers/Kconfig b/drivers/Kconfig
index 6e973b8..3992902 100644
--- a/drivers/Kconfig
+++ b/drivers/Kconfig
@@ -42,6 +42,8 @@ source "drivers/net/Kconfig"
source "drivers/isdn/Kconfig"
+source "drivers/lightnvm/Kconfig"
+
# input before char - char/joystick depends on it. As does USB.
source "drivers/input/Kconfig"
diff --git a/drivers/Makefile b/drivers/Makefile
index b64b49f..75978ab 100644
--- a/drivers/Makefile
+++ b/drivers/Makefile
@@ -63,6 +63,10 @@ obj-$(CONFIG_FB_I810) += video/fbdev/i810/
obj-$(CONFIG_FB_INTEL) += video/fbdev/intelfb/
obj-$(CONFIG_PARPORT) += parport/
+
+# lightnvm/ comes before block to initialize bm before usage
+obj-$(CONFIG_NVM) += lightnvm/
+
obj-y += base/ block/ misc/ mfd/ nfc/
obj-$(CONFIG_LIBNVDIMM) += nvdimm/
obj-$(CONFIG_DMA_SHARED_BUFFER) += dma-buf/
@@ -165,3 +169,4 @@ obj-$(CONFIG_RAS) += ras/
obj-$(CONFIG_THUNDERBOLT) += thunderbolt/
obj-$(CONFIG_CORESIGHT) += hwtracing/coresight/
obj-$(CONFIG_ANDROID) += android/
+
diff --git a/drivers/lightnvm/Kconfig b/drivers/lightnvm/Kconfig
new file mode 100644
index 0000000..1f8412c
--- /dev/null
+++ b/drivers/lightnvm/Kconfig
@@ -0,0 +1,16 @@
+#
+# Open-Channel SSD NVM configuration
+#
+
+menuconfig NVM
+ bool "Open-Channel SSD target support"
+ depends on BLOCK
+ help
+ Say Y here to get to enable Open-channel SSDs.
+
+ Open-Channel SSDs implement a set of extension to SSDs, that
+ exposes direct access to the underlying non-volatile memory.
+
+ If you say N, all options in this submenu will be skipped and disabled
+ only do this if you know what you are doing.
+
diff --git a/drivers/lightnvm/Makefile b/drivers/lightnvm/Makefile
new file mode 100644
index 0000000..38185e9
--- /dev/null
+++ b/drivers/lightnvm/Makefile
@@ -0,0 +1,5 @@
+#
+# Makefile for Open-Channel SSDs.
+#
+
+obj-$(CONFIG_NVM) := core.o
diff --git a/drivers/lightnvm/core.c b/drivers/lightnvm/core.c
new file mode 100644
index 0000000..c00b4b4
--- /dev/null
+++ b/drivers/lightnvm/core.c
@@ -0,0 +1,598 @@
+/*
+ * core.c - Open-channel SSD integration core
+ *
+ * Copyright (C) 2015 IT University of Copenhagen
+ * Initial release: Matias Bjorling <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License version
+ * 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; see the file COPYING. If not, write to
+ * the Free Software Foundation, 675 Mass Ave, Cambridge, MA 02139,
+ * USA.
+ *
+ */
+
+#include <linux/blkdev.h>
+#include <linux/blk-mq.h>
+#include <linux/list.h>
+#include <linux/types.h>
+#include <linux/sem.h>
+#include <linux/bitmap.h>
+#include <linux/module.h>
+
+#include <linux/lightnvm.h>
+
+static LIST_HEAD(_targets);
+static LIST_HEAD(_bms);
+static LIST_HEAD(_devices);
+static DECLARE_RWSEM(_lock);
+
+struct nvm_tgt_type *nvm_find_target_type(const char *name)
+{
+ struct nvm_tgt_type *tt;
+
+ list_for_each_entry(tt, &_targets, list)
+ if (!strcmp(name, tt->name))
+ return tt;
+
+ return NULL;
+}
+
+int nvm_register_target(struct nvm_tgt_type *tt)
+{
+ int ret = 0;
+
+ down_write(&_lock);
+ if (nvm_find_target_type(tt->name))
+ ret = -EEXIST;
+ else
+ list_add(&tt->list, &_targets);
+ up_write(&_lock);
+
+ return ret;
+}
+EXPORT_SYMBOL(nvm_register_target);
+
+void nvm_unregister_target(struct nvm_tgt_type *tt)
+{
+ if (!tt)
+ return;
+
+ down_write(&_lock);
+ list_del(&tt->list);
+ up_write(&_lock);
+}
+EXPORT_SYMBOL(nvm_unregister_target);
+
+struct nvm_bm_type *nvm_find_bm_type(const char *name)
+{
+ struct nvm_bm_type *bt;
+
+ list_for_each_entry(bt, &_bms, list)
+ if (!strcmp(name, bt->name))
+ return bt;
+
+ return NULL;
+}
+
+int nvm_register_bm(struct nvm_bm_type *bt)
+{
+ int ret = 0;
+
+ down_write(&_lock);
+ if (nvm_find_bm_type(bt->name))
+ ret = -EEXIST;
+ else
+ list_add(&bt->list, &_bms);
+ up_write(&_lock);
+
+ return ret;
+}
+EXPORT_SYMBOL(nvm_register_bm);
+
+void nvm_unregister_bm(struct nvm_bm_type *bt)
+{
+ if (!bt)
+ return;
+
+ down_write(&_lock);
+ list_del(&bt->list);
+ up_write(&_lock);
+}
+EXPORT_SYMBOL(nvm_unregister_bm);
+
+struct nvm_dev *nvm_find_nvm_dev(const char *name)
+{
+ struct nvm_dev *dev;
+
+ list_for_each_entry(dev, &_devices, devices)
+ if (!strcmp(name, dev->name))
+ return dev;
+
+ return NULL;
+}
+
+
+sector_t nvm_alloc_addr(struct nvm_block *block)
+{
+ sector_t addr = ADDR_EMPTY;
+
+ spin_lock(&block->lock);
+ if (block_is_full(block))
+ goto out;
+
+ addr = block_to_addr(block) + block->next_page;
+
+ block->next_page++;
+out:
+ spin_unlock(&block->lock);
+ return addr;
+}
+EXPORT_SYMBOL(nvm_alloc_addr);
+
+
+struct nvm_block *nvm_get_blk(struct nvm_dev *dev, struct nvm_lun *lun,
+ unsigned long flags)
+{
+ return dev->bm->get_blk(dev, lun, flags);
+}
+EXPORT_SYMBOL(nvm_get_blk);
+
+/* Assumes that all valid pages have already been moved on release to bm */
+void nvm_put_blk(struct nvm_dev *dev, struct nvm_block *blk)
+{
+ return dev->bm->put_blk(dev, blk);
+}
+EXPORT_SYMBOL(nvm_put_blk);
+
+int nvm_submit_io(struct nvm_dev *dev, struct nvm_rq *rqd)
+{
+ return dev->ops->submit_io(dev->q, rqd);
+}
+EXPORT_SYMBOL(nvm_submit_io);
+
+/* Send erase command to device */
+int nvm_erase_blk(struct nvm_dev *dev, struct nvm_block *blk)
+{
+ return dev->bm->erase_blk(dev, blk);
+}
+EXPORT_SYMBOL(nvm_erase_blk);
+
+static void nvm_core_free(struct nvm_dev *dev)
+{
+ kfree(dev->identity.chnls);
+ kfree(dev);
+}
+
+static int nvm_core_init(struct nvm_dev *dev)
+{
+ dev->nr_luns = dev->identity.nchannels;
+ dev->sector_size = EXPOSED_PAGE_SIZE;
+ INIT_LIST_HEAD(&dev->online_targets);
+
+ return 0;
+}
+
+static void nvm_free(struct nvm_dev *dev)
+{
+ if (!dev)
+ return;
+
+ if (dev->bm)
+ dev->bm->unregister_bm(dev);
+
+ nvm_core_free(dev);
+}
+
+int nvm_validate_features(struct nvm_dev *dev)
+{
+ struct nvm_get_features gf;
+ int ret;
+
+ ret = dev->ops->get_features(dev->q, &gf);
+ if (ret)
+ return ret;
+
+ dev->features = gf;
+
+ return 0;
+}
+
+int nvm_validate_responsibility(struct nvm_dev *dev)
+{
+ if (!dev->ops->set_responsibility)
+ return 0;
+
+ return dev->ops->set_responsibility(dev->q, 0);
+}
+
+int nvm_init(struct nvm_dev *dev)
+{
+ struct nvm_bm_type *bt;
+ int ret = 0;
+
+ if (!dev->q || !dev->ops)
+ return -EINVAL;
+
+ if (dev->ops->identify(dev->q, &dev->identity)) {
+ pr_err("nvm: device could not be identified\n");
+ ret = -EINVAL;
+ goto err;
+ }
+
+ pr_debug("nvm dev: ver %u type %u chnls %u\n",
+ dev->identity.ver_id,
+ dev->identity.nvm_type,
+ dev->identity.nchannels);
+
+ ret = nvm_validate_features(dev);
+ if (ret) {
+ pr_err("nvm: disk features are not supported.");
+ goto err;
+ }
+
+ ret = nvm_validate_responsibility(dev);
+ if (ret) {
+ pr_err("nvm: disk responsibilities are not supported.");
+ goto err;
+ }
+
+ ret = nvm_core_init(dev);
+ if (ret) {
+ pr_err("nvm: could not initialize core structures.\n");
+ goto err;
+ }
+
+ if (!dev->nr_luns) {
+ pr_err("nvm: device did not expose any luns.\n");
+ goto err;
+ }
+
+ /* register with device with a supported BM */
+ list_for_each_entry(bt, &_bms, list) {
+ ret = bt->register_bm(dev);
+ if (ret < 0)
+ goto err; /* initialization failed */
+ if (ret > 0) {
+ dev->bm = bt;
+ break; /* successfully initialized */
+ }
+ }
+
+ if (!ret) {
+ pr_info("nvm: no compatible bm was found.\n");
+ return 0;
+ }
+
+ pr_info("nvm: registered %s with luns: %u blocks: %lu sector size: %d\n",
+ dev->name, dev->nr_luns, dev->total_blocks, dev->sector_size);
+
+ return 0;
+err:
+ nvm_free(dev);
+ pr_err("nvm: failed to initialize nvm\n");
+ return ret;
+}
+
+void nvm_exit(struct nvm_dev *dev)
+{
+ nvm_free(dev);
+
+ pr_info("nvm: successfully unloaded\n");
+}
+
+static int nvm_ioctl(struct block_device *bdev, fmode_t mode, unsigned int cmd,
+ unsigned long arg)
+{
+ return 0;
+}
+
+static int nvm_open(struct block_device *bdev, fmode_t mode)
+{
+ return 0;
+}
+
+static void nvm_release(struct gendisk *disk, fmode_t mode)
+{
+}
+
+static const struct block_device_operations nvm_fops = {
+ .owner = THIS_MODULE,
+ .ioctl = nvm_ioctl,
+ .open = nvm_open,
+ .release = nvm_release,
+};
+
+static int nvm_create_target(struct nvm_dev *dev, char *ttname, char *tname,
+ int lun_begin, int lun_end)
+{
+ struct request_queue *tqueue;
+ struct gendisk *tdisk;
+ struct nvm_tgt_type *tt;
+ struct nvm_target *t;
+ void *targetdata;
+
+ tt = nvm_find_target_type(ttname);
+ if (!tt) {
+ pr_err("nvm: target type %s not found\n", ttname);
+ return -EINVAL;
+ }
+
+ down_write(&_lock);
+ list_for_each_entry(t, &dev->online_targets, list) {
+ if (!strcmp(tname, t->disk->disk_name)) {
+ pr_err("nvm: target name already exists.\n");
+ up_write(&_lock);
+ return -EINVAL;
+ }
+ }
+ up_write(&_lock);
+
+ t = kmalloc(sizeof(struct nvm_target), GFP_KERNEL);
+ if (!t)
+ return -ENOMEM;
+
+ tqueue = blk_alloc_queue_node(GFP_KERNEL, dev->q->node);
+ if (!tqueue)
+ goto err_t;
+ blk_queue_make_request(tqueue, tt->make_rq);
+
+ tdisk = alloc_disk(0);
+ if (!tdisk)
+ goto err_queue;
+
+ sprintf(tdisk->disk_name, "%s", tname);
+ tdisk->flags = GENHD_FL_EXT_DEVT;
+ tdisk->major = 0;
+ tdisk->first_minor = 0;
+ tdisk->fops = &nvm_fops;
+ tdisk->queue = tqueue;
+
+ targetdata = tt->init(dev, tdisk, lun_begin, lun_end);
+ if (IS_ERR(targetdata))
+ goto err_init;
+
+ tdisk->private_data = targetdata;
+ tqueue->queuedata = targetdata;
+
+ /* missing support for multi-page IOs. */
+ blk_queue_max_hw_sectors(tqueue, 8);
+
+ set_capacity(tdisk, tt->capacity(targetdata));
+ add_disk(tdisk);
+
+ t->type = tt;
+ t->disk = tdisk;
+
+ down_write(&_lock);
+ list_add_tail(&t->list, &dev->online_targets);
+ up_write(&_lock);
+
+ return 0;
+err_init:
+ put_disk(tdisk);
+err_queue:
+ blk_cleanup_queue(tqueue);
+err_t:
+ kfree(t);
+ return -ENOMEM;
+}
+
+/* _lock must be taken */
+static void nvm_remove_target(struct nvm_target *t)
+{
+ struct nvm_tgt_type *tt = t->type;
+ struct gendisk *tdisk = t->disk;
+ struct request_queue *q = tdisk->queue;
+
+ del_gendisk(tdisk);
+ if (tt->exit)
+ tt->exit(tdisk->private_data);
+
+ blk_cleanup_queue(q);
+
+ put_disk(tdisk);
+
+ list_del(&t->list);
+ kfree(t);
+}
+
+static int nvm_configure_show(const char *val)
+{
+ struct nvm_dev *dev;
+ char opcode, devname[DISK_NAME_LEN];
+ int ret;
+
+ ret = sscanf(val, "%c %s", &opcode, devname);
+ if (ret != 2) {
+ pr_err("nvm: invalid command. Use \"opcode devicename\".\n");
+ return -EINVAL;
+ }
+
+ dev = nvm_find_nvm_dev(devname);
+ if (!dev) {
+ pr_err("nvm: device not found\n");
+ return -EINVAL;
+ }
+
+ if (!dev->bm)
+ return 0;
+
+ dev->bm->free_blocks_print(dev);
+
+ return 0;
+}
+
+static int nvm_configure_del(const char *val)
+{
+ struct nvm_target *t = NULL;
+ struct nvm_dev *dev;
+ char opcode, tname[255];
+ int ret;
+
+ ret = sscanf(val, "%c %s", &opcode, tname);
+ if (ret != 2) {
+ pr_err("nvm: invalid command. Use \"d targetname\".\n");
+ return -EINVAL;
+ }
+
+ down_write(&_lock);
+ list_for_each_entry(dev, &_devices, devices)
+ list_for_each_entry(t, &dev->online_targets, list) {
+ if (!strcmp(tname, t->disk->disk_name)) {
+ nvm_remove_target(t);
+ ret = 0;
+ break;
+ }
+ }
+ up_write(&_lock);
+
+ if (ret) {
+ pr_err("nvm: target \"%s\" doesn't exist.\n", tname);
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int nvm_configure_add(const char *val)
+{
+ struct nvm_dev *dev;
+ char opcode, devname[DISK_NAME_LEN], tgtengine[255], tname[255];
+ int lun_begin, lun_end, ret;
+
+ ret = sscanf(val, "%c %s %s %s %u:%u", &opcode, devname, tgtengine,
+ tname, &lun_begin, &lun_end);
+ if (ret != 6) {
+ pr_err("nvm: invalid command. Use \"opcode device name tgtengine lun_begin:lun_end\".\n");
+ return -EINVAL;
+ }
+
+ dev = nvm_find_nvm_dev(devname);
+ if (!dev) {
+ pr_err("nvm: device not found\n");
+ return -EINVAL;
+ }
+
+ if (lun_begin > lun_end || lun_end > dev->nr_luns) {
+ pr_err("nvm: lun out of bound (%u:%u > %u)\n",
+ lun_begin, lun_end, dev->nr_luns);
+ return -EINVAL;
+ }
+
+ return nvm_create_target(dev, tname, tgtengine, lun_begin, lun_end);
+}
+
+/* Exposes administrative interface through /sys/module/lnvm/configure_by_str */
+static int nvm_configure_by_str_event(const char *val,
+ const struct kernel_param *kp)
+{
+ char opcode;
+ int ret;
+
+ ret = sscanf(val, "%c", &opcode);
+ if (ret != 1) {
+ pr_err("nvm: configure must be in the format of \"opcode ...\"\n");
+ return -EINVAL;
+ }
+
+ switch (opcode) {
+ case 'a':
+ return nvm_configure_add(val);
+ case 'd':
+ return nvm_configure_del(val);
+ case 's':
+ return nvm_configure_show(val);
+ default:
+ pr_err("nvm: invalid opcode.\n");
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int nvm_configure_get(char *buf, const struct kernel_param *kp)
+{
+ int sz = 0;
+ char *buf_start = buf;
+ struct nvm_dev *dev;
+
+ buf += sprintf(buf, "available devices:\n");
+ list_for_each_entry(dev, &_devices, devices) {
+ if (sz > 4095 - DISK_NAME_LEN)
+ break;
+ buf += sprintf(buf, " %s\n", dev->name);
+ }
+
+ return buf - buf_start - 1;
+}
+
+static const struct kernel_param_ops nvm_configure_by_str_event_param_ops = {
+ .set = nvm_configure_by_str_event,
+ .get = nvm_configure_get,
+};
+
+#undef MODULE_PARAM_PREFIX
+#define MODULE_PARAM_PREFIX "lnvm."
+
+module_param_cb(configure_debug, &nvm_configure_by_str_event_param_ops, NULL,
+ 0644);
+
+int nvm_register(struct request_queue *q, struct gendisk *disk,
+ struct nvm_dev_ops *ops)
+{
+ struct nvm_dev *dev;
+ int ret;
+
+ if (!ops->identify || !ops->get_features)
+ return -EINVAL;
+
+ dev = kzalloc(sizeof(struct nvm_dev), GFP_KERNEL);
+ if (!dev)
+ return -ENOMEM;
+
+ dev->q = q;
+ dev->ops = ops;
+ dev->disk = disk;
+ strncpy(dev->name, disk->disk_name, DISK_NAME_LEN);
+
+ ret = nvm_init(dev);
+ if (ret)
+ goto err_init;
+
+ down_write(&_lock);
+ list_add(&dev->devices, &_devices);
+ up_write(&_lock);
+
+ return 0;
+err_init:
+ kfree(dev);
+ return ret;
+}
+EXPORT_SYMBOL(nvm_register);
+
+void nvm_unregister(struct gendisk *disk)
+{
+ struct nvm_dev *dev = nvm_find_nvm_dev(disk->disk_name);
+
+ if (!dev) {
+ pr_err("nvm: could not find device %s on unregister\n",
+ disk->disk_name);
+ return;
+ }
+
+ nvm_exit(dev);
+
+ down_write(&_lock);
+ list_del(&dev->devices);
+ up_write(&_lock);
+}
+EXPORT_SYMBOL(nvm_unregister);
diff --git a/include/linux/lightnvm.h b/include/linux/lightnvm.h
new file mode 100644
index 0000000..b7fbb07
--- /dev/null
+++ b/include/linux/lightnvm.h
@@ -0,0 +1,393 @@
+#ifndef NVM_H
+#define NVM_H
+
+enum {
+ NVM_IO_OK = 0,
+ NVM_IO_REQUEUE = 1,
+ NVM_IO_DONE = 2,
+ NVM_IO_ERR = 3,
+
+ NVM_IOTYPE_NONE = 0,
+ NVM_IOTYPE_GC = 1,
+};
+
+#ifdef CONFIG_NVM
+
+#include <linux/blkdev.h>
+#include <linux/types.h>
+#include <linux/file.h>
+
+enum {
+ /* HW Responsibilities */
+ NVM_RSP_L2P = 1 << 0,
+ NVM_RSP_GC = 1 << 1,
+ NVM_RSP_ECC = 1 << 2,
+
+ /* Physical NVM Type */
+ NVM_NVMT_BLK = 0,
+ NVM_NVMT_BYTE = 1,
+
+ /* Internal IO Scheduling algorithm */
+ NVM_IOSCHED_CHANNEL = 0,
+ NVM_IOSCHED_CHIP = 1,
+
+ /* Status codes */
+ NVM_SUCCESS = 0,
+ NVM_RSP_NOT_CHANGEABLE = 1,
+};
+
+struct nvm_id_chnl {
+ u64 laddr_begin;
+ u64 laddr_end;
+ u32 oob_size;
+ u32 queue_size;
+ u32 gran_read;
+ u32 gran_write;
+ u32 gran_erase;
+ u32 t_r;
+ u32 t_sqr;
+ u32 t_w;
+ u32 t_sqw;
+ u32 t_e;
+ u16 chnl_parallelism;
+ u8 io_sched;
+ u8 res[133];
+};
+
+struct nvm_id {
+ u8 ver_id;
+ u8 nvm_type;
+ u16 nchannels;
+ struct nvm_id_chnl *chnls;
+};
+
+struct nvm_get_features {
+ u64 rsp;
+ u64 ext;
+};
+
+struct nvm_target {
+ struct list_head list;
+ struct nvm_tgt_type *type;
+ struct gendisk *disk;
+};
+
+struct nvm_tgt_instance {
+ struct nvm_tgt_type *tt;
+};
+
+struct nvm_rq {
+ struct nvm_tgt_instance *ins;
+ struct bio *bio;
+ sector_t phys_sector;
+};
+
+static inline struct nvm_rq *nvm_rq_from_pdu(void *pdu)
+{
+ return pdu - sizeof(struct nvm_rq);
+}
+
+static inline void *nvm_rq_to_pdu(struct nvm_rq *rqdata)
+{
+ return rqdata + 1;
+}
+
+struct nvm_block;
+
+extern void nvm_unregister(struct gendisk *);
+
+typedef int (nvm_l2p_update_fn)(u64, u64, u64 *, void *);
+typedef int (nvm_bb_update_fn)(u32, void *, unsigned int, void *);
+typedef int (nvm_id_fn)(struct request_queue *, struct nvm_id *);
+typedef int (nvm_get_features_fn)(struct request_queue *,
+ struct nvm_get_features *);
+typedef int (nvm_set_rsp_fn)(struct request_queue *, u64);
+typedef int (nvm_get_l2p_tbl_fn)(struct request_queue *, u64, u64,
+ nvm_l2p_update_fn *, void *);
+typedef int (nvm_op_bb_tbl_fn)(struct request_queue *, int, unsigned int,
+ nvm_bb_update_fn *, void *);
+typedef int (nvm_submit_io_fn)(struct request_queue *, struct nvm_rq *);
+typedef int (nvm_erase_blk_fn)(struct request_queue *, sector_t);
+
+struct nvm_dev_ops {
+ nvm_id_fn *identify;
+ nvm_get_features_fn *get_features;
+ nvm_set_rsp_fn *set_responsibility;
+ nvm_get_l2p_tbl_fn *get_l2p_tbl;
+ nvm_op_bb_tbl_fn *set_bb_tbl;
+ nvm_op_bb_tbl_fn *get_bb_tbl;
+
+ nvm_submit_io_fn *submit_io;
+ nvm_erase_blk_fn *erase_block;
+};
+
+/*
+ * We assume that the device exposes its channels as a linear address
+ * space. A lun therefore have a phy_addr_start and phy_addr_end that
+ * denotes the start and end. This abstraction is used to let the
+ * open-channel SSD (or any other device) expose its read/write/erase
+ * interface and be administrated by the host system.
+ */
+struct nvm_lun {
+ struct nvm_dev *dev;
+
+ /* lun block lists */
+ struct list_head used_list; /* In-use blocks */
+ struct list_head free_list; /* Not used blocks i.e. released
+ * and ready for use */
+ struct list_head bb_list; /* Bad blocks. Mutually exclusive with
+ free_list and used_list */
+
+ struct {
+ spinlock_t lock;
+ } ____cacheline_aligned_in_smp;
+
+ struct nvm_block *blocks;
+ struct nvm_id_chnl *chnl;
+
+ int id;
+ int reserved_blocks;
+
+ unsigned int nr_blocks; /* end_block - start_block. */
+ unsigned int nr_free_blocks; /* Number of unused blocks */
+
+ int nr_pages_per_blk;
+};
+
+struct nvm_block {
+ /* Management structures */
+ struct list_head list;
+ struct nvm_lun *lun;
+
+ spinlock_t lock;
+
+#define MAX_INVALID_PAGES_STORAGE 8
+ /* Bitmap for invalid page intries */
+ unsigned long invalid_pages[MAX_INVALID_PAGES_STORAGE];
+ /* points to the next writable page within a block */
+ unsigned int next_page;
+ /* number of pages that are invalid, wrt host page size */
+ unsigned int nr_invalid_pages;
+
+ unsigned int id;
+ int type;
+ /* Persistent data structures */
+ atomic_t data_cmnt_size; /* data pages committed to stable storage */
+};
+
+struct nvm_dev {
+ struct nvm_dev_ops *ops;
+
+ struct list_head devices;
+ struct list_head online_targets;
+
+ /* Block manager */
+ struct nvm_bm_type *bm;
+ void *bmp;
+
+ /* Target information */
+ int nr_luns;
+
+ /* Calculated/Cached values. These do not reflect the actual usable
+ * blocks at run-time. */
+ unsigned long total_pages;
+ unsigned long total_blocks;
+ unsigned max_pages_per_blk;
+
+ uint32_t sector_size;
+
+ /* Identity */
+ struct nvm_id identity;
+ struct nvm_get_features features;
+
+ /* Backend device */
+ struct request_queue *q;
+ struct gendisk *disk;
+ char name[DISK_NAME_LEN];
+};
+
+/* Logical to physical mapping */
+struct nvm_addr {
+ sector_t addr;
+ struct nvm_block *block;
+};
+
+/* Physical to logical mapping */
+struct nvm_rev_addr {
+ sector_t addr;
+};
+
+typedef void (nvm_tgt_make_rq_fn)(struct request_queue *, struct bio *);
+typedef sector_t (nvm_tgt_capacity_fn)(void *);
+typedef void (nvm_tgt_end_io_fn)(struct nvm_rq *, int);
+typedef void *(nvm_tgt_init_fn)(struct nvm_dev *, struct gendisk *, int, int);
+typedef void (nvm_tgt_exit_fn)(void *);
+
+struct nvm_tgt_type {
+ const char *name;
+ unsigned int version[3];
+
+ /* target entry points */
+ nvm_tgt_make_rq_fn *make_rq;
+ nvm_tgt_capacity_fn *capacity;
+ nvm_tgt_end_io_fn *end_io;
+
+ /* module-specific init/teardown */
+ nvm_tgt_init_fn *init;
+ nvm_tgt_exit_fn *exit;
+
+ /* For internal use */
+ struct list_head list;
+};
+
+extern int nvm_register_target(struct nvm_tgt_type *);
+extern void nvm_unregister_target(struct nvm_tgt_type *);
+
+typedef int (nvm_bm_register_fn)(struct nvm_dev *);
+typedef void (nvm_bm_unregister_fn)(struct nvm_dev *);
+typedef struct nvm_block *(nvm_bm_get_blk_fn)(struct nvm_dev *,
+ struct nvm_lun *, unsigned long);
+typedef void (nvm_bm_put_blk_fn)(struct nvm_dev *, struct nvm_block *);
+typedef int (nvm_bm_open_blk_fn)(struct nvm_dev *, struct nvm_block *);
+typedef int (nvm_bm_close_blk_fn)(struct nvm_dev *, struct nvm_block *);
+typedef void (nvm_bm_flush_blk_fn)(struct nvm_dev *, struct nvm_block *);
+typedef int (nvm_bm_submit_io_fn)(struct nvm_dev *, struct nvm_rq *);
+typedef void (nvm_bm_end_io_fn)(struct nvm_rq *, int);
+typedef int (nvm_bm_erase_blk_fn)(struct nvm_dev *, struct nvm_block *);
+typedef int (nvm_bm_register_prog_err_fn)(struct nvm_dev *,
+ void (prog_err_fn)(struct nvm_dev *, struct nvm_block *));
+typedef int (nvm_bm_save_state_fn)(struct file *);
+typedef int (nvm_bm_restore_state_fn)(struct file *);
+typedef struct nvm_lun *(nvm_bm_get_luns_fn)(struct nvm_dev *, int, int);
+typedef void (nvm_bm_free_blocks_print_fn)(struct nvm_dev *);
+
+struct nvm_bm_type {
+ const char *name;
+ unsigned int version[3];
+
+ nvm_bm_register_fn *register_bm;
+ nvm_bm_unregister_fn *unregister_bm;
+
+ /* Block administration callbacks */
+ nvm_bm_get_blk_fn *get_blk;
+ nvm_bm_put_blk_fn *put_blk;
+ nvm_bm_open_blk_fn *open_blk;
+ nvm_bm_close_blk_fn *close_blk;
+ nvm_bm_flush_blk_fn *flush_blk;
+
+ nvm_bm_submit_io_fn *submit_io;
+ nvm_bm_end_io_fn *end_io;
+ nvm_bm_erase_blk_fn *erase_blk;
+
+ /* State management for debugging purposes */
+ nvm_bm_save_state_fn *save_state;
+ nvm_bm_restore_state_fn *restore_state;
+
+ /* Configuration management */
+ nvm_bm_get_luns_fn *get_luns;
+
+ /* Statistics */
+ nvm_bm_free_blocks_print_fn *free_blocks_print;
+ struct list_head list;
+};
+
+extern int nvm_register_bm(struct nvm_bm_type *);
+extern void nvm_unregister_bm(struct nvm_bm_type *);
+
+extern struct nvm_block *nvm_get_blk(struct nvm_dev *, struct nvm_lun *,
+ unsigned long);
+extern void nvm_put_blk(struct nvm_dev *, struct nvm_block *);
+extern int nvm_erase_blk(struct nvm_dev *, struct nvm_block *);
+
+extern int nvm_register(struct request_queue *, struct gendisk *,
+ struct nvm_dev_ops *);
+extern void nvm_unregister(struct gendisk *);
+
+extern int nvm_submit_io(struct nvm_dev *, struct nvm_rq *);
+
+extern sector_t nvm_alloc_addr(struct nvm_block *);
+
+#define lun_for_each_block(p, b, i) \
+ for ((i) = 0, b = &(p)->blocks[0]; \
+ (i) < (p)->nr_blocks; (i)++, b = &(p)->blocks[(i)])
+
+#define block_for_each_page(b, p) \
+ for ((p)->addr = block_to_addr((b)), (p)->block = (b); \
+ (p)->addr < block_to_addr((b)) \
+ + (b)->lun->dev->nr_pages_per_blk; \
+ (p)->addr++)
+
+/* We currently assume that we the lightnvm device is accepting data in 512
+ * bytes chunks. This should be set to the smallest command size available for a
+ * given device.
+ */
+#define NVM_SECTOR (512)
+#define EXPOSED_PAGE_SIZE (4096)
+
+#define NR_PHY_IN_LOG (EXPOSED_PAGE_SIZE / NVM_SECTOR)
+
+#define NVM_MSG_PREFIX "nvm"
+#define ADDR_EMPTY (~0ULL)
+
+static inline int block_is_full(struct nvm_block *block)
+{
+ struct nvm_lun *lun = block->lun;
+
+ return block->next_page == lun->nr_pages_per_blk;
+}
+
+static inline sector_t block_to_addr(struct nvm_block *block)
+{
+ struct nvm_lun *lun = block->lun;
+
+ return block->id * lun->nr_pages_per_blk;
+}
+
+static inline unsigned long nvm_get_rq_flags(struct request *rq)
+{
+ return (unsigned long)rq->cmd;
+}
+
+#else /* CONFIG_NVM */
+
+struct nvm_dev_ops;
+struct nvm_dev;
+struct nvm_lun;
+struct nvm_block;
+struct nvm_rq {
+};
+struct nvm_tgt_type;
+struct nvm_tgt_instance;
+
+static inline struct nvm_tgt_type *nvm_find_target_type(const char *c)
+{
+ return NULL;
+}
+static inline int nvm_register_target(struct nvm_tgt_type *tt)
+{
+ return -EINVAL;
+}
+static inline void nvm_unregister_target(struct nvm_tgt_type *tt) {}
+static inline int nvm_register(struct request_queue *q, struct gendisk *disk,
+ struct nvm_dev_ops *ops, void *dev_private)
+{
+ return -EINVAL;
+}
+static inline void nvm_unregister(struct gendisk *disk) {}
+static inline struct nvm_block *nvm_get_blk(struct nvm_dev *dev,
+ struct nvm_lun *lun, unsigned long flags)
+{
+ return NULL;
+}
+static inline void nvm_put_blk(struct nvm_dev *dev, struct nvm_block *blk) {}
+static inline int nvm_erase_blk(struct nvm_dev *dev, struct nvm_block *blk)
+{
+ return -EINVAL;
+}
+static inline sector_t nvm_alloc_addr(struct nvm_block *blk)
+{
+ return 0;
+}
+
+#endif /* CONFIG_NVM */
+#endif /* LIGHTNVM.H */
--
2.1.4
This target implements a simple strategy FTL for Open-Channel SSDs.
It does round-robin selection across channels and luns. It uses a
simple greedy cost-based garbage collector and exposes the physical
flash as a block device.
Signed-off-by: Javier González <[email protected]>
Signed-off-by: Matias Bjørling <[email protected]>
---
drivers/lightnvm/Kconfig | 10 +
drivers/lightnvm/Makefile | 1 +
drivers/lightnvm/rrpc.c | 1190 +++++++++++++++++++++++++++++++++++++++++++++
drivers/lightnvm/rrpc.h | 220 +++++++++
4 files changed, 1421 insertions(+)
create mode 100644 drivers/lightnvm/rrpc.c
create mode 100644 drivers/lightnvm/rrpc.h
diff --git a/drivers/lightnvm/Kconfig b/drivers/lightnvm/Kconfig
index 1f8412c..ab1fe57 100644
--- a/drivers/lightnvm/Kconfig
+++ b/drivers/lightnvm/Kconfig
@@ -14,3 +14,13 @@ menuconfig NVM
If you say N, all options in this submenu will be skipped and disabled
only do this if you know what you are doing.
+if NVM
+
+config NVM_RRPC
+ tristate "Round-robin Hybrid Open-Channel SSD target"
+ ---help---
+ Allows an open-channel SSD to be exposed as a block device to the
+ host. The target is implemented using a linear mapping table and
+ cost-based garbage collection. It is optimized for 4K IO sizes.
+
+endif # NVM
diff --git a/drivers/lightnvm/Makefile b/drivers/lightnvm/Makefile
index 38185e9..b2a39e2 100644
--- a/drivers/lightnvm/Makefile
+++ b/drivers/lightnvm/Makefile
@@ -3,3 +3,4 @@
#
obj-$(CONFIG_NVM) := core.o
+obj-$(CONFIG_NVM_RRPC) += rrpc.o
diff --git a/drivers/lightnvm/rrpc.c b/drivers/lightnvm/rrpc.c
new file mode 100644
index 0000000..ccfc6b0
--- /dev/null
+++ b/drivers/lightnvm/rrpc.c
@@ -0,0 +1,1190 @@
+/*
+ * Copyright (C) 2015 IT University of Copenhagen
+ * Initial release: Matias Bjorling <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License version
+ * 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * Implementation of a Round-robin page-based Hybrid FTL for Open-channel SSDs.
+ */
+
+#include "rrpc.h"
+
+static struct kmem_cache *_gcb_cache, *_rq_cache;
+static DECLARE_RWSEM(_lock);
+
+static int rrpc_submit_io(struct rrpc *rrpc, struct bio *bio,
+ struct nvm_rq *rqd, unsigned long flags);
+
+#define rrpc_for_each_lun(rrpc, rlun, i) \
+ for ((i) = 0, rlun = &(rrpc)->luns[0]; \
+ (i) < (rrpc)->nr_luns; (i)++, rlun = &(rrpc)->luns[(i)])
+
+static void invalidate_block_page(struct nvm_addr *p)
+{
+ struct nvm_block *block = p->block;
+ unsigned int page_offset;
+
+ if (!block)
+ return;
+
+ spin_lock(&block->lock);
+ page_offset = p->addr % block->lun->nr_pages_per_blk;
+ WARN_ON(test_and_set_bit(page_offset, block->invalid_pages));
+ block->nr_invalid_pages++;
+ spin_unlock(&block->lock);
+}
+
+static inline void __nvm_page_invalidate(struct rrpc *rrpc, struct nvm_addr *a)
+{
+ BUG_ON(!spin_is_locked(&rrpc->rev_lock));
+ if (a->addr == ADDR_EMPTY)
+ return;
+
+ invalidate_block_page(a);
+ rrpc->rev_trans_map[a->addr - rrpc->poffset].addr = ADDR_EMPTY;
+}
+
+static void rrpc_invalidate_range(struct rrpc *rrpc, sector_t slba,
+ unsigned len)
+{
+ sector_t i;
+
+ spin_lock(&rrpc->rev_lock);
+ for (i = slba; i < slba + len; i++) {
+ struct nvm_addr *gp = &rrpc->trans_map[i];
+
+ __nvm_page_invalidate(rrpc, gp);
+ gp->block = NULL;
+ }
+ spin_unlock(&rrpc->rev_lock);
+}
+
+static struct nvm_rq *rrpc_inflight_laddr_acquire(struct rrpc *rrpc,
+ sector_t laddr, unsigned int pages)
+{
+ struct nvm_rq *rqd;
+ struct rrpc_inflight_rq *inf;
+
+ rqd = mempool_alloc(rrpc->rq_pool, GFP_ATOMIC);
+ if (!rqd)
+ return ERR_PTR(-ENOMEM);
+
+ inf = rrpc_get_inflight_rq(rqd);
+ if (rrpc_lock_laddr(rrpc, laddr, pages, inf)) {
+ mempool_free(rqd, rrpc->rq_pool);
+ return NULL;
+ }
+
+ return rqd;
+}
+
+static void rrpc_inflight_laddr_release(struct rrpc *rrpc, struct nvm_rq *rqd)
+{
+ struct rrpc_inflight_rq *inf = rrpc_get_inflight_rq(rqd);
+
+ rrpc_unlock_laddr(rrpc, inf->l_start, inf);
+
+ mempool_free(rqd, rrpc->rq_pool);
+}
+
+static void rrpc_discard(struct rrpc *rrpc, struct bio *bio)
+{
+ sector_t slba = bio->bi_iter.bi_sector / NR_PHY_IN_LOG;
+ sector_t len = bio->bi_iter.bi_size / EXPOSED_PAGE_SIZE;
+ struct nvm_rq *rqd;
+
+ do {
+ rqd = rrpc_inflight_laddr_acquire(rrpc, slba, len);
+ schedule();
+ } while (!rqd);
+
+ if (IS_ERR(rqd)) {
+ bio_io_error(bio);
+ return;
+ }
+
+ rrpc_invalidate_range(rrpc, slba, len);
+ rrpc_inflight_laddr_release(rrpc, rqd);
+}
+
+/* requires lun->lock taken */
+static void rrpc_set_lun_cur(struct rrpc_lun *rlun, struct nvm_block *block)
+{
+ BUG_ON(!block);
+
+ if (rlun->cur) {
+ spin_lock(&rlun->cur->lock);
+ WARN_ON(!block_is_full(rlun->cur));
+ spin_unlock(&rlun->cur->lock);
+ }
+ rlun->cur = block;
+}
+
+static struct rrpc_lun *get_next_lun(struct rrpc *rrpc)
+{
+ int next = atomic_inc_return(&rrpc->next_lun);
+
+ return &rrpc->luns[next % rrpc->nr_luns];
+}
+
+static void rrpc_gc_kick(struct rrpc *rrpc)
+{
+ struct rrpc_lun *rlun;
+ unsigned int i;
+
+ for (i = 0; i < rrpc->nr_luns; i++) {
+ rlun = &rrpc->luns[i];
+ queue_work(rrpc->krqd_wq, &rlun->ws_gc);
+ }
+}
+
+/**
+ * rrpc_gc_timer - default gc timer function.
+ * @data: ptr to the 'nvm' structure
+ *
+ * Description:
+ * rrpc configures a timer to kick the GC to force proactive behavior.
+ *
+ **/
+static void rrpc_gc_timer(unsigned long data)
+{
+ struct rrpc *rrpc = (struct rrpc *)data;
+
+ rrpc_gc_kick(rrpc);
+ mod_timer(&rrpc->gc_timer, jiffies + msecs_to_jiffies(10));
+}
+
+static void rrpc_end_sync_bio(struct bio *bio, int error)
+{
+ struct completion *waiting = bio->bi_private;
+
+ if (error)
+ pr_err("nvm: gc request failed (%u).\n", error);
+
+ complete(waiting);
+}
+
+/*
+ * rrpc_move_valid_pages -- migrate live data off the block
+ * @rrpc: the 'rrpc' structure
+ * @block: the block from which to migrate live pages
+ *
+ * Description:
+ * GC algorithms may call this function to migrate remaining live
+ * pages off the block prior to erasing it. This function blocks
+ * further execution until the operation is complete.
+ */
+static int rrpc_move_valid_pages(struct rrpc *rrpc, struct nvm_block *block)
+{
+ struct request_queue *q = rrpc->dev->q;
+ struct nvm_lun *lun = block->lun;
+ struct nvm_rev_addr *rev;
+ struct nvm_rq *rqd;
+ struct bio *bio;
+ struct page *page;
+ int slot;
+ sector_t phys_addr;
+ DECLARE_COMPLETION_ONSTACK(wait);
+
+ if (bitmap_full(block->invalid_pages, lun->nr_pages_per_blk))
+ return 0;
+
+ bio = bio_alloc(GFP_NOIO, 1);
+ if (!bio) {
+ pr_err("nvm: could not alloc bio to gc\n");
+ return -ENOMEM;
+ }
+
+ page = mempool_alloc(rrpc->page_pool, GFP_NOIO);
+
+ while ((slot = find_first_zero_bit(block->invalid_pages,
+ lun->nr_pages_per_blk)) <
+ lun->nr_pages_per_blk) {
+
+ /* Lock laddr */
+ phys_addr = block_to_addr(block) + slot;
+
+try:
+ spin_lock(&rrpc->rev_lock);
+ /* Get logical address from physical to logical table */
+ rev = &rrpc->rev_trans_map[phys_addr - rrpc->poffset];
+ /* already updated by previous regular write */
+ if (rev->addr == ADDR_EMPTY) {
+ spin_unlock(&rrpc->rev_lock);
+ continue;
+ }
+
+ rqd = rrpc_inflight_laddr_acquire(rrpc, rev->addr, 1);
+ if (IS_ERR_OR_NULL(rqd)) {
+ spin_unlock(&rrpc->rev_lock);
+ schedule();
+ goto try;
+ }
+
+ spin_unlock(&rrpc->rev_lock);
+
+ /* Perform read to do GC */
+ bio->bi_iter.bi_sector = nvm_get_sector(rev->addr);
+ bio->bi_rw = READ;
+ bio->bi_private = &wait;
+ bio->bi_end_io = rrpc_end_sync_bio;
+
+ /* TODO: may fail when EXP_PG_SIZE > PAGE_SIZE */
+ bio_add_pc_page(q, bio, page, EXPOSED_PAGE_SIZE, 0);
+
+ if (rrpc_submit_io(rrpc, bio, rqd, NVM_IOTYPE_GC)) {
+ pr_err("rrpc: gc read failed.\n");
+ rrpc_inflight_laddr_release(rrpc, rqd);
+ goto finished;
+ }
+ wait_for_completion_io(&wait);
+
+ bio_reset(bio);
+ reinit_completion(&wait);
+
+ bio->bi_iter.bi_sector = nvm_get_sector(rev->addr);
+ bio->bi_rw = WRITE;
+ bio->bi_private = &wait;
+ bio->bi_end_io = rrpc_end_sync_bio;
+
+ bio_add_pc_page(q, bio, page, EXPOSED_PAGE_SIZE, 0);
+
+ /* turn the command around and write the data back to a new
+ * address */
+ if (rrpc_submit_io(rrpc, bio, rqd, NVM_IOTYPE_GC)) {
+ pr_err("rrpc: gc write failed.\n");
+ rrpc_inflight_laddr_release(rrpc, rqd);
+ goto finished;
+ }
+ wait_for_completion_io(&wait);
+
+ rrpc_inflight_laddr_release(rrpc, rqd);
+
+ bio_reset(bio);
+ }
+
+finished:
+ mempool_free(page, rrpc->page_pool);
+ bio_put(bio);
+
+ if (!bitmap_full(block->invalid_pages, lun->nr_pages_per_blk)) {
+ pr_err("nvm: failed to garbage collect block\n");
+ return -EIO;
+ }
+
+ return 0;
+}
+
+static void rrpc_block_gc(struct work_struct *work)
+{
+ struct rrpc_block_gc *gcb = container_of(work, struct rrpc_block_gc,
+ ws_gc);
+ struct rrpc *rrpc = gcb->rrpc;
+ struct nvm_block *block = gcb->block;
+ struct nvm_dev *dev = rrpc->dev;
+
+ pr_debug("nvm: block '%d' being reclaimed\n", block->id);
+
+ if (rrpc_move_valid_pages(rrpc, block))
+ goto done;
+
+ nvm_erase_blk(dev, block);
+ nvm_put_blk(dev, block);
+done:
+ mempool_free(gcb, rrpc->gcb_pool);
+}
+
+/* the block with highest number of invalid pages, will be in the beginning
+ * of the list */
+static struct rrpc_block *rblock_max_invalid(struct rrpc_block *ra,
+ struct rrpc_block *rb)
+{
+ struct nvm_block *a = ra->parent;
+ struct nvm_block *b = rb->parent;
+
+ BUG_ON(!a || !b);
+
+ if (a->nr_invalid_pages == b->nr_invalid_pages)
+ return ra;
+
+ return (a->nr_invalid_pages < b->nr_invalid_pages) ? rb : ra;
+}
+
+/* linearly find the block with highest number of invalid pages
+ * requires lun->lock */
+static struct rrpc_block *block_prio_find_max(struct rrpc_lun *rlun)
+{
+ struct list_head *prio_list = &rlun->prio_list;
+ struct rrpc_block *rblock, *max;
+
+ BUG_ON(list_empty(prio_list));
+
+ max = list_first_entry(prio_list, struct rrpc_block, prio);
+ list_for_each_entry(rblock, prio_list, prio)
+ max = rblock_max_invalid(max, rblock);
+
+ return max;
+}
+
+static void rrpc_lun_gc(struct work_struct *work)
+{
+ struct rrpc_lun *rlun = container_of(work, struct rrpc_lun, ws_gc);
+ struct rrpc *rrpc = rlun->rrpc;
+ struct nvm_lun *lun = rlun->parent;
+ struct rrpc_block_gc *gcb;
+ unsigned int nr_blocks_need;
+
+ nr_blocks_need = lun->nr_blocks / GC_LIMIT_INVERSE;
+
+ if (nr_blocks_need < rrpc->nr_luns)
+ nr_blocks_need = rrpc->nr_luns;
+
+ spin_lock(&lun->lock);
+ while (nr_blocks_need > lun->nr_free_blocks &&
+ !list_empty(&rlun->prio_list)) {
+ struct rrpc_block *rblock = block_prio_find_max(rlun);
+ struct nvm_block *block = rblock->parent;
+
+ if (!block->nr_invalid_pages)
+ break;
+
+ list_del_init(&rblock->prio);
+
+ BUG_ON(!block_is_full(block));
+
+ pr_debug("rrpc: selected block '%d' for GC\n", block->id);
+
+ gcb = mempool_alloc(rrpc->gcb_pool, GFP_ATOMIC);
+ if (!gcb)
+ break;
+
+ gcb->rrpc = rrpc;
+ gcb->block = rblock->parent;
+ INIT_WORK(&gcb->ws_gc, rrpc_block_gc);
+
+ queue_work(rrpc->kgc_wq, &gcb->ws_gc);
+
+ nr_blocks_need--;
+ }
+ spin_unlock(&lun->lock);
+
+ /* TODO: Hint that request queue can be started again */
+}
+
+static void rrpc_gc_queue(struct work_struct *work)
+{
+ struct rrpc_block_gc *gcb = container_of(work, struct rrpc_block_gc,
+ ws_gc);
+ struct rrpc *rrpc = gcb->rrpc;
+ struct nvm_block *block = gcb->block;
+ struct nvm_lun *lun = block->lun;
+ struct rrpc_lun *rlun = &rrpc->luns[lun->id - rrpc->lun_offset];
+ struct rrpc_block *rblock =
+ &rlun->blocks[block->id % lun->nr_blocks];
+
+ spin_lock(&rlun->lock);
+ list_add_tail(&rblock->prio, &rlun->prio_list);
+ spin_unlock(&rlun->lock);
+
+ mempool_free(gcb, rrpc->gcb_pool);
+ pr_debug("nvm: block '%d' is full, allow GC (sched)\n", block->id);
+}
+
+static int rrpc_ioctl(struct block_device *bdev, fmode_t mode, unsigned int cmd,
+ unsigned long arg)
+{
+ return 0;
+}
+
+static int rrpc_open(struct block_device *bdev, fmode_t mode)
+{
+ return 0;
+}
+
+static void rrpc_release(struct gendisk *disk, fmode_t mode)
+{
+}
+
+static const struct block_device_operations rrpc_fops = {
+ .owner = THIS_MODULE,
+ .ioctl = rrpc_ioctl,
+ .open = rrpc_open,
+ .release = rrpc_release,
+};
+
+static struct rrpc_lun *__rrpc_get_lun_rr(struct rrpc *rrpc, int is_gc)
+{
+ unsigned int i;
+ struct rrpc_lun *rlun, *max_free;
+
+ if (!is_gc)
+ return get_next_lun(rrpc);
+
+ /* during GC, we don't care about RR, instead we want to make
+ * sure that we maintain evenness between the block luns. */
+ max_free = &rrpc->luns[0];
+ /* prevent GC-ing lun from devouring pages of a lun with
+ * little free blocks. We don't take the lock as we only need an
+ * estimate. */
+ rrpc_for_each_lun(rrpc, rlun, i) {
+ if (rlun->parent->nr_free_blocks >
+ max_free->parent->nr_free_blocks)
+ max_free = rlun;
+ }
+
+ return max_free;
+}
+
+static inline void __rrpc_page_invalidate(struct rrpc *rrpc,
+ struct nvm_addr *gp)
+{
+ BUG_ON(!spin_is_locked(&rrpc->rev_lock));
+ if (gp->addr == ADDR_EMPTY)
+ return;
+
+ invalidate_block_page(gp);
+ rrpc->rev_trans_map[gp->addr - rrpc->poffset].addr = ADDR_EMPTY;
+}
+
+struct nvm_addr *nvm_update_map(struct rrpc *rrpc, sector_t l_addr,
+ struct nvm_block *p_block, sector_t p_addr, int is_gc)
+{
+ struct nvm_addr *gp;
+ struct nvm_rev_addr *rev;
+
+ BUG_ON(l_addr >= rrpc->nr_pages);
+
+ gp = &rrpc->trans_map[l_addr];
+ spin_lock(&rrpc->rev_lock);
+ if (gp->block)
+ __nvm_page_invalidate(rrpc, gp);
+
+ gp->addr = p_addr;
+ gp->block = p_block;
+
+ rev = &rrpc->rev_trans_map[gp->addr - rrpc->poffset];
+ rev->addr = l_addr;
+ spin_unlock(&rrpc->rev_lock);
+
+ return gp;
+}
+
+/* Simple round-robin Logical to physical address translation.
+ *
+ * Retrieve the mapping using the active append point. Then update the ap for
+ * the next write to the disk.
+ *
+ * Returns nvm_addr with the physical address and block. Remember to return to
+ * rrpc->addr_cache when request is finished.
+ */
+static struct nvm_addr *rrpc_map_page(struct rrpc *rrpc, sector_t laddr,
+ int is_gc)
+{
+ struct rrpc_lun *rlun;
+ struct nvm_lun *lun;
+ struct nvm_block *p_block;
+ sector_t p_addr;
+
+ rlun = __rrpc_get_lun_rr(rrpc, is_gc);
+ lun = rlun->parent;
+
+ if (!is_gc && lun->nr_free_blocks < rrpc->nr_luns * 4)
+ return NULL;
+
+ spin_lock(&rlun->lock);
+
+ p_block = rlun->cur;
+ p_addr = nvm_alloc_addr(p_block);
+
+ if (p_addr == ADDR_EMPTY) {
+ p_block = nvm_get_blk(rrpc->dev, lun, 0);
+
+ if (!p_block) {
+ if (is_gc) {
+ p_addr = nvm_alloc_addr(rlun->gc_cur);
+ if (p_addr == ADDR_EMPTY) {
+ p_block =
+ nvm_get_blk(rrpc->dev, lun, 1);
+ if (!p_block) {
+ pr_err("rrpc: no more blocks");
+ goto finished;
+ } else {
+ rlun->gc_cur = p_block;
+ p_addr =
+ nvm_alloc_addr(rlun->gc_cur);
+ }
+ }
+ p_block = rlun->gc_cur;
+ }
+ goto finished;
+ }
+
+ rrpc_set_lun_cur(rlun, p_block);
+ p_addr = nvm_alloc_addr(p_block);
+ }
+
+finished:
+ if (p_addr == ADDR_EMPTY)
+ goto err;
+
+ if (!p_block)
+ WARN_ON(is_gc);
+
+ spin_unlock(&rlun->lock);
+ return nvm_update_map(rrpc, laddr, p_block, p_addr, is_gc);
+err:
+ spin_unlock(&rlun->lock);
+ return NULL;
+}
+
+static void rrpc_end_io_write(struct rrpc *rrpc, struct rrpc_rq *rrqd)
+{
+ struct nvm_addr *p = rrqd->addr;
+ struct nvm_block *block = p->block;
+ struct nvm_lun *lun = block->lun;
+ struct rrpc_block_gc *gcb;
+ int cmnt_size;
+
+ cmnt_size = atomic_inc_return(&block->data_cmnt_size);
+ if (likely(cmnt_size != lun->nr_pages_per_blk))
+ return;
+
+ gcb = mempool_alloc(rrpc->gcb_pool, GFP_ATOMIC);
+ if (!gcb) {
+ pr_err("rrpc: unable to queue block for gc.");
+ return;
+ }
+
+ gcb->rrpc = rrpc;
+ gcb->block = block;
+ INIT_WORK(&gcb->ws_gc, rrpc_gc_queue);
+
+ queue_work(rrpc->kgc_wq, &gcb->ws_gc);
+}
+
+static void rrpc_end_io(struct nvm_rq *rqd, int error)
+{
+ struct rrpc *rrpc = container_of(rqd->ins, struct rrpc, instance);
+ struct rrpc_rq *rrqd = nvm_rq_to_pdu(rqd);
+
+ if (bio_data_dir(rqd->bio) == WRITE)
+ rrpc_end_io_write(rrpc, rrqd);
+
+ if (rrqd->flags & NVM_IOTYPE_GC)
+ return;
+
+ rrpc_unlock_rq(rrpc, rqd->bio, rqd);
+ mempool_free(rqd, rrpc->rq_pool);
+}
+
+static int rrpc_read_rq(struct rrpc *rrpc, struct bio *bio, struct nvm_rq *rqd,
+ unsigned long flags)
+{
+ struct rrpc_rq *rrqd = nvm_rq_to_pdu(rqd);
+ int is_gc = flags & NVM_IOTYPE_GC;
+ sector_t l_addr = nvm_get_laddr(bio);
+ struct nvm_addr *gp;
+
+ if (!is_gc && rrpc_lock_rq(rrpc, bio, rqd))
+ return NVM_IO_REQUEUE;
+
+ BUG_ON(!(l_addr >= 0 && l_addr < rrpc->nr_pages));
+ gp = &rrpc->trans_map[l_addr];
+
+ if (gp->block) {
+ rqd->phys_sector = nvm_get_sector(gp->addr);
+ } else {
+ BUG_ON(is_gc);
+ rrpc_unlock_rq(rrpc, bio, rqd);
+ return NVM_IO_DONE;
+ }
+
+ rrqd->addr = gp;
+
+ return NVM_IO_OK;
+}
+
+static int rrpc_write_rq(struct rrpc *rrpc, struct bio *bio,
+ struct nvm_rq *rqd, unsigned long flags)
+{
+ struct rrpc_rq *rrqd = nvm_rq_to_pdu(rqd);
+ struct nvm_addr *p;
+ int is_gc = flags & NVM_IOTYPE_GC;
+ sector_t l_addr = nvm_get_laddr(bio);
+
+ if (!is_gc && rrpc_lock_rq(rrpc, bio, rqd))
+ return NVM_IO_REQUEUE;
+
+ p = rrpc_map_page(rrpc, l_addr, is_gc);
+ if (!p) {
+ BUG_ON(is_gc);
+ rrpc_unlock_rq(rrpc, bio, rqd);
+ rrpc_gc_kick(rrpc);
+ return NVM_IO_REQUEUE;
+ }
+
+ rqd->phys_sector = nvm_get_sector(p->addr);
+ rrqd->addr = p;
+
+ return NVM_IO_OK;
+}
+
+static int rrpc_setup_rq(struct rrpc *rrpc, struct bio *bio,
+ struct nvm_rq *rqd, unsigned long flags)
+{
+ if (bio_rw(bio) == WRITE)
+ return rrpc_write_rq(rrpc, bio, rqd, flags);
+
+ return rrpc_read_rq(rrpc, bio, rqd, flags);
+}
+
+static int rrpc_submit_io(struct rrpc *rrpc, struct bio *bio,
+ struct nvm_rq *rqd, unsigned long flags)
+{
+ int err;
+ struct rrpc_rq *rrq = nvm_rq_to_pdu(rqd);
+
+ err = rrpc_setup_rq(rrpc, bio, rqd, flags);
+ if (err)
+ return err;
+
+ rqd->bio = bio;
+ rqd->ins = &rrpc->instance;
+ rrq->flags = flags;
+
+ err = nvm_submit_io(rrpc->dev, rqd);
+ if (err)
+ return NVM_IO_ERR;
+
+ return NVM_IO_OK;
+}
+
+static void rrpc_make_rq(struct request_queue *q, struct bio *bio)
+{
+ struct rrpc *rrpc = q->queuedata;
+ struct nvm_rq *rqd;
+ int err;
+
+ if (bio->bi_rw & REQ_DISCARD) {
+ rrpc_discard(rrpc, bio);
+ return;
+ }
+
+ rqd = mempool_alloc(rrpc->rq_pool, GFP_KERNEL);
+ if (!rqd) {
+ pr_err_ratelimited("rrpc: not able to queue bio.");
+ bio_io_error(bio);
+ return;
+ }
+
+ err = rrpc_submit_io(rrpc, bio, rqd, NVM_IOTYPE_NONE);
+ switch (err) {
+ case NVM_IO_OK:
+ return;
+ case NVM_IO_ERR:
+ bio_io_error(bio);
+ break;
+ case NVM_IO_DONE:
+ bio_endio(bio, 0);
+ break;
+ case NVM_IO_REQUEUE:
+ spin_lock(&rrpc->bio_lock);
+ bio_list_add(&rrpc->requeue_bios, bio);
+ spin_unlock(&rrpc->bio_lock);
+ queue_work(rrpc->kgc_wq, &rrpc->ws_requeue);
+ break;
+ }
+
+ mempool_free(rqd, rrpc->rq_pool);
+}
+
+static void rrpc_requeue(struct work_struct *work)
+{
+ struct rrpc *rrpc = container_of(work, struct rrpc, ws_requeue);
+ struct bio_list bios;
+ struct bio *bio;
+
+ bio_list_init(&bios);
+
+ spin_lock(&rrpc->bio_lock);
+ bio_list_merge(&bios, &rrpc->requeue_bios);
+ bio_list_init(&rrpc->requeue_bios);
+ spin_unlock(&rrpc->bio_lock);
+
+ while ((bio = bio_list_pop(&bios)))
+ rrpc_make_rq(rrpc->disk->queue, bio);
+}
+
+static void rrpc_gc_free(struct rrpc *rrpc)
+{
+ struct rrpc_lun *rlun;
+ int i;
+
+ if (rrpc->krqd_wq)
+ destroy_workqueue(rrpc->krqd_wq);
+
+ if (rrpc->kgc_wq)
+ destroy_workqueue(rrpc->kgc_wq);
+
+ if (!rrpc->luns)
+ return;
+
+ for (i = 0; i < rrpc->nr_luns; i++) {
+ rlun = &rrpc->luns[i];
+
+ if (!rlun->blocks)
+ break;
+ vfree(rlun->blocks);
+ }
+}
+
+static int rrpc_gc_init(struct rrpc *rrpc)
+{
+ rrpc->krqd_wq = alloc_workqueue("rrpc-lun", WQ_MEM_RECLAIM|WQ_UNBOUND,
+ rrpc->nr_luns);
+ if (!rrpc->krqd_wq)
+ return -ENOMEM;
+
+ rrpc->kgc_wq = alloc_workqueue("rrpc-bg", WQ_MEM_RECLAIM, 1);
+ if (!rrpc->kgc_wq)
+ return -ENOMEM;
+
+ setup_timer(&rrpc->gc_timer, rrpc_gc_timer, (unsigned long)rrpc);
+
+ return 0;
+}
+
+static void rrpc_map_free(struct rrpc *rrpc)
+{
+ vfree(rrpc->rev_trans_map);
+ vfree(rrpc->trans_map);
+}
+
+static int rrpc_l2p_update(u64 slba, u64 nlb, u64 *entries, void *private)
+{
+ struct rrpc *rrpc = (struct rrpc *)private;
+ struct nvm_dev *dev = rrpc->dev;
+ struct nvm_addr *addr = rrpc->trans_map + slba;
+ struct nvm_rev_addr *raddr = rrpc->rev_trans_map;
+ sector_t max_pages = dev->total_pages * (dev->sector_size >> 9);
+ u64 elba = slba + nlb;
+ u64 i;
+
+ if (unlikely(elba > dev->total_pages)) {
+ pr_err("nvm: L2P data from device is out of bounds!\n");
+ return -EINVAL;
+ }
+
+ for (i = 0; i < nlb; i++) {
+ u64 pba = le64_to_cpu(entries[i]);
+ /* LNVM treats address-spaces as silos, LBA and PBA are
+ * equally large and zero-indexed. */
+ if (unlikely(pba >= max_pages && pba != U64_MAX)) {
+ pr_err("nvm: L2P data entry is out of bounds!\n");
+ return -EINVAL;
+ }
+
+ /* Address zero is a special one. The first page on a disk is
+ * protected. As it often holds internal device boot
+ * information. */
+ if (!pba)
+ continue;
+
+ addr[i].addr = pba;
+ raddr[pba].addr = slba + i;
+ }
+
+ return 0;
+}
+
+static int rrpc_map_init(struct rrpc *rrpc)
+{
+ struct nvm_dev *dev = rrpc->dev;
+ sector_t i;
+ int ret;
+
+ rrpc->trans_map = vzalloc(sizeof(struct nvm_addr) * rrpc->nr_pages);
+ if (!rrpc->trans_map)
+ return -ENOMEM;
+
+ rrpc->rev_trans_map = vmalloc(sizeof(struct nvm_rev_addr)
+ * rrpc->nr_pages);
+ if (!rrpc->rev_trans_map)
+ return -ENOMEM;
+
+ for (i = 0; i < rrpc->nr_pages; i++) {
+ struct nvm_addr *p = &rrpc->trans_map[i];
+ struct nvm_rev_addr *r = &rrpc->rev_trans_map[i];
+
+ p->addr = ADDR_EMPTY;
+ r->addr = ADDR_EMPTY;
+ }
+
+ if (!dev->ops->get_l2p_tbl)
+ return 0;
+
+ /* Bring up the mapping table from device */
+ ret = dev->ops->get_l2p_tbl(dev->q, 0, dev->total_pages,
+ rrpc_l2p_update, rrpc);
+ if (ret) {
+ pr_err("nvm: rrpc: could not read L2P table.\n");
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+
+/* Minimum pages needed within a lun */
+#define PAGE_POOL_SIZE 16
+#define ADDR_POOL_SIZE 64
+
+static int rrpc_core_init(struct rrpc *rrpc)
+{
+ int i;
+
+ down_write(&_lock);
+ if (!_gcb_cache) {
+ _gcb_cache = kmem_cache_create("rrpc_gcb",
+ sizeof(struct rrpc_block_gc), 0, 0, NULL);
+ if (!_gcb_cache) {
+ up_write(&_lock);
+ return -ENOMEM;
+ }
+
+ _rq_cache = kmem_cache_create("rrpc_rq",
+ sizeof(struct nvm_rq) + sizeof(struct rrpc_rq),
+ 0, 0, NULL);
+ if (!_rq_cache) {
+ kmem_cache_destroy(_gcb_cache);
+ up_write(&_lock);
+ return -ENOMEM;
+ }
+ }
+ up_write(&_lock);
+
+ rrpc->page_pool = mempool_create_page_pool(PAGE_POOL_SIZE, 0);
+ if (!rrpc->page_pool)
+ return -ENOMEM;
+
+ rrpc->gcb_pool = mempool_create_slab_pool(rrpc->dev->nr_luns,
+ _gcb_cache);
+ if (!rrpc->gcb_pool)
+ return -ENOMEM;
+
+ rrpc->rq_pool = mempool_create_slab_pool(64, _rq_cache);
+ if (!rrpc->rq_pool)
+ return -ENOMEM;
+
+ for (i = 0; i < NVM_INFLIGHT_PARTITIONS; i++) {
+ struct nvm_inflight *map = &rrpc->inflight_map[i];
+
+ spin_lock_init(&map->lock);
+ INIT_LIST_HEAD(&map->reqs);
+ }
+
+ return 0;
+}
+
+static void rrpc_core_free(struct rrpc *rrpc)
+{
+ if (rrpc->page_pool)
+ mempool_destroy(rrpc->page_pool);
+ if (rrpc->gcb_pool)
+ mempool_destroy(rrpc->gcb_pool);
+ if (rrpc->rq_pool)
+ mempool_destroy(rrpc->rq_pool);
+}
+
+static void rrpc_luns_free(struct rrpc *rrpc)
+{
+ kfree(rrpc->luns);
+}
+
+static int rrpc_luns_init(struct rrpc *rrpc, int lun_begin, int lun_end)
+{
+ struct nvm_dev *dev = rrpc->dev;
+ struct nvm_lun *luns;
+ struct nvm_block *block;
+ struct rrpc_lun *rlun;
+ int i, j;
+
+ spin_lock_init(&rrpc->rev_lock);
+
+ luns = dev->bm->get_luns(dev, lun_begin, lun_end);
+ if (!luns)
+ return -EINVAL;
+
+ rrpc->luns = kcalloc(rrpc->nr_luns, sizeof(struct rrpc_lun),
+ GFP_KERNEL);
+ if (!rrpc->luns)
+ return -ENOMEM;
+
+ /* 1:1 mapping */
+ for (i = 0; i < rrpc->nr_luns; i++) {
+ struct nvm_lun *lun = &luns[i];
+
+ rlun = &rrpc->luns[i];
+ rlun->rrpc = rrpc;
+ rlun->parent = lun;
+ rlun->nr_blocks = lun->nr_blocks;
+
+ rrpc->total_blocks += lun->nr_blocks;
+ rrpc->nr_pages += lun->nr_blocks * lun->nr_pages_per_blk;
+
+ INIT_LIST_HEAD(&rlun->prio_list);
+ INIT_WORK(&rlun->ws_gc, rrpc_lun_gc);
+ spin_lock_init(&rlun->lock);
+
+ rlun->blocks = vzalloc(sizeof(struct rrpc_block) *
+ rlun->nr_blocks);
+ if (!rlun->blocks)
+ goto err;
+
+ lun_for_each_block(lun, block, j) {
+ struct rrpc_block *rblock = &rlun->blocks[j];
+
+ rblock->parent = block;
+ INIT_LIST_HEAD(&rblock->prio);
+ }
+ }
+
+ return 0;
+err:
+ return -ENOMEM;
+}
+
+static void rrpc_free(struct rrpc *rrpc)
+{
+ rrpc_gc_free(rrpc);
+ rrpc_map_free(rrpc);
+ rrpc_core_free(rrpc);
+ rrpc_luns_free(rrpc);
+
+ kfree(rrpc);
+}
+
+static void rrpc_exit(void *private)
+{
+ struct rrpc *rrpc = private;
+
+ del_timer(&rrpc->gc_timer);
+
+ flush_workqueue(rrpc->krqd_wq);
+ flush_workqueue(rrpc->kgc_wq);
+
+ rrpc_free(rrpc);
+}
+
+static sector_t rrpc_capacity(void *private)
+{
+ struct rrpc *rrpc = private;
+ struct nvm_dev *dev = rrpc->dev;
+ sector_t reserved;
+
+ /* cur, gc, and two emergency blocks for each lun */
+ reserved = rrpc->nr_luns * dev->max_pages_per_blk * 4;
+
+ if (reserved > rrpc->nr_pages) {
+ pr_err("rrpc: not enough space available to expose storage.\n");
+ return 0;
+ }
+
+ return ((rrpc->nr_pages - reserved) / 10) * 9 * NR_PHY_IN_LOG;
+}
+
+/*
+ * Looks up the logical address from reverse trans map and check if its valid by
+ * comparing the logical to physical address with the physical address.
+ * Returns 0 on free, otherwise 1 if in use
+ */
+static void rrpc_block_map_update(struct rrpc *rrpc, struct nvm_block *block)
+{
+ struct nvm_lun *lun = block->lun;
+ int offset;
+ struct nvm_addr *laddr;
+ sector_t paddr, pladdr;
+
+ for (offset = 0; offset < lun->nr_pages_per_blk; offset++) {
+ paddr = block_to_addr(block) + offset;
+
+ pladdr = rrpc->rev_trans_map[paddr].addr;
+ if (pladdr == ADDR_EMPTY)
+ continue;
+
+ laddr = &rrpc->trans_map[pladdr];
+
+ if (paddr == laddr->addr) {
+ laddr->block = block;
+ } else {
+ set_bit(offset, block->invalid_pages);
+ block->nr_invalid_pages++;
+ }
+ }
+}
+
+static int rrpc_blocks_init(struct rrpc *rrpc)
+{
+ struct nvm_dev *dev = rrpc->dev;
+ struct nvm_lun *lun, *luns;
+ struct nvm_block *blk;
+ sector_t lun_iter, blk_iter;
+
+ luns = dev->bm->get_luns(dev, rrpc->lun_offset, rrpc->lun_offset +
+ rrpc->nr_luns);
+
+ if (!luns)
+ return -EINVAL;
+
+ for (lun_iter = 0; lun_iter < rrpc->nr_luns; lun_iter++) {
+ lun = &luns[lun_iter];
+
+ lun_for_each_block(lun, blk, blk_iter)
+ rrpc_block_map_update(rrpc, blk);
+ }
+
+ return 0;
+}
+
+static int rrpc_luns_configure(struct rrpc *rrpc)
+{
+ struct rrpc_lun *rlun;
+ struct nvm_block *blk;
+ int i;
+
+ for (i = 0; i < rrpc->nr_luns; i++) {
+ rlun = &rrpc->luns[i];
+
+ blk = nvm_get_blk(rrpc->dev, rlun->parent, 0);
+ if (!blk)
+ return -EINVAL;
+
+ rrpc_set_lun_cur(rlun, blk);
+
+ /* Emergency gc block */
+ blk = nvm_get_blk(rrpc->dev, rlun->parent, 1);
+ if (!blk)
+ return -EINVAL;
+ rlun->gc_cur = blk;
+ }
+
+ return 0;
+}
+
+static struct nvm_tgt_type tt_rrpc;
+
+static void *rrpc_init(struct nvm_dev *dev, struct gendisk *tdisk,
+ int lun_begin, int lun_end)
+{
+ struct request_queue *bqueue = dev->q;
+ struct request_queue *tqueue = tdisk->queue;
+ struct rrpc *rrpc;
+ int ret;
+
+ rrpc = kzalloc(sizeof(struct rrpc), GFP_KERNEL);
+ if (!rrpc) {
+ ret = -ENOMEM;
+ goto err;
+ }
+
+ rrpc->instance.tt = &tt_rrpc;
+ rrpc->dev = dev;
+ rrpc->disk = tdisk;
+
+ bio_list_init(&rrpc->requeue_bios);
+ spin_lock_init(&rrpc->bio_lock);
+ INIT_WORK(&rrpc->ws_requeue, rrpc_requeue);
+
+ rrpc->nr_luns = lun_end - lun_begin + 1;
+
+ /* simple round-robin strategy */
+ atomic_set(&rrpc->next_lun, -1);
+
+ ret = rrpc_luns_init(rrpc, lun_begin, lun_end);
+ if (ret) {
+ pr_err("nvm: could not initialize luns\n");
+ goto err;
+ }
+
+ rrpc->poffset = rrpc->luns[0].parent->nr_blocks *
+ rrpc->luns[0].parent->nr_pages_per_blk * lun_begin;
+ rrpc->lun_offset = lun_begin;
+
+ ret = rrpc_core_init(rrpc);
+ if (ret) {
+ pr_err("nvm: rrpc: could not initialize core\n");
+ goto err;
+ }
+
+ ret = rrpc_map_init(rrpc);
+ if (ret) {
+ pr_err("nvm: rrpc: could not initialize maps\n");
+ goto err;
+ }
+
+ ret = rrpc_blocks_init(rrpc);
+ if (ret) {
+ pr_err("nvm: rrpc: could not initialize state for blocks\n");
+ goto err;
+ }
+
+ ret = rrpc_luns_configure(rrpc);
+ if (ret) {
+ pr_err("nvm: rrpc: not enough blocks available in LUNs.\n");
+ goto err;
+ }
+
+ ret = rrpc_gc_init(rrpc);
+ if (ret) {
+ pr_err("nvm: rrpc: could not initialize gc\n");
+ goto err;
+ }
+
+ /* inherit the size from the underlying device */
+ blk_queue_logical_block_size(tqueue, queue_physical_block_size(bqueue));
+ blk_queue_max_hw_sectors(tqueue, queue_max_hw_sectors(bqueue));
+
+ pr_info("nvm: rrpc initialized with %u luns and %llu pages.\n",
+ rrpc->nr_luns, (unsigned long long)rrpc->nr_pages);
+
+ mod_timer(&rrpc->gc_timer, jiffies + msecs_to_jiffies(10));
+
+ return rrpc;
+err:
+ rrpc_free(rrpc);
+ return ERR_PTR(ret);
+}
+
+/* round robin, page-based FTL, and cost-based GC */
+static struct nvm_tgt_type tt_rrpc = {
+ .name = "rrpc",
+
+ .make_rq = rrpc_make_rq,
+ .capacity = rrpc_capacity,
+ .end_io = rrpc_end_io,
+
+ .init = rrpc_init,
+ .exit = rrpc_exit,
+};
+
+static int __init rrpc_module_init(void)
+{
+ return nvm_register_target(&tt_rrpc);
+}
+
+static void rrpc_module_exit(void)
+{
+ nvm_unregister_target(&tt_rrpc);
+}
+
+module_init(rrpc_module_init);
+module_exit(rrpc_module_exit);
+MODULE_LICENSE("GPL v2");
+MODULE_DESCRIPTION("Hybrid Target for Open-Channel SSDs");
diff --git a/drivers/lightnvm/rrpc.h b/drivers/lightnvm/rrpc.h
new file mode 100644
index 0000000..f8a5f19
--- /dev/null
+++ b/drivers/lightnvm/rrpc.h
@@ -0,0 +1,220 @@
+/*
+ * Copyright (C) 2015 IT University of Copenhagen
+ * Initial release: Matias Bjorling <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License version
+ * 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * Implementation of a Round-robin page-based Hybrid FTL for Open-channel SSDs.
+ */
+
+#ifndef RRPC_H_
+#define RRPC_H_
+
+#include <linux/blkdev.h>
+#include <linux/blk-mq.h>
+#include <linux/bio.h>
+#include <linux/module.h>
+#include <linux/kthread.h>
+#include <linux/vmalloc.h>
+
+#include <linux/lightnvm.h>
+
+/* We partition the namespace of translation map into these pieces for tracking
+ * in-flight addresses. */
+#define NVM_INFLIGHT_PARTITIONS 1
+
+/* Run only GC if less than 1/X blocks are free */
+#define GC_LIMIT_INVERSE 10
+#define GC_TIME_SECS 100
+
+struct nvm_inflight {
+ spinlock_t lock;
+ struct list_head reqs;
+};
+
+struct rrpc_inflight_rq {
+ struct list_head list;
+ sector_t l_start;
+ sector_t l_end;
+};
+
+struct rrpc_rq {
+ struct rrpc_inflight_rq inflight_rq;
+ struct nvm_addr *addr;
+ unsigned long flags;
+};
+
+struct rrpc_block {
+ struct nvm_block *parent;
+ struct list_head prio;
+};
+
+struct rrpc_lun {
+ struct rrpc *rrpc;
+ struct nvm_lun *parent;
+ struct nvm_block *cur, *gc_cur;
+ struct rrpc_block *blocks; /* Reference to block allocation */
+ struct list_head prio_list; /* Blocks that may be GC'ed */
+ struct work_struct ws_gc;
+
+ int nr_blocks;
+ spinlock_t lock;
+};
+
+struct rrpc {
+ /* instance must be kept in top to resolve rrpc in unprep */
+ struct nvm_tgt_instance instance;
+
+ struct nvm_dev *dev;
+ struct gendisk *disk;
+
+ struct bio_list requeue_bios;
+ spinlock_t bio_lock;
+ struct work_struct ws_requeue;
+
+ int nr_luns;
+ int lun_offset;
+ sector_t poffset; /* physical page offset */
+
+ struct rrpc_lun *luns;
+
+ /* calculated values */
+ unsigned long nr_pages;
+ unsigned long total_blocks;
+
+ /* Write strategy variables. Move these into each for structure for each
+ * strategy */
+ atomic_t next_lun; /* Whenever a page is written, this is updated
+ * to point to the next write lun */
+
+ /* Simple translation map of logical addresses to physical addresses.
+ * The logical addresses is known by the host system, while the physical
+ * addresses are used when writing to the disk block device. */
+ struct nvm_addr *trans_map;
+ /* also store a reverse map for garbage collection */
+ struct nvm_rev_addr *rev_trans_map;
+ spinlock_t rev_lock;
+
+ struct nvm_inflight inflight_map[NVM_INFLIGHT_PARTITIONS];
+
+ mempool_t *addr_pool;
+ mempool_t *page_pool;
+ mempool_t *gcb_pool;
+ mempool_t *rq_pool;
+
+ struct timer_list gc_timer;
+ struct workqueue_struct *krqd_wq;
+ struct workqueue_struct *kgc_wq;
+};
+
+struct rrpc_block_gc {
+ struct rrpc *rrpc;
+ struct nvm_block *block;
+ struct work_struct ws_gc;
+};
+
+static inline sector_t nvm_get_laddr(struct bio *bio)
+{
+ return bio->bi_iter.bi_sector / NR_PHY_IN_LOG;
+}
+
+static inline unsigned int nvm_get_pages(struct bio *bio)
+{
+ return bio->bi_iter.bi_size / EXPOSED_PAGE_SIZE;
+}
+
+static inline sector_t nvm_get_sector(sector_t laddr)
+{
+ return laddr * NR_PHY_IN_LOG;
+}
+
+static inline int request_intersects(struct rrpc_inflight_rq *r,
+ sector_t laddr_start, sector_t laddr_end)
+{
+ return (laddr_end >= r->l_start && laddr_end <= r->l_end) &&
+ (laddr_start >= r->l_start && laddr_start <= r->l_end);
+}
+
+static int __rrpc_lock_laddr(struct rrpc *rrpc, sector_t laddr,
+ unsigned pages, struct rrpc_inflight_rq *r)
+{
+ struct nvm_inflight *map =
+ &rrpc->inflight_map[laddr % NVM_INFLIGHT_PARTITIONS];
+ sector_t laddr_end = laddr + pages - 1;
+ struct rrpc_inflight_rq *rtmp;
+
+ spin_lock_irq(&map->lock);
+ list_for_each_entry(rtmp, &map->reqs, list) {
+ if (unlikely(request_intersects(rtmp, laddr, laddr_end))) {
+ /* existing, overlapping request, come back later */
+ spin_unlock_irq(&map->lock);
+ return 1;
+ }
+ }
+
+ r->l_start = laddr;
+ r->l_end = laddr_end;
+
+ list_add_tail(&r->list, &map->reqs);
+ spin_unlock_irq(&map->lock);
+ return 0;
+}
+
+static inline int rrpc_lock_laddr(struct rrpc *rrpc, sector_t laddr,
+ unsigned pages,
+ struct rrpc_inflight_rq *r)
+{
+ BUG_ON((laddr + pages) > rrpc->nr_pages);
+
+ return __rrpc_lock_laddr(rrpc, laddr, pages, r);
+}
+
+static inline struct rrpc_inflight_rq *rrpc_get_inflight_rq(struct nvm_rq *rqd)
+{
+ struct rrpc_rq *rrqd = nvm_rq_to_pdu(rqd);
+
+ return &rrqd->inflight_rq;
+}
+
+static inline int rrpc_lock_rq(struct rrpc *rrpc, struct bio *bio,
+ struct nvm_rq *rqd)
+{
+ sector_t laddr = nvm_get_laddr(bio);
+ unsigned int pages = nvm_get_pages(bio);
+ struct rrpc_inflight_rq *r = rrpc_get_inflight_rq(rqd);
+
+ return rrpc_lock_laddr(rrpc, laddr, pages, r);
+}
+
+static inline void rrpc_unlock_laddr(struct rrpc *rrpc, sector_t laddr,
+ struct rrpc_inflight_rq *r)
+{
+ struct nvm_inflight *map =
+ &rrpc->inflight_map[laddr % NVM_INFLIGHT_PARTITIONS];
+ unsigned long flags;
+
+ spin_lock_irqsave(&map->lock, flags);
+ list_del_init(&r->list);
+ spin_unlock_irqrestore(&map->lock, flags);
+}
+
+static inline void rrpc_unlock_rq(struct rrpc *rrpc, struct bio *bio,
+ struct nvm_rq *rqd)
+{
+ sector_t laddr = nvm_get_laddr(bio);
+ unsigned int pages = nvm_get_pages(bio);
+ struct rrpc_inflight_rq *r = rrpc_get_inflight_rq(rqd);
+
+ BUG_ON((laddr + pages) > rrpc->nr_pages);
+
+ rrpc_unlock_laddr(rrpc, laddr, r);
+}
+
+#endif /* RRPC_H_ */
--
2.1.4
The host implementation for Open-Channel SSDs is divided into block
management and targets. This patch implements the block manager for
hybrid open-channel SSDs. On top a target, such as rrpc is initialized.
Signed-off-by: Matias Bjørling <[email protected]>
---
drivers/lightnvm/Kconfig | 7 +
drivers/lightnvm/Makefile | 1 +
drivers/lightnvm/bm_hb.c | 379 ++++++++++++++++++++++++++++++++++++++++++++++
drivers/lightnvm/bm_hb.h | 31 ++++
4 files changed, 418 insertions(+)
create mode 100644 drivers/lightnvm/bm_hb.c
create mode 100644 drivers/lightnvm/bm_hb.h
diff --git a/drivers/lightnvm/Kconfig b/drivers/lightnvm/Kconfig
index ab1fe57..37b00ae 100644
--- a/drivers/lightnvm/Kconfig
+++ b/drivers/lightnvm/Kconfig
@@ -23,4 +23,11 @@ config NVM_RRPC
host. The target is implemented using a linear mapping table and
cost-based garbage collection. It is optimized for 4K IO sizes.
+config NVM_BM_HB
+ tristate "Block manager for Hybrid Open-Channel SSD"
+ ---help---
+ Block manager for SSDs that offload block management off to the device,
+ while keeping data placement and garbage collection decisions on the
+ host.
+
endif # NVM
diff --git a/drivers/lightnvm/Makefile b/drivers/lightnvm/Makefile
index b2a39e2..9ff4669 100644
--- a/drivers/lightnvm/Makefile
+++ b/drivers/lightnvm/Makefile
@@ -4,3 +4,4 @@
obj-$(CONFIG_NVM) := core.o
obj-$(CONFIG_NVM_RRPC) += rrpc.o
+obj-$(CONFIG_NVM_BM_HB) += bm_hb.o
diff --git a/drivers/lightnvm/bm_hb.c b/drivers/lightnvm/bm_hb.c
new file mode 100644
index 0000000..4272b7e
--- /dev/null
+++ b/drivers/lightnvm/bm_hb.c
@@ -0,0 +1,379 @@
+/*
+ * Copyright: Matias Bjorling <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License version
+ * 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * Implementation of a block manager for hybrid open-channel SSD.
+ */
+
+#include "bm_hb.h"
+
+static void hb_blocks_free(struct nvm_dev *dev)
+{
+ struct bm_hb *bm = dev->bmp;
+ struct nvm_lun *lun;
+ int i;
+
+ bm_for_each_lun(dev, bm, lun, i) {
+ if (!lun->blocks)
+ break;
+ vfree(lun->blocks);
+ }
+}
+
+static void hb_luns_free(struct nvm_dev *dev)
+{
+ struct bm_hb *bm = dev->bmp;
+
+ kfree(bm->luns);
+}
+
+static int hb_luns_init(struct nvm_dev *dev, struct bm_hb *bm)
+{
+ struct nvm_lun *lun;
+ struct nvm_id_chnl *chnl;
+ int i;
+
+ bm->luns = kcalloc(dev->nr_luns, sizeof(struct nvm_lun), GFP_KERNEL);
+ if (!bm->luns)
+ return -ENOMEM;
+
+ bm_for_each_lun(dev, bm, lun, i) {
+ chnl = &dev->identity.chnls[i];
+ pr_info("bm_hb: p %u qsize %u gr %u ge %u begin %llu end %llu\n",
+ i, chnl->queue_size, chnl->gran_read, chnl->gran_erase,
+ chnl->laddr_begin, chnl->laddr_end);
+
+ spin_lock_init(&lun->lock);
+
+ INIT_LIST_HEAD(&lun->free_list);
+ INIT_LIST_HEAD(&lun->used_list);
+ INIT_LIST_HEAD(&lun->bb_list);
+
+ lun->id = i;
+ lun->dev = dev;
+ lun->chnl = chnl;
+ lun->reserved_blocks = 2; /* for GC only */
+ lun->nr_blocks =
+ (chnl->laddr_end - chnl->laddr_begin + 1) /
+ (chnl->gran_erase / chnl->gran_read);
+ lun->nr_free_blocks = lun->nr_blocks;
+ lun->nr_pages_per_blk = chnl->gran_erase / chnl->gran_write *
+ (chnl->gran_write / dev->sector_size);
+
+ if (lun->nr_pages_per_blk > dev->max_pages_per_blk)
+ dev->max_pages_per_blk = lun->nr_pages_per_blk;
+
+ dev->total_pages += lun->nr_blocks * lun->nr_pages_per_blk;
+ dev->total_blocks += lun->nr_blocks;
+
+ if (lun->nr_pages_per_blk >
+ MAX_INVALID_PAGES_STORAGE * BITS_PER_LONG) {
+ pr_err("bm_hb: number of pages per block too high.");
+ return -EINVAL;
+ }
+ }
+
+ return 0;
+}
+
+static int hb_block_bb(u32 lun_id, void *bb_bitmap, unsigned int nr_blocks,
+ void *private)
+{
+ struct bm_hb *bm = private;
+ struct nvm_lun *lun = &bm->luns[lun_id];
+ struct nvm_block *block;
+ int i;
+
+ if (unlikely(bitmap_empty(bb_bitmap, nr_blocks)))
+ return 0;
+
+ i = -1;
+ while ((i = find_next_bit(bb_bitmap, nr_blocks, i + 1)) <
+ nr_blocks) {
+ block = &lun->blocks[i];
+ if (!block) {
+ pr_err("bm_hb: BB data is out of bounds!\n");
+ return -EINVAL;
+ }
+ list_move_tail(&block->list, &lun->bb_list);
+ }
+
+ return 0;
+}
+
+static int hb_block_map(u64 slba, u64 nlb, u64 *entries, void *private)
+{
+ struct nvm_dev *dev = private;
+ struct bm_hb *bm = dev->bmp;
+ sector_t max_pages = dev->total_pages * (dev->sector_size >> 9);
+ u64 elba = slba + nlb;
+ struct nvm_lun *lun;
+ struct nvm_block *blk;
+ sector_t total_pgs_per_lun = /* each lun have the same configuration */
+ bm->luns[0].nr_blocks * bm->luns[0].nr_pages_per_blk;
+ u64 i;
+ int lun_id;
+
+ if (unlikely(elba > dev->total_pages)) {
+ pr_err("bm_hb: L2P data from device is out of bounds!\n");
+ return -EINVAL;
+ }
+
+ for (i = 0; i < nlb; i++) {
+ u64 pba = le64_to_cpu(entries[i]);
+
+ if (unlikely(pba >= max_pages && pba != U64_MAX)) {
+ pr_err("bm_hb: L2P data entry is out of bounds!\n");
+ return -EINVAL;
+ }
+
+ /* Address zero is a special one. The first page on a disk is
+ * protected. As it often holds internal device boot
+ * information. */
+ if (!pba)
+ continue;
+
+ /* resolve block from physical address */
+ lun_id = pba / total_pgs_per_lun;
+ lun = &bm->luns[lun_id];
+
+ /* Calculate block offset into lun */
+ pba = pba - (total_pgs_per_lun * lun_id);
+ blk = &lun->blocks[pba / lun->nr_pages_per_blk];
+
+ if (!blk->type) {
+ /* at this point, we don't know anything about the
+ * block. It's up to the FTL on top to re-etablish the
+ * block state */
+ list_move_tail(&blk->list, &lun->used_list);
+ blk->type = 1;
+ lun->nr_free_blocks--;
+ }
+ }
+
+ return 0;
+}
+
+static int hb_blocks_init(struct nvm_dev *dev, struct bm_hb *bm)
+{
+ struct nvm_lun *lun;
+ struct nvm_block *block;
+ sector_t lun_iter, block_iter, cur_block_id = 0;
+ int ret;
+
+ bm_for_each_lun(dev, bm, lun, lun_iter) {
+ lun->blocks = vzalloc(sizeof(struct nvm_block) *
+ lun->nr_blocks);
+ if (!lun->blocks)
+ return -ENOMEM;
+
+ lun_for_each_block(lun, block, block_iter) {
+ spin_lock_init(&block->lock);
+ INIT_LIST_HEAD(&block->list);
+
+ block->lun = lun;
+ block->id = cur_block_id++;
+
+ /* First block is reserved for device */
+ if (unlikely(lun_iter == 0 && block_iter == 0))
+ continue;
+
+ list_add_tail(&block->list, &lun->free_list);
+ }
+
+ if (dev->ops->get_bb_tbl) {
+ ret = dev->ops->get_bb_tbl(dev->q, lun->id,
+ lun->nr_blocks, hb_block_bb, bm);
+ if (ret)
+ pr_err("bm_hb: could not read BB table\n");
+ }
+ }
+
+ if (dev->ops->get_l2p_tbl) {
+ ret = dev->ops->get_l2p_tbl(dev->q, 0, dev->total_pages,
+ hb_block_map, dev);
+ if (ret) {
+ pr_err("bm_hb: could not read L2P table.\n");
+ pr_warn("bm_hb: default block initialization");
+ }
+ }
+
+ return 0;
+}
+
+static int hb_register(struct nvm_dev *dev)
+{
+ struct bm_hb *bm;
+ int ret;
+
+ if (!dev->features.rsp & NVM_RSP_L2P)
+ return 0;
+
+ bm = kzalloc(sizeof(struct bm_hb), GFP_KERNEL);
+ if (!bm)
+ return -ENOMEM;
+
+ dev->bmp = bm;
+
+ ret = hb_luns_init(dev, bm);
+ if (ret) {
+ pr_err("bm_hb: could not initialize luns\n");
+ goto err;
+ }
+
+ ret = hb_blocks_init(dev, bm);
+ if (ret) {
+ pr_err("bm_hb: could not initialize blocks\n");
+ goto err;
+ }
+
+ return 1;
+err:
+ kfree(bm);
+ return ret;
+}
+
+static void hb_unregister(struct nvm_dev *dev)
+{
+ hb_blocks_free(dev);
+ hb_luns_free(dev);
+ kfree(dev->bmp);
+ dev->bmp = NULL;
+}
+
+static void nvm_reset_block(struct nvm_lun *lun, struct nvm_block *block)
+{
+ spin_lock(&block->lock);
+ bitmap_zero(block->invalid_pages, lun->nr_pages_per_blk);
+ block->next_page = 0;
+ block->nr_invalid_pages = 0;
+ atomic_set(&block->data_cmnt_size, 0);
+ spin_unlock(&block->lock);
+}
+
+static struct nvm_block *hb_get_blk(struct nvm_dev *dev, struct nvm_lun *lun,
+ unsigned long flags)
+{
+ struct nvm_block *block = NULL;
+ int is_gc = flags & NVM_IOTYPE_GC;
+
+ BUG_ON(!lun);
+
+ spin_lock(&lun->lock);
+
+ if (list_empty(&lun->free_list)) {
+ pr_err_ratelimited("bm_hb: lun %u have no free pages available",
+ lun->id);
+ spin_unlock(&lun->lock);
+ goto out;
+ }
+
+ while (!is_gc && lun->nr_free_blocks < lun->reserved_blocks) {
+ spin_unlock(&lun->lock);
+ goto out;
+ }
+
+ block = list_first_entry(&lun->free_list, struct nvm_block, list);
+ list_move_tail(&block->list, &lun->used_list);
+
+ lun->nr_free_blocks--;
+
+ spin_unlock(&lun->lock);
+
+ nvm_reset_block(lun, block);
+
+out:
+ return block;
+}
+
+static void hb_put_blk(struct nvm_dev *dev, struct nvm_block *blk)
+{
+ struct nvm_lun *lun = blk->lun;
+
+ spin_lock(&lun->lock);
+
+ list_move_tail(&blk->list, &lun->free_list);
+ lun->nr_free_blocks++;
+
+ spin_unlock(&lun->lock);
+}
+
+static int hb_submit_io(struct nvm_dev *dev, struct nvm_rq *rqd)
+{
+ if (!dev->ops->submit_io)
+ return 0;
+
+ return dev->ops->submit_io(dev->q, rqd);
+}
+
+static void hb_end_io(struct nvm_rq *rqd, int error)
+{
+ struct nvm_tgt_instance *ins = rqd->ins;
+
+ ins->tt->end_io(rqd, error);
+}
+
+static int hb_erase_blk(struct nvm_dev *dev, struct nvm_block *blk)
+{
+ if (!dev->ops->erase_block)
+ return 0;
+
+ return dev->ops->erase_block(dev->q, blk->id);
+}
+
+static struct nvm_lun *hb_get_luns(struct nvm_dev *dev, int begin, int end)
+{
+ struct bm_hb *bm = dev->bmp;
+
+ return bm->luns + begin;
+}
+
+static void hb_free_blocks_print(struct nvm_dev *dev)
+{
+ struct bm_hb *bm = dev->bmp;
+ struct nvm_lun *lun;
+ unsigned int i;
+
+ bm_for_each_lun(dev, bm, lun, i)
+ pr_info("%s: lun%8u\t%u\n", dev->name, i, lun->nr_free_blocks);
+}
+
+static struct nvm_bm_type bm_hb = {
+ .name = "hb",
+
+ .register_bm = hb_register,
+ .unregister_bm = hb_unregister,
+
+ .get_blk = hb_get_blk,
+ .put_blk = hb_put_blk,
+
+ .submit_io = hb_submit_io,
+ .end_io = hb_end_io,
+ .erase_blk = hb_erase_blk,
+
+ .get_luns = hb_get_luns,
+ .free_blocks_print = hb_free_blocks_print,
+};
+
+static int __init hb_module_init(void)
+{
+ return nvm_register_bm(&bm_hb);
+}
+
+static void hb_module_exit(void)
+{
+ nvm_unregister_bm(&bm_hb);
+}
+
+module_init(hb_module_init);
+module_exit(hb_module_exit);
+MODULE_LICENSE("GPL v2");
+MODULE_DESCRIPTION("Block manager for Hybrid Open-Channel SSDs");
diff --git a/drivers/lightnvm/bm_hb.h b/drivers/lightnvm/bm_hb.h
new file mode 100644
index 0000000..13171af
--- /dev/null
+++ b/drivers/lightnvm/bm_hb.h
@@ -0,0 +1,31 @@
+/*
+ * Copyright: Matias Bjorling <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License version
+ * 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ */
+
+#ifndef BM_HB_H_
+#define BM_HB_H_
+
+#include <linux/module.h>
+#include <linux/vmalloc.h>
+
+#include <linux/lightnvm.h>
+
+struct bm_hb {
+ struct nvm_lun *luns;
+};
+
+#define bm_for_each_lun(dev, bm, lun, i) \
+ for ((i) = 0, lun = &(bm)->luns[0]; \
+ (i) < (dev)->nr_luns; (i)++, lun = &(bm)->luns[(i)])
+
+#endif /* BM_HB_H_ */
--
2.1.4
Initial support for LightNVM. The support can be used to benchmark
performance of targets and core implementation.
Signed-off-by: Matias Bjørling <[email protected]>
---
Documentation/block/null_blk.txt | 8 +++
drivers/block/null_blk.c | 138 +++++++++++++++++++++++++++++++++++++--
2 files changed, 140 insertions(+), 6 deletions(-)
diff --git a/Documentation/block/null_blk.txt b/Documentation/block/null_blk.txt
index 2f6c6ff..a34f50a 100644
--- a/Documentation/block/null_blk.txt
+++ b/Documentation/block/null_blk.txt
@@ -70,3 +70,11 @@ use_per_node_hctx=[0/1]: Default: 0
parameter.
1: The multi-queue block layer is instantiated with a hardware dispatch
queue for each CPU node in the system.
+
+IV: LightNVM specific parameters
+
+nvm_enable=[x]: Default: 0
+ Enable LightNVM for null block devices. Requires blk-mq to be used.
+
+nvm_num_channels=[x]: Default: 1
+ Number of LightNVM channels that is exposed to the LightNVM driver.
diff --git a/drivers/block/null_blk.c b/drivers/block/null_blk.c
index 69de41a..6531250 100644
--- a/drivers/block/null_blk.c
+++ b/drivers/block/null_blk.c
@@ -8,6 +8,7 @@
#include <linux/slab.h>
#include <linux/blk-mq.h>
#include <linux/hrtimer.h>
+#include <linux/lightnvm.h>
struct nullb_cmd {
struct list_head list;
@@ -147,6 +148,14 @@ static bool use_per_node_hctx = false;
module_param(use_per_node_hctx, bool, S_IRUGO);
MODULE_PARM_DESC(use_per_node_hctx, "Use per-node allocation for hardware context queues. Default: false");
+static bool nvm_enable;
+module_param(nvm_enable, bool, S_IRUGO);
+MODULE_PARM_DESC(nvm_enable, "Enable Open-channel SSD. Default: false");
+
+static int nvm_num_channels = 1;
+module_param(nvm_num_channels, int, S_IRUGO);
+MODULE_PARM_DESC(nvm_num_channels, "Number of channels to be exposed from the Open-Channel SSD. Default: 1");
+
static void put_tag(struct nullb_queue *nq, unsigned int tag)
{
clear_bit_unlock(tag, nq->tag_map);
@@ -363,6 +372,110 @@ static void null_request_fn(struct request_queue *q)
}
}
+#ifdef CONFIG_NVM
+static int null_nvm_id(struct request_queue *q, struct nvm_id *id)
+{
+ sector_t size = gb * 1024 * 1024 * 1024ULL;
+ unsigned long per_chnl_size =
+ size / bs / nvm_num_channels;
+ struct nvm_id_chnl *chnl;
+ int i;
+
+ id->ver_id = 0x1;
+ id->nvm_type = NVM_NVMT_BLK;
+ id->nchannels = nvm_num_channels;
+
+ id->chnls = kmalloc_array(id->nchannels, sizeof(struct nvm_id_chnl),
+ GFP_KERNEL);
+ if (!id->chnls)
+ return -ENOMEM;
+
+ for (i = 0; i < id->nchannels; i++) {
+ chnl = &id->chnls[i];
+ chnl->queue_size = hw_queue_depth;
+ chnl->gran_read = bs;
+ chnl->gran_write = bs;
+ chnl->gran_erase = bs * 256;
+ chnl->oob_size = 0;
+ chnl->t_r = chnl->t_sqr = 25000; /* 25us */
+ chnl->t_w = chnl->t_sqw = 500000; /* 500us */
+ chnl->t_e = 1500000; /* 1.500us */
+ chnl->io_sched = NVM_IOSCHED_CHANNEL;
+ chnl->laddr_begin = per_chnl_size * i;
+ chnl->laddr_end = per_chnl_size * (i + 1) - 1;
+ }
+
+ return 0;
+}
+
+static int null_nvm_get_features(struct request_queue *q,
+ struct nvm_get_features *gf)
+{
+ gf->rsp = NVM_RSP_L2P;
+ gf->ext = 0;
+
+ return 0;
+}
+
+static void null_nvm_end_io(struct request *rq, int error)
+{
+ struct nvm_rq *rqd = rq->end_io_data;
+ struct nvm_tgt_instance *ins = rqd->ins;
+
+ ins->tt->end_io(rq->end_io_data, error);
+
+ blk_put_request(rq);
+}
+
+static int null_nvm_submit_io(struct request_queue *q, struct nvm_rq *rqd)
+{
+ struct request *rq;
+ struct bio *bio = rqd->bio;
+
+ rq = blk_mq_alloc_request(q, bio_rw(bio), GFP_KERNEL, 0);
+ if (IS_ERR(rq))
+ return -ENOMEM;
+
+ rq->cmd_type = REQ_TYPE_DRV_PRIV;
+ rq->__sector = bio->bi_iter.bi_sector;
+ rq->ioprio = bio_prio(bio);
+
+ if (bio_has_data(bio))
+ rq->nr_phys_segments = bio_phys_segments(q, bio);
+
+ rq->__data_len = bio->bi_iter.bi_size;
+ rq->bio = rq->biotail = bio;
+
+ rq->end_io_data = rqd;
+
+ blk_execute_rq_nowait(q, NULL, rq, 0, null_nvm_end_io);
+
+ return 0;
+}
+
+static struct nvm_dev_ops null_nvm_dev_ops = {
+ .identify = null_nvm_id,
+ .get_features = null_nvm_get_features,
+ .submit_io = null_nvm_submit_io,
+};
+
+static int null_nvm_register(struct nullb *nullb, struct gendisk *disk)
+{
+ return nvm_register(nullb->q, disk, &null_nvm_dev_ops);
+}
+
+static void null_nvm_unregister(struct gendisk *disk)
+{
+ nvm_unregister(disk);
+}
+#else
+static int null_nvm_register(struct nullb *nullb, struct gendisk *disk)
+{
+ return -EINVAL;
+}
+static void null_nvm_unregister(struct gendisk *disk) { }
+#endif /* CONFIG_NVM */
+
static int null_queue_rq(struct blk_mq_hw_ctx *hctx,
const struct blk_mq_queue_data *bd)
{
@@ -410,6 +523,9 @@ static void null_del_dev(struct nullb *nullb)
{
list_del_init(&nullb->list);
+ if (nvm_enable)
+ null_nvm_unregister(nullb->disk);
+
del_gendisk(nullb->disk);
blk_cleanup_queue(nullb->q);
if (queue_mode == NULL_Q_MQ)
@@ -579,11 +695,6 @@ static int null_add_dev(void)
goto out_cleanup_blk_queue;
}
- mutex_lock(&lock);
- list_add_tail(&nullb->list, &nullb_list);
- nullb->index = nullb_indexes++;
- mutex_unlock(&lock);
-
blk_queue_logical_block_size(nullb->q, bs);
blk_queue_physical_block_size(nullb->q, bs);
@@ -598,9 +709,24 @@ static int null_add_dev(void)
disk->private_data = nullb;
disk->queue = nullb->q;
sprintf(disk->disk_name, "nullb%d", nullb->index);
- add_disk(disk);
+
+ if (nvm_enable) {
+ rv = null_nvm_register(nullb, disk);
+ if (rv)
+ goto out_cleanup_disk;
+ } else {
+ add_disk(disk);
+ }
+
+ mutex_lock(&lock);
+ list_add_tail(&nullb->list, &nullb_list);
+ nullb->index = nullb_indexes++;
+ mutex_unlock(&lock);
+
return 0;
+out_cleanup_disk:
+ put_disk(disk);
out_cleanup_blk_queue:
blk_cleanup_queue(nullb->q);
out_cleanup_tags:
--
2.1.4
The first generation of Open-Channel SSDs will be based on NVMe. The
integration requires that a NVMe device exposes itself as a LightNVM
device. The way this is done currently is by hooking into the
Controller Capabilities (CAP register) and a bit in NSFEAT for each
namespace.
After detection, vendor specific codes are used to identify the device
and enumerate supported features.
Signed-off-by: Javier González <[email protected]>
Signed-off-by: Matias Bjørling <[email protected]>
---
drivers/block/Makefile | 2 +-
drivers/block/nvme-core.c | 22 +-
drivers/block/nvme-lightnvm.c | 504 ++++++++++++++++++++++++++++++++++++++++++
include/linux/nvme.h | 6 +
include/uapi/linux/nvme.h | 3 +
5 files changed, 533 insertions(+), 4 deletions(-)
create mode 100644 drivers/block/nvme-lightnvm.c
diff --git a/drivers/block/Makefile b/drivers/block/Makefile
index 02b688d..a01d7d8 100644
--- a/drivers/block/Makefile
+++ b/drivers/block/Makefile
@@ -44,6 +44,6 @@ obj-$(CONFIG_BLK_DEV_RSXX) += rsxx/
obj-$(CONFIG_BLK_DEV_NULL_BLK) += null_blk.o
obj-$(CONFIG_ZRAM) += zram/
-nvme-y := nvme-core.o nvme-scsi.o
+nvme-y := nvme-core.o nvme-scsi.o nvme-lightnvm.o
skd-y := skd_main.o
swim_mod-y := swim.o swim_asm.o
diff --git a/drivers/block/nvme-core.c b/drivers/block/nvme-core.c
index 666e994..ab799b0 100644
--- a/drivers/block/nvme-core.c
+++ b/drivers/block/nvme-core.c
@@ -40,6 +40,7 @@
#include <linux/slab.h>
#include <linux/t10-pi.h>
#include <linux/types.h>
+#include <linux/lightnvm.h>
#include <scsi/sg.h>
#include <asm-generic/io-64-nonatomic-lo-hi.h>
@@ -1751,7 +1752,8 @@ static int nvme_configure_admin_queue(struct nvme_dev *dev)
dev->page_size = 1 << page_shift;
- dev->ctrl_config = NVME_CC_CSS_NVM;
+ dev->ctrl_config = NVME_CAP_LIGHTNVM(cap) ?
+ NVME_CC_CSS_LIGHTNVM : NVME_CC_CSS_NVM;
dev->ctrl_config |= (page_shift - 12) << NVME_CC_MPS_SHIFT;
dev->ctrl_config |= NVME_CC_ARB_RR | NVME_CC_SHN_NONE;
dev->ctrl_config |= NVME_CC_IOSQES | NVME_CC_IOCQES;
@@ -1997,6 +1999,16 @@ static int nvme_revalidate_disk(struct gendisk *disk)
return -ENODEV;
}
+ if ((dev->ctrl_config & NVME_CC_CSS_LIGHTNVM) &&
+ id->nsfeat & NVME_NS_FEAT_NVM && ns->type != NVME_NS_NVM) {
+ if (nvme_nvm_register(ns->queue, disk)) {
+ dev_warn(dev->dev,
+ "%s: LightNVM init failure\n", __func__);
+ return -ENODEV;
+ }
+ ns->type = NVME_NS_NVM;
+ }
+
old_ms = ns->ms;
lbaf = id->flbas & NVME_NS_FLBAS_LBA_MASK;
ns->lba_shift = id->lbaf[lbaf].ds;
@@ -2028,7 +2040,7 @@ static int nvme_revalidate_disk(struct gendisk *disk)
!ns->ext)
nvme_init_integrity(ns);
- if (ns->ms && !blk_get_integrity(disk))
+ if ((ns->ms && !blk_get_integrity(disk)) || ns->type == NVME_NS_NVM)
set_capacity(disk, 0);
else
set_capacity(disk, le64_to_cpup(&id->nsze) << (ns->lba_shift - 9));
@@ -2146,7 +2158,8 @@ static void nvme_alloc_ns(struct nvme_dev *dev, unsigned nsid)
if (nvme_revalidate_disk(ns->disk))
goto out_free_disk;
- add_disk(ns->disk);
+ if (ns->type != NVME_NS_NVM)
+ add_disk(ns->disk);
if (ns->ms) {
struct block_device *bd = bdget_disk(ns->disk, 0);
if (!bd)
@@ -2345,6 +2358,9 @@ static void nvme_free_namespace(struct nvme_ns *ns)
{
list_del(&ns->list);
+ if (ns->type == NVME_NS_NVM)
+ nvm_unregister(ns->disk);
+
spin_lock(&dev_list_lock);
ns->disk->private_data = NULL;
spin_unlock(&dev_list_lock);
diff --git a/drivers/block/nvme-lightnvm.c b/drivers/block/nvme-lightnvm.c
new file mode 100644
index 0000000..397a66e
--- /dev/null
+++ b/drivers/block/nvme-lightnvm.c
@@ -0,0 +1,504 @@
+/*
+ * nvme-lightnvm.c - LightNVM NVMe device
+ *
+ * Copyright (C) 2014-2015 IT University of Copenhagen
+ * Initial release: Matias Bjorling <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License version
+ * 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; see the file COPYING. If not, write to
+ * the Free Software Foundation, 675 Mass Ave, Cambridge, MA 02139,
+ * USA.
+ *
+ */
+
+#include <linux/nvme.h>
+#include <linux/bitops.h>
+#include <linux/lightnvm.h>
+
+#ifdef CONFIG_NVM
+
+enum nvme_nvm_opcode {
+ nvme_nvm_cmd_hb_write = 0x81,
+ nvme_nvm_cmd_hb_read = 0x02,
+ nvme_nvm_cmd_phys_write = 0x91,
+ nvme_nvm_cmd_phys_read = 0x92,
+ nvme_nvm_cmd_erase = 0x90,
+};
+
+enum nvme_nvm_admin_opcode {
+ nvme_nvm_admin_identify = 0xe2,
+ nvme_nvm_admin_get_features = 0xe6,
+ nvme_nvm_admin_set_resp = 0xe5,
+ nvme_nvm_admin_get_l2p_tbl = 0xea,
+ nvme_nvm_admin_get_bb_tbl = 0xf2,
+ nvme_nvm_admin_set_bb_tbl = 0xf1,
+};
+
+struct nvme_nvm_hb_rw {
+ __u8 opcode;
+ __u8 flags;
+ __u16 command_id;
+ __le32 nsid;
+ __u64 rsvd2;
+ __le64 metadata;
+ __le64 prp1;
+ __le64 prp2;
+ __le64 slba;
+ __le16 length;
+ __le16 control;
+ __le32 dsmgmt;
+ __le64 phys_addr;
+};
+
+struct nvme_nvm_identify {
+ __u8 opcode;
+ __u8 flags;
+ __u16 command_id;
+ __le32 nsid;
+ __u64 rsvd[2];
+ __le64 prp1;
+ __le64 prp2;
+ __le32 chnl_off;
+ __u32 rsvd11[5];
+};
+
+struct nvme_nvm_l2ptbl {
+ __u8 opcode;
+ __u8 flags;
+ __u16 command_id;
+ __le32 nsid;
+ __le32 cdw2[4];
+ __le64 prp1;
+ __le64 prp2;
+ __le64 slba;
+ __le32 nlb;
+ __le16 cdw14[6];
+};
+
+struct nvme_nvm_bbtbl {
+ __u8 opcode;
+ __u8 flags;
+ __u16 command_id;
+ __le32 nsid;
+ __u64 rsvd[2];
+ __le64 prp1;
+ __le64 prp2;
+ __le32 prp1_len;
+ __le32 prp2_len;
+ __le32 lbb;
+ __u32 rsvd11[3];
+};
+
+struct nvme_nvm_set_resp {
+ __u8 opcode;
+ __u8 flags;
+ __u16 command_id;
+ __le32 nsid;
+ __u64 rsvd[2];
+ __le64 prp1;
+ __le64 prp2;
+ __le64 resp;
+ __u32 rsvd11[4];
+};
+
+struct nvme_nvm_erase_blk {
+ __u8 opcode;
+ __u8 flags;
+ __u16 command_id;
+ __le32 nsid;
+ __u64 rsvd[2];
+ __le64 prp1;
+ __le64 prp2;
+ __le64 blk_addr;
+ __u32 rsvd11[4];
+};
+
+struct nvme_nvm_command {
+ union {
+ struct nvme_common_command common;
+ struct nvme_nvm_identify nvm_identify;
+ struct nvme_nvm_hb_rw nvm_hb_rw;
+ struct nvme_nvm_l2ptbl nvm_l2p;
+ struct nvme_nvm_bbtbl nvm_get_bb;
+ struct nvme_nvm_bbtbl nvm_set_bb;
+ struct nvme_nvm_set_resp nvm_resp;
+ struct nvme_nvm_erase_blk nvm_erase;
+ };
+};
+
+/*
+ * Check we didin't inadvertently grow the command struct
+ */
+static inline void _nvme_nvm_check_size(void)
+{
+ BUILD_BUG_ON(sizeof(struct nvme_nvm_identify) != 64);
+ BUILD_BUG_ON(sizeof(struct nvme_nvm_hb_rw) != 64);
+ BUILD_BUG_ON(sizeof(struct nvme_nvm_l2ptbl) != 64);
+ BUILD_BUG_ON(sizeof(struct nvme_nvm_bbtbl) != 64);
+ BUILD_BUG_ON(sizeof(struct nvme_nvm_set_resp) != 64);
+ BUILD_BUG_ON(sizeof(struct nvme_nvm_erase_blk) != 64);
+}
+
+struct nvme_nvm_id_chnl {
+ __le64 laddr_begin;
+ __le64 laddr_end;
+ __le32 oob_size;
+ __le32 queue_size;
+ __le32 gran_read;
+ __le32 gran_write;
+ __le32 gran_erase;
+ __le32 t_r;
+ __le32 t_sqr;
+ __le32 t_w;
+ __le32 t_sqw;
+ __le32 t_e;
+ __le16 chnl_parallelism;
+ __u8 io_sched;
+ __u8 reserved[133];
+} __packed;
+
+struct nvme_nvm_id {
+ __u8 ver_id;
+ __u8 nvm_type;
+ __le16 nchannels;
+ __u8 reserved[252];
+ struct nvme_nvm_id_chnl chnls[];
+} __packed;
+
+#define NVME_NVM_CHNLS_PR_REQ ((4096U - sizeof(struct nvme_nvm_id)) \
+ / sizeof(struct nvme_nvm_id_chnl))
+
+
+static int init_chnls(struct request_queue *q, struct nvm_id *nvm_id,
+ struct nvme_nvm_id *nvme_nvm_id)
+{
+ struct nvme_nvm_id_chnl *src = nvme_nvm_id->chnls;
+ struct nvm_id_chnl *dst = nvm_id->chnls;
+ struct nvme_ns *ns = q->queuedata;
+ struct nvme_nvm_command c = {
+ .nvm_identify.opcode = nvme_nvm_admin_identify,
+ .nvm_identify.nsid = cpu_to_le32(ns->ns_id),
+ };
+ unsigned int len = nvm_id->nchannels;
+ int i, end, ret, off = 0;
+
+ while (len) {
+ end = min_t(u32, NVME_NVM_CHNLS_PR_REQ, len);
+
+ for (i = 0; i < end; i++, dst++, src++) {
+ dst->laddr_begin = le64_to_cpu(src->laddr_begin);
+ dst->laddr_end = le64_to_cpu(src->laddr_end);
+ dst->oob_size = le32_to_cpu(src->oob_size);
+ dst->queue_size = le32_to_cpu(src->queue_size);
+ dst->gran_read = le32_to_cpu(src->gran_read);
+ dst->gran_write = le32_to_cpu(src->gran_write);
+ dst->gran_erase = le32_to_cpu(src->gran_erase);
+ dst->t_r = le32_to_cpu(src->t_r);
+ dst->t_sqr = le32_to_cpu(src->t_sqr);
+ dst->t_w = le32_to_cpu(src->t_w);
+ dst->t_sqw = le32_to_cpu(src->t_sqw);
+ dst->t_e = le32_to_cpu(src->t_e);
+ dst->io_sched = src->io_sched;
+ }
+
+ len -= end;
+ if (!len)
+ break;
+
+ off += end;
+
+ c.nvm_identify.chnl_off = off;
+
+ ret = nvme_submit_sync_cmd(q, (struct nvme_command *)&c,
+ nvme_nvm_id, 4096);
+ if (ret)
+ return ret;
+ }
+ return 0;
+}
+
+static int nvme_nvm_identify(struct request_queue *q, struct nvm_id *nvm_id)
+{
+ struct nvme_ns *ns = q->queuedata;
+ struct nvme_nvm_id *nvme_nvm_id;
+ struct nvme_nvm_command c = {
+ .nvm_identify.opcode = nvme_nvm_admin_identify,
+ .nvm_identify.nsid = cpu_to_le32(ns->ns_id),
+ .nvm_identify.chnl_off = 0,
+ };
+ int ret;
+
+ nvme_nvm_id = kmalloc(4096, GFP_KERNEL);
+ if (!nvme_nvm_id)
+ return -ENOMEM;
+
+ ret = nvme_submit_sync_cmd(q, (struct nvme_command *)&c, nvme_nvm_id,
+ 4096);
+ if (ret) {
+ ret = -EIO;
+ goto out;
+ }
+
+ nvm_id->ver_id = nvme_nvm_id->ver_id;
+ nvm_id->nvm_type = nvme_nvm_id->nvm_type;
+ nvm_id->nchannels = le16_to_cpu(nvme_nvm_id->nchannels);
+
+ if (!nvm_id->chnls)
+ nvm_id->chnls = kmalloc(sizeof(struct nvm_id_chnl)
+ * nvm_id->nchannels, GFP_KERNEL);
+ if (!nvm_id->chnls) {
+ ret = -ENOMEM;
+ goto out;
+ }
+
+ ret = init_chnls(q, nvm_id, nvme_nvm_id);
+out:
+ kfree(nvme_nvm_id);
+ return ret;
+}
+
+static int nvme_nvm_get_features(struct request_queue *q,
+ struct nvm_get_features *gf)
+{
+ struct nvme_ns *ns = q->queuedata;
+ struct nvme_nvm_command c = {
+ .common.opcode = nvme_nvm_admin_get_features,
+ .common.nsid = ns->ns_id,
+ };
+ int sz = sizeof(struct nvm_get_features);
+ int ret;
+ u64 *resp;
+
+ resp = kmalloc(sz, GFP_KERNEL);
+ if (!resp)
+ return -ENOMEM;
+
+ ret = nvme_submit_sync_cmd(q, (struct nvme_command *)&c, resp, sz);
+ if (ret)
+ goto done;
+
+ gf->rsp = le64_to_cpu(resp[0]);
+ gf->ext = le64_to_cpu(resp[1]);
+
+done:
+ kfree(resp);
+ return ret;
+}
+
+static int nvme_nvm_set_resp(struct request_queue *q, u64 resp)
+{
+ struct nvme_ns *ns = q->queuedata;
+ struct nvme_nvm_command c = {
+ .nvm_resp.opcode = nvme_nvm_admin_set_resp,
+ .nvm_resp.nsid = cpu_to_le32(ns->ns_id),
+ .nvm_resp.resp = cpu_to_le64(resp),
+ };
+
+ return nvme_submit_sync_cmd(q, (struct nvme_command *)&c, NULL, 0);
+}
+
+static int nvme_nvm_get_l2p_tbl(struct request_queue *q, u64 slba, u64 nlb,
+ nvm_l2p_update_fn *update_l2p, void *priv)
+{
+ struct nvme_ns *ns = q->queuedata;
+ struct nvme_dev *dev = ns->dev;
+ struct nvme_nvm_command c = {
+ .nvm_l2p.opcode = nvme_nvm_admin_get_l2p_tbl,
+ .nvm_l2p.nsid = cpu_to_le32(ns->ns_id),
+ };
+ u32 len = queue_max_hw_sectors(q) << 9;
+ u64 nlb_pr_rq = len / sizeof(u64);
+ u64 cmd_slba = slba;
+ void *entries;
+ int ret = 0;
+
+ entries = kmalloc(len, GFP_KERNEL);
+ if (!entries)
+ return -ENOMEM;
+
+ while (nlb) {
+ u64 cmd_nlb = min_t(u64, nlb_pr_rq, nlb);
+
+ c.nvm_l2p.slba = cmd_slba;
+ c.nvm_l2p.nlb = cmd_nlb;
+
+ ret = nvme_submit_sync_cmd(q, (struct nvme_command *)&c,
+ entries, len);
+ if (ret) {
+ dev_err(dev->dev, "L2P table transfer failed (%d)\n",
+ ret);
+ ret = -EIO;
+ goto out;
+ }
+
+ if (update_l2p(cmd_slba, cmd_nlb, entries, priv)) {
+ ret = -EINTR;
+ goto out;
+ }
+
+ cmd_slba += cmd_nlb;
+ nlb -= cmd_nlb;
+ }
+
+out:
+ kfree(entries);
+ return ret;
+}
+
+static int nvme_nvm_set_bb_tbl(struct request_queue *q, int lunid,
+ unsigned int nr_blocks, nvm_bb_update_fn *update_bbtbl, void *priv)
+{
+ return 0;
+}
+
+static int nvme_nvm_get_bb_tbl(struct request_queue *q, int lunid,
+ unsigned int nr_blocks, nvm_bb_update_fn *update_bbtbl, void *priv)
+{
+ struct nvme_ns *ns = q->queuedata;
+ struct nvme_dev *dev = ns->dev;
+ struct nvme_nvm_command c = {
+ .nvm_get_bb.opcode = nvme_nvm_admin_get_bb_tbl,
+ .nvm_get_bb.nsid = cpu_to_le32(ns->ns_id),
+ .nvm_get_bb.lbb = cpu_to_le32(lunid),
+ };
+ void *bb_bitmap;
+ u16 bb_bitmap_size;
+ int ret = 0;
+
+ bb_bitmap_size = ((nr_blocks >> 15) + 1) * PAGE_SIZE;
+ bb_bitmap = kmalloc(bb_bitmap_size, GFP_KERNEL);
+ if (!bb_bitmap)
+ return -ENOMEM;
+
+ bitmap_zero(bb_bitmap, nr_blocks);
+
+ ret = nvme_submit_sync_cmd(q, (struct nvme_command *)&c, bb_bitmap,
+ bb_bitmap_size);
+ if (ret) {
+ dev_err(dev->dev, "get bad block table failed (%d)\n", ret);
+ ret = -EIO;
+ goto out;
+ }
+
+ ret = update_bbtbl(lunid, bb_bitmap, nr_blocks, priv);
+ if (ret) {
+ ret = -EINTR;
+ goto out;
+ }
+
+out:
+ kfree(bb_bitmap);
+ return ret;
+}
+
+static inline void nvme_nvm_rqtocmd(struct request *rq, struct nvm_rq *rqd,
+ struct nvme_ns *ns, struct nvme_nvm_command *c)
+{
+ c->nvm_hb_rw.opcode = (rq_data_dir(rq) ?
+ nvme_nvm_cmd_hb_write : nvme_nvm_cmd_hb_read);
+ c->nvm_hb_rw.nsid = cpu_to_le32(ns->ns_id);
+ c->nvm_hb_rw.slba = cpu_to_le64(nvme_block_nr(ns,
+ rqd->bio->bi_iter.bi_sector));
+ c->nvm_hb_rw.length = cpu_to_le16(
+ (blk_rq_bytes(rq) >> ns->lba_shift) - 1);
+ c->nvm_hb_rw.phys_addr =
+ cpu_to_le64(nvme_block_nr(ns, rqd->phys_sector));
+}
+
+static void nvme_nvm_end_io(struct request *rq, int error)
+{
+ struct nvm_rq *rqd = rq->end_io_data;
+ struct nvm_tgt_instance *ins = rqd->ins;
+
+ ins->tt->end_io(rq->end_io_data, error);
+
+ kfree(rq->cmd);
+ blk_mq_free_request(rq);
+}
+
+static int nvme_nvm_submit_io(struct request_queue *q, struct nvm_rq *rqd)
+{
+ struct nvme_ns *ns = q->queuedata;
+ struct request *rq;
+ struct bio *bio = rqd->bio;
+ struct nvme_nvm_command *cmd;
+
+ rq = blk_mq_alloc_request(q, bio_rw(bio), GFP_KERNEL, 0);
+ if (IS_ERR(rq))
+ return -ENOMEM;
+
+ cmd = kzalloc(sizeof(struct nvme_nvm_command), GFP_KERNEL);
+ if (!cmd) {
+ blk_mq_free_request(rq);
+ return -ENOMEM;
+ }
+
+ rq->cmd_type = REQ_TYPE_DRV_PRIV;
+ rq->ioprio = bio_prio(bio);
+
+ if (bio_has_data(bio))
+ rq->nr_phys_segments = bio_phys_segments(q, bio);
+
+ rq->__data_len = bio->bi_iter.bi_size;
+ rq->bio = rq->biotail = bio;
+
+ nvme_nvm_rqtocmd(rq, rqd, ns, cmd);
+
+ rq->cmd = (unsigned char *)cmd;
+ rq->cmd_len = sizeof(struct nvme_nvm_command);
+ rq->special = (void *)0;
+
+ rq->end_io_data = rqd;
+
+ blk_execute_rq_nowait(q, NULL, rq, 0, nvme_nvm_end_io);
+
+ return 0;
+}
+
+static int nvme_nvm_erase_block(struct request_queue *q, sector_t block_id)
+{
+ struct nvme_ns *ns = q->queuedata;
+ struct nvme_nvm_command c = {
+ .nvm_erase.opcode = nvme_nvm_cmd_erase,
+ .nvm_erase.nsid = cpu_to_le32(ns->ns_id),
+ .nvm_erase.blk_addr = cpu_to_le64(block_id),
+ };
+
+ return nvme_submit_sync_cmd(q, (struct nvme_command *)&c, NULL, 0);
+}
+
+static struct nvm_dev_ops nvme_nvm_dev_ops = {
+ .identify = nvme_nvm_identify,
+
+ .get_features = nvme_nvm_get_features,
+ .set_responsibility = nvme_nvm_set_resp,
+
+ .get_l2p_tbl = nvme_nvm_get_l2p_tbl,
+
+ .set_bb_tbl = nvme_nvm_set_bb_tbl,
+ .get_bb_tbl = nvme_nvm_get_bb_tbl,
+
+ .submit_io = nvme_nvm_submit_io,
+ .erase_block = nvme_nvm_erase_block,
+};
+
+int nvme_nvm_register(struct request_queue *q, struct gendisk *disk)
+{
+ return nvm_register(q, disk, &nvme_nvm_dev_ops);
+}
+#else
+int nvme_nvm_register(struct request_queue *q, struct gendisk *disk)
+{
+ return 0;
+}
+
+#endif /* CONFIG_NVM */
diff --git a/include/linux/nvme.h b/include/linux/nvme.h
index fa3fe16..fa58242 100644
--- a/include/linux/nvme.h
+++ b/include/linux/nvme.h
@@ -19,6 +19,7 @@
#include <linux/pci.h>
#include <linux/kref.h>
#include <linux/blk-mq.h>
+#include <linux/lightnvm.h>
struct nvme_bar {
__u64 cap; /* Controller Capabilities */
@@ -41,6 +42,7 @@ struct nvme_bar {
#define NVME_CAP_STRIDE(cap) (((cap) >> 32) & 0xf)
#define NVME_CAP_MPSMIN(cap) (((cap) >> 48) & 0xf)
#define NVME_CAP_MPSMAX(cap) (((cap) >> 52) & 0xf)
+#define NVME_CAP_LIGHTNVM(cap) (((cap) >> 38) & 0x1)
#define NVME_CMB_BIR(cmbloc) ((cmbloc) & 0x7)
#define NVME_CMB_OFST(cmbloc) (((cmbloc) >> 12) & 0xfffff)
@@ -56,6 +58,7 @@ struct nvme_bar {
enum {
NVME_CC_ENABLE = 1 << 0,
NVME_CC_CSS_NVM = 0 << 4,
+ NVME_CC_CSS_LIGHTNVM = 1 << 4,
NVME_CC_MPS_SHIFT = 7,
NVME_CC_ARB_RR = 0 << 11,
NVME_CC_ARB_WRRU = 1 << 11,
@@ -138,6 +141,7 @@ struct nvme_ns {
u16 ms;
bool ext;
u8 pi_type;
+ int type;
u64 mode_select_num_blocks;
u32 mode_select_block_len;
};
@@ -184,4 +188,6 @@ int nvme_sg_io(struct nvme_ns *ns, struct sg_io_hdr __user *u_hdr);
int nvme_sg_io32(struct nvme_ns *ns, unsigned long arg);
int nvme_sg_get_version_num(int __user *ip);
+int nvme_nvm_register(struct request_queue *q, struct gendisk *disk);
+
#endif /* _LINUX_NVME_H */
diff --git a/include/uapi/linux/nvme.h b/include/uapi/linux/nvme.h
index 732b32e..0374f11 100644
--- a/include/uapi/linux/nvme.h
+++ b/include/uapi/linux/nvme.h
@@ -130,6 +130,7 @@ struct nvme_id_ns {
enum {
NVME_NS_FEAT_THIN = 1 << 0,
+ NVME_NS_FEAT_NVM = 1 << 3,
NVME_NS_FLBAS_LBA_MASK = 0xf,
NVME_NS_FLBAS_META_EXT = 0x10,
NVME_LBAF_RP_BEST = 0,
@@ -146,6 +147,8 @@ enum {
NVME_NS_DPS_PI_TYPE1 = 1,
NVME_NS_DPS_PI_TYPE2 = 2,
NVME_NS_DPS_PI_TYPE3 = 3,
+
+ NVME_NS_NVM = 1,
};
struct nvme_smart_log {
--
2.1.4
Hi Matias,
I like this new architecture.
Minor nitpicks below:
> @@ -0,0 +1,598 @@
> +/*
> + * core.c - Open-channel SSD integration core
No need to state the file name in the top of the file comments, it
doesn't add value and get stale easily.
> +static LIST_HEAD(_targets);
> +static LIST_HEAD(_bms);
> +static LIST_HEAD(_devices);
> +static DECLARE_RWSEM(_lock);
Please add a nvm_ prefix to these.
> +static int nvm_ioctl(struct block_device *bdev, fmode_t mode, unsigned int cmd,
> + unsigned long arg)
> +{
> + return 0;
> +}
> +
> +static int nvm_open(struct block_device *bdev, fmode_t mode)
> +{
> + return 0;
> +}
> +
> +static void nvm_release(struct gendisk *disk, fmode_t mode)
> +{
> +}
No need to implement these empty methods.
> +/* _lock must be taken */
Turn this into an lockdep_assert_held in the code, please.
> +int nvm_register(struct request_queue *q, struct gendisk *disk,
> + struct nvm_dev_ops *ops)
> +{
No need to pass a gendisk here, in fact LighNVM low level driver
shouldn't even need to allocate one. The only thing you seem to be
using here is the name anyway.
> +void nvm_unregister(struct gendisk *disk)
Same here.
Any reason you're adding this to null_blk instead of having a separate
null_nvm driver?
On 07/23/2015 11:53 AM, Christoph Hellwig wrote:
> Hi Matias,
>
> I like this new architecture.
Thanks, great to hear. :)
I will get the nitpicks fixed.
On 07/23/2015 11:53 AM, Christoph Hellwig wrote:
> Any reason you're adding this to null_blk instead of having a separate
> null_nvm driver?
>
Only reason was to not duplicate the I/O submission/completion flow.
With the simple code at the moment, it is little to add. However, if
someone decides to put backing store, simulation, etc. into it, then it
should properly go into another module.
Do you want me to move it into a new driver for this patchset?
On Thu, Jul 23, 2015 at 12:48:00PM +0200, Matias Bj?rling wrote:
> Only reason was to not duplicate the I/O submission/completion flow.
> With the simple code at the moment, it is little to add. However, if
> someone decides to put backing store, simulation, etc. into it, then it
> should properly go into another module.
>
> Do you want me to move it into a new driver for this patchset?
I'd prefer to have it separate unless there's a good reason to merge it
into null_blk. From your above descriptions it sounds like it really
should be separate.