2019-07-03 23:04:16

by Logan Gunthorpe

[permalink] [raw]
Subject: [PATCH v2 1/2] nvmet: Fix use-after-free bug when a port is removed

When a port is removed through configfs, any connected controllers
are still active and can still send commands. This causes a
use-after-free bug which is detected by KASAN for any admin command
that dereferences req->port (like in nvmet_execute_identify_ctrl).

To fix this, disconnect all active controllers when a subsystem is
removed from a port. This ensures there are no active controllers
when the port is eventually removed.

Signed-off-by: Logan Gunthorpe <[email protected]>
---
drivers/nvme/target/configfs.c | 1 +
drivers/nvme/target/core.c | 12 ++++++++++++
drivers/nvme/target/nvmet.h | 3 +++
3 files changed, 16 insertions(+)

diff --git a/drivers/nvme/target/configfs.c b/drivers/nvme/target/configfs.c
index 08dd5af357f7..3854363118cc 100644
--- a/drivers/nvme/target/configfs.c
+++ b/drivers/nvme/target/configfs.c
@@ -673,6 +673,7 @@ static void nvmet_port_subsys_drop_link(struct config_item *parent,

found:
list_del(&p->entry);
+ nvmet_port_del_ctrls(port, subsys);
nvmet_port_disc_changed(port, subsys);

if (list_empty(&port->subsystems))
diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
index 7734a6acff85..e4db9a441168 100644
--- a/drivers/nvme/target/core.c
+++ b/drivers/nvme/target/core.c
@@ -277,6 +277,18 @@ void nvmet_unregister_transport(const struct nvmet_fabrics_ops *ops)
}
EXPORT_SYMBOL_GPL(nvmet_unregister_transport);

+void nvmet_port_del_ctrls(struct nvmet_port *port, struct nvmet_subsys *subsys)
+{
+ struct nvmet_ctrl *ctrl;
+
+ mutex_lock(&subsys->lock);
+ list_for_each_entry(ctrl, &subsys->ctrls, subsys_entry) {
+ if (ctrl->port == port)
+ ctrl->ops->delete_ctrl(ctrl);
+ }
+ mutex_unlock(&subsys->lock);
+}
+
int nvmet_enable_port(struct nvmet_port *port)
{
const struct nvmet_fabrics_ops *ops;
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index c25d88fc9dec..b6b0d483e0c5 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -415,6 +415,9 @@ void nvmet_port_send_ana_event(struct nvmet_port *port);
int nvmet_register_transport(const struct nvmet_fabrics_ops *ops);
void nvmet_unregister_transport(const struct nvmet_fabrics_ops *ops);

+void nvmet_port_del_ctrls(struct nvmet_port *port,
+ struct nvmet_subsys *subsys);
+
int nvmet_enable_port(struct nvmet_port *port);
void nvmet_disable_port(struct nvmet_port *port);

--
2.20.1


2019-07-04 21:03:17

by Logan Gunthorpe

[permalink] [raw]
Subject: Re: [PATCH v2 1/2] nvmet: Fix use-after-free bug when a port is removed



On 2019-07-04 3:00 p.m., Max Gurtovoy wrote:
> Hi Logan,
>
> On 7/4/2019 2:03 AM, Logan Gunthorpe wrote:
>> When a port is removed through configfs, any connected controllers
>> are still active and can still send commands. This causes a
>> use-after-free bug which is detected by KASAN for any admin command
>> that dereferences req->port (like in nvmet_execute_identify_ctrl).
>>
>> To fix this, disconnect all active controllers when a subsystem is
>> removed from a port. This ensures there are no active controllers
>> when the port is eventually removed.
>
> so now we are enforcing controller existence with port configfs, right ?
> sounds reasonable.

Correct.

> Did you run your patches with other transport (RDMA/TCP/FC) ?

Just RDMA and loop. I suppose I could test with TCP but I don't have FC
hardware.

Logan

2019-07-04 21:27:52

by Max Gurtovoy

[permalink] [raw]
Subject: Re: [PATCH v2 1/2] nvmet: Fix use-after-free bug when a port is removed

Hi Logan,

On 7/4/2019 2:03 AM, Logan Gunthorpe wrote:
> When a port is removed through configfs, any connected controllers
> are still active and can still send commands. This causes a
> use-after-free bug which is detected by KASAN for any admin command
> that dereferences req->port (like in nvmet_execute_identify_ctrl).
>
> To fix this, disconnect all active controllers when a subsystem is
> removed from a port. This ensures there are no active controllers
> when the port is eventually removed.

so now we are enforcing controller existence with port configfs, right ?
sounds reasonable.

Did you run your patches with other transport (RDMA/TCP/FC) ?


2019-07-05 12:19:05

by Max Gurtovoy

[permalink] [raw]
Subject: Re: [PATCH v2 1/2] nvmet: Fix use-after-free bug when a port is removed


On 7/5/2019 12:01 AM, Logan Gunthorpe wrote:
>
> On 2019-07-04 3:00 p.m., Max Gurtovoy wrote:
>> Hi Logan,
>>
>> On 7/4/2019 2:03 AM, Logan Gunthorpe wrote:
>>> When a port is removed through configfs, any connected controllers
>>> are still active and can still send commands. This causes a
>>> use-after-free bug which is detected by KASAN for any admin command
>>> that dereferences req->port (like in nvmet_execute_identify_ctrl).
>>>
>>> To fix this, disconnect all active controllers when a subsystem is
>>> removed from a port. This ensures there are no active controllers
>>> when the port is eventually removed.
>> so now we are enforcing controller existence with port configfs, right ?
>> sounds reasonable.
> Correct.
>
>> Did you run your patches with other transport (RDMA/TCP/FC) ?
> Just RDMA and loop. I suppose I could test with TCP but I don't have FC
> hardware.

Great.

the code looks good:

Reviewed-by: Max Gurtovoy <[email protected]>

>
> Logan