This patch set acts as the second part of the new version of [1] (The first
part can be referred from [2]), the updated things of this version are listed
at the end.
- Background
SMC-D is now used in IBM z with ISM function to optimize network interconnect
for intra-CPC communications. Inspired by this, we try to make SMC-D available
on the non-s390 architecture through a software-implemented Emulated-ISM device,
that is the loopback-ism device here, to accelerate inter-process or
inter-containers communication within the same OS instance.
- Design
This patch set includes 3 parts:
- Patch #1-#2: some prepare work for loopback-ism.
- Patch #3-#7: implement loopback-ism device. Noted that loopback-ism now
serves only SMC and no userspace interface exposed.
- Patch #10-#15: memory copy optimization for intra-OS scenario.
The loopback-ism device is designed as an ISMv2 device and not be limited to
a specific net namespace, ends of both inter-process connection (1/1' in diagram
below) or inter-container connection (2/2' in diagram below) can find the same
available loopback-ism and choose it during the CLC handshake.
Container 1 (ns1) Container 2 (ns2)
+-----------------------------------------+ +-------------------------+
| +-------+ +-------+ +-------+ | | +-------+ |
| | App A | | App B | | App C | | | | App D |<-+ |
| +-------+ +---^---+ +-------+ | | +-------+ |(2') |
| |127.0.0.1 (1')| |192.168.0.11 192.168.0.12| |
| (1)| +--------+ | +--------+ |(2) | | +--------+ +--------+ |
| `-->| lo |-` | eth0 |<-` | | | lo | | eth0 | |
+---------+--|---^-+---+-----|--+---------+ +-+--------+---+-^------+-+
| | | |
Kernel | | | |
+----+-------v---+-----------v----------------------------------+---+----+
| | TCP | |
| | | |
| +--------------------------------------------------------------+ |
| |
| +--------------+ |
| | smc loopback | |
+---------------------------+--------------+-----------------------------+
loopback-ism device creates DMBs (shared memory) for each connection peer.
Since data transfer occurs within the same kernel, the sndbuf of each peer
is only a descriptor and point to the same memory region as peer DMB, so that
the data copy from sndbuf to peer DMB can be avoided in loopback-ism case.
Container 1 (ns1) Container 2 (ns2)
+-----------------------------------------+ +-------------------------+
| +-------+ | | +-------+ |
| | App C |-----+ | | | App D | |
| +-------+ | | | +-^-----+ |
| | | | | |
| (2) | | | (2') | |
| | | | | |
+---------------|-------------------------+ +----------|--------------+
| |
Kernel | |
+---------------|-----------------------------------------|--------------+
| +--------+ +--v-----+ +--------+ +--------+ |
| |dmb_desc| |snd_desc| |dmb_desc| |snd_desc| |
| +-----|--+ +--|-----+ +-----|--+ +--------+ |
| +-----|--+ | +-----|--+ |
| | DMB C | +---------------------------------| DMB D | |
| +--------+ +--------+ |
| |
| +--------------+ |
| | smc loopback | |
+---------------------------+--------------+-----------------------------+
- Benchmark Test
* Test environments:
- VM with Intel Xeon Platinum 8 core 2.50GHz, 16 GiB mem.
- SMC sndbuf/DMB size 1MB.
* Test object:
- TCP: run on TCP loopback.
- SMC lo: run on SMC loopback-ism.
1. ipc-benchmark (see [3])
- ./<foo> -c 1000000 -s 100
TCP SMC-lo
Message
rate (msg/s) 79287 148946(+87.86%)
2. sockperf
- serv: <smc_run> taskset -c <cpu> sockperf sr --tcp
- clnt: <smc_run> taskset -c <cpu> sockperf { tp | pp } --tcp --msg-size={ 64000 for tp | 14 for pp } -i 127.0.0.1 -t 30
TCP SMC-lo
Bandwidth(MBps) 5053.47 8195.07(+62.17%)
Latency(us) 6.108 3.404(-44.27%)
3. nginx/wrk
- serv: <smc_run> nginx
- clnt: <smc_run> wrk -t 8 -c 1000 -d 30 http://127.0.0.1:80
TCP SMC-lo
Requests/s 178187.56 247970.92(+39.16%)
4. redis-benchmark
- serv: <smc_run> redis-server
- clnt: <smc_run> redis-benchmark -h 127.0.0.1 -q -t set,get -n 400000 -c 200 -d 1024
TCP SMC-lo
GET(Requests/s) 89067.02 123877.36(+39.08%)
SET(Requests/s) 87700.07 131319.77(+49.73%)
Change log:
v2->v1:
- All the patches: changed the term virtual-ISM to Emulated-ISM as defined by SMCv2.1.
- Patch #3: optimized the description of SMC_LO config. Avoid exposing loopback-ism
to sysfs and remove all the knobs until future definition clear.
- Patch #3: try to make lockdep happy by using read_lock_bh() in smc_lo_move_data().
- Patch #6: defaultly use physical contiguous DMB buffers.
- Patch #11: defaultly enable DMB no-copy for loopback-ism and free the DMB in
unregister_dmb or detach_dmb when dmb_node->refcnt reaches 0, instead of using
wait_event to keep waiting in unregister_dmb.
v1->RFC:
- Patch #9: merge rx_bytes and tx_bytes as xfer_bytes statistics:
/sys/devices/virtual/smc/loopback-ism/xfer_bytes
- Patch #10: add support_dmb_nocopy operation to check if SMC-D device supports
merging sndbuf with peer DMB.
- Patch #13 & #14: introduce loopback-ism device control of DMB memory type and
control of whether to merge sndbuf and DMB. They can be respectively set by:
/sys/devices/virtual/smc/loopback-ism/dmb_type
/sys/devices/virtual/smc/loopback-ism/dmb_copy
The motivation for these two control is that a performance bottleneck was
found when using vzalloced DMB and sndbuf is merged with DMB, and there are
many CPUs and CONFIG_HARDENED_USERCOPY is set [4]. The bottleneck is caused
by the lock contention in vmap_area_lock [5] which is involved in memcpy_from_msg()
or memcpy_to_msg(). Currently, Uladzislau Rezki is working on mitigating the
vmap lock contention [6]. It has significant effects, but using virtual memory
still has additional overhead compared to using physical memory.
So this new version provides controls of dmb_type and dmb_copy to suit
different scenarios.
- Some minor changes and comments improvements.
RFC->old version([1]):
Link: https://lore.kernel.org/netdev/[email protected]/
- Patch #1: improve the loopback-ism dump, it shows as follows now:
# smcd d
FID Type PCI-ID PCHID InUse #LGs PNET-ID
0000 0 loopback-ism ffff No 0
- Patch #3: introduce the smc_ism_set_v2_capable() helper and set
smc_ism_v2_capable when ISMv2 or virtual ISM is registered,
regardless of whether there is already a device in smcd device list.
- Patch #3: loopback-ism will be added into /sys/devices/virtual/smc/loopback-ism/.
- Patch #8: introduce the runtime switch /sys/devices/virtual/smc/loopback-ism/active
to activate or deactivate the loopback-ism.
- Patch #9: introduce the statistics of loopback-ism by
/sys/devices/virtual/smc/loopback-ism/{{tx|rx}_tytes|dmbs_cnt}.
- Some minor changes and comments improvements.
[1] https://lore.kernel.org/netdev/[email protected]/
[2] https://lore.kernel.org/netdev/[email protected]/
[3] https://github.com/goldsborough/ipc-bench
[4] https://lore.kernel.org/all/[email protected]/
[5] https://lore.kernel.org/all/[email protected]/
[6] https://lore.kernel.org/all/[email protected]/
Wen Gu (11):
net/smc: adapt SMC-D device dump for Emulated-ISM
net/smc: decouple ism_client from SMC-D DMB registration
net/smc: introduce loopback-ism for SMC intra-OS shortcut
net/smc: implement ID-related operations of loopback-ism
net/smc: implement some unsupported operations of loopback-ism
net/smc: implement DMB-related operations of loopback-ism
net/smc: register loopback-ism into SMC-D device list
net/smc: add operations to merge sndbuf with peer DMB
net/smc: attach or detach ghost sndbuf to peer DMB
net/smc: adapt cursor update when sndbuf and peer DMB are merged
net/smc: implement DMB-merged operations of loopback-ism
drivers/s390/net/ism_drv.c | 2 +-
include/net/smc.h | 7 +-
net/smc/Kconfig | 13 +
net/smc/Makefile | 2 +-
net/smc/af_smc.c | 28 ++-
net/smc/smc_cdc.c | 58 ++++-
net/smc/smc_cdc.h | 1 +
net/smc/smc_core.c | 61 ++++-
net/smc/smc_core.h | 1 +
net/smc/smc_ism.c | 71 +++++-
net/smc/smc_ism.h | 5 +
net/smc/smc_loopback.c | 469 +++++++++++++++++++++++++++++++++++++
net/smc/smc_loopback.h | 52 ++++
13 files changed, 741 insertions(+), 29 deletions(-)
create mode 100644 net/smc/smc_loopback.c
create mode 100644 net/smc/smc_loopback.h
--
2.32.0.3.g01195cf9f
The ghost sndbuf descriptor will be created and attached to peer DMB
once peer token is obtained and it will be detach and freed when the
connection is freed.
Signed-off-by: Wen Gu <[email protected]>
---
net/smc/af_smc.c | 16 ++++++++++++
net/smc/smc_core.c | 61 +++++++++++++++++++++++++++++++++++++++++++++-
net/smc/smc_core.h | 1 +
3 files changed, 77 insertions(+), 1 deletion(-)
diff --git a/net/smc/af_smc.c b/net/smc/af_smc.c
index 751a8fdd6eb0..fce7a5b2ce5c 100644
--- a/net/smc/af_smc.c
+++ b/net/smc/af_smc.c
@@ -1438,6 +1438,14 @@ static int smc_connect_ism(struct smc_sock *smc,
}
smc_conn_save_peer_info(smc, aclc);
+
+ if (smc_ism_support_dmb_nocopy(smc->conn.lgr->smcd)) {
+ rc = smcd_buf_attach(smc);
+ if (rc) {
+ rc = SMC_CLC_DECL_MEM; /* try to fallback */
+ goto connect_abort;
+ }
+ }
smc_close_init(smc);
smc_rx_init(smc);
smc_tx_init(smc);
@@ -2542,6 +2550,14 @@ static void smc_listen_work(struct work_struct *work)
mutex_unlock(&smc_server_lgr_pending);
}
smc_conn_save_peer_info(new_smc, cclc);
+
+ if (ini->is_smcd &&
+ smc_ism_support_dmb_nocopy(new_smc->conn.lgr->smcd)) {
+ rc = smcd_buf_attach(new_smc);
+ if (rc)
+ goto out_decl;
+ }
+
smc_listen_out_connected(new_smc);
SMC_STAT_SERV_SUCC_INC(sock_net(newclcsock->sk), ini);
goto out_free;
diff --git a/net/smc/smc_core.c b/net/smc/smc_core.c
index 9b84d5897aa5..fafdb97adfad 100644
--- a/net/smc/smc_core.c
+++ b/net/smc/smc_core.c
@@ -1149,6 +1149,20 @@ static void smcr_buf_unuse(struct smc_buf_desc *buf_desc, bool is_rmb,
}
}
+static void smcd_buf_detach(struct smc_connection *conn)
+{
+ struct smcd_dev *smcd = conn->lgr->smcd;
+ u64 peer_token = conn->peer_token;
+
+ if (!conn->sndbuf_desc)
+ return;
+
+ smc_ism_detach_dmb(smcd, peer_token);
+
+ kfree(conn->sndbuf_desc);
+ conn->sndbuf_desc = NULL;
+}
+
static void smc_buf_unuse(struct smc_connection *conn,
struct smc_link_group *lgr)
{
@@ -1192,6 +1206,8 @@ void smc_conn_free(struct smc_connection *conn)
if (lgr->is_smcd) {
if (!list_empty(&lgr->list))
smc_ism_unset_conn(conn);
+ if (smc_ism_support_dmb_nocopy(lgr->smcd))
+ smcd_buf_detach(conn);
tasklet_kill(&conn->rx_tsklet);
} else {
smc_cdc_wait_pend_tx_wr(conn);
@@ -1445,6 +1461,8 @@ static void smc_conn_kill(struct smc_connection *conn, bool soft)
smc_sk_wake_ups(smc);
if (conn->lgr->is_smcd) {
smc_ism_unset_conn(conn);
+ if (smc_ism_support_dmb_nocopy(conn->lgr->smcd))
+ smcd_buf_detach(conn);
if (soft)
tasklet_kill(&conn->rx_tsklet);
else
@@ -2464,12 +2482,18 @@ int smc_buf_create(struct smc_sock *smc, bool is_smcd)
int rc;
/* create send buffer */
+ if (is_smcd &&
+ smc_ism_support_dmb_nocopy(smc->conn.lgr->smcd))
+ goto create_rmb;
+
rc = __smc_buf_create(smc, is_smcd, false);
if (rc)
return rc;
+
+create_rmb:
/* create rmb */
rc = __smc_buf_create(smc, is_smcd, true);
- if (rc) {
+ if (rc && smc->conn.sndbuf_desc) {
down_write(&smc->conn.lgr->sndbufs_lock);
list_del(&smc->conn.sndbuf_desc->list);
up_write(&smc->conn.lgr->sndbufs_lock);
@@ -2479,6 +2503,41 @@ int smc_buf_create(struct smc_sock *smc, bool is_smcd)
return rc;
}
+int smcd_buf_attach(struct smc_sock *smc)
+{
+ struct smc_connection *conn = &smc->conn;
+ struct smcd_dev *smcd = conn->lgr->smcd;
+ u64 peer_token = conn->peer_token;
+ struct smc_buf_desc *buf_desc;
+ int rc;
+
+ buf_desc = kzalloc(sizeof(*buf_desc), GFP_KERNEL);
+ if (!buf_desc)
+ return -ENOMEM;
+
+ /* The ghost sndbuf_desc describes the same memory region as
+ * peer RMB. Its lifecycle is consistent with the connection's
+ * and it will be freed with the connections instead of the
+ * link group.
+ */
+ rc = smc_ism_attach_dmb(smcd, peer_token, buf_desc);
+ if (rc)
+ goto free;
+
+ smc->sk.sk_sndbuf = buf_desc->len;
+ buf_desc->cpu_addr =
+ (u8 *)buf_desc->cpu_addr + sizeof(struct smcd_cdc_msg);
+ buf_desc->len -= sizeof(struct smcd_cdc_msg);
+ conn->sndbuf_desc = buf_desc;
+ conn->sndbuf_desc->used = 1;
+ atomic_set(&conn->sndbuf_space, conn->sndbuf_desc->len);
+ return 0;
+
+free:
+ kfree(buf_desc);
+ return rc;
+}
+
static inline int smc_rmb_reserve_rtoken_idx(struct smc_link_group *lgr)
{
int i;
diff --git a/net/smc/smc_core.h b/net/smc/smc_core.h
index 1f175376037b..d93cf51dbd7c 100644
--- a/net/smc/smc_core.h
+++ b/net/smc/smc_core.h
@@ -557,6 +557,7 @@ void smc_smcd_terminate(struct smcd_dev *dev, struct smcd_gid *peer_gid,
void smc_smcd_terminate_all(struct smcd_dev *dev);
void smc_smcr_terminate_all(struct smc_ib_device *smcibdev);
int smc_buf_create(struct smc_sock *smc, bool is_smcd);
+int smcd_buf_attach(struct smc_sock *smc);
int smc_uncompress_bufsize(u8 compressed);
int smc_rmb_rtoken_handling(struct smc_connection *conn, struct smc_link *link,
struct smc_clc_msg_accept_confirm *clc);
--
2.32.0.3.g01195cf9f
vlan operations are not supported currently since the need for vlan in
loopback-ism situation does not seem to be strong.
signal_event operation is not supported since no event now needs to be
processed by loopback-ism device.
Signed-off-by: Wen Gu <[email protected]>
---
net/smc/smc_loopback.c | 36 +++++++++++++++++++++++++++++++-----
1 file changed, 31 insertions(+), 5 deletions(-)
diff --git a/net/smc/smc_loopback.c b/net/smc/smc_loopback.c
index 7656a2474500..253128c77208 100644
--- a/net/smc/smc_loopback.c
+++ b/net/smc/smc_loopback.c
@@ -49,6 +49,32 @@ static int smc_lo_query_rgid(struct smcd_dev *smcd, struct smcd_gid *rgid,
return 0;
}
+static int smc_lo_add_vlan_id(struct smcd_dev *smcd, u64 vlan_id)
+{
+ return -EOPNOTSUPP;
+}
+
+static int smc_lo_del_vlan_id(struct smcd_dev *smcd, u64 vlan_id)
+{
+ return -EOPNOTSUPP;
+}
+
+static int smc_lo_set_vlan_required(struct smcd_dev *smcd)
+{
+ return -EOPNOTSUPP;
+}
+
+static int smc_lo_reset_vlan_required(struct smcd_dev *smcd)
+{
+ return -EOPNOTSUPP;
+}
+
+static int smc_lo_signal_event(struct smcd_dev *dev, struct smcd_gid *rgid,
+ u32 trigger_irq, u32 event_code, u64 info)
+{
+ return 0;
+}
+
static int smc_lo_supports_v2(void)
{
return SMC_LO_V2_CAPABLE;
@@ -77,11 +103,11 @@ static const struct smcd_ops lo_ops = {
.query_remote_gid = smc_lo_query_rgid,
.register_dmb = NULL,
.unregister_dmb = NULL,
- .add_vlan_id = NULL,
- .del_vlan_id = NULL,
- .set_vlan_required = NULL,
- .reset_vlan_required = NULL,
- .signal_event = NULL,
+ .add_vlan_id = smc_lo_add_vlan_id,
+ .del_vlan_id = smc_lo_del_vlan_id,
+ .set_vlan_required = smc_lo_set_vlan_required,
+ .reset_vlan_required = smc_lo_reset_vlan_required,
+ .signal_event = smc_lo_signal_event,
.move_data = NULL,
.supports_v2 = smc_lo_supports_v2,
.get_local_gid = smc_lo_get_local_gid,
--
2.32.0.3.g01195cf9f
In some scenarios using Emulated-ISM device, sndbuf can share the same
physical memory region with peer DMB to avoid data copy from one side
to the other. In such case the sndbuf is only a descriptor that
describes the shared memory and does not actually occupy memory, it's
more like a ghost buffer.
+----------+ +----------+
| socket A | | socket B |
+----------+ +----------+
| |
+--------+ +--------+
| sndbuf | | DMB |
| desc | | desc |
+--------+ +--------+
| |
| +----v-----+
+--------------------------> memory |
+----------+
So here introduces three new SMC-D device operations to check if this
feature is supported by device, and to {attach|detach} ghost sndbuf to
peer DMB. For now only loopback-ism supports this.
Signed-off-by: Wen Gu <[email protected]>
---
include/net/smc.h | 3 +++
net/smc/smc_ism.c | 40 ++++++++++++++++++++++++++++++++++++++++
net/smc/smc_ism.h | 4 ++++
3 files changed, 47 insertions(+)
diff --git a/include/net/smc.h b/include/net/smc.h
index 6273c3a8b24a..01387631d8a6 100644
--- a/include/net/smc.h
+++ b/include/net/smc.h
@@ -62,6 +62,9 @@ struct smcd_ops {
int (*register_dmb)(struct smcd_dev *dev, struct smcd_dmb *dmb,
void *client);
int (*unregister_dmb)(struct smcd_dev *dev, struct smcd_dmb *dmb);
+ int (*support_dmb_nocopy)(struct smcd_dev *dev);
+ int (*attach_dmb)(struct smcd_dev *dev, struct smcd_dmb *dmb);
+ int (*detach_dmb)(struct smcd_dev *dev, u64 token);
int (*add_vlan_id)(struct smcd_dev *dev, u64 vlan_id);
int (*del_vlan_id)(struct smcd_dev *dev, u64 vlan_id);
int (*set_vlan_required)(struct smcd_dev *dev);
diff --git a/net/smc/smc_ism.c b/net/smc/smc_ism.c
index c50910300a03..67ab9bce202a 100644
--- a/net/smc/smc_ism.c
+++ b/net/smc/smc_ism.c
@@ -246,6 +246,46 @@ int smc_ism_register_dmb(struct smc_link_group *lgr, int dmb_len,
return rc;
}
+bool smc_ism_support_dmb_nocopy(struct smcd_dev *smcd)
+{
+ /* for now only loopback-ism supports
+ * merging sndbuf with peer DMB to avoid
+ * data copies between them.
+ */
+ return (smcd->ops->support_dmb_nocopy &&
+ smcd->ops->support_dmb_nocopy(smcd));
+}
+
+int smc_ism_attach_dmb(struct smcd_dev *dev, u64 token,
+ struct smc_buf_desc *dmb_desc)
+{
+ struct smcd_dmb dmb;
+ int rc = 0;
+
+ if (!dev->ops->attach_dmb)
+ return -EINVAL;
+
+ memset(&dmb, 0, sizeof(dmb));
+ dmb.dmb_tok = token;
+ rc = dev->ops->attach_dmb(dev, &dmb);
+ if (!rc) {
+ dmb_desc->sba_idx = dmb.sba_idx;
+ dmb_desc->token = dmb.dmb_tok;
+ dmb_desc->cpu_addr = dmb.cpu_addr;
+ dmb_desc->dma_addr = dmb.dma_addr;
+ dmb_desc->len = dmb.dmb_len;
+ }
+ return rc;
+}
+
+int smc_ism_detach_dmb(struct smcd_dev *dev, u64 token)
+{
+ if (!dev->ops->detach_dmb)
+ return -EINVAL;
+
+ return dev->ops->detach_dmb(dev, token);
+}
+
static int smc_nl_handle_smcd_dev(struct smcd_dev *smcd,
struct sk_buff *skb,
struct netlink_callback *cb)
diff --git a/net/smc/smc_ism.h b/net/smc/smc_ism.h
index 2ea169c1301c..83cf99265160 100644
--- a/net/smc/smc_ism.h
+++ b/net/smc/smc_ism.h
@@ -48,6 +48,10 @@ int smc_ism_put_vlan(struct smcd_dev *dev, unsigned short vlan_id);
int smc_ism_register_dmb(struct smc_link_group *lgr, int buf_size,
struct smc_buf_desc *dmb_desc);
int smc_ism_unregister_dmb(struct smcd_dev *dev, struct smc_buf_desc *dmb_desc);
+bool smc_ism_support_dmb_nocopy(struct smcd_dev *smcd);
+int smc_ism_attach_dmb(struct smcd_dev *dev, u64 token,
+ struct smc_buf_desc *dmb_desc);
+int smc_ism_detach_dmb(struct smcd_dev *dev, u64 token);
int smc_ism_signal_shutdown(struct smc_link_group *lgr);
void smc_ism_get_system_eid(u8 **eid);
u16 smc_ism_get_chid(struct smcd_dev *dev);
--
2.32.0.3.g01195cf9f
This implements operations related to merging sndbuf with peer DMB in
loopback-ism. The DMB won't be freed until no sndbuf is attached to it.
Signed-off-by: Wen Gu <[email protected]>
---
net/smc/smc_loopback.c | 136 +++++++++++++++++++++++++++++++++++------
net/smc/smc_loopback.h | 3 +
2 files changed, 119 insertions(+), 20 deletions(-)
diff --git a/net/smc/smc_loopback.c b/net/smc/smc_loopback.c
index 6828e0ad3e90..7e772f3772de 100644
--- a/net/smc/smc_loopback.c
+++ b/net/smc/smc_loopback.c
@@ -21,6 +21,7 @@
#if IS_ENABLED(CONFIG_SMC_LO)
#define SMC_LO_V2_CAPABLE 0x1 /* loopback-ism acts as ISMv2 */
+#define SMC_LO_SUPPORT_NOCOPY 0x1
#define SMC_DMA_ADDR_INVALID (~(dma_addr_t)0)
static const char smc_lo_dev_name[] = "loopback-ism";
@@ -82,6 +83,7 @@ static int smc_lo_register_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb,
goto err_node;
}
dmb_node->dma_addr = SMC_DMA_ADDR_INVALID;
+ refcount_set(&dmb_node->refcnt, 1);
again:
/* add new dmb into hash table */
@@ -95,6 +97,7 @@ static int smc_lo_register_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb,
}
hash_add(ldev->dmb_ht, &dmb_node->list, dmb_node->token);
write_unlock(&ldev->dmb_ht_lock);
+ atomic_inc(&ldev->dmb_cnt);
dmb->sba_idx = dmb_node->sba_idx;
dmb->dmb_tok = dmb_node->token;
@@ -111,13 +114,29 @@ static int smc_lo_register_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb,
return rc;
}
+static void __smc_lo_unregister_dmb(struct smc_lo_dev *ldev,
+ struct smc_lo_dmb_node *dmb_node)
+{
+ /* remove dmb from hash table */
+ write_lock(&ldev->dmb_ht_lock);
+ hash_del(&dmb_node->list);
+ write_unlock(&ldev->dmb_ht_lock);
+
+ clear_bit(dmb_node->sba_idx, ldev->sba_idx_mask);
+ kvfree(dmb_node->cpu_addr);
+ kfree(dmb_node);
+
+ if (atomic_dec_and_test(&ldev->dmb_cnt))
+ wake_up(&ldev->ldev_release);
+}
+
static int smc_lo_unregister_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb)
{
struct smc_lo_dmb_node *dmb_node = NULL, *tmp_node;
struct smc_lo_dev *ldev = smcd->priv;
- /* remove dmb from hash table */
- write_lock(&ldev->dmb_ht_lock);
+ /* find dmb from hash table */
+ read_lock(&ldev->dmb_ht_lock);
hash_for_each_possible(ldev->dmb_ht, tmp_node, list, dmb->dmb_tok) {
if (tmp_node->token == dmb->dmb_tok) {
dmb_node = tmp_node;
@@ -125,16 +144,76 @@ static int smc_lo_unregister_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb)
}
}
if (!dmb_node) {
- write_unlock(&ldev->dmb_ht_lock);
+ read_unlock(&ldev->dmb_ht_lock);
return -EINVAL;
}
- hash_del(&dmb_node->list);
- write_unlock(&ldev->dmb_ht_lock);
+ read_unlock(&ldev->dmb_ht_lock);
- clear_bit(dmb_node->sba_idx, ldev->sba_idx_mask);
- kfree(dmb_node->cpu_addr);
- kfree(dmb_node);
+ if (refcount_dec_and_test(&dmb_node->refcnt))
+ __smc_lo_unregister_dmb(ldev, dmb_node);
+ return 0;
+}
+
+static int smc_lo_support_dmb_nocopy(struct smcd_dev *smcd)
+{
+ return SMC_LO_SUPPORT_NOCOPY;
+}
+
+static int smc_lo_attach_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb)
+{
+ struct smc_lo_dmb_node *dmb_node = NULL, *tmp_node;
+ struct smc_lo_dev *ldev = smcd->priv;
+ /* find dmb_node according to dmb->dmb_tok */
+ read_lock(&ldev->dmb_ht_lock);
+ hash_for_each_possible(ldev->dmb_ht, tmp_node, list, dmb->dmb_tok) {
+ if (tmp_node->token == dmb->dmb_tok) {
+ dmb_node = tmp_node;
+ break;
+ }
+ }
+ if (!dmb_node) {
+ read_unlock(&ldev->dmb_ht_lock);
+ return -EINVAL;
+ }
+ read_unlock(&ldev->dmb_ht_lock);
+
+ if (!refcount_inc_not_zero(&dmb_node->refcnt))
+ /* the dmb is being unregistered, but has
+ * not been removed from the hash table.
+ */
+ return -EINVAL;
+
+ /* provide dmb information */
+ dmb->sba_idx = dmb_node->sba_idx;
+ dmb->dmb_tok = dmb_node->token;
+ dmb->cpu_addr = dmb_node->cpu_addr;
+ dmb->dma_addr = dmb_node->dma_addr;
+ dmb->dmb_len = dmb_node->len;
+ return 0;
+}
+
+static int smc_lo_detach_dmb(struct smcd_dev *smcd, u64 token)
+{
+ struct smc_lo_dmb_node *dmb_node = NULL, *tmp_node;
+ struct smc_lo_dev *ldev = smcd->priv;
+
+ /* find dmb_node according to dmb->dmb_tok */
+ read_lock(&ldev->dmb_ht_lock);
+ hash_for_each_possible(ldev->dmb_ht, tmp_node, list, token) {
+ if (tmp_node->token == token) {
+ dmb_node = tmp_node;
+ break;
+ }
+ }
+ if (!dmb_node) {
+ read_unlock(&ldev->dmb_ht_lock);
+ return -EINVAL;
+ }
+ read_unlock(&ldev->dmb_ht_lock);
+
+ if (refcount_dec_and_test(&dmb_node->refcnt))
+ __smc_lo_unregister_dmb(ldev, dmb_node);
return 0;
}
@@ -170,8 +249,22 @@ static int smc_lo_move_data(struct smcd_dev *smcd, u64 dmb_tok,
{
struct smc_lo_dmb_node *rmb_node = NULL, *tmp_node;
struct smc_lo_dev *ldev = smcd->priv;
-
- read_lock(&ldev->dmb_ht_lock);
+ struct smc_connection *conn;
+
+ if (!sf)
+ /* since sndbuf is merged with peer DMB, there is
+ * no need to copy data from sndbuf to peer DMB.
+ */
+ return 0;
+
+ /* read_lock_bh() is used here just to make lockdep
+ * happy, because spin_(un)lock_bh(&conn->send_lock) wraps
+ * smc_lo_move_data() and if we use read_lock() here, lockdep
+ * will complain about SOFTIRQ-safe -> SOFTIRQ-unsafe lock
+ * order detected, but in fact ldev->dmb_ht_lock will never
+ * be held in bh context.
+ */
+ read_lock_bh(&ldev->dmb_ht_lock);
hash_for_each_possible(ldev->dmb_ht, tmp_node, list, dmb_tok) {
if (tmp_node->token == dmb_tok) {
rmb_node = tmp_node;
@@ -182,19 +275,14 @@ static int smc_lo_move_data(struct smcd_dev *smcd, u64 dmb_tok,
read_unlock(&ldev->dmb_ht_lock);
return -EINVAL;
}
- read_unlock(&ldev->dmb_ht_lock);
+ read_unlock_bh(&ldev->dmb_ht_lock);
memcpy((char *)rmb_node->cpu_addr + offset, data, size);
- if (sf) {
- struct smc_connection *conn =
- smcd->conn[rmb_node->sba_idx];
-
- if (conn && !conn->killed)
- smcd_cdc_rx_handler(conn);
- else
- return -EPIPE;
- }
+ conn = smcd->conn[rmb_node->sba_idx];
+ if (!conn || conn->killed)
+ return -EPIPE;
+ smcd_cdc_rx_handler(conn);
return 0;
}
@@ -226,6 +314,9 @@ static const struct smcd_ops lo_ops = {
.query_remote_gid = smc_lo_query_rgid,
.register_dmb = smc_lo_register_dmb,
.unregister_dmb = smc_lo_unregister_dmb,
+ .support_dmb_nocopy = smc_lo_support_dmb_nocopy,
+ .attach_dmb = smc_lo_attach_dmb,
+ .detach_dmb = smc_lo_detach_dmb,
.add_vlan_id = smc_lo_add_vlan_id,
.del_vlan_id = smc_lo_del_vlan_id,
.set_vlan_required = smc_lo_set_vlan_required,
@@ -304,12 +395,17 @@ static int smc_lo_dev_init(struct smc_lo_dev *ldev)
smc_lo_generate_id(ldev);
rwlock_init(&ldev->dmb_ht_lock);
hash_init(ldev->dmb_ht);
+ atomic_set(&ldev->dmb_cnt, 0);
+ init_waitqueue_head(&ldev->ldev_release);
+
return smcd_lo_register_dev(ldev);
}
static void smc_lo_dev_exit(struct smc_lo_dev *ldev)
{
smcd_lo_unregister_dev(ldev);
+ if (atomic_read(&ldev->dmb_cnt))
+ wait_event(ldev->ldev_release, !atomic_read(&ldev->dmb_cnt));
}
static void smc_lo_dev_release(struct device *dev)
diff --git a/net/smc/smc_loopback.h b/net/smc/smc_loopback.h
index 24ab9d747613..9156a6c37e65 100644
--- a/net/smc/smc_loopback.h
+++ b/net/smc/smc_loopback.h
@@ -30,6 +30,7 @@ struct smc_lo_dmb_node {
u32 sba_idx;
void *cpu_addr;
dma_addr_t dma_addr;
+ refcount_t refcnt;
};
struct smc_lo_dev {
@@ -37,9 +38,11 @@ struct smc_lo_dev {
struct device dev;
u16 chid;
struct smcd_gid local_gid;
+ atomic_t dmb_cnt;
rwlock_t dmb_ht_lock;
DECLARE_BITMAP(sba_idx_mask, SMC_LO_MAX_DMBS);
DECLARE_HASHTABLE(dmb_ht, SMC_LO_DMBS_HASH_BITS);
+ wait_queue_head_t ldev_release;
};
#endif
--
2.32.0.3.g01195cf9f
On 07/03/2024 10:55, Wen Gu wrote:
> This patch set acts as the second part of the new version of [1] (The first
> part can be referred from [2]), the updated things of this version are listed
> at the end.
Hi Wen Gu,
thanks for the updated patch. Please givr me some time to review and
test the latest version.
Thanks
- Jan
>
> - Background
>
> SMC-D is now used in IBM z with ISM function to optimize network interconnect
> for intra-CPC communications. Inspired by this, we try to make SMC-D available
> on the non-s390 architecture through a software-implemented Emulated-ISM device,
> that is the loopback-ism device here, to accelerate inter-process or
> inter-containers communication within the same OS instance.
>
> - Design
>
> This patch set includes 3 parts:
>
> - Patch #1-#2: some prepare work for loopback-ism.
> - Patch #3-#7: implement loopback-ism device. Noted that loopback-ism now
> serves only SMC and no userspace interface exposed.
> - Patch #10-#15: memory copy optimization for intra-OS scenario.
>
> The loopback-ism device is designed as an ISMv2 device and not be limited to
> a specific net namespace, ends of both inter-process connection (1/1' in diagram
> below) or inter-container connection (2/2' in diagram below) can find the same
> available loopback-ism and choose it during the CLC handshake.
>
> Container 1 (ns1) Container 2 (ns2)
> +-----------------------------------------+ +-------------------------+
> | +-------+ +-------+ +-------+ | | +-------+ |
> | | App A | | App B | | App C | | | | App D |<-+ |
> | +-------+ +---^---+ +-------+ | | +-------+ |(2') |
> | |127.0.0.1 (1')| |192.168.0.11 192.168.0.12| |
> | (1)| +--------+ | +--------+ |(2) | | +--------+ +--------+ |
> | `-->| lo |-` | eth0 |<-` | | | lo | | eth0 | |
> +---------+--|---^-+---+-----|--+---------+ +-+--------+---+-^------+-+
> | | | |
> Kernel | | | |
> +----+-------v---+-----------v----------------------------------+---+----+
> | | TCP | |
> | | | |
> | +--------------------------------------------------------------+ |
> | |
> | +--------------+ |
> | | smc loopback | |
> +---------------------------+--------------+-----------------------------+
>
> loopback-ism device creates DMBs (shared memory) for each connection peer.
> Since data transfer occurs within the same kernel, the sndbuf of each peer
> is only a descriptor and point to the same memory region as peer DMB, so that
> the data copy from sndbuf to peer DMB can be avoided in loopback-ism case.
>
> Container 1 (ns1) Container 2 (ns2)
> +-----------------------------------------+ +-------------------------+
> | +-------+ | | +-------+ |
> | | App C |-----+ | | | App D | |
> | +-------+ | | | +-^-----+ |
> | | | | | |
> | (2) | | | (2') | |
> | | | | | |
> +---------------|-------------------------+ +----------|--------------+
> | |
> Kernel | |
> +---------------|-----------------------------------------|--------------+
> | +--------+ +--v-----+ +--------+ +--------+ |
> | |dmb_desc| |snd_desc| |dmb_desc| |snd_desc| |
> | +-----|--+ +--|-----+ +-----|--+ +--------+ |
> | +-----|--+ | +-----|--+ |
> | | DMB C | +---------------------------------| DMB D | |
> | +--------+ +--------+ |
> | |
> | +--------------+ |
> | | smc loopback | |
> +---------------------------+--------------+-----------------------------+
>
> - Benchmark Test
>
> * Test environments:
> - VM with Intel Xeon Platinum 8 core 2.50GHz, 16 GiB mem.
> - SMC sndbuf/DMB size 1MB.
>
> * Test object:
> - TCP: run on TCP loopback.
> - SMC lo: run on SMC loopback-ism.
>
> 1. ipc-benchmark (see [3])
>
> - ./<foo> -c 1000000 -s 100
>
> TCP SMC-lo
> Message
> rate (msg/s) 79287 148946(+87.86%)
>
> 2. sockperf
>
> - serv: <smc_run> taskset -c <cpu> sockperf sr --tcp
> - clnt: <smc_run> taskset -c <cpu> sockperf { tp | pp } --tcp --msg-size={ 64000 for tp | 14 for pp } -i 127.0.0.1 -t 30
>
> TCP SMC-lo
> Bandwidth(MBps) 5053.47 8195.07(+62.17%)
> Latency(us) 6.108 3.404(-44.27%)
>
> 3. nginx/wrk
>
> - serv: <smc_run> nginx
> - clnt: <smc_run> wrk -t 8 -c 1000 -d 30 http://127.0.0.1:80
>
> TCP SMC-lo
> Requests/s 178187.56 247970.92(+39.16%)
>
> 4. redis-benchmark
>
> - serv: <smc_run> redis-server
> - clnt: <smc_run> redis-benchmark -h 127.0.0.1 -q -t set,get -n 400000 -c 200 -d 1024
>
> TCP SMC-lo
> GET(Requests/s) 89067.02 123877.36(+39.08%)
> SET(Requests/s) 87700.07 131319.77(+49.73%)
>
>
> Change log:
>
> v2->v1:
> - All the patches: changed the term virtual-ISM to Emulated-ISM as defined by SMCv2.1.
> - Patch #3: optimized the description of SMC_LO config. Avoid exposing loopback-ism
> to sysfs and remove all the knobs until future definition clear.
> - Patch #3: try to make lockdep happy by using read_lock_bh() in smc_lo_move_data().
> - Patch #6: defaultly use physical contiguous DMB buffers.
> - Patch #11: defaultly enable DMB no-copy for loopback-ism and free the DMB in
> unregister_dmb or detach_dmb when dmb_node->refcnt reaches 0, instead of using
> wait_event to keep waiting in unregister_dmb.
>
> v1->RFC:
> - Patch #9: merge rx_bytes and tx_bytes as xfer_bytes statistics:
> /sys/devices/virtual/smc/loopback-ism/xfer_bytes
> - Patch #10: add support_dmb_nocopy operation to check if SMC-D device supports
> merging sndbuf with peer DMB.
> - Patch #13 & #14: introduce loopback-ism device control of DMB memory type and
> control of whether to merge sndbuf and DMB. They can be respectively set by:
> /sys/devices/virtual/smc/loopback-ism/dmb_type
> /sys/devices/virtual/smc/loopback-ism/dmb_copy
> The motivation for these two control is that a performance bottleneck was
> found when using vzalloced DMB and sndbuf is merged with DMB, and there are
> many CPUs and CONFIG_HARDENED_USERCOPY is set [4]. The bottleneck is caused
> by the lock contention in vmap_area_lock [5] which is involved in memcpy_from_msg()
> or memcpy_to_msg(). Currently, Uladzislau Rezki is working on mitigating the
> vmap lock contention [6]. It has significant effects, but using virtual memory
> still has additional overhead compared to using physical memory.
> So this new version provides controls of dmb_type and dmb_copy to suit
> different scenarios.
> - Some minor changes and comments improvements.
>
> RFC->old version([1]):
> Link: https://lore.kernel.org/netdev/[email protected]/
> - Patch #1: improve the loopback-ism dump, it shows as follows now:
> # smcd d
> FID Type PCI-ID PCHID InUse #LGs PNET-ID
> 0000 0 loopback-ism ffff No 0
> - Patch #3: introduce the smc_ism_set_v2_capable() helper and set
> smc_ism_v2_capable when ISMv2 or virtual ISM is registered,
> regardless of whether there is already a device in smcd device list.
> - Patch #3: loopback-ism will be added into /sys/devices/virtual/smc/loopback-ism/.
> - Patch #8: introduce the runtime switch /sys/devices/virtual/smc/loopback-ism/active
> to activate or deactivate the loopback-ism.
> - Patch #9: introduce the statistics of loopback-ism by
> /sys/devices/virtual/smc/loopback-ism/{{tx|rx}_tytes|dmbs_cnt}.
> - Some minor changes and comments improvements.
>
> [1] https://lore.kernel.org/netdev/[email protected]/
> [2] https://lore.kernel.org/netdev/[email protected]/
> [3] https://github.com/goldsborough/ipc-bench
> [4] https://lore.kernel.org/all/[email protected]/
> [5] https://lore.kernel.org/all/[email protected]/
> [6] https://lore.kernel.org/all/[email protected]/
>
> Wen Gu (11):
> net/smc: adapt SMC-D device dump for Emulated-ISM
> net/smc: decouple ism_client from SMC-D DMB registration
> net/smc: introduce loopback-ism for SMC intra-OS shortcut
> net/smc: implement ID-related operations of loopback-ism
> net/smc: implement some unsupported operations of loopback-ism
> net/smc: implement DMB-related operations of loopback-ism
> net/smc: register loopback-ism into SMC-D device list
> net/smc: add operations to merge sndbuf with peer DMB
> net/smc: attach or detach ghost sndbuf to peer DMB
> net/smc: adapt cursor update when sndbuf and peer DMB are merged
> net/smc: implement DMB-merged operations of loopback-ism
>
> drivers/s390/net/ism_drv.c | 2 +-
> include/net/smc.h | 7 +-
> net/smc/Kconfig | 13 +
> net/smc/Makefile | 2 +-
> net/smc/af_smc.c | 28 ++-
> net/smc/smc_cdc.c | 58 ++++-
> net/smc/smc_cdc.h | 1 +
> net/smc/smc_core.c | 61 ++++-
> net/smc/smc_core.h | 1 +
> net/smc/smc_ism.c | 71 +++++-
> net/smc/smc_ism.h | 5 +
> net/smc/smc_loopback.c | 469 +++++++++++++++++++++++++++++++++++++
> net/smc/smc_loopback.h | 52 ++++
> 13 files changed, 741 insertions(+), 29 deletions(-)
> create mode 100644 net/smc/smc_loopback.c
> create mode 100644 net/smc/smc_loopback.h
>
On 07/03/2024 10:55, Wen Gu wrote:
> This implements operations related to merging sndbuf with peer DMB in
> loopback-ism. The DMB won't be freed until no sndbuf is attached to it.
Hi Wen Gu,
while I'm still reviewing let me drop a lockdep finding.
>
> Signed-off-by: Wen Gu <[email protected]>
> ---
> net/smc/smc_loopback.c | 136 +++++++++++++++++++++++++++++++++++------
> net/smc/smc_loopback.h | 3 +
> 2 files changed, 119 insertions(+), 20 deletions(-)
>
> diff --git a/net/smc/smc_loopback.c b/net/smc/smc_loopback.c
> index 6828e0ad3e90..7e772f3772de 100644
> --- a/net/smc/smc_loopback.c
> +++ b/net/smc/smc_loopback.c
[...]
>
> @@ -170,8 +249,22 @@ static int smc_lo_move_data(struct smcd_dev *smcd, u64 dmb_tok,
> {
> struct smc_lo_dmb_node *rmb_node = NULL, *tmp_node;
> struct smc_lo_dev *ldev = smcd->priv;
> -
> - read_lock(&ldev->dmb_ht_lock);
> + struct smc_connection *conn;
> +
> + if (!sf)
> + /* since sndbuf is merged with peer DMB, there is
> + * no need to copy data from sndbuf to peer DMB.
> + */
> + return 0;
> +
> + /* read_lock_bh() is used here just to make lockdep
> + * happy, because spin_(un)lock_bh(&conn->send_lock) wraps
> + * smc_lo_move_data() and if we use read_lock() here, lockdep
> + * will complain about SOFTIRQ-safe -> SOFTIRQ-unsafe lock
> + * order detected, but in fact ldev->dmb_ht_lock will never
> + * be held in bh context.
> + */
> + read_lock_bh(&ldev->dmb_ht_lock);
> hash_for_each_possible(ldev->dmb_ht, tmp_node, list, dmb_tok) {
> if (tmp_node->token == dmb_tok) {
> rmb_node = tmp_node;
> @@ -182,19 +275,14 @@ static int smc_lo_move_data(struct smcd_dev *smcd, u64 dmb_tok,
> read_unlock(&ldev->dmb_ht_lock);
> return -EINVAL;
> }
> - read_unlock(&ldev->dmb_ht_lock);
> + read_unlock_bh(&ldev->dmb_ht_lock);
>
> memcpy((char *)rmb_node->cpu_addr + offset, data, size);
>
> - if (sf) {
> - struct smc_connection *conn =
> - smcd->conn[rmb_node->sba_idx];
> -
> - if (conn && !conn->killed)
> - smcd_cdc_rx_handler(conn);
> - else
> - return -EPIPE;
> - }
> + conn = smcd->conn[rmb_node->sba_idx];
> + if (!conn || conn->killed)
> + return -EPIPE;
> + smcd_cdc_rx_handler(conn);
[ 2385.528515] ============================================
[ 2385.528517] WARNING: possible recursive locking detected
[ 2385.528519] 6.8.0-loopback_ism-g30af186e8a18-dirty #12 Not tainted
[ 2385.528521] --------------------------------------------
[ 2385.528522] smcapp/51326 is trying to acquire lock:
[ 2385.528524] 000000018707a128 (&smc->conn.send_lock){+...}-{2:2}, at:
smc_tx_sndbuf_nonempty+0xba/0x1c0 [smc]
[ 2385.528552]
but task is already holding lock:
[ 2385.528554] 0000000187078728 (&smc->conn.send_lock){+...}-{2:2}, at:
smc_cdc_get_slot_and_msg_send+0x66/0xa0 [smc]
[ 2385.528568]
other info that might help us debug this:
[ 2385.528570] Possible unsafe locking scenario:
[ 2385.528572] CPU0
[ 2385.528573] ----
[ 2385.528574] lock(&smc->conn.send_lock);
[ 2385.528576] lock(&smc->conn.send_lock);
[ 2385.528579]
*** DEADLOCK ***
[ 2385.528580] May be due to missing lock nesting notation
[ 2385.528582] 3 locks held by smcapp/51326:
[ 2385.528584] #0: 0000000187078378 (sk_lock-AF_SMC){+.+.}-{0:0}, at:
smc_recvmsg+0x3c/0x2b0 [smc]
[ 2385.528598] #1: 0000000187078728 (&smc->conn.send_lock){+...}-{2:2},
at: smc_cdc_get_slot_and_msg_send+0x66/0xa0 [smc]
[ 2385.528613] #2: 0000000187079ce8 (slock-AF_SMC){+...}-{2:2}, at:
smc_cdc_msg_recv+0x56/0xe0 [smc]
[ 2385.528627]
stack backtrace:
[ 2385.528660] CPU: 3 PID: 51326 Comm: smcapp Not tainted
6.8.0-loopback_ism-g30af186e8a18-dirty #12
[ 2385.528663] Hardware name: IBM 3906 M04 704 (LPAR)
[ 2385.528664] Call Trace:
[ 2385.528666] [<000000012db60788>] dump_stack_lvl+0x90/0x120
[ 2385.528671] [<000000012cc6d088>] validate_chain+0x560/0x960
[ 2385.528677] [<000000012cc6f644>] __lock_acquire+0x654/0xd58
[ 2385.528680] [<000000012cc70a04>] lock_acquire.part.0+0xec/0x260
[ 2385.528683] [<000000012cc70c24>] lock_acquire+0xac/0x170
[ 2385.528687] [<000000012dba4ccc>] _raw_spin_lock_bh+0x5c/0xb0
[ 2385.528690] [<000003ff80453b32>] smc_tx_sndbuf_nonempty+0xba/0x1c0 [smc]
[ 2385.528702] [<000003ff8045428a>] smc_tx_pending+0x32/0x60 [smc]
[ 2385.528712] [<000003ff80451f02>] smc_cdc_msg_recv_action+0x3c2/0x528
[smc]
[ 2385.528723] [<000003ff804520cc>] smc_cdc_msg_recv+0x64/0xe0 [smc]
[ 2385.528734] [<000003ff80452a4c>] smcd_cdc_rx_handler+0x64/0x70 [smc]
[ 2385.528745] [<000003ff80459f7e>] smc_lo_move_data+0xde/0x100 [smc]
[ 2385.528755] [<000003ff804533e0>] smcd_tx_ism_write+0x68/0x90 [smc]
[ 2385.528766] [<000003ff804528a4>] smcd_cdc_msg_send+0x74/0x118 [smc]
[ 2385.528776] [<000003ff804529b8>]
smc_cdc_get_slot_and_msg_send+0x70/0xa0 [smc]
[ 2385.528788] [<000003ff804543ec>] smc_tx_consumer_update+0xe4/0x1b0 [smc]
[ 2385.528798] [<000003ff8045458e>] smc_rx_update_consumer+0x86/0x170 [smc]
[ 2385.528809] [<000003ff80455ba8>] smc_rx_recvmsg+0x3b8/0x6e8 [smc]
[ 2385.528820] [<000003ff804388a4>] smc_recvmsg+0xdc/0x2b0 [smc]
[ 2385.528831] [<000000012d8a6d58>] sock_recvmsg+0x70/0xb0
[ 2385.528837] [<000000012d8aa0c8>] __sys_recvfrom+0xa8/0x128
[ 2385.528840] [<000000012d8ab3ca>] __do_sys_socketcall+0x1ca/0x398
[ 2385.528844] [<000000012db8d4c4>] __do_syscall+0x244/0x308
[ 2385.528847] [<000000012dba6140>] system_call+0x70/0x98
[ 2385.528850] INFO: lockdep is turned off.
I did not investigate deeper, yet. Just an early heads up that there
might be something broken.
Thank you
- Jan
> return 0;
> }
>
> @@ -226,6 +314,9 @@ static const struct smcd_ops lo_ops = {
> .query_remote_gid = smc_lo_query_rgid,
> .register_dmb = smc_lo_register_dmb,
> .unregister_dmb = smc_lo_unregister_dmb,
> + .support_dmb_nocopy = smc_lo_support_dmb_nocopy,
> + .attach_dmb = smc_lo_attach_dmb,
> + .detach_dmb = smc_lo_detach_dmb,
> .add_vlan_id = smc_lo_add_vlan_id,
> .del_vlan_id = smc_lo_del_vlan_id,
> .set_vlan_required = smc_lo_set_vlan_required,
> @@ -304,12 +395,17 @@ static int smc_lo_dev_init(struct smc_lo_dev *ldev)
> smc_lo_generate_id(ldev);
> rwlock_init(&ldev->dmb_ht_lock);
> hash_init(ldev->dmb_ht);
> + atomic_set(&ldev->dmb_cnt, 0);
> + init_waitqueue_head(&ldev->ldev_release);
> +
> return smcd_lo_register_dev(ldev);
> }
>
> static void smc_lo_dev_exit(struct smc_lo_dev *ldev)
> {
> smcd_lo_unregister_dev(ldev);
> + if (atomic_read(&ldev->dmb_cnt))
> + wait_event(ldev->ldev_release, !atomic_read(&ldev->dmb_cnt));
> }
>
> static void smc_lo_dev_release(struct device *dev)
> diff --git a/net/smc/smc_loopback.h b/net/smc/smc_loopback.h
> index 24ab9d747613..9156a6c37e65 100644
> --- a/net/smc/smc_loopback.h
> +++ b/net/smc/smc_loopback.h
> @@ -30,6 +30,7 @@ struct smc_lo_dmb_node {
> u32 sba_idx;
> void *cpu_addr;
> dma_addr_t dma_addr;
> + refcount_t refcnt;
> };
>
> struct smc_lo_dev {
> @@ -37,9 +38,11 @@ struct smc_lo_dev {
> struct device dev;
> u16 chid;
> struct smcd_gid local_gid;
> + atomic_t dmb_cnt;
> rwlock_t dmb_ht_lock;
> DECLARE_BITMAP(sba_idx_mask, SMC_LO_MAX_DMBS);
> DECLARE_HASHTABLE(dmb_ht, SMC_LO_DMBS_HASH_BITS);
> + wait_queue_head_t ldev_release;
> };
> #endif
>
On 2024/3/12 16:06, Jan Karcher wrote:
>
>
> On 07/03/2024 10:55, Wen Gu wrote:
>> This implements operations related to merging sndbuf with peer DMB in
>> loopback-ism. The DMB won't be freed until no sndbuf is attached to it.
>
> Hi Wen Gu,
>
> while I'm still reviewing let me drop a lockdep finding.
>
>>
>> Signed-off-by: Wen Gu <[email protected]>
>> ---
>> net/smc/smc_loopback.c | 136 +++++++++++++++++++++++++++++++++++------
>> net/smc/smc_loopback.h | 3 +
>> 2 files changed, 119 insertions(+), 20 deletions(-)
>>
>> diff --git a/net/smc/smc_loopback.c b/net/smc/smc_loopback.c
>> index 6828e0ad3e90..7e772f3772de 100644
>> --- a/net/smc/smc_loopback.c
>> +++ b/net/smc/smc_loopback.c
>
> [...]
>
>> @@ -170,8 +249,22 @@ static int smc_lo_move_data(struct smcd_dev *smcd, u64 dmb_tok,
>> {
>> struct smc_lo_dmb_node *rmb_node = NULL, *tmp_node;
>> struct smc_lo_dev *ldev = smcd->priv;
>> -
>> - read_lock(&ldev->dmb_ht_lock);
>> + struct smc_connection *conn;
>> +
>> + if (!sf)
>> + /* since sndbuf is merged with peer DMB, there is
>> + * no need to copy data from sndbuf to peer DMB.
>> + */
>> + return 0;
>> +
>> + /* read_lock_bh() is used here just to make lockdep
>> + * happy, because spin_(un)lock_bh(&conn->send_lock) wraps
>> + * smc_lo_move_data() and if we use read_lock() here, lockdep
>> + * will complain about SOFTIRQ-safe -> SOFTIRQ-unsafe lock
>> + * order detected, but in fact ldev->dmb_ht_lock will never
>> + * be held in bh context.
>> + */
>> + read_lock_bh(&ldev->dmb_ht_lock);
>> hash_for_each_possible(ldev->dmb_ht, tmp_node, list, dmb_tok) {
>> if (tmp_node->token == dmb_tok) {
>> rmb_node = tmp_node;
>> @@ -182,19 +275,14 @@ static int smc_lo_move_data(struct smcd_dev *smcd, u64 dmb_tok,
>> read_unlock(&ldev->dmb_ht_lock);
>> return -EINVAL;
>> }
>> - read_unlock(&ldev->dmb_ht_lock);
>> + read_unlock_bh(&ldev->dmb_ht_lock);
>> memcpy((char *)rmb_node->cpu_addr + offset, data, size);
>> - if (sf) {
>> - struct smc_connection *conn =
>> - smcd->conn[rmb_node->sba_idx];
>> -
>> - if (conn && !conn->killed)
>> - smcd_cdc_rx_handler(conn);
>> - else
>> - return -EPIPE;
>> - }
>> + conn = smcd->conn[rmb_node->sba_idx];
>> + if (!conn || conn->killed)
>> + return -EPIPE;
>> + smcd_cdc_rx_handler(conn);
>
> [ 2385.528515] ============================================
> [ 2385.528517] WARNING: possible recursive locking detected
> [ 2385.528519] 6.8.0-loopback_ism-g30af186e8a18-dirty #12 Not tainted
> [ 2385.528521] --------------------------------------------
> [ 2385.528522] smcapp/51326 is trying to acquire lock:
> [ 2385.528524] 000000018707a128 (&smc->conn.send_lock){+...}-{2:2}, at: smc_tx_sndbuf_nonempty+0xba/0x1c0 [smc]
> [ 2385.528552]
> but task is already holding lock:
> [ 2385.528554] 0000000187078728 (&smc->conn.send_lock){+...}-{2:2}, at: smc_cdc_get_slot_and_msg_send+0x66/0xa0 [smc]
> [ 2385.528568]
> other info that might help us debug this:
> [ 2385.528570] Possible unsafe locking scenario:
>
> [ 2385.528572] CPU0
> [ 2385.528573] ----
> [ 2385.528574] lock(&smc->conn.send_lock);
> [ 2385.528576] lock(&smc->conn.send_lock);
> [ 2385.528579]
> *** DEADLOCK ***
>
> [ 2385.528580] May be due to missing lock nesting notation
>
> [ 2385.528582] 3 locks held by smcapp/51326:
> [ 2385.528584] #0: 0000000187078378 (sk_lock-AF_SMC){+.+.}-{0:0}, at: smc_recvmsg+0x3c/0x2b0 [smc]
> [ 2385.528598] #1: 0000000187078728 (&smc->conn.send_lock){+...}-{2:2}, at: smc_cdc_get_slot_and_msg_send+0x66/0xa0 [smc]
> [ 2385.528613] #2: 0000000187079ce8 (slock-AF_SMC){+...}-{2:2}, at: smc_cdc_msg_recv+0x56/0xe0 [smc]
> [ 2385.528627]
> stack backtrace:
> [ 2385.528660] CPU: 3 PID: 51326 Comm: smcapp Not tainted 6.8.0-loopback_ism-g30af186e8a18-dirty #12
> [ 2385.528663] Hardware name: IBM 3906 M04 704 (LPAR)
> [ 2385.528664] Call Trace:
> [ 2385.528666] [<000000012db60788>] dump_stack_lvl+0x90/0x120
> [ 2385.528671] [<000000012cc6d088>] validate_chain+0x560/0x960
> [ 2385.528677] [<000000012cc6f644>] __lock_acquire+0x654/0xd58
> [ 2385.528680] [<000000012cc70a04>] lock_acquire.part.0+0xec/0x260
> [ 2385.528683] [<000000012cc70c24>] lock_acquire+0xac/0x170
> [ 2385.528687] [<000000012dba4ccc>] _raw_spin_lock_bh+0x5c/0xb0
> [ 2385.528690] [<000003ff80453b32>] smc_tx_sndbuf_nonempty+0xba/0x1c0 [smc]
> [ 2385.528702] [<000003ff8045428a>] smc_tx_pending+0x32/0x60 [smc]
> [ 2385.528712] [<000003ff80451f02>] smc_cdc_msg_recv_action+0x3c2/0x528 [smc]
> [ 2385.528723] [<000003ff804520cc>] smc_cdc_msg_recv+0x64/0xe0 [smc]
> [ 2385.528734] [<000003ff80452a4c>] smcd_cdc_rx_handler+0x64/0x70 [smc]
> [ 2385.528745] [<000003ff80459f7e>] smc_lo_move_data+0xde/0x100 [smc]
> [ 2385.528755] [<000003ff804533e0>] smcd_tx_ism_write+0x68/0x90 [smc]
> [ 2385.528766] [<000003ff804528a4>] smcd_cdc_msg_send+0x74/0x118 [smc]
> [ 2385.528776] [<000003ff804529b8>] smc_cdc_get_slot_and_msg_send+0x70/0xa0 [smc]
> [ 2385.528788] [<000003ff804543ec>] smc_tx_consumer_update+0xe4/0x1b0 [smc]
> [ 2385.528798] [<000003ff8045458e>] smc_rx_update_consumer+0x86/0x170 [smc]
> [ 2385.528809] [<000003ff80455ba8>] smc_rx_recvmsg+0x3b8/0x6e8 [smc]
> [ 2385.528820] [<000003ff804388a4>] smc_recvmsg+0xdc/0x2b0 [smc]
> [ 2385.528831] [<000000012d8a6d58>] sock_recvmsg+0x70/0xb0
> [ 2385.528837] [<000000012d8aa0c8>] __sys_recvfrom+0xa8/0x128
> [ 2385.528840] [<000000012d8ab3ca>] __do_sys_socketcall+0x1ca/0x398
> [ 2385.528844] [<000000012db8d4c4>] __do_syscall+0x244/0x308
> [ 2385.528847] [<000000012dba6140>] system_call+0x70/0x98
> [ 2385.528850] INFO: lockdep is turned off.
>
>
> I did not investigate deeper, yet. Just an early heads up that there might be something broken.
>
Thank you for reminding, Jan. I think it is because that I used smcd_cdc_rx_handler(),
which may acquire conn->send_lock, in smc_lo_move_data() where the send_lock has been
held. I reproduced this issue and will fix it in v3.
Thanks!
> Thank you
> - Jan
>
>
>> return 0;
>> }
>> @@ -226,6 +314,9 @@ static const struct smcd_ops lo_ops = {
>> .query_remote_gid = smc_lo_query_rgid,
>> .register_dmb = smc_lo_register_dmb,
>> .unregister_dmb = smc_lo_unregister_dmb,
>> + .support_dmb_nocopy = smc_lo_support_dmb_nocopy,
>> + .attach_dmb = smc_lo_attach_dmb,
>> + .detach_dmb = smc_lo_detach_dmb,
>> .add_vlan_id = smc_lo_add_vlan_id,
>> .del_vlan_id = smc_lo_del_vlan_id,
>> .set_vlan_required = smc_lo_set_vlan_required,
>> @@ -304,12 +395,17 @@ static int smc_lo_dev_init(struct smc_lo_dev *ldev)
>> smc_lo_generate_id(ldev);
>> rwlock_init(&ldev->dmb_ht_lock);
>> hash_init(ldev->dmb_ht);
>> + atomic_set(&ldev->dmb_cnt, 0);
>> + init_waitqueue_head(&ldev->ldev_release);
>> +
>> return smcd_lo_register_dev(ldev);
>> }
>> static void smc_lo_dev_exit(struct smc_lo_dev *ldev)
>> {
>> smcd_lo_unregister_dev(ldev);
>> + if (atomic_read(&ldev->dmb_cnt))
>> + wait_event(ldev->ldev_release, !atomic_read(&ldev->dmb_cnt));
>> }
>> static void smc_lo_dev_release(struct device *dev)
>> diff --git a/net/smc/smc_loopback.h b/net/smc/smc_loopback.h
>> index 24ab9d747613..9156a6c37e65 100644
>> --- a/net/smc/smc_loopback.h
>> +++ b/net/smc/smc_loopback.h
>> @@ -30,6 +30,7 @@ struct smc_lo_dmb_node {
>> u32 sba_idx;
>> void *cpu_addr;
>> dma_addr_t dma_addr;
>> + refcount_t refcnt;
>> };
>> struct smc_lo_dev {
>> @@ -37,9 +38,11 @@ struct smc_lo_dev {
>> struct device dev;
>> u16 chid;
>> struct smcd_gid local_gid;
>> + atomic_t dmb_cnt;
>> rwlock_t dmb_ht_lock;
>> DECLARE_BITMAP(sba_idx_mask, SMC_LO_MAX_DMBS);
>> DECLARE_HASHTABLE(dmb_ht, SMC_LO_DMBS_HASH_BITS);
>> + wait_queue_head_t ldev_release;
>> };
>> #endif