When using the max_mw_size parameter of ntb_transport to limit the size of
the Memory windows, communication cannot be established and the queues
freeze.
This is because the mw_size that's reported to the peer is correctly
limited but the size used locally is not. So the MW is initialized
with a buffer smaller than the window but the TX side is using the
full window. This means the TX side will be writing to a region of the
window that points nowhere.
This is easily fixed by applying the same limit to tx_size in
ntb_transport_init_queue().
Fixes: e26a5843f7f5 ("NTB: Split ntb_hw_intel and ntb_transport drivers")
Signed-off-by: Logan Gunthorpe <[email protected]>
Acked-by: Allen Hubbe <[email protected]>
Cc: Jon Mason <[email protected]>
Cc: Dave Jiang <[email protected]>
---
drivers/ntb/ntb_transport.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/drivers/ntb/ntb_transport.c b/drivers/ntb/ntb_transport.c
index 045e3dd4750e..9878c48826e3 100644
--- a/drivers/ntb/ntb_transport.c
+++ b/drivers/ntb/ntb_transport.c
@@ -1003,6 +1003,9 @@ static int ntb_transport_init_queue(struct ntb_transport_ctx *nt,
mw_base = nt->mw_vec[mw_num].phys_addr;
mw_size = nt->mw_vec[mw_num].phys_size;
+ if (max_mw_size && mw_size > max_mw_size)
+ mw_size = max_mw_size;
+
tx_size = (unsigned int)mw_size / num_qps_mw;
qp_offset = tx_size * (qp_num / mw_count);
--
2.11.0
With Switchtec hardware, the buffer used for a memory window must be
aligned to its size (the hardware only replaces the lower bits). In
certain circumstances dma_alloc_coherent() will not provide a buffer
that adheres to this requirement like when using the CMA and
CONFIG_CMA_ALIGNMENT is set lower than the buffer size.
When we get an unaligned buffer mw_set_trans() should return an error.
We also log an error so we know the cause of the problem.
Signed-off-by: Logan Gunthorpe <[email protected]>
Cc: Jon Mason <[email protected]>
---
v2: Use IS_ALIGNED macro as suggested by Allen
drivers/ntb/hw/mscc/ntb_hw_switchtec.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/drivers/ntb/hw/mscc/ntb_hw_switchtec.c b/drivers/ntb/hw/mscc/ntb_hw_switchtec.c
index bcd5b6fb3800..6c6f991999b5 100644
--- a/drivers/ntb/hw/mscc/ntb_hw_switchtec.c
+++ b/drivers/ntb/hw/mscc/ntb_hw_switchtec.c
@@ -320,6 +320,19 @@ static int switchtec_ntb_mw_set_trans(struct ntb_dev *ntb, int pidx, int widx,
if (xlate_pos < 12)
return -EINVAL;
+ if (!IS_ALIGNED(addr, BIT_ULL(xlate_pos))) {
+ /*
+ * In certain circumstances we can get a buffer that is
+ * not aligned to its size. (Most of the time
+ * dma_alloc_coherent ensures this). This can happen when
+ * using large buffers allocated by the CMA
+ * (see CMA_CONFIG_ALIGNMENT)
+ */
+ dev_err(&sndev->stdev->dev,
+ "ERROR: Memory window address is not aligned to it's size!\n");
+ return -EINVAL;
+ }
+
rc = switchtec_ntb_part_op(sndev, ctl, NTB_CTRL_PART_OP_LOCK,
NTB_CTRL_PART_STATUS_LOCKED);
if (rc)
--
2.11.0
On Mon, Dec 18, 2017 at 11:25:06AM -0700, Logan Gunthorpe wrote:
> With Switchtec hardware, the buffer used for a memory window must be
> aligned to its size (the hardware only replaces the lower bits). In
> certain circumstances dma_alloc_coherent() will not provide a buffer
> that adheres to this requirement like when using the CMA and
> CONFIG_CMA_ALIGNMENT is set lower than the buffer size.
>
> When we get an unaligned buffer mw_set_trans() should return an error.
> We also log an error so we know the cause of the problem.
>
Applied to ntb-next.
Thanks,
Jon
> Signed-off-by: Logan Gunthorpe <[email protected]>
> Cc: Jon Mason <[email protected]>
> ---
>
> v2: Use IS_ALIGNED macro as suggested by Allen
>
> drivers/ntb/hw/mscc/ntb_hw_switchtec.c | 13 +++++++++++++
> 1 file changed, 13 insertions(+)
>
> diff --git a/drivers/ntb/hw/mscc/ntb_hw_switchtec.c b/drivers/ntb/hw/mscc/ntb_hw_switchtec.c
> index bcd5b6fb3800..6c6f991999b5 100644
> --- a/drivers/ntb/hw/mscc/ntb_hw_switchtec.c
> +++ b/drivers/ntb/hw/mscc/ntb_hw_switchtec.c
> @@ -320,6 +320,19 @@ static int switchtec_ntb_mw_set_trans(struct ntb_dev *ntb, int pidx, int widx,
> if (xlate_pos < 12)
> return -EINVAL;
>
> + if (!IS_ALIGNED(addr, BIT_ULL(xlate_pos))) {
> + /*
> + * In certain circumstances we can get a buffer that is
> + * not aligned to its size. (Most of the time
> + * dma_alloc_coherent ensures this). This can happen when
> + * using large buffers allocated by the CMA
> + * (see CMA_CONFIG_ALIGNMENT)
> + */
> + dev_err(&sndev->stdev->dev,
> + "ERROR: Memory window address is not aligned to it's size!\n");
> + return -EINVAL;
> + }
> +
> rc = switchtec_ntb_part_op(sndev, ctl, NTB_CTRL_PART_OP_LOCK,
> NTB_CTRL_PART_STATUS_LOCKED);
> if (rc)
> --
> 2.11.0
>
> --
> You received this message because you are subscribed to the Google Groups "linux-ntb" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
> To post to this group, send email to [email protected].
> To view this discussion on the web visit https://groups.google.com/d/msgid/linux-ntb/20171218182506.5219-2-logang%40deltatee.com.
> For more options, visit https://groups.google.com/d/optout.
On Mon, Dec 18, 2017 at 11:25:05AM -0700, Logan Gunthorpe wrote:
> When using the max_mw_size parameter of ntb_transport to limit the size of
> the Memory windows, communication cannot be established and the queues
> freeze.
>
> This is because the mw_size that's reported to the peer is correctly
> limited but the size used locally is not. So the MW is initialized
> with a buffer smaller than the window but the TX side is using the
> full window. This means the TX side will be writing to a region of the
> window that points nowhere.
>
> This is easily fixed by applying the same limit to tx_size in
> ntb_transport_init_queue().
Applied to ntb-next.
Thanks,
Jon
>
> Fixes: e26a5843f7f5 ("NTB: Split ntb_hw_intel and ntb_transport drivers")
> Signed-off-by: Logan Gunthorpe <[email protected]>
> Acked-by: Allen Hubbe <[email protected]>
> Cc: Jon Mason <[email protected]>
> Cc: Dave Jiang <[email protected]>
> ---
> drivers/ntb/ntb_transport.c | 3 +++
> 1 file changed, 3 insertions(+)
>
> diff --git a/drivers/ntb/ntb_transport.c b/drivers/ntb/ntb_transport.c
> index 045e3dd4750e..9878c48826e3 100644
> --- a/drivers/ntb/ntb_transport.c
> +++ b/drivers/ntb/ntb_transport.c
> @@ -1003,6 +1003,9 @@ static int ntb_transport_init_queue(struct ntb_transport_ctx *nt,
> mw_base = nt->mw_vec[mw_num].phys_addr;
> mw_size = nt->mw_vec[mw_num].phys_size;
>
> + if (max_mw_size && mw_size > max_mw_size)
> + mw_size = max_mw_size;
> +
> tx_size = (unsigned int)mw_size / num_qps_mw;
> qp_offset = tx_size * (qp_num / mw_count);
>
> --
> 2.11.0
>
> --
> You received this message because you are subscribed to the Google Groups "linux-ntb" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
> To post to this group, send email to [email protected].
> To view this discussion on the web visit https://groups.google.com/d/msgid/linux-ntb/20171218182506.5219-1-logang%40deltatee.com.
> For more options, visit https://groups.google.com/d/optout.