Subject: Re: [PATCH] nvme-rdma: remove race conditions from IB signalling
To: Marta Rybczynska <mrybczyn@kalray.eu>, <keith.busch@intel.com>,
        <axboe@fb.com>, <hch@lst.de>, <sagi@grimberg.me>,
        <linux-nvme@lists.infradead.org>, <linux-kernel@vger.kernel.org>
References: <1785464880.73761005.1496655938570.JavaMail.zimbra@kalray.eu>
CC: Jason Gunthorpe <jgunthorpe@obsidianresearch.com>,
        Bart Van Assche <bart.vanassche@sandisk.com>,
        Doug Ledford <dledford@redhat.com>,
        "Leon Romanovsky" <leon@kernel.org>
From: Max Gurtovoy <maxg@mellanox.com>
Message-ID: <f732eb30-4582-1daa-21e2-c7c03b9af9e0@mellanox.com>
Date: Mon, 5 Jun 2017 17:08:16 +0300
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101
 Thunderbird/45.8.0
MIME-Version: 1.0
In-Reply-To: <1785464880.73761005.1496655938570.JavaMail.zimbra@kalray.eu>
Content-Type: text/plain; charset="windows-1252"; format=flowed
Content-Transfer-Encoding: 7bit
SpamDiagnosticOutput: 1:99
SpamDiagnosticMetadata: NSPM
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Jun 2017 14:08:20.9135
 (UTC)
X-MS-Exchange-CrossTenant-Id: a652971c-7d2e-4d9b-a6a4-d149256f461b
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=a652971c-7d2e-4d9b-a6a4-d149256f461b;Ip=[193.47.165.134];Helo=[mtlcas13.mtl.com]
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB5PR05MB1352
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
Content-Length: 3025
Lines: 82


On 6/5/2017 12:45 PM, Marta Rybczynska wrote:
> This patch improves the way the RDMA IB signalling is done
> by using atomic operations for the signalling variable. This
> avoids race conditions on sig_count.
>
> The signalling interval changes slightly and is now the
> largest power of two not larger than queue depth / 2.
>
> ilog() usage idea by Bart Van Assche.
>
> Signed-off-by: Marta Rybczynska <marta.rybczynska@kalray.eu>
> ---
>  drivers/nvme/host/rdma.c | 31 +++++++++++++++++++++----------
>  1 file changed, 21 insertions(+), 10 deletions(-)
>
> diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
> index 28bd255..80682f7 100644
> --- a/drivers/nvme/host/rdma.c
> +++ b/drivers/nvme/host/rdma.c
> @@ -88,7 +88,7 @@ enum nvme_rdma_queue_flags {
>
>  struct nvme_rdma_queue {
>         struct nvme_rdma_qe     *rsp_ring;
> -       u8                      sig_count;
> +       atomic_t                sig_count;
>         int                     queue_size;
>         size_t                  cmnd_capsule_len;
>         struct nvme_rdma_ctrl   *ctrl;
> @@ -554,6 +554,8 @@ static int nvme_rdma_init_queue(struct nvme_rdma_ctrl *ctrl,
>
>         queue->queue_size = queue_size;
>
> +       atomic_set(&queue->sig_count, 0);
> +
>         queue->cm_id = rdma_create_id(&init_net, nvme_rdma_cm_handler, queue,
>                         RDMA_PS_TCP, IB_QPT_RC);
>         if (IS_ERR(queue->cm_id)) {
> @@ -1038,17 +1040,26 @@ static void nvme_rdma_send_done(struct ib_cq *cq, struct ib_wc *wc)
>                 nvme_rdma_wr_error(cq, wc, "SEND");
>  }
>
> -static inline int nvme_rdma_queue_sig_limit(struct nvme_rdma_queue *queue)
> +static inline int nvme_rdma_init_sig_count(int queue_size)
>  {
> -       int sig_limit;
> -
> -       /*
> -        * We signal completion every queue depth/2 and also handle the
> -        * degenerated case of a  device with queue_depth=1, where we
> -        * would need to signal every message.
> +       /* We want to signal completion at least every queue depth/2.
> +        * This returns the largest power of two that is not above half
> +        * of (queue size + 1) to optimize (avoid divisions).
>          */
> -       sig_limit = max(queue->queue_size / 2, 1);
> -       return (++queue->sig_count % sig_limit) == 0;
> +       return 1 << ilog2((queue_size + 1) / 2);
> +}
> +
> +static inline bool nvme_rdma_queue_sig_limit(struct nvme_rdma_queue *queue)
> +{
> +       int count, limit;
> +
> +       limit = nvme_rdma_init_sig_count(queue->queue_size);

Maybe I'm missing something, but why we need to calc limit each time in 
data path (not a big deal but we can avoid that)?
Can you add queue->sig_limit that will be calculated once at queue 
allocation ?

> +       count = atomic_inc_return(&queue->sig_count);
> +
> +       /* Signal if count is a multiple of limit */
> +       if ((count & (limit - 1)) == 0)
> +               return true;
> +       return false;
>  }
>
>  static int nvme_rdma_post_send(struct nvme_rdma_queue *queue,
>