Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Received-SPF: Pass (protection.outlook.com: domain of mellanox.com designates
 193.47.165.134 as permitted sender) receiver=protection.outlook.com;
 client-ip=193.47.165.134; helo=mtlcas13.mtl.com;
Subject: Re: [PATCH V2] nvme-rdma: fix double free in nvme_rdma_free_queue
To:     Jianchao Wang <jianchao.w.wang@oracle.com>,
        <keith.busch@intel.com>, <axboe@fb.com>, <hch@lst.de>,
        <sagi@grimberg.me>
CC:     <linux-kernel@vger.kernel.org>, <linux-nvme@lists.infradead.org>
References: <1525945328-1908-1-git-send-email-jianchao.w.wang@oracle.com>
From:   Max Gurtovoy <maxg@mellanox.com>
Message-ID: <c32eef8a-ae1e-ce7e-ee39-eee60192e517@mellanox.com>
Date:   Wed, 16 May 2018 15:18:47 +0300
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101
 Thunderbird/52.7.0
MIME-Version: 1.0
In-Reply-To: <1525945328-1908-1-git-send-email-jianchao.w.wang@oracle.com>
Content-Type: text/plain; charset="windows-1255"; format=flowed
Content-Language: he
Content-Transfer-Encoding: 7bit
SpamDiagnosticOutput: 1:99
SpamDiagnosticMetadata: NSPM
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 May 2018 12:19:34.4830
 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: 63054344-bb50-4403-320e-08d5bb274923
X-MS-Exchange-CrossTenant-Id: a652971c-7d2e-4d9b-a6a4-d149256f461b
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=a652971c-7d2e-4d9b-a6a4-d149256f461b;Ip=[193.47.165.134];Helo=[mtlcas13.mtl.com]
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR05MB1760
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Jianchao,

On 5/10/2018 12:42 PM, Jianchao Wang wrote:
> BUG: KASAN: double-free or invalid-free in nvme_rdma_free_queue+0xf6/0x110 [nvme_rdma]
> Workqueue: nvme-reset-wq nvme_rdma_reset_ctrl_work [nvme_rdma]
> Call Trace:
>   dump_stack+0x91/0xeb
>   print_address_description+0x6b/0x290
>   kasan_report_invalid_free+0x55/0x80
>   __kasan_slab_free+0x176/0x190
>   kfree+0xeb/0x310
>   nvme_rdma_free_queue+0xf6/0x110 [nvme_rdma]
>   nvme_rdma_configure_admin_queue+0x1a3/0x4d0 [nvme_rdma]
>   nvme_rdma_reset_ctrl_work+0x4e/0xd0 [nvme_rdma]
>   process_one_work+0x3ca/0xaa0
>   worker_thread+0x4e2/0x6c0
>   kthread+0x18d/0x1e0
>   ret_from_fork+0x24/0x30
> 
> The double free is on ctrl->async_event_sqe.
> If any case fails before ctrl->async_event_sqe is allocated in
> nvme_rdma_configure_admin_queue, nvme_rdma_free_queue will be
> invoked. However, at the moment, the ctrl->async_event_sqe has
> not been allocated because it has been freed in
> nvme_rdma_reset_ctrl_work
>    -> nvme_rdma_shutdown_ctrl
>      ->nvme_rdma_destroy_admin_queue
>        -> nvme_rdma_free_queue
> 
> Signed-off-by: Jianchao Wang <jianchao.w.wang@oracle.com>
> ---
> 
> V2:
> handle it in nvme_rdma_free_queue and add some comment to explain it.

I don't know exactly what Christoph meant but IMO the best place to 
allocate it is in nvme_rdma_alloc_queue just before calling

"set_bit(NVME_RDMA_Q_ALLOCATED, &queue->flags);"

then you will never get to double free since we clear the 
NVME_RDMA_Q_ALLOCATED bit in the beginning of nvme_rdma_free_queue.

> 
>   drivers/nvme/host/rdma.c | 15 ++++++++++++---
>   1 file changed, 12 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
> index 966e0dd..fa5cf87 100644
> --- a/drivers/nvme/host/rdma.c
> +++ b/drivers/nvme/host/rdma.c
> @@ -561,9 +561,18 @@ static void nvme_rdma_free_queue(struct nvme_rdma_queue *queue)
>   		return;
>   
>   	if (nvme_rdma_queue_idx(queue) == 0) {
> -		nvme_rdma_free_qe(queue->device->dev,
> -			&queue->ctrl->async_event_sqe,
> -			sizeof(struct nvme_command), DMA_TO_DEVICE);
> +		/*
> +		 * async_event_sqe is not allocated in nvme_rdma_alloc_queue.
> +		 * so there are cases where NVME_RDMA_Q_ALLOCATED is set, but
> +		 * async_event_sqe is not allocated. To avoid double free, set
> +		 * async_event_sqe.data to NULL to indicate it has been freed.
> +		 */
> +		if (queue->ctrl->async_event_sqe.data) {
> +			nvme_rdma_free_qe(queue->device->dev,
> +				&queue->ctrl->async_event_sqe,
> +				sizeof(struct nvme_command), DMA_TO_DEVICE);
> +			queue->ctrl->async_event_sqe.data = NULL;
> +		}
>   	}
>   
>   	nvme_rdma_destroy_queue_ib(queue);
> 


-Max.