Received: by 2002:a05:6a10:8c0a:0:0:0:0 with SMTP id go10csp1015146pxb; Thu, 25 Feb 2021 23:39:37 -0800 (PST) X-Google-Smtp-Source: ABdhPJzXUCmcK15SH1V+rGADv4s5307/A/gSHdKU4vqwp6LuxcLsgEEvjf4RkAEumYeHvdf5nejj X-Received: by 2002:a17:906:b4c:: with SMTP id v12mr1957857ejg.330.1614325177571; Thu, 25 Feb 2021 23:39:37 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1614325177; cv=none; d=google.com; s=arc-20160816; b=SSXcvZFh9uWJLAFH0omKxLo/bg2Yh4wmwolnbPWoJoQIW1gadbRQhw9Meh0QJL8JD1 y5xOiwXx/jK8h/5yi4N63PrxUfGYSKQSn9dMtf+lDGN6r/hpLuWv8tvI0vjTqbgiXLmw lyyG/YKXmp2JmlxxMAV+3LKH708dddZcztiPF4r9sj+Uh6LWaknK2TIediv8aClUzjN6 MRdVETxgo32TJ7gV2mWINsbwkdTwh3HhumqNGgzSnSGu3fMoly7KMi6o/VntVJU+N4Np FcYfDzyjFN+VZM4Hq/sFErB5SPCDU3qULi/rJhpNqs6lbs5HMxPZ128olYa0Z2gAjUa3 TURg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:dlp-filter:cms-type :content-transfer-encoding:date:message-id:in-reply-to:cc:to:from :sender:reply-to:subject:mime-version:dkim-signature:dkim-filter; bh=aJROdop8zf/XvGt8nTrnpWGLxMJZ8KkNWLDt7LVy0IY=; b=EgRO0xMQ95Hit4f8bqI3MFpVLU01CNKCZdO6qvLiBd/MFuGb0zzpxXUj/P/ANebseH CL5lXupKhB3mn55jTHWO/0u049TwqIbOQcozpxmyl4UnM6l27UQRlYg1n+x6LAVuW5NM Ftee53tlpKdI7HiGuNNoMaw/KNKqHhmcocIsVk3mpnwDJo+W6XWqWVSnNzCohT+T2gpD dgxNNdQSabc4NPfiKkmYSKFJ6szpS4fbSS60W3lWfrQZgJQtbFrAqmfFTAUiJZjL54Jh Lxc0tg11YHvULIMzeKI3MEHLfuaotA1FUwDe7YkmKYKauz0uQsVrwKedXLTC5Td2/I7P fBmQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@samsung.com header.s=mail20170921 header.b=LeBdkNDa; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=samsung.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id g7si5115672edh.31.2021.02.25.23.39.14; Thu, 25 Feb 2021 23:39:37 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@samsung.com header.s=mail20170921 header.b=LeBdkNDa; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=samsung.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230132AbhBZHgo (ORCPT + 99 others); Fri, 26 Feb 2021 02:36:44 -0500 Received: from mailout1.samsung.com ([203.254.224.24]:62289 "EHLO mailout1.samsung.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230087AbhBZHft (ORCPT ); Fri, 26 Feb 2021 02:35:49 -0500 Received: from epcas2p3.samsung.com (unknown [182.195.41.55]) by mailout1.samsung.com (KnoxPortal) with ESMTP id 20210226073501epoutp0182a22237ead909aaa51895627beb9f61~nO-HpDT4A0666306663epoutp01C for ; Fri, 26 Feb 2021 07:35:01 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.samsung.com 20210226073501epoutp0182a22237ead909aaa51895627beb9f61~nO-HpDT4A0666306663epoutp01C DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1614324901; bh=aJROdop8zf/XvGt8nTrnpWGLxMJZ8KkNWLDt7LVy0IY=; h=Subject:Reply-To:From:To:CC:In-Reply-To:Date:References:From; b=LeBdkNDajCS1FuevPImegy11NfXahFqvSXMM3y8bhp9Q6kLxi1W2Q3an55pApJ3my yN58nVwV+dVADh911qsvtn3Wu2DZZMf4l8Mn5Wnjcfy+fLhBPmQHLKElf0bPP8fifP vmSOhVd3A8c9JbAycg6qJX2D58MCzrApo1C2HQfE= Received: from epsnrtp1.localdomain (unknown [182.195.42.162]) by epcas2p1.samsung.com (KnoxPortal) with ESMTP id 20210226073500epcas2p1c4c3fe0576b838435c518adedc17056c~nO-HAtcSL2984029840epcas2p10; Fri, 26 Feb 2021 07:35:00 +0000 (GMT) Received: from epsmges2p2.samsung.com (unknown [182.195.40.183]) by epsnrtp1.localdomain (Postfix) with ESMTP id 4Dn1dZ5XcCz4x9Q2; Fri, 26 Feb 2021 07:34:58 +0000 (GMT) X-AuditID: b6c32a46-1d9ff7000000dbf8-d8-6038a4a29782 Received: from epcas2p3.samsung.com ( [182.195.41.55]) by epsmges2p2.samsung.com (Symantec Messaging Gateway) with SMTP id 4B.55.56312.2A4A8306; Fri, 26 Feb 2021 16:34:58 +0900 (KST) Mime-Version: 1.0 Subject: [PATCH v25 3/4] scsi: ufs: Prepare HPB read for cached sub-region Reply-To: daejun7.park@samsung.com Sender: Daejun Park From: Daejun Park To: Daejun Park , Greg KH , "avri.altman@wdc.com" , "jejb@linux.ibm.com" , "martin.petersen@oracle.com" , "asutoshd@codeaurora.org" , "stanley.chu@mediatek.com" , "cang@codeaurora.org" , "bvanassche@acm.org" , "huobean@gmail.com" , ALIM AKHTAR , Javier Gonzalez CC: "linux-scsi@vger.kernel.org" , "linux-kernel@vger.kernel.org" , JinHwan Park , SEUNGUK SHIN , Sung-Jun Park , yongmyung lee , Jinyoung CHOI , BoRam Shin X-Priority: 3 X-Content-Kind-Code: NORMAL In-Reply-To: <20210226073233epcms2p80fca2dffabea03143a9414838f757633@epcms2p8> X-CPGS-Detection: blocking_info_exchange X-Drm-Type: N,general X-Msg-Generator: Mail X-Msg-Type: PERSONAL X-Reply-Demand: N Message-ID: <20210226073458epcms2p1789d223a2b19e641783dbbcaad3bbd4d@epcms2p1> Date: Fri, 26 Feb 2021 16:34:58 +0900 X-CMS-MailID: 20210226073458epcms2p1789d223a2b19e641783dbbcaad3bbd4d Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="utf-8" X-Sendblock-Type: AUTO_CONFIDENTIAL X-CPGSPASS: Y X-CPGSPASS: Y CMS-TYPE: 102P X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrIJsWRmVeSWpSXmKPExsWy7bCmue6iJRYJBnvWq1o8mLeNzWJv2wl2 i5c/r7JZHL79jt1i2oefzBaf1i9jtXh5SNNi1YNwi+bF69ks5pxtYLLo7d/KZvH4zmd2i0U3 tjFZ9P9rZ7G4vGsOm0X39R1sFsuP/2OyuL2Fy2Lp1puMFp3T17BYLFq4m8VB1OPyFW+Py329 TB47Z91l95iw6ACjx/65a9g9Wk7uZ/H4+PQWi0ffllWMHp83yXm0H+hmCuCKyrHJSE1MSS1S SM1Lzk/JzEu3VfIOjneONzUzMNQ1tLQwV1LIS8xNtVVy8QnQdcvMAfpQSaEsMacUKBSQWFys pG9nU5RfWpKqkJFfXGKrlFqQklNgaFigV5yYW1yal66XnJ9rZWhgYGQKVJmQk9E9YTtjwVX/ ir/N/A2Mrx26GDk4JARMJJr+WXUxcnEICexglGhe8pQZJM4rICjxd4dwFyMnh7CAl0Rn60Y2 EFtIQEli/cVZ7BBxPYlbD9cwgthsAjoS00/cZweZIyKwiUXi3oIOVhCHWeAXk8SJxx/AqiQE eCVmtD9lgbClJbYv38oIsoxTwE/iyit7iLCGxI9lvcwQtqjEzdVv2WHs98fmQ40RkWi9dxaq RlDiwc/dUHFJiWO7PzBB2PUSW+/8YgS5QUKgh1Hi8M5brBAJfYlrHRvBbuAV8JW48e8EmM0i oCpxbMp1qNtcJHY9ugQ2lFlAXmL72zngQGEW0JRYv0sfEm7KEkduscB81bDxNzs6m1mAT6Lj 8F+4+I55T6BOU5NY93M90wRG5VmIkJ6FZNcshF0LGJlXMYqlFhTnpqcWGxUYIUftJkZwWtdy 28E45e0HvUOMTByMhxglOJiVRHg3/zNNEOJNSaysSi3Kjy8qzUktPsRoCvTlRGYp0eR8YGbJ K4k3NDUyMzOwNLUwNTOyUBLnLTZ4EC8kkJ5YkpqdmlqQWgTTx8TBKdXAtGS1v+Fi3d+ys56u yQ+7vcFq75EZc7+ujdCSdy+4s2Re0U67+eqemR6PkpS7t+svdFQMY4w84br7xlL1zj3Xbs21 0py18bfxFMdpdqeeVXktnXpD7cF0w4TktDNCXIfmvPI80H/o91H15/l5Rl/vJqzK1rvGdrRu 2V2pyx7zzWse/1v+Uix746+726z6t1+M85qT4rNbtOXElmbLzM53851WyD2JF03mYew5tPza 3h+Wl07ax03y4WtNtdr2qOK+nYL3xqay9bnJGluVTcMY7UvEz6oV1wT0sCSk1NdprFwwoZ69 4NEx+xw2h2TbpP6Q1Ye8Cw6dDwg4mx/+ICulXM+cUZJtZST/x1yrSTU9SizFGYmGWsxFxYkA wl87RXQEAAA= DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20210226073233epcms2p80fca2dffabea03143a9414838f757633 References: <20210226073233epcms2p80fca2dffabea03143a9414838f757633@epcms2p8> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org This patch changes the read I/O to the HPB read I/O. If the logical address of the read I/O belongs to active sub-region, the HPB driver modifies the read I/O command to HPB read. It modifies the UPIU command of UFS instead of modifying the existing SCSI command. In the HPB version 1.0, the maximum read I/O size that can be converted to HPB read is 4KB. The dirty map of the active sub-region prevents an incorrect HPB read that has stale physical page number which is updated by previous write I/O. Reviewed-by: Can Guo Reviewed-by: Bart Van Assche Acked-by: Avri Altman Tested-by: Bean Huo Signed-off-by: Daejun Park --- drivers/scsi/ufs/ufshcd.c | 2 + drivers/scsi/ufs/ufshpb.c | 253 +++++++++++++++++++++++++++++++++++++- drivers/scsi/ufs/ufshpb.h | 2 + 3 files changed, 254 insertions(+), 3 deletions(-) diff --git a/drivers/scsi/ufs/ufshcd.c b/drivers/scsi/ufs/ufshcd.c index 5852ff44c3cc..851c01a26207 100644 --- a/drivers/scsi/ufs/ufshcd.c +++ b/drivers/scsi/ufs/ufshcd.c @@ -2656,6 +2656,8 @@ static int ufshcd_queuecommand(struct Scsi_Host *host, struct scsi_cmnd *cmd) lrbp->req_abort_skip = false; + ufshpb_prep(hba, lrbp); + ufshcd_comp_scsi_upiu(hba, lrbp); err = ufshcd_map_sg(hba, lrbp); diff --git a/drivers/scsi/ufs/ufshpb.c b/drivers/scsi/ufs/ufshpb.c index 8abadb0e010a..c75a6816a03f 100644 --- a/drivers/scsi/ufs/ufshpb.c +++ b/drivers/scsi/ufs/ufshpb.c @@ -46,6 +46,29 @@ static void ufshpb_set_state(struct ufshpb_lu *hpb, int state) atomic_set(&hpb->hpb_state, state); } +static int ufshpb_is_valid_srgn(struct ufshpb_region *rgn, + struct ufshpb_subregion *srgn) +{ + return rgn->rgn_state != HPB_RGN_INACTIVE && + srgn->srgn_state == HPB_SRGN_VALID; +} + +static bool ufshpb_is_read_cmd(struct scsi_cmnd *cmd) +{ + return req_op(cmd->request) == REQ_OP_READ; +} + +static bool ufshpb_is_write_or_discard_cmd(struct scsi_cmnd *cmd) +{ + return op_is_write(req_op(cmd->request)) || + op_is_discard(req_op(cmd->request)); +} + +static bool ufshpb_is_support_chunk(int transfer_len) +{ + return transfer_len <= HPB_MULTI_CHUNK_HIGH; +} + static bool ufshpb_is_general_lun(int lun) { return lun < UFS_UPIU_MAX_UNIT_NUM_ID; @@ -80,8 +103,8 @@ static void ufshpb_kick_map_work(struct ufshpb_lu *hpb) } static bool ufshpb_is_hpb_rsp_valid(struct ufs_hba *hba, - struct ufshcd_lrb *lrbp, - struct utp_hpb_rsp *rsp_field) + struct ufshcd_lrb *lrbp, + struct utp_hpb_rsp *rsp_field) { /* Check HPB_UPDATE_ALERT */ if (!(lrbp->ucd_rsp_ptr->header.dword_2 & @@ -107,6 +130,230 @@ static bool ufshpb_is_hpb_rsp_valid(struct ufs_hba *hba, return true; } +static void ufshpb_set_ppn_dirty(struct ufshpb_lu *hpb, int rgn_idx, + int srgn_idx, int srgn_offset, int cnt) +{ + struct ufshpb_region *rgn; + struct ufshpb_subregion *srgn; + int set_bit_len; + int bitmap_len; + +next_srgn: + rgn = hpb->rgn_tbl + rgn_idx; + srgn = rgn->srgn_tbl + srgn_idx; + + if (likely(!srgn->is_last)) + bitmap_len = hpb->entries_per_srgn; + else + bitmap_len = hpb->last_srgn_entries; + + if ((srgn_offset + cnt) > bitmap_len) + set_bit_len = bitmap_len - srgn_offset; + else + set_bit_len = cnt; + + if (rgn->rgn_state != HPB_RGN_INACTIVE && + srgn->srgn_state == HPB_SRGN_VALID) + bitmap_set(srgn->mctx->ppn_dirty, srgn_offset, set_bit_len); + + srgn_offset = 0; + if (++srgn_idx == hpb->srgns_per_rgn) { + srgn_idx = 0; + rgn_idx++; + } + + cnt -= set_bit_len; + if (cnt > 0) + goto next_srgn; +} + +static bool ufshpb_test_ppn_dirty(struct ufshpb_lu *hpb, int rgn_idx, + int srgn_idx, int srgn_offset, int cnt) +{ + struct ufshpb_region *rgn; + struct ufshpb_subregion *srgn; + int bitmap_len; + int bit_len; + +next_srgn: + rgn = hpb->rgn_tbl + rgn_idx; + srgn = rgn->srgn_tbl + srgn_idx; + + if (likely(!srgn->is_last)) + bitmap_len = hpb->entries_per_srgn; + else + bitmap_len = hpb->last_srgn_entries; + + if (!ufshpb_is_valid_srgn(rgn, srgn)) + return true; + + /* + * If the region state is active, mctx must be allocated. + * In this case, check whether the region is evicted or + * mctx allcation fail. + */ + if (unlikely(!srgn->mctx)) { + dev_err(&hpb->sdev_ufs_lu->sdev_dev, + "no mctx in region %d subregion %d.\n", + srgn->rgn_idx, srgn->srgn_idx); + return true; + } + + if ((srgn_offset + cnt) > bitmap_len) + bit_len = bitmap_len - srgn_offset; + else + bit_len = cnt; + + if (find_next_bit(srgn->mctx->ppn_dirty, + bit_len, srgn_offset) >= srgn_offset) + return true; + + srgn_offset = 0; + if (++srgn_idx == hpb->srgns_per_rgn) { + srgn_idx = 0; + rgn_idx++; + } + + cnt -= bit_len; + if (cnt > 0) + goto next_srgn; + + return false; +} + +static int ufshpb_fill_ppn_from_page(struct ufshpb_lu *hpb, + struct ufshpb_map_ctx *mctx, int pos, + int len, u64 *ppn_buf) +{ + struct page *page; + int index, offset; + int copied; + + index = pos / (PAGE_SIZE / HPB_ENTRY_SIZE); + offset = pos % (PAGE_SIZE / HPB_ENTRY_SIZE); + + if ((offset + len) <= (PAGE_SIZE / HPB_ENTRY_SIZE)) + copied = len; + else + copied = (PAGE_SIZE / HPB_ENTRY_SIZE) - offset; + + page = mctx->m_page[index]; + if (unlikely(!page)) { + dev_err(&hpb->sdev_ufs_lu->sdev_dev, + "error. cannot find page in mctx\n"); + return -ENOMEM; + } + + memcpy(ppn_buf, page_address(page) + (offset * HPB_ENTRY_SIZE), + copied * HPB_ENTRY_SIZE); + + return copied; +} + +static void +ufshpb_get_pos_from_lpn(struct ufshpb_lu *hpb, unsigned long lpn, int *rgn_idx, + int *srgn_idx, int *offset) +{ + int rgn_offset; + + *rgn_idx = lpn >> hpb->entries_per_rgn_shift; + rgn_offset = lpn & hpb->entries_per_rgn_mask; + *srgn_idx = rgn_offset >> hpb->entries_per_srgn_shift; + *offset = rgn_offset & hpb->entries_per_srgn_mask; +} + +static void +ufshpb_set_hpb_read_to_upiu(struct ufshpb_lu *hpb, struct ufshcd_lrb *lrbp, + u32 lpn, u64 ppn, unsigned int transfer_len) +{ + unsigned char *cdb = lrbp->cmd->cmnd; + + cdb[0] = UFSHPB_READ; + + /* ppn value is stored as big-endian in the host memory */ + memcpy(&cdb[6], &ppn, sizeof(u64)); + cdb[14] = transfer_len; + + lrbp->cmd->cmd_len = UFS_CDB_SIZE; +} + +/* + * This function will set up HPB read command using host-side L2P map data. + * In HPB v1.0, maximum size of HPB read command is 4KB. + */ +void ufshpb_prep(struct ufs_hba *hba, struct ufshcd_lrb *lrbp) +{ + struct ufshpb_lu *hpb; + struct ufshpb_region *rgn; + struct ufshpb_subregion *srgn; + struct scsi_cmnd *cmd = lrbp->cmd; + u32 lpn; + u64 ppn; + unsigned long flags; + int transfer_len, rgn_idx, srgn_idx, srgn_offset; + int err = 0; + + hpb = ufshpb_get_hpb_data(cmd->device); + if (!hpb) + return; + + if (ufshpb_get_state(hpb) != HPB_PRESENT) { + dev_notice(&hpb->sdev_ufs_lu->sdev_dev, + "%s: ufshpb state is not PRESENT", __func__); + return; + } + + if (!ufshpb_is_write_or_discard_cmd(cmd) && + !ufshpb_is_read_cmd(cmd)) + return; + + transfer_len = sectors_to_logical(cmd->device, + blk_rq_sectors(cmd->request)); + if (unlikely(!transfer_len)) + return; + + lpn = sectors_to_logical(cmd->device, blk_rq_pos(cmd->request)); + ufshpb_get_pos_from_lpn(hpb, lpn, &rgn_idx, &srgn_idx, &srgn_offset); + rgn = hpb->rgn_tbl + rgn_idx; + srgn = rgn->srgn_tbl + srgn_idx; + + /* If command type is WRITE or DISCARD, set bitmap as drity */ + if (ufshpb_is_write_or_discard_cmd(cmd)) { + spin_lock_irqsave(&hpb->rgn_state_lock, flags); + ufshpb_set_ppn_dirty(hpb, rgn_idx, srgn_idx, srgn_offset, + transfer_len); + spin_unlock_irqrestore(&hpb->rgn_state_lock, flags); + return; + } + + if (!ufshpb_is_support_chunk(transfer_len)) + return; + + spin_lock_irqsave(&hpb->rgn_state_lock, flags); + if (ufshpb_test_ppn_dirty(hpb, rgn_idx, srgn_idx, srgn_offset, + transfer_len)) { + hpb->stats.miss_cnt++; + spin_unlock_irqrestore(&hpb->rgn_state_lock, flags); + return; + } + + err = ufshpb_fill_ppn_from_page(hpb, srgn->mctx, srgn_offset, 1, &ppn); + spin_unlock_irqrestore(&hpb->rgn_state_lock, flags); + if (unlikely(err < 0)) { + /* + * In this case, the region state is active, + * but the ppn table is not allocated. + * Make sure that ppn table must be allocated on + * active state. + */ + dev_err(hba->dev, "get ppn failed. err %d\n", err); + return; + } + + ufshpb_set_hpb_read_to_upiu(hpb, lrbp, lpn, ppn, transfer_len); + + hpb->stats.hit_cnt++; +} static struct ufshpb_req *ufshpb_get_map_req(struct ufshpb_lu *hpb, struct ufshpb_subregion *srgn) { @@ -153,7 +400,7 @@ static struct ufshpb_req *ufshpb_get_map_req(struct ufshpb_lu *hpb, } static void ufshpb_put_map_req(struct ufshpb_lu *hpb, - struct ufshpb_req *map_req) + struct ufshpb_req *map_req) { bio_put(map_req->bio); blk_put_request(map_req->req); diff --git a/drivers/scsi/ufs/ufshpb.h b/drivers/scsi/ufs/ufshpb.h index aaffc8968afd..c70e73546e35 100644 --- a/drivers/scsi/ufs/ufshpb.h +++ b/drivers/scsi/ufs/ufshpb.h @@ -200,6 +200,7 @@ struct ufs_hba; struct ufshcd_lrb; #ifndef CONFIG_SCSI_UFS_HPB +static void ufshpb_prep(struct ufs_hba *hba, struct ufshcd_lrb *lrbp) {} static void ufshpb_rsp_upiu(struct ufs_hba *hba, struct ufshcd_lrb *lrbp) {} static void ufshpb_resume(struct ufs_hba *hba) {} static void ufshpb_suspend(struct ufs_hba *hba) {} @@ -213,6 +214,7 @@ static bool ufshpb_is_allowed(struct ufs_hba *hba) { return false; } static void ufshpb_get_geo_info(struct ufs_hba *hba, u8 *geo_buf) {} static void ufshpb_get_dev_info(struct ufs_hba *hba, u8 *desc_buf) {} #else +void ufshpb_prep(struct ufs_hba *hba, struct ufshcd_lrb *lrbp); void ufshpb_rsp_upiu(struct ufs_hba *hba, struct ufshcd_lrb *lrbp); void ufshpb_resume(struct ufs_hba *hba); void ufshpb_suspend(struct ufs_hba *hba); -- 2.25.1