Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932552AbdHWSJS (ORCPT ); Wed, 23 Aug 2017 14:09:18 -0400 Received: from mail-co1nam03on0098.outbound.protection.outlook.com ([104.47.40.98]:20223 "EHLO NAM03-CO1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S932111AbdHWSJQ (ORCPT ); Wed, 23 Aug 2017 14:09:16 -0400 From: Long Li To: Leon Romanovsky CC: Steve French , "linux-cifs@vger.kernel.org" , "samba-technical@lists.samba.org" , "linux-kernel@vger.kernel.org" , "linux-rdma@vger.kernel.org" , Christoph Hellwig , "Tom Talpey" , Matthew Wilcox Subject: RE: [Patch v2 13/19] CIFS: SMBD: Use registered memory RDMA read for SMB write Thread-Topic: [Patch v2 13/19] CIFS: SMBD: Use registered memory RDMA read for SMB write Thread-Index: AQHTGed6LlK55yJ0X0KVYBzzxyxVlKKR+i0AgABF2FA= Date: Wed, 23 Aug 2017 18:09:11 +0000 Message-ID: References: <1503255883-3041-1-git-send-email-longli@exchange.microsoft.com> <1503255883-3041-14-git-send-email-longli@exchange.microsoft.com> <20170823135200.GP1724@mtr-leonro.local> In-Reply-To: <20170823135200.GP1724@mtr-leonro.local> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Enabled=True; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SiteId=72f988bf-86f1-41af-91ab-2d7cd011db47; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Ref=https://api.informationprotection.azure.com/api/72f988bf-86f1-41af-91ab-2d7cd011db47; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Owner=longli@microsoft.com; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SetDate=2017-08-23T11:09:09.7572028-07:00; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Name=General; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Application=Microsoft Azure Information Protection; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Extended_MSFT_Method=Automatic; Sensitivity=General x-originating-ip: [2001:4898:80e8:f::735] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1;MWHPR21MB0141;6:YRaZPBfIBUbONF7PQCo/0b9upUh2jIV/7pMgHhOfRpp5GblTWJlmSmaLhyr/eV+CiiCkDBJY4VmvXUdTfV8KLrJPponPapSqFJosjG4dhkdmOsfepcv+IlsESCO1zxurYVj2kl5u4XgzkYqfpX6SiXlFS0dz+MAkSaDIXrgt9QWgMuDsKD1G0RSjUzVmSVciYnveqJWu017bllpboiiW3RHNqDdFET2gBxckLl2IJA+bcvLFC3/HHQlDmnbyyk7YJjtXJF38eVRkRfTr+nPPBpQ7fn9m4CPC2x4+bIYgkrnYQrSyaatLQo0zP5UnqUHkDAv+XAKOwCuxDDWCWv4QyQ==;5:OoONpVCgg0GGSw2PC3t7/mWI5WoA8eSTWyPUSHUP9x5GuxPO//bfeBdxGdF9zHJ/OKZBpPO8INQzQ32XO5iHUbpDLCJNLO0+BZ029FQfgeOfhtq/IR0C3DRtArPiyMFtfdEFTO6wh1jNW7J2vXiFSQ==;24:Q3XoMsu1lcTyyfjnZClytHE7scvld9jth0PBe+vSrvw53hihmP5aDWdGkkpyQi55NrPUZWs+d6fgih78hjngB2Io9JPzQCuK97FS5AZwaZs=;7:GwwzbdT1RqvgdQ0JNbzJKH4H/3wkxvb2NoF6O5CtKVAqfxaCgrttsqM4D2BraCOQCVIKEJUFTmdt/UCSUp9ZHe8t3xdNBJALj2eBJy3Ta/gW8aN0X7K4S3nm68zgVd7qwBKqKjOVBDHrlGaaG9EK8ePp4t4nliHWighBWSaWAF5d9Jz2EWeeWXwDFItB0nozxyF53V44dUyPO27NN2zoU2fYvJUxv7YZC9mj9FGnU2U= x-ms-exchange-antispam-srfa-diagnostics: SSOS;SSOR; x-forefront-antispam-report: SFV:SKI;SCL:-1;SFV:NSPM;SFS:(10019020)(6009001)(39860400002)(47760400005)(189002)(377454003)(13464003)(24454002)(199003)(69234005)(6506006)(25786009)(3280700002)(86612001)(9686003)(6436002)(8990500004)(6246003)(10090500001)(105586002)(106356001)(189998001)(5005710100001)(3660700001)(6116002)(77096006)(2950100002)(6916009)(102836003)(86362001)(55016002)(2900100001)(229853002)(53936002)(4326008)(54906002)(99286003)(2906002)(74316002)(7696004)(97736004)(5660300001)(76176999)(7736002)(50986999)(54356999)(33656002)(101416001)(107886003)(110136004)(8676002)(10290500003)(478600001)(53546010)(305945005)(68736007)(81166006)(81156014)(8936002)(14454004);DIR:OUT;SFP:1102;SCL:1;SRVR:MWHPR21MB0141;H:MWHPR21MB0190.namprd21.prod.outlook.com;FPR:;SPF:None;PTR:InfoNoRecords;MX:1;A:1;LANG:en; x-ms-office365-filtering-correlation-id: 80e7996b-7b75-402b-d285-08d4ea520eb0 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(300000500095)(300135000095)(300000501095)(300135300095)(300000502095)(300135100095)(22001)(2017030254152)(48565401081)(300000503095)(300135400095)(2017052603187)(201703131423075)(201703031133081)(201702281549075)(300000504095)(300135200095)(300000505095)(300135600095)(300000506095)(300135500095);SRVR:MWHPR21MB0141; x-ms-traffictypediagnostic: MWHPR21MB0141: x-exchange-antispam-report-test: UriScan:(158342451672863)(89211679590171)(9452136761055); x-microsoft-antispam-prvs: x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(61425038)(6040450)(601004)(2401047)(5005006)(8121501046)(3002001)(93006095)(93001095)(10201501046)(100000703101)(100105400095)(6055026)(61426038)(61427038)(6041248)(20161123558100)(20161123560025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123564025)(20161123562025)(20161123555025)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095);SRVR:MWHPR21MB0141;BCL:0;PCL:0;RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095);SRVR:MWHPR21MB0141; x-forefront-prvs: 040866B734 authentication-results: spf=none (sender IP is ) smtp.mailfrom=longli@microsoft.com; spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-originalarrivaltime: 23 Aug 2017 18:09:11.8425 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR21MB0141 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by nfs id v7NI9Pnt032629 Content-Length: 4089 Lines: 118 > -----Original Message----- > From: Leon Romanovsky [mailto:leon@kernel.org] > Sent: Wednesday, August 23, 2017 6:52 AM > To: Long Li > Cc: Steve French ; linux-cifs@vger.kernel.org; samba- > technical@lists.samba.org; linux-kernel@vger.kernel.org; linux- > rdma@vger.kernel.org; Christoph Hellwig ; Tom Talpey > ; Matthew Wilcox ; > Long Li > Subject: Re: [Patch v2 13/19] CIFS: SMBD: Use registered memory RDMA > read for SMB write > > On Sun, Aug 20, 2017 at 12:04:37PM -0700, Long Li wrote: > > From: Long Li > > > > When sending I/O, if size is larger than rdma_readwrite_threshold we > prepare to send SMB WRITE packet for a RDMA read via memory registration. > The actual I/O is done out-of-the-band, so modify the relevant fields in the > packet accordingly. > > > > Signed-off-by: Long Li > > --- > > fs/cifs/smb2pdu.c | 45 > ++++++++++++++++++++++++++++++++++++++++++++- > > 1 file changed, 44 insertions(+), 1 deletion(-) > > > > diff --git a/fs/cifs/smb2pdu.c b/fs/cifs/smb2pdu.c index > > 5cc5f6c..5581afd 100644 > > --- a/fs/cifs/smb2pdu.c > > +++ b/fs/cifs/smb2pdu.c > > @@ -48,6 +48,7 @@ > > #include "smb2glob.h" > > #include "cifspdu.h" > > #include "cifs_spnego.h" > > +#include "smbdirect.h" > > > > /* > > * The following table defines the expected "StructureSize" of SMB2 > > requests @@ -2716,6 +2717,41 @@ smb2_async_writev(struct > cifs_writedata *wdata, > > offsetof(struct smb2_write_req, Buffer) - 4); > > req->RemainingBytes = 0; > > > > + /* > > + * If we want to do a server RDMA read, fill in and append > > + * smbd_buffer_descriptor_v1 to the end of write request > > + */ > > + if (server->rdma && wdata->bytes > > > + server->smbd_conn->rdma_readwrite_threshold) { > > + > > + struct smbd_buffer_descriptor_v1 *v1; > > + bool need_invalidate = server->dialect == SMB30_PROT_ID; > > + > > + wdata->mr = smbd_register_mr( > > + server->smbd_conn, wdata->pages, > > + wdata->nr_pages, wdata->tailsz, > > + false, need_invalidate); > > + if (!wdata->mr) { > > + rc = -ENOBUFS; > > + goto async_writev_out; > > + } > > + req->Length = 0; > > + req->DataOffset = 0; > > + req->RemainingBytes = > > Wow, we have CamelCase variables in linux kernel. It will help if you start > your patchset with small cleanup to convert those variables from CamelCase > to normal names. They are used everywhere in the upper layer code for packet definitions, written a long time ago. (most in fs/cifs/smb2pdu.h and fs/cifs/cifspdu.h) I suggest we do another cleanup patch to clean things up. > > Thanks > > > + (wdata->nr_pages-1)*PAGE_SIZE + wdata->tailsz; > > + req->Channel = SMB2_CHANNEL_RDMA_V1_INVALIDATE; > > + if (need_invalidate) > > + req->Channel = SMB2_CHANNEL_RDMA_V1; > > + req->WriteChannelInfoOffset = > > + offsetof(struct smb2_write_req, Buffer) - 4; > > + req->WriteChannelInfoLength = > > + sizeof(struct smbd_buffer_descriptor_v1); > > + v1 = (struct smbd_buffer_descriptor_v1 *) &req->Buffer[0]; > > + v1->offset = wdata->mr->mr->iova; > > + v1->token = wdata->mr->mr->rkey; > > + v1->length = wdata->mr->mr->length; > > + } > > + > > /* 4 for rfc1002 length field and 1 for Buffer */ > > iov[0].iov_len = 4; > > iov[0].iov_base = req; > > @@ -2729,10 +2765,17 @@ smb2_async_writev(struct cifs_writedata > *wdata, > > rqst.rq_pagesz = wdata->pagesz; > > rqst.rq_tailsz = wdata->tailsz; > > > > + if (wdata->mr) { > > + iov[1].iov_len += sizeof(struct smbd_buffer_descriptor_v1); > > + rqst.rq_npages = 0; > > + } > > + > > cifs_dbg(FYI, "async write at %llu %u bytes\n", > > wdata->offset, wdata->bytes); > > > > - req->Length = cpu_to_le32(wdata->bytes); > > + /* For RDMA read, I/O size is in RemainingBytes not in Length */ > > + if (!wdata->mr) > > + req->Length = cpu_to_le32(wdata->bytes); > > > > inc_rfc1001_len(&req->hdr, wdata->bytes - 1 /* Buffer */); > > > > -- > > 2.7.4 > >