Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932567AbdHWTKp (ORCPT ); Wed, 23 Aug 2017 15:10:45 -0400 Received: from mail-cys01nam02on0102.outbound.protection.outlook.com ([104.47.37.102]:37664 "EHLO NAM02-CY1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S932458AbdHWTKn (ORCPT ); Wed, 23 Aug 2017 15:10:43 -0400 From: Long Li To: Leon Romanovsky CC: Steve French , "linux-cifs@vger.kernel.org" , "samba-technical@lists.samba.org" , "linux-kernel@vger.kernel.org" , "linux-rdma@vger.kernel.org" , Christoph Hellwig , "Tom Talpey" , Matthew Wilcox Subject: RE: [Patch v2 13/19] CIFS: SMBD: Use registered memory RDMA read for SMB write Thread-Topic: [Patch v2 13/19] CIFS: SMBD: Use registered memory RDMA read for SMB write Thread-Index: AQHTGed6LlK55yJ0X0KVYBzzxyxVlKKR+i0AgABF2FCAABDVAIAAAKOA Date: Wed, 23 Aug 2017 19:10:38 +0000 Message-ID: References: <1503255883-3041-1-git-send-email-longli@exchange.microsoft.com> <1503255883-3041-14-git-send-email-longli@exchange.microsoft.com> <20170823135200.GP1724@mtr-leonro.local> <20170823190214.GY1724@mtr-leonro.local> In-Reply-To: <20170823190214.GY1724@mtr-leonro.local> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Enabled=True; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SiteId=72f988bf-86f1-41af-91ab-2d7cd011db47; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Ref=https://api.informationprotection.azure.com/api/72f988bf-86f1-41af-91ab-2d7cd011db47; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Owner=longli@microsoft.com; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SetDate=2017-08-23T12:10:36.9087734-07:00; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Name=General; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Application=Microsoft Azure Information Protection; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Extended_MSFT_Method=Automatic; Sensitivity=General x-originating-ip: [2001:4898:80e8:f::735] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1;MWHPR21MB0479;6:OLz3AkuSGwLYFPMnis5f+D+7SbfsQZkDnpHJj/xo5n8be79MWpik/Rp50K5T1qvN/Tzup0oF/BczE0bFMqsFSlB5gH8kwGDJ34i/Ihc7cKDCJvtrqePuGsvpVA2MHs6i16gaPxvSw6FI0hOfIfgqGWD83PvC6DabyX8i52x+xccngKZRR+yBK5Z/vlLD3F9KVVqWtaejW2ieQ9fTCGS/BCacMpBpa2ffbt56NhDm6DF50V1UkI3+1o9t7Fluw/py7XlGZfxzd5bpUv/dH2s0YPyKc8ALavCn4Rlt9qTO5lsk7r4RqoO013WjdgRGttoOqmDCRHq3AQLHvyOC+R8tjw==;5:aRg19+mG3q/7n251GSJjr186zX3SyRn4gYRTQEVHPmYzOQQ3et/YNZEYrH89+9AjghVVcK/HIzGSkrXPuM3/JI/++fHgBqk8U967qimE9t25nTtCfLVk7lTIwRud56vazQ7pFCb3xuBKmEIvJmlGKQ==;24:SEsMavaVVI9dxtLmWPgFgsHj7wi6/WPpFCeAPSz6EucVksRtG5w+NLWvOuKDaY7metVJBVgNlnxdbTA8ZkncNhZPfBgQi/vrJyysmvXh3K8=;7:/jqBOdt6hw1FAImR6OwcXdi01DTSMvzO5po8G+DJWpnOeURjYB9UNRQhqOCVhtCiGTkycnHpmypYZHumR+BQrb/SHEuqE4633KIHfuMuGU01wFzKQGdS1UylODyJ6pKhdVvSDTjg4/qjp/a/ZMpW214wE2vD7nx+8+bBiKW1+k/K1P4vGEOG+oOP042Fti2O/XWM0Zt+QS/zJYXM6m/z8rgwLcDyNI5XhT2VByo4U5Q= x-ms-exchange-antispam-srfa-diagnostics: SSOS;SSOR; x-forefront-antispam-report: SFV:SKI;SCL:-1;SFV:NSPM;SFS:(10019020)(6009001)(39860400002)(47760400005)(189002)(13464003)(377454003)(199003)(24454002)(3280700002)(33656002)(4326008)(93886005)(6916009)(2950100002)(2906002)(8936002)(6116002)(102836003)(81166006)(106356001)(6506006)(105586002)(54356999)(77096006)(3660700001)(6436002)(50986999)(76176999)(229853002)(107886003)(110136004)(101416001)(6246003)(7736002)(54906002)(9686003)(305945005)(10290500003)(53936002)(99286003)(55016002)(81156014)(8676002)(189998001)(5005710100001)(68736007)(10090500001)(478600001)(53546010)(74316002)(14454004)(5660300001)(86612001)(86362001)(25786009)(8990500004)(97736004)(7696004)(2900100001);DIR:OUT;SFP:1102;SCL:1;SRVR:MWHPR21MB0479;H:MWHPR21MB0190.namprd21.prod.outlook.com;FPR:;SPF:None;PTR:InfoNoRecords;MX:1;A:1;LANG:en; x-ms-office365-filtering-correlation-id: 43f177b6-32fd-4dbe-0a82-08d4ea5aa437 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(300000500095)(300135000095)(300000501095)(300135300095)(300000502095)(300135100095)(22001)(2017030254152)(48565401081)(300000503095)(300135400095)(2017052603188)(201703131423075)(201703031133081)(201702281549075)(300000504095)(300135200095)(300000505095)(300135600095)(300000506095)(300135500095);SRVR:MWHPR21MB0479; x-ms-traffictypediagnostic: MWHPR21MB0479: authentication-results: spf=none (sender IP is ) smtp.mailfrom=longli@microsoft.com; x-exchange-antispam-report-test: UriScan:(158342451672863)(89211679590171)(192374486261705)(9452136761055)(788757137089); x-microsoft-antispam-prvs: x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(61425038)(6040450)(601004)(2401047)(8121501046)(5005006)(100000703101)(100105400095)(10201501046)(3002001)(93006095)(93001095)(6055026)(61426038)(61427038)(6041248)(20161123564025)(20161123562025)(20161123555025)(20161123558100)(20161123560025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095);SRVR:MWHPR21MB0479;BCL:0;PCL:0;RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095);SRVR:MWHPR21MB0479; x-forefront-prvs: 040866B734 spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-originalarrivaltime: 23 Aug 2017 19:10:38.6296 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR21MB0479 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from base64 to 8bit by nfs id v7NJAq6F003827 Content-Length: 5327 Lines: 135 > -----Original Message----- > From: Leon Romanovsky [mailto:leon@kernel.org] > Sent: Wednesday, August 23, 2017 12:02 PM > To: Long Li > Cc: Steve French ; linux-cifs@vger.kernel.org; samba- > technical@lists.samba.org; linux-kernel@vger.kernel.org; linux- > rdma@vger.kernel.org; Christoph Hellwig ; Tom Talpey > ; Matthew Wilcox > Subject: Re: [Patch v2 13/19] CIFS: SMBD: Use registered memory RDMA > read for SMB write > > On Wed, Aug 23, 2017 at 06:09:11PM +0000, Long Li wrote: > > > > > > > -----Original Message----- > > > From: Leon Romanovsky [mailto:leon@kernel.org] > > > Sent: Wednesday, August 23, 2017 6:52 AM > > > To: Long Li > > > Cc: Steve French ; linux-cifs@vger.kernel.org; > > > samba- technical@lists.samba.org; linux-kernel@vger.kernel.org; > > > linux- rdma@vger.kernel.org; Christoph Hellwig ; > > > Tom Talpey ; Matthew Wilcox > > > ; Long Li > > > Subject: Re: [Patch v2 13/19] CIFS: SMBD: Use registered memory RDMA > > > read for SMB write > > > > > > On Sun, Aug 20, 2017 at 12:04:37PM -0700, Long Li wrote: > > > > From: Long Li > > > > > > > > When sending I/O, if size is larger than rdma_readwrite_threshold > > > > we > > > prepare to send SMB WRITE packet for a RDMA read via memory > registration. > > > The actual I/O is done out-of-the-band, so modify the relevant > > > fields in the packet accordingly. > > > > > > > > Signed-off-by: Long Li > > > > --- > > > > fs/cifs/smb2pdu.c | 45 > > > ++++++++++++++++++++++++++++++++++++++++++++- > > > > 1 file changed, 44 insertions(+), 1 deletion(-) > > > > > > > > diff --git a/fs/cifs/smb2pdu.c b/fs/cifs/smb2pdu.c index > > > > 5cc5f6c..5581afd 100644 > > > > --- a/fs/cifs/smb2pdu.c > > > > +++ b/fs/cifs/smb2pdu.c > > > > @@ -48,6 +48,7 @@ > > > > #include "smb2glob.h" > > > > #include "cifspdu.h" > > > > #include "cifs_spnego.h" > > > > +#include "smbdirect.h" > > > > > > > > /* > > > > * The following table defines the expected "StructureSize" of > > > > SMB2 requests @@ -2716,6 +2717,41 @@ smb2_async_writev(struct > > > cifs_writedata *wdata, > > > > offsetof(struct smb2_write_req, Buffer) - 4); > > > > req->RemainingBytes = 0; > > > > > > > > + /* > > > > + * If we want to do a server RDMA read, fill in and append > > > > + * smbd_buffer_descriptor_v1 to the end of write request > > > > + */ > > > > + if (server->rdma && wdata->bytes > > > > > + server->smbd_conn->rdma_readwrite_threshold) { > > > > + > > > > + struct smbd_buffer_descriptor_v1 *v1; > > > > + bool need_invalidate = server->dialect == SMB30_PROT_ID; > > > > + > > > > + wdata->mr = smbd_register_mr( > > > > + server->smbd_conn, wdata->pages, > > > > + wdata->nr_pages, wdata->tailsz, > > > > + false, need_invalidate); > > > > + if (!wdata->mr) { > > > > + rc = -ENOBUFS; > > > > + goto async_writev_out; > > > > + } > > > > + req->Length = 0; > > > > + req->DataOffset = 0; > > > > + req->RemainingBytes = > > > > > > Wow, we have CamelCase variables in linux kernel. It will help if > > > you start your patchset with small cleanup to convert those > > > variables from CamelCase to normal names. > > > > They are used everywhere in the upper layer code for packet > > definitions, written a long time ago. (most in fs/cifs/smb2pdu.h and > > fs/cifs/cifspdu.h) > > "everywhere" is a little bit over estimated in this case. > ➜ linux-rdma git:(master) git grep RemainingBytes > fs/cifs/smb2pdu.c: req->RemainingBytes = > cpu_to_le32(remaining_bytes); > fs/cifs/smb2pdu.c: req->RemainingBytes = 0; > fs/cifs/smb2pdu.c: req->RemainingBytes = 0; > fs/cifs/smb2pdu.c: req->RemainingBytes = 0; > fs/cifs/smb2pdu.h: __le32 RemainingBytes; > fs/cifs/smb2pdu.h: __le32 RemainingBytes; > > One simple "sed -i" will replace all them in one shot and it doesn't look like > undoable task. I mean cifspdu.h and smb2pdu.h. use CamelCase for all packet definitions. For example another one in smb2pdu.h: struct smb2_negotiate_rsp { struct smb2_hdr hdr; __le16 StructureSize; /* Must be 65 */ __le16 SecurityMode; __le16 DialectRevision; __le16 NegotiateContextCount; /* Prior to SMB3.1.1 was Reserved & MBZ */ __u8 ServerGUID[16]; __le32 Capabilities; __le32 MaxTransactSize; __le32 MaxReadSize; __le32 MaxWriteSize; __le64 SystemTime; /* MBZ */ __le64 ServerStartTime; __le16 SecurityBufferOffset; __le16 SecurityBufferLength; __le32 NegotiateContextOffset; /* Pre:SMB3.1.1 was reserved/ignored */ __u8 Buffer[1]; /* variable length GSS security buffer */ } __packed; We may want to change them all together to keep naming consistent, that's a lot of changes. > > > > > I suggest we do another cleanup patch to clean things up. > > Yes, another cleanup patch is needed before your patches. You are adding > your code in 2017 and you are expected to follow present coding standards > like everyone else in the kernel. > > Thanks