Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751906AbdH3CaK (ORCPT ); Tue, 29 Aug 2017 22:30:10 -0400 Received: from mail-bn3nam01on0130.outbound.protection.outlook.com ([104.47.33.130]:24255 "EHLO NAM01-BN3-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751297AbdH3CaI (ORCPT ); Tue, 29 Aug 2017 22:30:08 -0400 From: Long Li To: Tom Talpey , Steve French , "linux-cifs@vger.kernel.org" , "samba-technical@lists.samba.org" , "linux-kernel@vger.kernel.org" , "linux-rdma@vger.kernel.org" Subject: RE: [[PATCH v1] 18/37] [CIFS] SMBD: Implement API for upper layer to send data Thread-Topic: [[PATCH v1] 18/37] [CIFS] SMBD: Implement API for upper layer to send data Thread-Index: AQHTC8upaPAPpH+nEUu8Y+46XWD9J6KEZJsAgBfxEtA= Date: Wed, 30 Aug 2017 02:30:04 +0000 Message-ID: References: <1501704648-20159-1-git-send-email-longli@exchange.microsoft.com> <1501704648-20159-19-git-send-email-longli@exchange.microsoft.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Enabled=True; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SiteId=72f988bf-86f1-41af-91ab-2d7cd011db47; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Ref=https://api.informationprotection.azure.com/api/72f988bf-86f1-41af-91ab-2d7cd011db47; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Owner=longli@microsoft.com; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SetDate=2017-08-29T19:30:02.7034595-07:00; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Name=General; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Application=Microsoft Azure Information Protection; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Extended_MSFT_Method=Automatic; Sensitivity=General x-originating-ip: [2001:4898:80e8:c::735] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1;MWHPR21MB0749;6:7xBS6TBdmLjtXrpSI4H95gxKKi5TNYByZVE+3m4Kum0IkHSbURQGAeC6pSeGWoLOZkC+dJ6tF9k2+qK1bXn0YPX46wvIk9YDKz4s/pr977ckianH7x1sD5YPNulzcXA5sAv7nVqpOYlHUGug/8gbG5svqylSk0OWeGdkz4DDCooUD+QI37ji5hH9DGWy//88C7zMEo5WXnqXtFk1PLmS3XuPyLjljSlfvAKwX55NCdmVD6cdzyxkhSAZe9lI2jJBX6m5f/CtEZW12R0/ZRWkWKm+Ke7YXTsQCAMFXVqoIiJ5MNEcNrrYEuhGEJYU08nKhk2wVjkXV7tjeWVUno85LA==;5:/aj8l2ZnGxVKxYv1nsD4dr0vVDbKpnlSmP1x+D8uYf5iNh9PS/d8yzxV/2DrjTzI6OuxYja/vWs6/qLKkaIVeW9G7L49zQDEnn9LDg77ofl2FUSj8iwspuLxiEw0QG1/BedLg1wqLEcTkORxgjX1DQ==;24:x+puKdm47nnlYVR+4UogagQrIInmMbR2AeR6MvVbzFjhiQZJmpmphAU8McINxclk/GuaLHkvxGzEBUQv/rUyEtGE9YUfNjcNIBMKeze6gNU=;7:hX27aPzUoimD3DOCobPQLSliiTAnocnvvLyxwYTvzEjpAB5SAu/OdIsv9ZzvPG6gO17uzTtrvFXn7GggExvkYs4iTGu8tOe7+Q8fqcBp2IwCFHDSYGNDK8Ep98IJ74OiZkYUoHC11dns5GecpeLrotMxk0Os4wpZuqP7enKrpa8zVEs/7rEzT7cLbEgExJQn1sUwSUx8gfWshmQd2zuCC34dCG8UdC2BgcRHtZ5UjN8= x-ms-exchange-antispam-srfa-diagnostics: SOS; x-ms-office365-filtering-correlation-id: 7209584e-008d-4b9a-77f8-08d4ef4f0623 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(300000500095)(300135000095)(300000501095)(300135300095)(22001)(300000502095)(300135100095)(2017030254152)(48565401081)(300000503095)(300135400095)(2017052603199)(201703131423075)(201703031133081)(201702281549075)(300000504095)(300135200095)(300000505095)(300135600095)(300000506095)(300135500095);SRVR:MWHPR21MB0749; x-ms-traffictypediagnostic: MWHPR21MB0749: x-exchange-antispam-report-test: UriScan:(89211679590171)(9452136761055); x-microsoft-antispam-prvs: x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(61425038)(6040450)(601004)(2401047)(8121501046)(5005006)(100000703101)(100105400095)(93006095)(93001095)(10201501046)(3002001)(6055026)(61426038)(61427038)(6041248)(20161123564025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123555025)(20161123558100)(20161123562025)(20161123560025)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095);SRVR:MWHPR21MB0749;BCL:0;PCL:0;RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095);SRVR:MWHPR21MB0749; x-forefront-prvs: 041517DFAB x-forefront-antispam-report: SFV:NSPM;SFS:(10019020)(6009001)(39860400002)(47760400005)(189002)(377454003)(13464003)(199003)(54356999)(6246003)(86362001)(1511001)(86612001)(6436002)(101416001)(102836003)(6116002)(53936002)(106356001)(105586002)(2421001)(99286003)(55016002)(9686003)(2906002)(53546010)(189998001)(25786009)(10290500003)(2561002)(5660300001)(229853002)(7696004)(2900100001)(68736007)(6506006)(76176999)(33656002)(2950100002)(478600001)(14454004)(77096006)(50986999)(305945005)(3280700002)(97736004)(5005710100001)(8936002)(7736002)(81166006)(2201001)(10090500001)(74316002)(8676002)(81156014)(8990500004)(2501003)(3660700001);DIR:OUT;SFP:1102;SCL:1;SRVR:MWHPR21MB0749;H:MWHPR21MB0190.namprd21.prod.outlook.com;FPR:;SPF:None;PTR:InfoNoRecords;MX:1;A:1;LANG:en; authentication-results: spf=none (sender IP is ) smtp.mailfrom=longli@microsoft.com; spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-originalarrivaltime: 30 Aug 2017 02:30:04.9431 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR21MB0749 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by nfs id v7U2UG6T001088 Content-Length: 3223 Lines: 86 > -----Original Message----- > From: Tom Talpey > Sent: Monday, August 14, 2017 1:44 PM > To: Long Li ; Steve French ; > linux-cifs@vger.kernel.org; samba-technical@lists.samba.org; linux- > kernel@vger.kernel.org; linux-rdma@vger.kernel.org > Subject: RE: [[PATCH v1] 18/37] [CIFS] SMBD: Implement API for upper layer > to send data > > > -----Original Message----- > > From: linux-cifs-owner@vger.kernel.org [mailto:linux-cifs- > > owner@vger.kernel.org] On Behalf Of Long Li > > Sent: Wednesday, August 2, 2017 4:10 PM > > To: Steve French ; linux-cifs@vger.kernel.org; > > samba- technical@lists.samba.org; linux-kernel@vger.kernel.org > > Cc: Long Li > > Subject: [[PATCH v1] 18/37] [CIFS] SMBD: Implement API for upper layer > > to send data > > > > +/* > > + * Write data to transport > > + * Each rqst is transported as a SMBDirect payload > > + * rqst: the data to write > > + * return value: 0 if successfully write, otherwise error code */ > > +int cifs_rdma_write(struct cifs_rdma_info *info, struct smb_rqst > > +*rqst) { > > !!! > This is a VERY confusing name. It is not sending an RDMA Write, which will > confuse any RDMA-enlightened reader. It's performing an RDMA Send, so > that name is perhaps one possibility. I have fixed that in v3. > > > + if (info->transport_status != CIFS_RDMA_CONNECTED) { > > + log_cifs_write("disconnected returning -EIO\n"); > > + return -EIO; > > + } > > Isn't this optimizing the error case? There's no guarantee it's still connected > by the time the following request construction occurs. Why not just proceed > without the check? I rearranged the shutdown logic in v3. Checking for transport status is still needed, but it checks after checking for other counters on pending activities. For example, on sending code: info->smbd_send_pending++; if (info->transport_status != SMBD_CONNECTED) { info->smbd_send_pending--; wake_up(&info->wait_smbd_send_pending); } On transport shutdown code: info->transport_status = SMBD_DISCONNECTING; ....... ....... ....... log_rdma_event(INFO, "wait for all send to finish\n"); wait_event(info->wait_smbd_send_pending, info->smbd_send_pending == 0); It guarantees no sending code can enter transport after shutdown is finished. Shutdown is running on a separate work queue, so it is needed. > > > + /* Strip the first 4 bytes MS-SMB2 section 2.1 > > + * they are used only for TCP transport */ > > + iov[0].iov_base = (char*)rqst->rq_iov[0].iov_base + 4; > > + iov[0].iov_len = rqst->rq_iov[0].iov_len - 4; > > + buflen += iov[0].iov_len; > > Ok, that layering choice in the cifs.ko client code needs to be corrected. After > all, it will need to be RDMA-aware to build the SMB3 read/write channel > structures. > And, the code in cifs_post_send_data() is allocating and building a structure > that could have been accounted for much earlier, avoiding the extra > overhead. > > That change could happen later, the hack is mostly ok for now. But > something needs to be said in a comment. > > Tom.