Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752491AbdHNSLB (ORCPT ); Mon, 14 Aug 2017 14:11:01 -0400 Received: from mail-by2nam01on0130.outbound.protection.outlook.com ([104.47.34.130]:4905 "EHLO NAM01-BY2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1752133AbdHNSK7 (ORCPT ); Mon, 14 Aug 2017 14:10:59 -0400 From: Long Li To: Stefan Metzmacher , Steve French , "linux-cifs@vger.kernel.org" , "samba-technical@lists.samba.org" , "linux-kernel@vger.kernel.org" , "linux-rdma@vger.kernel.org" , Christoph Hellwig Subject: RE: [[PATCH v1] 02/37] [CIFS] SMBD: Add structure for SMBD transport Thread-Topic: [[PATCH v1] 02/37] [CIFS] SMBD: Add structure for SMBD transport Thread-Index: AQHTC8ujB24zac3NZEiE9YSsVOxDeKJ6D8MAgAZYRbCAA4ZTAIAASKsg Date: Mon, 14 Aug 2017 18:10:46 +0000 Message-ID: References: <1501704648-20159-1-git-send-email-longli@exchange.microsoft.com> <1501704648-20159-3-git-send-email-longli@exchange.microsoft.com> <9632c208-d6a5-2c47-b583-274d046d97bd@samba.org> <61ab1564-d699-e8f2-631e-67a6c28b678b@samba.org> In-Reply-To: <61ab1564-d699-e8f2-631e-67a6c28b678b@samba.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Enabled=True; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SiteId=72f988bf-86f1-41af-91ab-2d7cd011db47; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Ref=https://api.informationprotection.azure.com/api/72f988bf-86f1-41af-91ab-2d7cd011db47; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Owner=longli@microsoft.com; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SetDate=2017-08-14T11:10:43.9429804-07:00; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Name=General; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Application=Microsoft Azure Information Protection; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Extended_MSFT_Method=Automatic; Sensitivity=General x-originating-ip: [2001:4898:80e8:c::735] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1;CY4PR21MB0119;6:J8v2Xu8sJjmjGt063A6q1akbFIxd6jjGttNxvTBDlpDyUFi9mqOK4eCni1+Hm8Id8j+C+VvSmI1KQ+LB8Mi3IitU8IDrtSJmzv17YOPa6V3obfSxb2U0So1PuHXYikgLtadmjIMLwYvgwdZXTKKY0jLxmQPj6+vg/dYA1TayONVacLX6Wd3VsyPMbcP4I9i3Sk+8oh/Vcw8KVaPLyU4OLRA23EaDw9e6+Lb8I71kltKswWdwIKOoIG26nMwAjhy0sHRUJMZ3+4093SVRtItzyCl0YXHk190XaQkcpTOMqMzigs5YHyjAyMsaEpEQnVPUW3/Zamh9MIyblanzfSiCog==;5:H2K2Dc2P8mr8WtvJFCUMMX8Ph7C4P5UEh28fks+uPZs1jmxNDs8iVrmH5cCy0MqJ7GDHBOfBlCP2qys2Vq5YIUAKpR8ZQECj/y4F60HIb17GSdv66L1w0NPx2FHLoLIbsIk+N4tc1dH5AyCU1bja2w==;24:f44JtPgi3OGScqBS2MHZ6TR5jexA+9TP2U7dqfC+SssxWWd1EjA+YpvIQLsFdlAgs3WILgfIOAZsGax3pdBhXwRyvCIm/R8Qzupf2s2qr0Y=;7:Nv29fphqA697UN7IREBNfUpcm9XYfwTtx7ums7Sgz4fbsQOtq188PIA4dPhTsCxxcVGaE8tgAGCDTptlW+FKu0AaJlx3Ujk9qsOPLlkV8IQUJ3By+5epeUw07KUG7h3a+DD2Cw4TDpwo6PXLKYmrHBm3OGYwJYDnPvooQeEJJw9ijgl4eD0q03EYBaZW3iAcdmpNR49jObtB/4Ou+Iej8DSnmo1UIR8WJq/1wXDaAJY= x-ms-office365-filtering-correlation-id: b30d1d27-319d-434b-1993-08d4e33fc927 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(300000500095)(300135000095)(300000501095)(300135300095)(22001)(300000502095)(300135100095)(2017030254152)(48565401081)(300000503095)(300135400095)(2017052603142)(201703131423075)(201703031133081)(201702281549075)(300000504095)(300135200095)(300000505095)(300135600095)(300000506095)(300135500095);SRVR:CY4PR21MB0119; x-ms-traffictypediagnostic: CY4PR21MB0119: x-exchange-antispam-report-test: UriScan:(158342451672863)(89211679590171)(9452136761055)(788757137089)(21532816269658); x-microsoft-antispam-prvs: x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(61425038)(6040450)(601004)(2401047)(8121501046)(5005006)(93006095)(93001095)(100000703101)(100105400095)(10201501046)(3002001)(6055026)(61426038)(61427038)(6041248)(20161123560025)(20161123555025)(201703131423075)(201703061421075)(201703161042150)(20161123564025)(20161123558100)(20161123562025)(6042181)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095);SRVR:CY4PR21MB0119;BCL:0;PCL:0;RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095);SRVR:CY4PR21MB0119; x-forefront-prvs: 039975700A x-forefront-antispam-report: SFV:NSPM;SFS:(10019020)(6009001)(47760400005)(13464003)(377454003)(199003)(189002)(51884002)(2900100001)(106356001)(81166006)(3280700002)(105586002)(81156014)(2906002)(7696004)(97736004)(498600001)(74316002)(345774005)(6246003)(2501003)(14454004)(50986999)(8676002)(53546010)(5005710100001)(8936002)(54356999)(76176999)(6436002)(10290500003)(5660300001)(53936002)(2201001)(8990500004)(3660700001)(33656002)(189998001)(68736007)(55016002)(10090500001)(93886004)(99286003)(77096006)(6506006)(229853002)(966005)(6116002)(102836003)(575784001)(86362001)(2950100002)(25786009)(86612001)(6306002)(9686003)(101416001)(305945005)(7736002);DIR:OUT;SFP:1102;SCL:1;SRVR:CY4PR21MB0119;H:CY4PR21MB0182.namprd21.prod.outlook.com;FPR:;SPF:None;PTR:InfoNoRecords;A:1;MX:1;LANG:en; authentication-results: spf=none (sender IP is ) smtp.mailfrom=longli@microsoft.com; spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-originalarrivaltime: 14 Aug 2017 18:10:46.1591 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR21MB0119 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from base64 to 8bit by nfs id v7EIB7TK024239 Content-Length: 4049 Lines: 95 > -----Original Message----- > From: Stefan Metzmacher [mailto:metze@samba.org] > Sent: Monday, August 14, 2017 6:41 AM > To: Long Li ; Steve French ; > linux-cifs@vger.kernel.org; samba-technical@lists.samba.org; linux- > kernel@vger.kernel.org; linux-rdma@vger.kernel.org; Christoph Hellwig > > Subject: Re: [[PATCH v1] 02/37] [CIFS] SMBD: Add structure for SMBD > transport > > Hi Long, > > >> It seems that the new transport is tied to it's caller regarding > >> structures and naming conventions. > >> > >> I think it would be better to strictly separate them, as I'd like to > >> use the SMBDirect transport also from the userspace for the client > >> side e.g. in Samba's '[lib]smbclient', but also in Samba's server side code > 'smbd'. > > > > Thank you for reviewing the patch set. > > > > I think it is possible to separate the common code that implements the > SMBDirect transport. There are some challenges to reuse the same code for > both kernel and user spaces. > > 1. Kernel mode RDMA verbs are similar but different to user-mode ones. > > 2. Some RDMA features (e.g Fast Registration Work Request) are not > available in user-mode. > > 3. Locking and synchronization mechanism is different 4. Memory > > management is different. > > 5. Process creation/scheduling and data sharing between processes are > different, and there is no user-mode code running in interrupt/softirq. > > > > Those needs to be abstracted through a layer, the rest of the code can be > shared. I can work on this after patch set is reviewed. > > I guess this is a misunderstanding... > > I don't want to use that code and run it in userspace, I have a userspace > prototype more or less working here, see > https://git.samba.org/?p=metze/samba/wip.git;a=shortlog;h=refs/heads/m > aster3-rdma > and > https://git.samba.org/?p=metze/samba/wip.git;a=blob;f=libcli/smb/smb_dir > ect.c;h=9cc0d861ccfcbb4df9ef6ad85a7fe3d262e549c0;hb=85d46de6fdbba041 > d3e8004af46865a72d2b8405 > > I goal is that we'll have an api that allows userspace code to use the kernel > code SMBDirect code. This userspace code would get a file descriptor from > the kernel and would be able to use it similar to a tcp socket. > If the kernel would simulate the 4 byte length header, it's trivial to get to a > stage were smbclient and smbd are able to support SMBDirect without much > changes. > We only need to replace connect(), listen(), accept() and a few more by > SMBDirect versions. This is possible. SMBDirect code can handle the first 4 bytes for upper layer. > > For the real data transfer we might be able to use memfd_create() or > something similar to share the buffers between userspace and kernel. You'll need to post RDMA read/write through the same QP created by SMBDirect in the kernel. I think this needs some more work but it's doable. > > I guess this is a long way, but having the basic SMBDirect code in dependently > in the kernel would help a lot. > > >> Would it be possible to isolate this in smb_direct.c and smb_direct.h > >> while using > >> smb_direct_* prefixes for structures and functions? Also avoiding the > >> usage of other headers from fs/cifs/*.h, expect for something generic like > nterr.h. > > > > Sure I will make naming changes and clean up the header files. > > Thanks! > > >> I guess 'struct cifs_rdma_info' would then be 'struct > smb_direct_connection'. > >> And it won't have a reference to struct TCP_Server_Info. > > > > I will look for ways to remove reference to struct TCP_Server_Info . The > reason why it has a reference to TCP_Server_Info is that: TCP_Server_Info > represents a transport connection, although it also has many other TCP > related code. SMBD needs to get to this connection TCP_Server_Info and set > the transport status on shutdown (and maybe other situations). > > > > Wouldn't it be better to provide a way to ask for the connection state and let > the caller ask for the state instead of changing the callers structure? > > metze