Received: by 10.223.185.116 with SMTP id b49csp3033810wrg; Mon, 5 Mar 2018 12:53:21 -0800 (PST) X-Google-Smtp-Source: AG47ELuv8iD7ximgTMW4v7w89QHTp35FNnFRDH/ZMsrN/bK0TLVWofm4cJVvysn7ZNxsIZ85mFvd X-Received: by 10.98.166.200 with SMTP id r69mr16678265pfl.205.1520283201050; Mon, 05 Mar 2018 12:53:21 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1520283201; cv=none; d=google.com; s=arc-20160816; b=wimKTp2ON41fxIAwzKlELBvt3wMRH86NL5Kr99y09oEZeVYfLqdJC+hDp5d+ZR7Pc7 a8AMiRAdVQ5+yHTWNawGyPPzKN3yunNhfC+1ziE7g10J2g4eRWn2I4C4zdyz+CyVd2Wo 2ed8/WmHR3e/cJhYVHcmk6nf8m2wkXIxjUXRUDFyItTz9PeiKONJ2FRkntOzsGrc2WkR D5FR5n9Jcf3W/L+L1WJ1fX0Ca6un5g+BOv8DuAMEotKAnAAvfIpNhjHH3wdClH2H/XfD laGqGguKsXMG/6gTCEe5bQvir/OVafookMY3WwV4TUYUCH0OParsQ5THA+9CO3pLml73 2psA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:spamdiagnosticmetadata :spamdiagnosticoutput:user-agent:in-reply-to:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :dkim-signature:arc-authentication-results; bh=+/3N3xuLsuVoirgniVhFlLzRWY5RE5RpHi4G/Bwl5w4=; b=YZc+1CqJ8Mj+NIIbLUsJI0DZu2bVeqZJkrDJbuvsFaGUQeAf7WoKXUCEIaVd8pemyl J0xw8p4io0IEwOM/3CwrY7YRgtr7lMMqRpLrnkAhUBD+vcZteTbPEltlNRTxE/L2edVT qN+9FTdbxwM5mrWfjV04tgFU1ohMK0N3yTyKc7j9DUyedJcU6W5qZfgc52olrTmuxKGP zQe1VojYbWYjAZfpuWwp5rVpk97s5Lf9prdBTY3VzdIRpVyDLnJWqethBwQb200BIUHh PX67rml5x5jhUPPWWj5y1GABXaFlGyUg/bEt9FvFQs3H7VhFv2SHCj1RFgUoExr7rOXm q6Qw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@Mellanox.com header.s=selector1 header.b=W67zaXsJ; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=mellanox.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id 3-v6si9868507pll.585.2018.03.05.12.53.06; Mon, 05 Mar 2018 12:53:21 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@Mellanox.com header.s=selector1 header.b=W67zaXsJ; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=mellanox.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932388AbeCEUud (ORCPT + 99 others); Mon, 5 Mar 2018 15:50:33 -0500 Received: from mail-db5eur01on0080.outbound.protection.outlook.com ([104.47.2.80]:32236 "EHLO EUR01-DB5-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1752040AbeCEUu3 (ORCPT ); Mon, 5 Mar 2018 15:50:29 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=+/3N3xuLsuVoirgniVhFlLzRWY5RE5RpHi4G/Bwl5w4=; b=W67zaXsJnnHD/3dQrsYVL+BHj/pfd3lfsS/TT7y0E67/XWdycha26F6Zts6kdpfLmaIq917oKCEIBPcZ2o/jf6k7Fa4Jvd9++JLzIRP7LgUHrykVYMauLcG7TaQ0lRZGo8bAlt0ppdSQOWNKNNKc0lD6TRQsH0P6KicjwntMtUY= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=jgg@mellanox.com; Received: from mlx.ziepe.ca (70.74.179.152) by DB6PR0501MB2854.eurprd05.prod.outlook.com (2603:10a6:4:83::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.548.13; Mon, 5 Mar 2018 20:50:24 +0000 Received: from jgg by mlx.ziepe.ca with local (Exim 4.86_2) (envelope-from ) id 1esx3n-00072H-KT; Mon, 05 Mar 2018 13:50:15 -0700 Date: Mon, 5 Mar 2018 13:50:15 -0700 From: Jason Gunthorpe To: Keith Busch Cc: Sagi Grimberg , Oliver , Jens Axboe , "linux-nvdimm@lists.01.org" , linux-rdma@vger.kernel.org, linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, Alex Williamson , =?utf-8?B?SsOpcsO0bWU=?= Glisse , Benjamin Herrenschmidt , Bjorn Helgaas , Max Gurtovoy , Christoph Hellwig Subject: Re: [PATCH v2 07/10] nvme-pci: Use PCI p2pmem subsystem to manage the CMB Message-ID: <20180305205015.GT11337@mellanox.com> References: <20180228234006.21093-1-logang@deltatee.com> <20180228234006.21093-8-logang@deltatee.com> <20180305160004.GA30975@localhost.localdomain> <36c78987-006a-a97f-1d18-b0a08cbea9d4@grimberg.me> <20180305201053.GK11337@mellanox.com> <20180305204212.GD30975@localhost.localdomain> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20180305204212.GD30975@localhost.localdomain> User-Agent: Mutt/1.5.24 (2015-08-30) X-Originating-IP: [70.74.179.152] X-ClientProxiedBy: HE1PR06CA0139.eurprd06.prod.outlook.com (2603:10a6:7:16::26) To DB6PR0501MB2854.eurprd05.prod.outlook.com (2603:10a6:4:83::17) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 5802d96b-7781-4568-d4ad-08d582dab852 X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(7020095)(4652020)(48565401081)(5600026)(4604075)(4534165)(4627221)(201703031133081)(201702281549075)(2017052603307)(7153060)(7193020);SRVR:DB6PR0501MB2854; X-Microsoft-Exchange-Diagnostics: 1;DB6PR0501MB2854;3:uJiBKSL9VER5v73AfJf6p17tgZCw7Bkuyg/43OyYTnjbnSYUjnBzWVtgFVbaqfb4zs+WSOGnx9vkR3TM47TyvIQRtMj/3k+16q5F0ODkv9aYsosBKEPkPCyRSth72CwRsxpGCd3Yg8xKTGZywkf6p51GgRcIT0gOfxy6MXCy4tficFaFTkJ7rz94LrNetW6IO2nSd5vtvUt5BnFh3CnCTVkfhkeAJ7Hl2bP8TrOdtPJGwS6zWWg/H114Vis0vX0w;25:85vam3P9uEJ/ej9gTgv7QVIm9T0BqmgiOjsRXB18kJgP+b4/jNgvexmpfoEtaZ8Hsr7zlWG2sTJTQHv/plEgfmj3M9LDXL4Li/EEG7lmt0nlHjYHV8fLLtV04wZnuHglrigwHzF5A/ZgfeSrfX4pYVBHiBCm2RYtQqga7gZQ6mVpdcfV+CPspSyaeYZZts6a4o44ZxEtBTW4Hbx52ZbMqLl9TFEMHwzDdgvdHrkd/m9VPvm8pj2iMxYH5SRhF+0qyZAmvYWKHRCfGBH8pXUfAb28gr9525TN2h/mVimYkVHfcbU9xGt/amOhbpn1+AgeZttjETLzIU5pVrPedg+ieQ==;31:8v7ccR5++awPL/RdFTCi0b6tFOI4BI0iJ2SNZUo5ZROGHzz4Rk4+WhNRAKhHySzhBv883OSW9wF/uTSHkjPe5jY4GimVbS1QI0cn6+CmPSegLoZoZnHeJlm4rPuWqm2aErqbp5MuIXOuXyIJkiuTfjqly87YV8i9V+OnXePCiTtxhrxC4qebzewiK85LpM5aG6NheLuP1G6aGJb88uZcZ+LOJ1CxYD6acy0XaTJLZ4A= X-MS-TrafficTypeDiagnostic: DB6PR0501MB2854: X-Microsoft-Exchange-Diagnostics: 1;DB6PR0501MB2854;20:SD3adGDkrZIYHInPzbv5P5b2mksaLcZ8zR6qbxU0JnV0xzsxWksOiwFrZBLkIQ3ws4QzP6Pt1YJ2aGj40JeGteZOZr1lBiVsYcYVXY8CS5SpMqMy28CkWW1NWsn4EQL2phZ+HiCcLlxmd5DV4qk6zucRbVtHyhCy69uZel7y0B6t+35ikOoYVTNaUpZBZcGAkcLDc9Z6NvPnUC88CpVRnz3lwBrO5FcHkDAf5VSOg4hvjHFzPkHcFNQFnMwiD9IQ/Nozm0Kq4r2xvha5cx4ECUyCynJGBSR/cfxMl+Y8fTItGnaHsT8zKeFDN4NvFqfliB2hdfpHGo4zE94dGkPiHTpTWMorsHUES5ouU2STwSEjizgjRXBOidrkiXcqffTY66tiPFn5HjwtFhQiFcFv5OOsG9fWjHLIrk3QIDKUwlTNr+Rmchn1d1ZscLyWCkA34BUtOm4FzP2/zv1m35hA1XITU37D2Aomely41f2IRbmXpnIVmJIsbS/RLBm/OUFN;4:aUynAYiExpiGkRRWSd25pcpDvxAFNvG/bDT60nt1dRjtDCsVMNARCv4BQyak3F4HlGkZ1DiGEdiY7G8xj472uV5FQVc3nJ6J0Wr/Q00Cd1/VKKKC/kXOqt7UXzgGLnK3fDVyTOstZ+Ny9C5AxLWMnn5FWIrjNC2UPDmV/43qqrmNWKV8wo/OFMMm/xL19WjbogOG1BLvulS+xxuoziWoRLAe4FEGMOhlc0Sex07hpVBEEH2jXvPEoS7g4uGVlws8i1cku+ZzuSOqcNqOrhpebw== X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(8211001083)(6040501)(2401047)(8121501046)(5005006)(3231220)(944501244)(52105095)(93006095)(93001095)(3002001)(10201501046)(6055026)(6041288)(20161123560045)(20161123564045)(20161123562045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123558120)(6072148)(201708071742011);SRVR:DB6PR0501MB2854;BCL:0;PCL:0;RULEID:;SRVR:DB6PR0501MB2854; X-Forefront-PRVS: 06022AA85F X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10009020)(39860400002)(39380400002)(396003)(376002)(346002)(366004)(189003)(199004)(6916009)(26005)(7736002)(6246003)(2906002)(9786002)(186003)(9746002)(83796002)(46656002)(66066001)(52116002)(316002)(58126008)(305945005)(86362001)(8936002)(7416002)(106356001)(16586007)(2950100002)(54906003)(8676002)(3846002)(6116002)(69596002)(386003)(39060400002)(1076002)(50466002)(23726003)(81166006)(4326008)(229853002)(33656002)(122856001)(97736004)(68736007)(93886005)(105586002)(53936002)(76176011)(5660300001)(47776003)(36756003)(57986006)(81156014)(478600001)(18370500001)(24400500001)(42262002);DIR:OUT;SFP:1101;SCL:1;SRVR:DB6PR0501MB2854;H:mlx.ziepe.ca;FPR:;SPF:None;PTR:InfoNoRecords;MX:1;A:1;LANG:en; Received-SPF: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1;DB6PR0501MB2854;23:6b468pxLAwxDwWeGOiN2xLbQ2ppl/nUYvDxU5uD?= =?us-ascii?Q?8/nyqcwL2Mqp+UYN9NUKQ5bN0vAfzuYN3XiNuN1+ODkw5XJ0Sfz57nz0o7aS?= =?us-ascii?Q?s2StuWko/XvpoBKgHvl/M28CP7/F0Y3Yjdh0Nhont0l6vKjc/LS1KzaIeVAa?= =?us-ascii?Q?yHJWNzkXiO2B7wISQH3z0gnHBYHc//4zC+ZBK2sFVOaAzQtsoD5Pw0Xg3rYv?= =?us-ascii?Q?QiDA8kWxIIpLJw9OVFIoXoHhgCerYyl3CulfWT1LKV2r56amHMPYJKafp2ie?= =?us-ascii?Q?XCDWWT187Ahzw76qnU3a8Wly5Cro/Kmfu5Gt14pse5Bo5IHCuvK5cf9y//Mk?= =?us-ascii?Q?A7Eky2qOKKgSETnnr+KNyexEQuMBXCQv+FOW9u/2tnBFhOkJTOmL6GqC4k/g?= =?us-ascii?Q?BcMXdwVgbcy+6FxYKzSyKw0JjbP/fjUlFwCY7ZBIOWtc9+8xWbGN8bUXoVsW?= =?us-ascii?Q?yZ7fQ5qcmfJfjBmQDjOLM+LuE5jE24i6JRBQdp2EkcsLMI1VV3PP5ns6sWom?= =?us-ascii?Q?+7xcwmeUbuY83aQ+sbYq3kLNO9CksETU0jycMhuSDiJhLZ3inrdRtSvb9KAy?= =?us-ascii?Q?NzPrcwCuT1r5Ws6NNyN8NY9o+d1698YZSGTRXeXqxZAj69wnnp2gT7Kb0fzp?= =?us-ascii?Q?0Z4bS8ieo5rMdtYYXLXonKWUlm6jeuXPe00RLb76o0Ok7nKU1c7quI1Auc6p?= =?us-ascii?Q?X7zzrYC0xCXmqk3l431PaION7qHLP7jrVGy2p4Ve/7DDVV+J6kPvTkj6m1mj?= =?us-ascii?Q?ss0Q2Sa6hImAFP456siQfikxQLT6Oza6MvntcUEnIbdVA+MZ5P9gXSdvFnkB?= =?us-ascii?Q?aGUQ1bc8y9BIDjqpkxrZKn8WSiarAH+96xUFTaKEEKHfhd22BN35j72C/teL?= =?us-ascii?Q?ovpbMXQyLMsZVDtd5lHYaY7eFMUOndyppHM8Dd2I3hWwVSq9mEKBAIWl813s?= =?us-ascii?Q?BsTHINJU83tomfzYu/DJZCEwLOHRKPw4axb03Pa2y1xCg2g4ergFOsYPBrg3?= =?us-ascii?Q?JqdAPNdIoCllw4dgy6HGwmY5cXgTc7mWsouqV/0SCJKm2okDsX/Ah5lBwAd1?= =?us-ascii?Q?Ylzj+dTqRT3hG+AyVUwSUMrn2Muvrd5dZ03hMFe4z/kAHCc5t28O9XHesm66?= =?us-ascii?Q?fHE1H5Jv2jF0JovuPty8tNY+OTQ3tSgR962gh+9eEFwCXvFQLlih8+kcSNal?= =?us-ascii?Q?bDwXSnze1/7jbwtpoLaLExwhx8e/P5uP4N9suxc+s0PnRUsTVUON7WI9KPc6?= =?us-ascii?Q?eGlOV1x0DOwOMVKN/CXl4Dj+ZCGMJSN99i53yM/hm0qM+/yRK2KXB6L3Vy/U?= =?us-ascii?Q?yEemwBgy3suCJOXI0E2fipQIuY2BhWGr+Bi5ogakA+EK17SR9rZLr83U0rxS?= =?us-ascii?Q?643WqLQ=3D=3D?= X-Microsoft-Antispam-Message-Info: W1wgmrM1RHTxCCp9Aessek4BtTDRQ7TkhQZEpKC+Un7rv549jcWVJjFg3TPzWv3wnOKWaDjoWPw1ka+nMFR7D9WCWJ6OZCgi+9rf8j0UgPrZMVTwyt+yUKs6vMUjopSgcBUt+yNI07e2iNJ6BE7oNSzWMnzJEVkUC65v8FETHuueTZp00e7iFLCDjnHZ2VJv X-Microsoft-Exchange-Diagnostics: 1;DB6PR0501MB2854;6:OEhKdhgTFwGaT+PqUMrMpQpcDBGpuA1idz8REA/+ky0Ktna6TE228Qk4Z8AT31683pS88HA1t57bK4qkVwCG6X2YB/s5KimQZYd0OOoizfF3MPu9Jge5lWjMGjn3B1LPIZBclOthZ6WChXNUvzQy1veEPKcXkGXuQq+9tizZYOcexqC/grroTzN3Y84jV2mimNFzLwWMYJ0dfRixS6KX/CCc1WA2WNPMzMVhx9rIiz9ZlTn4Fh1yNqSv9bXNXpfuNQ/wbilXGDyMYxMSlJDE1JisgVv4ZA6FM77fP5xBSr0eMBYvXPcmlXIiOovbepgzCQGxpZ7DholdyfLw18PJ7bagRYeBBIUqsDTDTIUk6Rg=;5:SXcts4bFH4or3RFeaSAxxq08BWC9kD5wpxwXF8VTKAUQwwc+um8IxmVEkFUxVvFhuGqNFfFZH8Y74ktTavhsyh5VNSMh/Urew7Q94yHfa1jthYHrelGv2BKrkupWI5lKeRvMCa+CXkhHOZOztE6ifGH7NbsH2E+aKe758caCKSY=;24:uNqqO81m/qDlAzdNLxrSho6SaU88VjaQLS94fDxAyVNhrMiEDouosQolbCqEYqHVSrhpRoVYOvprYIoauq/kw8EZ+w8HpvhtTew9oTtQrpg=;7:KwQpZFIp7RRfIeUkf/1DUER+sZlaRTPTglwBizCMcmUVrJyGVfK4q/gKID79i8nHExB608XR9RrmtXv/Q6UV93dijEd0wZ7ZlQu500YcJQyuCxx2qGcyRXYZBF++2PRYZqqAAHXRm4TYx2Yju1YGLlqePjJ64/h1BJxbjkARLnVtyOrKNbbUTyNhPdIwjUaIegqzmMiap9S+W0flYxGwXapQ0zHgbTw80fUzvhpL3XIGNH5Pgiopx5AvWQXG5vFC SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Mar 2018 20:50:24.7210 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 5802d96b-7781-4568-d4ad-08d582dab852 X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB6PR0501MB2854 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, Mar 05, 2018 at 01:42:12PM -0700, Keith Busch wrote: > On Mon, Mar 05, 2018 at 01:10:53PM -0700, Jason Gunthorpe wrote: > > So when reading the above mlx code, we see the first wmb() being used > > to ensure that CPU stores to cachable memory are visible to the DMA > > triggered by the doorbell ring. > > IIUC, we don't need a similar barrier for NVMe to ensure memory is > visibile to DMA since the SQE memory is allocated DMA coherent when the > SQ is not within a CMB. You still need it. DMA coherent just means you don't need to call the DMA API after writing, it says nothing about CPU ordering. eg on x86 DMA coherent is just normal system memory, and you do need the SFENCE betweeen system memory stores and DMA triggering MMIO, apparently. Jason