Received: by 2002:ad5:474a:0:0:0:0:0 with SMTP id i10csp3118904imu; Mon, 19 Nov 2018 10:54:49 -0800 (PST) X-Google-Smtp-Source: AFSGD/UYMKk1ndHm5jS4etp9djRUvESIJE/P6BM416f3BXOWHlTNhJ3zSGNHCePXQIA6V2j7NkR/ X-Received: by 2002:a17:902:8607:: with SMTP id f7mr7521744plo.123.1542653689868; Mon, 19 Nov 2018 10:54:49 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1542653689; cv=none; d=google.com; s=arc-20160816; b=IBSmPc9jEl+I50wNTOP/KkjcaMikqsHtrfG5EZWjPkhLew4abnc1xnJkYvYIMsDsDU UKIW/XhcRDJsFiXMuB1/eo8E6rBeBW73mFGinWKfIeU2XHq8huUzpnGe8RmqdjJGO0ca wtMTf9ra0U11XLRqIpjfkvyz+iSLkozQKYIQnyD4bpLPl0cDIE+8gAY/vQrCqOj6lGr7 yCcDDcVAB5SsQSWFGfINHVvRReDc4cKTaOTmufxArnuuvTG8n2GvpznxE3kuy4RKVVoP 3uhfMAKacllV7OFq5H53UYSz+b+G9Kt4tX676bm6+uxod/6DZniEg0EbaKeJS0P8g4ji t0Pw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:user-agent:in-reply-to :content-transfer-encoding:content-disposition:mime-version :references:message-id:subject:cc:to:from:date; bh=tzo5wuuqIt2mNGOx8ovO71PD8ZGxFATSwK/ZZ7o0fXY=; b=k3G2T8y6iNa8jQFkNJELUAUC129VXaRL8EEBop2kyHrz88vALB4Jap3zR0dJKpYt1J hDTvF2dDELLvP1RE9Ql3170kR1nH6677LoCvirRkydZnk8s+Fw3t/3xO6YIqrxjMbi9O OdRiJQKGG3THNqYxxMSjNoKOr088kAnRNMapLex5rIP32lsYwiRmoZnLz0dMWoJrdm8n l78nyg93bFK80bEtOa7TN95PX9FWfc+X1kQ38Tqb1X4AOTuugrEkQCN9PIlcIoPv8Xt/ b6hS5jeDElQ3zsaoQFArpeSwbKfW9giAwffv417tRyFkyXoTu0lxyAwgYuDCSzkQoNCN c5EQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=redhat.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id s198-v6si50501588pfs.248.2018.11.19.10.54.35; Mon, 19 Nov 2018 10:54:49 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726372AbeKTFHK (ORCPT + 99 others); Tue, 20 Nov 2018 00:07:10 -0500 Received: from mx1.redhat.com ([209.132.183.28]:48704 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726167AbeKTFHK (ORCPT ); Tue, 20 Nov 2018 00:07:10 -0500 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id AC48630842AA; Mon, 19 Nov 2018 18:42:22 +0000 (UTC) Received: from redhat.com (ovpn-124-1.rdu2.redhat.com [10.10.124.1]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 31E825D73F; Mon, 19 Nov 2018 18:42:18 +0000 (UTC) Date: Mon, 19 Nov 2018 13:42:16 -0500 From: Jerome Glisse To: Jason Gunthorpe Cc: Leon Romanovsky , Kenneth Lee , Tim Sell , linux-doc@vger.kernel.org, Alexander Shishkin , Zaibo Xu , zhangfei.gao@foxmail.com, linuxarm@huawei.com, haojian.zhuang@linaro.org, Christoph Lameter , Hao Fang , Gavin Schenk , RDMA mailing list , Zhou Wang , Doug Ledford , Uwe =?iso-8859-1?Q?Kleine-K=F6nig?= , David Kershner , Kenneth Lee , Johan Hovold , Cyrille Pitchen , Sagar Dharia , Jens Axboe , guodong.xu@linaro.org, linux-netdev , Randy Dunlap , linux-kernel@vger.kernel.org, Vinod Koul , linux-crypto@vger.kernel.org, Philippe Ombredanne , Sanyog Kale , "David S. Miller" , linux-accelerators@lists.ozlabs.org Subject: Re: [RFCv3 PATCH 1/6] uacce: Add documents for WarpDrive/uacce Message-ID: <20181119184215.GB4593@redhat.com> References: <20181113002354.GO3695@mtr-leonro.mtl.com> <95310df4-b32c-42f0-c750-3ad5eb89b3dd@gmail.com> <20181114160017.GI3759@mtr-leonro.mtl.com> <20181115085109.GD157308@Turing-Arch-b> <20181115145455.GN3759@mtr-leonro.mtl.com> <20181119091405.GE157308@Turing-Arch-b> <20181119091910.GF157308@Turing-Arch-b> <20181119104801.GF8268@mtr-leonro.mtl.com> <20181119164853.GA4593@redhat.com> <20181119182752.GA4890@ziepe.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20181119182752.GA4890@ziepe.ca> User-Agent: Mutt/1.10.1 (2018-07-13) X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.40]); Mon, 19 Nov 2018 18:42:23 +0000 (UTC) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, Nov 19, 2018 at 11:27:52AM -0700, Jason Gunthorpe wrote: > On Mon, Nov 19, 2018 at 11:48:54AM -0500, Jerome Glisse wrote: > > > Just to comment on this, any infiniband driver which use umem and do > > not have ODP (here ODP for me means listening to mmu notifier so all > > infiniband driver except mlx5) will be affected by same issue AFAICT. > > > > AFAICT there is no special thing happening after fork() inside any of > > those driver. So if parent create a umem mr before fork() and program > > hardware with it then after fork() the parent might start using new > > page for the umem range while the old memory is use by the child. The > > reverse is also true (parent using old memory and child new memory) > > bottom line you can not predict which memory the child or the parent > > will use for the range after fork(). > > > > So no matter what you consider the child or the parent, what the hw > > will use for the mr is unlikely to match what the CPU use for the > > same virtual address. In other word: > > > > Before fork: > > CPU parent: virtual addr ptr1 -> physical address = 0xCAFE > > HARDWARE: virtual addr ptr1 -> physical address = 0xCAFE > > > > Case 1: > > CPU parent: virtual addr ptr1 -> physical address = 0xCAFE > > CPU child: virtual addr ptr1 -> physical address = 0xDEAD > > HARDWARE: virtual addr ptr1 -> physical address = 0xCAFE > > > > Case 2: > > CPU parent: virtual addr ptr1 -> physical address = 0xBEEF > > CPU child: virtual addr ptr1 -> physical address = 0xCAFE > > HARDWARE: virtual addr ptr1 -> physical address = 0xCAFE > > IIRC this is solved in IB by automatically calling > madvise(MADV_DONTFORK) before creating the MR. > > MADV_DONTFORK > .. This is useful to prevent copy-on-write semantics from changing the > physical location of a page if the parent writes to it after a > fork(2) .. This would work around the issue but this is not transparent ie range marked with DONTFORK no longer behave as expected from the application point of view. Also it relies on userspace doing the right thing (which is not something i usualy trust :)). Cheers, J?r?me