Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Message-ID: <a1cd336e-60c0-b72c-022f-4b53c0b3fc51@nvidia.com>
Date:   Fri, 1 Oct 2021 16:27:43 -0700
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.1.1
Subject: Re: [PATCH v3 19/20] PCI/P2PDMA: introduce pci_mmap_p2pmem()
Content-Language: en-US
To:     Jason Gunthorpe <jgg@ziepe.ca>,
        Logan Gunthorpe <logang@deltatee.com>
Cc:     Alistair Popple <apopple@nvidia.com>,
        Felix Kuehling <Felix.Kuehling@amd.com>,
        Christoph Hellwig <hch@lst.de>,
        Dan Williams <dan.j.williams@intel.com>,
        linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org,
        linux-block@vger.kernel.org, linux-pci@vger.kernel.org,
        linux-mm@kvack.org, iommu@lists.linux-foundation.org,
        Stephen Bates <sbates@raithlin.com>,
        =?UTF-8?Q?Christian_K=c3=b6nig?= <christian.koenig@amd.com>,
        Don Dutile <ddutile@redhat.com>,
        Matthew Wilcox <willy@infradead.org>,
        Daniel Vetter <daniel.vetter@ffwll.ch>,
        Jakowski Andrzej <andrzej.jakowski@intel.com>,
        Minturn Dave B <dave.b.minturn@intel.com>,
        Jason Ekstrand <jason@jlekstrand.net>,
        Dave Hansen <dave.hansen@linux.intel.com>,
        Xiong Jianxin <jianxin.xiong@intel.com>,
        Bjorn Helgaas <helgaas@kernel.org>,
        Ira Weiny <ira.weiny@intel.com>,
        Robin Murphy <robin.murphy@arm.com>,
        Martin Oliveira <martin.oliveira@eideticom.com>,
        Chaitanya Kulkarni <ckulkarnilinux@gmail.com>
References: <32ce26d7-86e9-f8d5-f0cf-40497946efe9@deltatee.com>
 <20210929233540.GF3544071@ziepe.ca>
 <f9a83402-3d66-7437-ca47-77bac4108424@deltatee.com>
 <20210930003652.GH3544071@ziepe.ca> <20211001134856.GN3544071@ziepe.ca>
 <4fdd337b-fa35-a909-5eee-823bfd1e9dc4@deltatee.com>
 <20211001174511.GQ3544071@ziepe.ca>
 <95ada0ac-08cc-5b77-8675-b955b1b6d488@deltatee.com>
 <20211001221405.GR3544071@ziepe.ca>
 <8871549c-63b5-d062-87ea-9036605984d5@deltatee.com>
 <20211001224605.GS3544071@ziepe.ca>
From:   John Hubbard <jhubbard@nvidia.com>
In-Reply-To: <20211001224605.GS3544071@ziepe.ca>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?SnJEbTBnQWUwb1UySGhrRE4wTWJQSTJxU2wwY2h3V2Uzai9xUzFyamxQaFY5?=
 =?utf-8?B?NjBPRCs5ZkZVM0wxQUNIMmgvWldKS0FOaDlqb2Z6RDc0YTVUV0U2MmxZSS9T?=
 =?utf-8?B?WjZmY1I2S1RWM3dIWW44VEE2akh4aUdzV3ltMXg2S2RtbUc3U0pabWJObmNm?=
 =?utf-8?B?a21SQWIwQnlHd3NkYkNiTTRXcVBVUkFiZk93RW1XWWRBdHloR2w4MmlPWTI0?=
 =?utf-8?B?ZjRSMWgxajJpOWgzOUl6OU1mYW5DMHV1Qi9pVktWVGxVSTF2TlZ2aUxYT2hK?=
 =?utf-8?B?UkxNR2tsK3hBUllKZmY1RzVJazJXSE5wVG1IbHdESHhSRXN2TXU0TUNtSTVR?=
 =?utf-8?B?aFFxWGE5dG5vMW1keDhaVDZHeTZiZFhLSER2T3hsU1cwWkpxdk5ybkY4S1Ru?=
 =?utf-8?B?aHF0WlRuaW9Sd0ZCc0RLRGRjc3dCQmpaekJiV29vV1VaMnJOdVBDTVU2eVNE?=
 =?utf-8?B?alNHR3dZaHZrMnBrT2xQbllTMTJLRmt4MUlMNXQzRFJNYVVCZkNtNDh1aGg2?=
 =?utf-8?B?QXQ0bDVuRjZ1RFB4S1VDMG9Kc2Jac2wvQnBRSnZGeE9IOU1zWVFLVnhwNTRp?=
 =?utf-8?B?KzR6UzE3UHBrOXVQMjVTSlpUOHE1Y0I4SVQra2pwamo3dDJVYXlOZVl0N1Y0?=
 =?utf-8?B?d0hyM2RHaXRnN2JmbTVoVmE1M2lFTmJHcndONytrQlRxM05wMHNSNXhZYm9k?=
 =?utf-8?B?WjlnVGE5SXp3Z0kxcVJBK2RUWmZTd0V6TWQyaEUyazRJMm10aFdWOUlSN256?=
 =?utf-8?B?amhabG9XUkQ2YUI3djc1OFVkTldINzhTOUw5T2JrUi9KbHpMZ0dEVllYVVRM?=
 =?utf-8?B?bnQwYnc5UCtiVFk3UUpBN1lLeEZQMEliQVJicmNTeFpkQVVHcEpDQ2RDSDdy?=
 =?utf-8?B?TnVMYzU3VENBaGZuVHh3OFlLY0lkMHNQVkFkVzhGMzc3UFdwajYrdEhJV3Vi?=
 =?utf-8?B?d3ZOV29EMm93VXlyUXUrd1BUUzFtelg1VkVGWjNob1FQNFg1NDMrZ2hYdnhx?=
 =?utf-8?B?emZwZmtyWFJVbGFWajEzS01EL2pKUUVXZWdGTzZyQmlhU0hmalRPZHVZUTRm?=
 =?utf-8?B?N1hVMEFET1V2RnhJdWovUDUwOHpUTFVVQjdBZUJJQ0RFQzRUcmthUVhjYm1x?=
 =?utf-8?B?Tk8zQlZSU1pWUXVrUmxXcDRoT1QxWDVNZzU4UlhPNlk5bEdSOHRRelB4cmYw?=
 =?utf-8?B?L2hFb21wM2NSWHorNXowUEtoM0FlY205YkJHSUEyZXhIN2ZpWHBCMFhFSnd6?=
 =?utf-8?B?ZU9BSkdpc2ZZSlJSZm1LRTdYWDRWN1gzRHRWS0xqWisxblVkcmZxam1pTjQy?=
 =?utf-8?B?cWQ1ZU1iWFR4T2k2cU1VTHpMbHR4SzZINWRtenFRV29YRGdveGd1b0tRaGpI?=
 =?utf-8?B?bnk2WlBpWk5aUnhFUWNHTjg0RFk3YU8zVDVkM09zR3hKeUYvekN2dWNkbkp4?=
 =?utf-8?B?cTk1MURPemU1c1NsWDB3S0JMWkg4NXplbXdoK0NSbWhLKytyVU5OdnRqUCtP?=
 =?utf-8?B?WUk5d1ZYa0dnVC9CZDBtckJhVGw3MTNvMllsb2lLRjhCa09xK1U4WDhvWUc1?=
 =?utf-8?B?cFNMelNQNU9WWG1BVitNUTUrMWJsd1VwZ2s1UUwzQVo1bXIvbS9PTWU2RmZO?=
 =?utf-8?B?cHBnczVuenE1QzlRWFVualhjRndQNDhoOVF4OUxrR094VGhjTmdod0syRHIx?=
 =?utf-8?B?Ui80NjVMbGZzTmpSWnN5SzdWSm9yQ1ZPaG5LVFY3WXpmd29hZHJlMFFNbkpR?=
 =?utf-8?Q?NpZH3490Oh/k7yG704+tiRVll3GSORGEzqfPLBU?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: b8f04de5-bdf2-4f45-e510-08d985331268
X-MS-Exchange-CrossTenant-AuthSource: BY5PR12MB4130.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Oct 2021 23:27:44.9292
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: ZEJzYs6/A3tYihDqmP7YI7oSA9DXCFwZvXqao7qflcx3inwmHXmFnY0WjpKjEBTu+S4+7kewnko27NazOUzFlg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR12MB4789
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 10/1/21 15:46, Jason Gunthorpe wrote:
> On Fri, Oct 01, 2021 at 04:22:28PM -0600, Logan Gunthorpe wrote:
> 
>>> It would close this issue, however synchronize_rcu() is very slow
>>> (think > 1second) in some cases and thus cannot be inserted here.
>>
>> It shouldn't be *that* slow, at least not the vast majority of the
>> time... it seems a bit unreasonable that a CPU wouldn't schedule for
>> more than a second.
> 
> I've seen bug reports on exactly this, it is well known. Loaded
> big multi-cpu systems have high delays here, for whatever reason.
> 

So have I. One reason is that synchronize_rcu() doesn't merely wait
for a context switch on each CPU--it also waits for callbacks (such as
those set up by call_rcu(), if I understand correctly) to run.

These can really add up to something quite substantial. In fact, I don't
think there is an upper limit on the running times, anywhere.

thanks,
-- 
John Hubbard
NVIDIA