Date:   Thu, 6 Jan 2022 08:34:56 -0400
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Daniel Jordan <daniel.m.jordan@oracle.com>
Cc:     Alexander Duyck <alexanderduyck@fb.com>,
        Alex Williamson <alex.williamson@redhat.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        Ben Segall <bsegall@google.com>,
        Cornelia Huck <cohuck@redhat.com>,
        Dan Williams <dan.j.williams@intel.com>,
        Dave Hansen <dave.hansen@linux.intel.com>,
        Dietmar Eggemann <dietmar.eggemann@arm.com>,
        Herbert Xu <herbert@gondor.apana.org.au>,
        Ingo Molnar <mingo@redhat.com>,
        Johannes Weiner <hannes@cmpxchg.org>,
        Josh Triplett <josh@joshtriplett.org>,
        Michal Hocko <mhocko@suse.com>, Nico Pache <npache@redhat.com>,
        Pasha Tatashin <pasha.tatashin@soleen.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Steffen Klassert <steffen.klassert@secunet.com>,
        Steve Sistare <steven.sistare@oracle.com>,
        Tejun Heo <tj@kernel.org>,
        Tim Chen <tim.c.chen@linux.intel.com>,
        Vincent Guittot <vincent.guittot@linaro.org>,
        linux-mm@kvack.org, kvm@vger.kernel.org,
        linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org
Subject: Re: [RFC 08/16] vfio/type1: Cache locked_vm to ease mmap_lock
 contention
Message-ID: <20220106123456.GZ2328285@nvidia.com>
References: <20220106004656.126790-1-daniel.m.jordan@oracle.com>
 <20220106004656.126790-9-daniel.m.jordan@oracle.com>
 <20220106005339.GX2328285@nvidia.com>
 <20220106011708.6ajbhzgreevu62gl@oracle.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20220106011708.6ajbhzgreevu62gl@oracle.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?+VIJJ/crGtRev+Dysi81iO2Z2KOGwtnlcTR0cPn1wRFRInzJ9Gkkp/W0VBYP?=
 =?us-ascii?Q?MrNtosjv7Td204apZn2uPi0tRmwuGGKrPfmTpVGBI2+uSJMG/VXfJDUDyZty?=
 =?us-ascii?Q?ugRJOX8b1canAO6gDI8BwPNZ/B1E9nZQr4SZ2t9/2TLaQgGie8f1DNkcfkdA?=
 =?us-ascii?Q?DiSfVW2Ot0N19MGlJa4R0qwuqzmA3goB4rqGy+zrhnKlyg5kj3cruZ1CccK3?=
 =?us-ascii?Q?D0Zh0pHJtWM6Mc2i1RFEStELH6JuBacvInjSvNkMsr4XNz1CV4wmKvEIupYe?=
 =?us-ascii?Q?IrozhGWe3CxBk6P6pcMJX5sHiNQNWyFyoAGcDynLcBTx09Xg5DRC7sZsu7tz?=
 =?us-ascii?Q?HhDUuSZy9uEAozwyTpLapO0GXQK7vfCeIYuaJm53kx6mxA6VAfAt40SRPTAy?=
 =?us-ascii?Q?ibSjK5gT6BmJeKjc9N8JXnFo+tIzbJvvEM3zE+qvywGg/b1Sa1w+9D8iFkEE?=
 =?us-ascii?Q?w5MuuKPcXq3GPa2KIqSp9TB1qOMMVtaZ0FKcNdt0zAjngAQFi1nQH56Rw75W?=
 =?us-ascii?Q?giiYPADZ6E9+8Jm7g0j7ctHbEzZScFur/j0cnOJKhTxLo3Ky92O+XTC9ClHA?=
 =?us-ascii?Q?4BUQxgnFhyChdrkrRERSPlgohpUvh9zVlDHswUQPBweaddLY03scdYYl2CtA?=
 =?us-ascii?Q?XKw1pNK8QTlU5Nq0oVEu/AX/N1a0W+Dgh05+rPt+TORF+itTlYKKpUMGZNlo?=
 =?us-ascii?Q?3Js4a7oBx1KVmdT37U+gC9Q4n1Xld6AUaHeqGHOUFyDaVff5IOk1hMHHzRj8?=
 =?us-ascii?Q?aHsdjOwqFyAB7UXt8jEwl3tvhQ9C9yXQskPrSz6GwFlO3r2d+qSza352ir9a?=
 =?us-ascii?Q?GHywAkcD3Dt6Foxeh1sngvBQW+yDzEfWGu9Eaeoy+Vi78McztLTLSNxcyIrx?=
 =?us-ascii?Q?rljtwNYcTu7b58XFxZju8SFmz4b6kpPjwIk/bk1WVpJyu0xhLJjpnAFk2gUf?=
 =?us-ascii?Q?g49rU3OlXLzy6pg2J+GMU+X4Lx1kvjaTew1qfhMep4DEhMrkBg2LNjQkRygk?=
 =?us-ascii?Q?StmxvHziaQbwV4WdvdH01LSNkjPeL3pI/FCFOeQlwrqozPJojeXkq8CsYpsR?=
 =?us-ascii?Q?N6wsgFzF2HhCtfbZPIhkG1BFbxPmfz6nuDQOSui7kF1xGSiZ0EL4QpK7lAco?=
 =?us-ascii?Q?gn0rSl+hqFyOEO+3HbuqbHPSoDBz2TUQl6huRnjBOwkP+QUGWKjWcOWKabCm?=
 =?us-ascii?Q?zyK5qXhrKJka5XKjfWtbI+sH/3Bzirdu3Ew7jd8R/ZiKPU4VssbamKpk9PuO?=
 =?us-ascii?Q?TyJNSz68Abn+3hYFo0Fj/HP3LBmZEAO7H/fRIspCNdod3BPZeXE4R69ct3pQ?=
 =?us-ascii?Q?+PaGspPhkDZ2DgIPZ/lbZ1dsF9BJFhofvUoHEoRSC4I1EjC8s2BieVlU8LBZ?=
 =?us-ascii?Q?Qg1a6U7F2NnrGMD8MagrpbWEUYp/Ep0D5KligCdwQRMiXaF44xeyB3bShCpJ?=
 =?us-ascii?Q?lHcjSgDbTzLXSXomgeRLalQqyoiXTYSWDePQRhgkh3clclFYEoCvOTLWn6xk?=
 =?us-ascii?Q?xPfvBuHYgMqcpqxtjhSy00lrtjsIkTvzVv0Ti1CL/fr5N+p/q7eqoort2Sv7?=
 =?us-ascii?Q?+Pvph1xMQh/QrtVMINI=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 9a45fbcc-9c80-43df-29e6-08d9d110f3ee
X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB5506.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Jan 2022 12:34:59.5706
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: OvVUX+KVIV7YoFOZA1DF7W10bCwF24HDBHeiVZ9Il6QAILPGE11vz85bRVRzG6JT
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5208
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Wed, Jan 05, 2022 at 08:17:08PM -0500, Daniel Jordan wrote:
> On Wed, Jan 05, 2022 at 08:53:39PM -0400, Jason Gunthorpe wrote:
> > On Wed, Jan 05, 2022 at 07:46:48PM -0500, Daniel Jordan wrote:
> > > padata threads hold mmap_lock as reader for the majority of their
> > > runtime in order to call pin_user_pages_remote(), but they also
> > > periodically take mmap_lock as writer for short periods to adjust
> > > mm->locked_vm, hurting parallelism.
> > > 
> > > Alleviate the write-side contention with a per-thread cache of locked_vm
> > > which allows taking mmap_lock as writer far less frequently.
> > > 
> > > Failure to refill the cache due to insufficient locked_vm will not cause
> > > the entire pinning operation to error out.  This avoids spurious failure
> > > in case some pinned pages aren't accounted to locked_vm.
> > > 
> > > Cache size is limited to provide some protection in the unlikely event
> > > of a concurrent locked_vm accounting operation in the same address space
> > > needlessly failing in case the cache takes more locked_vm than it needs.
> > 
> > Why not just do the pinned page accounting once at the start? Why does
> > it have to be done incrementally?
> 
> Yeah, good question.  I tried doing it that way recently and it did
> improve performance a bit, but I thought it wasn't enough of a gain to
> justify how it overaccounted by the size of the entire pin.

Why would it over account?

Jason