Date:   Wed, 5 Jan 2022 20:17:08 -0500
From:   Daniel Jordan <daniel.m.jordan@oracle.com>
To:     Jason Gunthorpe <jgg@nvidia.com>
Cc:     Alexander Duyck <alexanderduyck@fb.com>,
        Alex Williamson <alex.williamson@redhat.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        Ben Segall <bsegall@google.com>,
        Cornelia Huck <cohuck@redhat.com>,
        Dan Williams <dan.j.williams@intel.com>,
        Dave Hansen <dave.hansen@linux.intel.com>,
        Dietmar Eggemann <dietmar.eggemann@arm.com>,
        Herbert Xu <herbert@gondor.apana.org.au>,
        Ingo Molnar <mingo@redhat.com>,
        Johannes Weiner <hannes@cmpxchg.org>,
        Josh Triplett <josh@joshtriplett.org>,
        Michal Hocko <mhocko@suse.com>, Nico Pache <npache@redhat.com>,
        Pasha Tatashin <pasha.tatashin@soleen.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Steffen Klassert <steffen.klassert@secunet.com>,
        Steve Sistare <steven.sistare@oracle.com>,
        Tejun Heo <tj@kernel.org>,
        Tim Chen <tim.c.chen@linux.intel.com>,
        Vincent Guittot <vincent.guittot@linaro.org>,
        linux-mm@kvack.org, kvm@vger.kernel.org,
        linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org
Subject: Re: [RFC 08/16] vfio/type1: Cache locked_vm to ease mmap_lock
 contention
Message-ID: <20220106011708.6ajbhzgreevu62gl@oracle.com>
References: <20220106004656.126790-1-daniel.m.jordan@oracle.com>
 <20220106004656.126790-9-daniel.m.jordan@oracle.com>
 <20220106005339.GX2328285@nvidia.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20220106005339.GX2328285@nvidia.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?f3nz0N/Q8WBuoFNEA1/Z+byvMepmKiF+ztGSEYTfmUlWDoNbSC6oPRcfiJSB?=
 =?us-ascii?Q?H7+4rms9JyyZrV2nm6G87Svin8DJI/zC8b5Bz5BFgLlNvfwuqEqPblZekHur?=
 =?us-ascii?Q?/J62HGzw3+JQPMeBIiytwkxzwNQZ5cCEmosbu7mvQlAirXb+ja3l9EdIqLZ3?=
 =?us-ascii?Q?bI3946bxf82gpagfpnS4nAF5FsdQcN3uqONO383V2LUxdIoaVVHdJBBKfod/?=
 =?us-ascii?Q?E43UcJ3bp6hBJofeLc4R7mfENxMok9OfFOKOLFHTaMVmj+Y/ZwAbUjW5LPVQ?=
 =?us-ascii?Q?LjucGesuohO7xGyzPyoepEcyh1Zn/SKJI8e3G+1GT3sNlLu8AC+vsE2/gmpO?=
 =?us-ascii?Q?xcrF2I1LqazojWNOTtprQi7wGzYGSq3gbtrvT8wXLlGS48RiXeR7I19kzJAg?=
 =?us-ascii?Q?yj8B7Stf/dy8fpUhHjIq6ae1Im7feAqwWYyz2vz2gEUtLI/mxKRAEWIQem0P?=
 =?us-ascii?Q?KluurfkLmgyWy3PxkDgIZq/nrno/UEQMjUzMuPWD+8WdHyjk296jnmBAha8q?=
 =?us-ascii?Q?e1B+gri/lm49EWdRLE7DOAwlm+Pf83ba+hDgm7Uz0Hp5kzKixGxMhf6kDkbB?=
 =?us-ascii?Q?ar50lDLs1wqPKCqyKKDwuyzo2+cw2DBIix0Gjobf7Oc7hhaxkLAmh1w3VOkI?=
 =?us-ascii?Q?BIJlhc8RLQq0lXHLAFaCEWX3A+Al4aDkeQfFYG5cvCKqPPEABu0bmUWxGVhT?=
 =?us-ascii?Q?R34vdn2mfPw3jS1357X/nJOHi1es+RPSSdaSMSILrEUK1R9cbfKorSmEdJku?=
 =?us-ascii?Q?Y3i/khAEIQdAQjFrzaLVXIrkLYPghmmIDe5jtwgi0MyxHjs9eT1sX3abCRbJ?=
 =?us-ascii?Q?eeSI4Wk9ItItVoZ4fdu9zn1+5KMmhYmJcpT0ImHC71yVddUES6UVS7HBWl7m?=
 =?us-ascii?Q?eKXOqgaBhqHoxBo7pdIczTGcxUNlbnfVX/NElndrOQwg/hmOh245HwseRIni?=
 =?us-ascii?Q?rh8qfOZtaDeyPjoSHEmqYpppuVq0cBipKfyaNYmnIigX3tZgDRYOza1e9Ipv?=
 =?us-ascii?Q?zgYoAupjh1xtXVyh80wKEs5soUWfmVj6lKcLJ/l4n2aSx5+aou3IqoybbLSp?=
 =?us-ascii?Q?Lhh/9UAqXJUBhRvmjpr8nv/+diE0D2pPy8+vIsacO1Kh1Vxt4zho06kAy9yg?=
 =?us-ascii?Q?CQibqK+uzyPGLdtCVCjhJ858MR+Kuhy0lzSTPh103P5ltg8Z4x6TxLd706+3?=
 =?us-ascii?Q?NLrK6b/p9fzJKtSnnBB238OyYqm9nUbaR0XrhWyEsmCk8xwiuvEChTEqCXtB?=
 =?us-ascii?Q?IXOy9Me/iNJmivSrnN2uaLXxQj+kjrs35nwR73hOkxgmsETauSkX6Gx5ATU4?=
 =?us-ascii?Q?Qjy0mXDRQbBFGCODBCZ2jwuKDJoKJwcECRuxiSTwRbzS2fgTFDoA3sLaH3cw?=
 =?us-ascii?Q?+s9zM70HhH7PlqCRGrGr1U4nW38+1QkFjH6O0ngIk74hr795R2+traXK1tRc?=
 =?us-ascii?Q?TC5DkXClt0JUlUdV821LQF0o9aURT9XQbJmfFF0foMixEd+J/nHJQDAVg2oZ?=
 =?us-ascii?Q?ei5fBTK+TCtxdqUg1qe5H9+W8nJ/xh8yL6Mky5OhbX2CuS1pA4ONoz7kTqz8?=
 =?us-ascii?Q?/p7xyuncrDX/nDMtKJ6MgJ8VLF9hz52eYJSeFv2Z4UEZnTSftlmLWcIF3lGH?=
 =?us-ascii?Q?OjVgPhpfcq7LIr8sIAUDJcZi+raqi2/x2ayM5rJjp/dBAd84eU5HRwWVpeRP?=
 =?us-ascii?Q?HKyGChYFJfOVNG6C6vrhvSKpCNA=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 4bfaae82-aae5-4361-bf2d-08d9d0b24579
X-MS-Exchange-CrossTenant-AuthSource: PH7PR10MB5698.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Jan 2022 01:17:13.9352
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: LYgcFNIK6ujN7etBeMj5JBetfbDYYDhTO+EMvgXpb5dybwOYDKSqvhJnXkmIfNwpf13rG49OiLMOfa5W/XgybFUvM7co+F0+cLdbIRMLXNI=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR10MB4597
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10218 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 bulkscore=0 mlxscore=0
 malwarescore=0 suspectscore=0 mlxlogscore=789 spamscore=0 phishscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2112160000
 definitions=main-2201060004
X-Proofpoint-ORIG-GUID: rekKIasEjWlNS_n95MWIVh5wrBpk0Zig
X-Proofpoint-GUID: rekKIasEjWlNS_n95MWIVh5wrBpk0Zig
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Wed, Jan 05, 2022 at 08:53:39PM -0400, Jason Gunthorpe wrote:
> On Wed, Jan 05, 2022 at 07:46:48PM -0500, Daniel Jordan wrote:
> > padata threads hold mmap_lock as reader for the majority of their
> > runtime in order to call pin_user_pages_remote(), but they also
> > periodically take mmap_lock as writer for short periods to adjust
> > mm->locked_vm, hurting parallelism.
> > 
> > Alleviate the write-side contention with a per-thread cache of locked_vm
> > which allows taking mmap_lock as writer far less frequently.
> > 
> > Failure to refill the cache due to insufficient locked_vm will not cause
> > the entire pinning operation to error out.  This avoids spurious failure
> > in case some pinned pages aren't accounted to locked_vm.
> > 
> > Cache size is limited to provide some protection in the unlikely event
> > of a concurrent locked_vm accounting operation in the same address space
> > needlessly failing in case the cache takes more locked_vm than it needs.
> 
> Why not just do the pinned page accounting once at the start? Why does
> it have to be done incrementally?

Yeah, good question.  I tried doing it that way recently and it did
improve performance a bit, but I thought it wasn't enough of a gain to
justify how it overaccounted by the size of the entire pin.

If the concurrent accounting I worried about above isn't really a
concern, though, I can reconsider this.