Date:   Thu, 6 Jan 2022 22:06:42 -0500
From:   Daniel Jordan <daniel.m.jordan@oracle.com>
To:     Jason Gunthorpe <jgg@nvidia.com>
Cc:     Alex Williamson <alex.williamson@redhat.com>,
        Alexander Duyck <alexanderduyck@fb.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        Ben Segall <bsegall@google.com>,
        Cornelia Huck <cohuck@redhat.com>,
        Dan Williams <dan.j.williams@intel.com>,
        Dave Hansen <dave.hansen@linux.intel.com>,
        Dietmar Eggemann <dietmar.eggemann@arm.com>,
        Herbert Xu <herbert@gondor.apana.org.au>,
        Ingo Molnar <mingo@redhat.com>,
        Johannes Weiner <hannes@cmpxchg.org>,
        Josh Triplett <josh@joshtriplett.org>,
        Michal Hocko <mhocko@suse.com>, Nico Pache <npache@redhat.com>,
        Pasha Tatashin <pasha.tatashin@soleen.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Steffen Klassert <steffen.klassert@secunet.com>,
        Steve Sistare <steven.sistare@oracle.com>,
        Tejun Heo <tj@kernel.org>,
        Tim Chen <tim.c.chen@linux.intel.com>,
        Vincent Guittot <vincent.guittot@linaro.org>,
        linux-mm@kvack.org, kvm@vger.kernel.org,
        linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org
Subject: Re: [RFC 08/16] vfio/type1: Cache locked_vm to ease mmap_lock
 contention
Message-ID: <20220107030642.re2d7gkfndbtzb6v@oracle.com>
References: <20220106004656.126790-1-daniel.m.jordan@oracle.com>
 <20220106004656.126790-9-daniel.m.jordan@oracle.com>
 <20220106005339.GX2328285@nvidia.com>
 <20220106011708.6ajbhzgreevu62gl@oracle.com>
 <20220106123456.GZ2328285@nvidia.com>
 <20220106140527.5c292d34.alex.williamson@redhat.com>
 <20220107001945.GN2328285@nvidia.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20220107001945.GN2328285@nvidia.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?6E7guNOm6x+rIfSAbgFYGuRwJom9XcHVR0YS9o3ZlTxlYk7qnFEroO5Q5o0U?=
 =?us-ascii?Q?4N3uDhyDeMTQ1W5PO70WW+9sKjCtTufk/aa0QQd2EQoDnPN1DOGbMolFhX4/?=
 =?us-ascii?Q?QxwTYPXzoLhASv0ZQRCgAeIiGGaPvE29V5tfNedRQftl2ANdejhC7AhXy7/7?=
 =?us-ascii?Q?poilQ4zWsPavNb15pFxReOVdhMxU1ubvGROsBoPcahQHAdX3rh/9K38iaDQy?=
 =?us-ascii?Q?ltJYI7u1LMMJ0BrKgBGjcdfGhir/vvJ2CBrKHnPuhJcIz9hPMKRfQjGwUlnl?=
 =?us-ascii?Q?PUY06RxqAQ5mfFxNzRogwsNDdxdrHoQ9nIfkIzUsDIdMvdPs2MDvnb+6FMv2?=
 =?us-ascii?Q?qePDEM90xwRtA9tNQZ6wmie68E7wlQEmdhOeDpHJ2BTHHuvPRVWrycH3m/Qa?=
 =?us-ascii?Q?iTH2Bcab6+kT9uypkUK34UeuC9SRPmi9/UqCOhxzJJa1posj/GG1DI4RKRWW?=
 =?us-ascii?Q?PgQ6qq3P9afwRHT7gKwtxKxz5y/FTCHMGFuldrXFJpR/6rZSlgACw4Mhs5fZ?=
 =?us-ascii?Q?PqV7v7auyg5KxdPm2vB8AlLW/Xjbvy2uoSwISMTBCtSysN24I3D9W0pqD+yV?=
 =?us-ascii?Q?3BP/Jh/PQWdlaV5feJOGSHtm0UZ0b9uWj/35AVxf0JMzPLpcNs+noNL41iQX?=
 =?us-ascii?Q?ZJhRf2xbC1P8pjQJ1xX7bgAQk/JN3NygRVPoWk0lz8yCQ0/X08BiUIohy+IW?=
 =?us-ascii?Q?2rMggN2Ow0lNPFvTyJtyAYVwOP/eLIKAo8tahjsDc87Fc4E8tvxRWXo0ueU5?=
 =?us-ascii?Q?bo836qtVYxTdSWN7FdnY70mWUCxwRFU4JKfFMzlAhtFLxr0mhks0zsCATeFd?=
 =?us-ascii?Q?geeH2ZvQxrOER3shaWXR8re1vKVdxyflgx9EB1YMXwaIPL2kDicFvPSmQ6YF?=
 =?us-ascii?Q?ACqOma4VkCRDjSaGR/EYomqhajG9sLAzBJywISNdIPDyX7pCcd03PmEogMFw?=
 =?us-ascii?Q?0B13qRo2JQ6MBLQZaekmZdQqLHbTPi56uoubgVYn7YpZF7hLlpqvfmo/jmhk?=
 =?us-ascii?Q?Cf4bLr7XJNriZ4cVPTseaBE6IxGp4PKFzNuL6Stx266/zMGcMIEIBMUsuFMf?=
 =?us-ascii?Q?EyOzgx065To6wzbOw60tkOfDRsLoRapCfz7dysizNhqg9RpZ/N+pMAjqFMwE?=
 =?us-ascii?Q?GAhxtoo4gadPrS4ajCO5K/httVITJptVMLbQ+eknc0ONlLBhP3WRC08N/THE?=
 =?us-ascii?Q?OIot5JdE0xdCJZmWGMG0NSkpNZ1+sQv7mWs1/wFWBWmcicvG6udTDES1js3+?=
 =?us-ascii?Q?HcIXcSLhg7zSGkxCdhKvVeP3TOfAzl7pmj73qWEGBESxV9W+b13vHcbU+bCf?=
 =?us-ascii?Q?wQc7s12n58/7kOEOkXOfF+2NuktTTz5TKyzwJHfVMp17z0bsMeREHqse7Cmn?=
 =?us-ascii?Q?/seqVwx3V221odsP855U42byW2ZlZiqyFEez/Hrhsq5ufN3L3+0kO6IhFMU3?=
 =?us-ascii?Q?7cSEQlDukEj4p/z9hFKDUs1Be447f0oEMEFNRZH/oq++7trgoe8V+v+09W9w?=
 =?us-ascii?Q?XUCSAS4jEDcSaUJ4st2MYEneRH7KXuiY8uZ3TFomfQYtmKRniCFvyjEejXJC?=
 =?us-ascii?Q?a0r36t7mpcNofeOzxu8S+/AeC/+zuPr1pYkFrFX+NFv3wrdGwbAux84bAYb7?=
 =?us-ascii?Q?aqwcDET2jFSkGiAOps3GYGtJ3S5PFY8SB1J2qqL4ocZCbj3k8EYRPYitZwIX?=
 =?us-ascii?Q?57h8bGaqDINhjSziLqzkOOQ+JNw=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 27800304-d008-4e63-3c79-08d9d18abd47
X-MS-Exchange-CrossTenant-AuthSource: SA1PR10MB5711.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Jan 2022 03:06:46.2295
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: kSqkpAJdWhMg2m1gDJWOJTxnzIEB8DYinKNe6v++iFDOiltaomuyBZNTY41kqdOPbBx92ahHx4ckopZ0dI8T0Rs6Kno7FpV7ajHRQrmOsuU=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN4PR10MB5542
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10219 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0
 mlxlogscore=885 adultscore=0 phishscore=0 mlxscore=0 spamscore=0
 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2110150000 definitions=main-2201070019
X-Proofpoint-ORIG-GUID: XV9TbI3A-uQHkC5gX9eU4PKcL8lLefj4
X-Proofpoint-GUID: XV9TbI3A-uQHkC5gX9eU4PKcL8lLefj4
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Thu, Jan 06, 2022 at 08:19:45PM -0400, Jason Gunthorpe wrote:
> On Thu, Jan 06, 2022 at 02:05:27PM -0700, Alex Williamson wrote:
> > > > Yeah, good question.  I tried doing it that way recently and it did
> > > > improve performance a bit, but I thought it wasn't enough of a gain to
> > > > justify how it overaccounted by the size of the entire pin.  
> > > 
> > > Why would it over account?
> > 
> > We'd be guessing that the entire virtual address mapping counts against
> > locked memory limits, but it might include PFNMAP pages or pages that
> > are already account via the page pinning interface that mdev devices
> > use.  At that point we're risking that the user isn't concurrently
> > doing something else that could fail as a result of pre-accounting and
> > fixup later schemes like this.  Thanks,

Yes, that's exactly what I was thinking.

> At least in iommufd I'm planning to keep the P2P ranges seperated from
> the normal page ranges. For user space compat we'd have to scan over
> the VA range looking for special VMAs. I expect in most cases there
> are few VMAs..
> 
> Computing the # pages pinned by mdevs requires a interval tree scan,
> in Daniel's target scenario the intervals will be empty so this costs
> nothing.
> 
> At least it seems like it is not an insurmountable problem if it makes
> an appreciable difference..

Ok, I can think more about this.

> After seeing Daniels's patches I've been wondering if the pin step in
> iommufd's draft could be parallized on a per-map basis without too
> much trouble. It might give Daniel a way to do a quick approach
> comparison..

Sorry, comparison between what?  I can take a look at iommufd tomorrow
though and see if your comment makes more sense.