Date:   Fri, 7 Jan 2022 11:39:38 -0500
From:   Daniel Jordan <daniel.m.jordan@oracle.com>
To:     Jason Gunthorpe <jgg@nvidia.com>
Cc:     Alex Williamson <alex.williamson@redhat.com>,
        Alexander Duyck <alexanderduyck@fb.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        Ben Segall <bsegall@google.com>,
        Cornelia Huck <cohuck@redhat.com>,
        Dan Williams <dan.j.williams@intel.com>,
        Dave Hansen <dave.hansen@linux.intel.com>,
        Dietmar Eggemann <dietmar.eggemann@arm.com>,
        Herbert Xu <herbert@gondor.apana.org.au>,
        Ingo Molnar <mingo@redhat.com>,
        Johannes Weiner <hannes@cmpxchg.org>,
        Josh Triplett <josh@joshtriplett.org>,
        Michal Hocko <mhocko@suse.com>, Nico Pache <npache@redhat.com>,
        Pasha Tatashin <pasha.tatashin@soleen.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Steffen Klassert <steffen.klassert@secunet.com>,
        Steve Sistare <steven.sistare@oracle.com>,
        Tejun Heo <tj@kernel.org>,
        Tim Chen <tim.c.chen@linux.intel.com>,
        Vincent Guittot <vincent.guittot@linaro.org>,
        linux-mm@kvack.org, kvm@vger.kernel.org,
        linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org
Subject: Re: [RFC 08/16] vfio/type1: Cache locked_vm to ease mmap_lock
 contention
Message-ID: <20220107163938.esft3w5nscav2gu2@oracle.com>
References: <20220106004656.126790-1-daniel.m.jordan@oracle.com>
 <20220106004656.126790-9-daniel.m.jordan@oracle.com>
 <20220106005339.GX2328285@nvidia.com>
 <20220106011708.6ajbhzgreevu62gl@oracle.com>
 <20220106123456.GZ2328285@nvidia.com>
 <20220106140527.5c292d34.alex.williamson@redhat.com>
 <20220107001945.GN2328285@nvidia.com>
 <20220107030642.re2d7gkfndbtzb6v@oracle.com>
 <20220107151807.GT2328285@nvidia.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20220107151807.GT2328285@nvidia.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?dqVqbkqnhI4XYe8WGnBUMDUZLr6PjSmpNqu62WKZyo+51Nrw+iUPXMxkc1lv?=
 =?us-ascii?Q?7gGW5lCoS+XVIPWZM/Qbbl5gnUqEemQIfwjueZWT2OuJbHcq8RhiTyj4mzFX?=
 =?us-ascii?Q?uG7DToGjbhZnHYsdi8pziueQzdGoMFILTj01rEmFt87ZcidfydClUH7ffCFz?=
 =?us-ascii?Q?01frsRABytkN443i1qqQl2xTd9aFH73kRf1F8C8K7FqrgkAMk+SsW1EyZZZ2?=
 =?us-ascii?Q?S4O9NHuREozbk+pPjULvFjIQT9NM/EiH7WFcEBgIMKbAGUMlneWa9HMAXQcL?=
 =?us-ascii?Q?POmcoo6Rbi9fwDRIZQWjThdHP0uKaWVEFeMysEYWTOGCbU7ukEtJkx7PwMz/?=
 =?us-ascii?Q?5BjvTfU28uut2g6HxF5r80NsNp85Bf2fI9AMFUlSCSlSIGQcFJOPwq+1uJAe?=
 =?us-ascii?Q?154b9v0hbpqRu97ChDko23q0gfvQ33GjoLQXSkdGZ18umHeVXorMLO68LpGD?=
 =?us-ascii?Q?u8N7VE5oLqb4FFD9+ccFosQ5Jk+TfzAra7Dh4uFuV7z7aoa4AFUq1j2DxH4g?=
 =?us-ascii?Q?6YS54xt5jsClIMCFCzSkCW3DetrXXmg7xbQEdOsNrXaGJTMqGZOgnKo8sNmz?=
 =?us-ascii?Q?5XcKmqrorzGLaChlwvP4khNr/EnFSCHN+6gTteK/KTIcOnJylV22dEHo60Kx?=
 =?us-ascii?Q?rHvMJb5kYpOn/LBWs1bWck0yH8193whZYaSc+Ffa4qWE2e4odQLbL6DrA+lp?=
 =?us-ascii?Q?9niphFbfgEN8INFS1TRGyjPHGJcu+yzuRkKuxot6LKY2EmtbTvUJgkEJ4MwN?=
 =?us-ascii?Q?MK9hZ54Sc6L4GZTSMKPj1gz3PZgXw8+HjMjYkgNPsc2zN5G0xYdhx3vDlqyf?=
 =?us-ascii?Q?2IGCUex88KiOzGCYX1TFOPqT5kM+3uKGQKNUX6P63AkCyt6YceC+rAuA+Ifq?=
 =?us-ascii?Q?+RZX3koaORothou03Hc8ngWiYvDdt1PgLk3q/d/ZaXcQxOzeaorVUcxM/4BA?=
 =?us-ascii?Q?e+C67kptipK0ua76mrru9cKaoyXbw8vc1dTYXJ2bHKdW2H8wB8gisPoDd7Lm?=
 =?us-ascii?Q?H0FFwkce0uD26dmDMuhqB9+HRlYyGp7ljwG/OyDadPdQz3EAI92ETnwf0cvc?=
 =?us-ascii?Q?BJYICNPoQSjsMv5qx2vfOFHuupoEG+knEFjeIOBLKvcb4G3xE/zHDj7HZYFx?=
 =?us-ascii?Q?AUbSc6ASUXLfP2ynx1D+kr8pzyqDKVa+L3FgYygK+q3HiDtqeba61nIyPcDE?=
 =?us-ascii?Q?afCihw4NEiZtlxzdHIaiVxenyxGGe6GG/p/xVVY5Jr4a+HSdFnOulJoQM3pC?=
 =?us-ascii?Q?PGT+nmh2UVeElNJCJzgVu7+Eilrlu0eKMHp2kHvbzMJAOk5bJsfo90A1gNUd?=
 =?us-ascii?Q?2P12JzI8sd53AFhxdhIt7wnJuDTUaXQnxKf5Y14hR01i3qA228SCbrFfsx4O?=
 =?us-ascii?Q?OdfMwZGSWavLoWwkN1yeaq88gbQBc6B4CWbLpuqMINZORBvNDeziwFNGyVnw?=
 =?us-ascii?Q?xcL+8m7O9DkdrD10PH5Uk0jc6kIYOWRRcVDgvgY4krh1QPDC5uoYfYAZGWGX?=
 =?us-ascii?Q?gCQfFntYvff6D4itLKpmP0KPFpy1/0XmmJ0fufaqYiqBrok0y0vFOSdGHzs8?=
 =?us-ascii?Q?b28WyXirJOkgm7ZBCcQbCNo7UbKlX+BidObDcskRw94FejmIXC21e2D//+aC?=
 =?us-ascii?Q?4rSxgeKaZRUh2VKj7RIiC8kVSIw19byuwrSjwWfrvW+NB888myuTTwRqEWEl?=
 =?us-ascii?Q?Cnen6g=3D=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: dca60111-0c14-4145-2230-08d9d1fc4e78
X-MS-Exchange-CrossTenant-AuthSource: SA1PR10MB5711.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Jan 2022 16:39:43.0750
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: r/N19XvDI4Qn9ifNamA+ADSOrrivtineeRISLwvWF2CPemEGsDVf5C6WD0OgNJHr7xGdY7YUPGoLCVHe7fnXHxOsDIwq/KW1rZAyta7yqRo=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN6PR10MB3021
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10220 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=811 bulkscore=0 mlxscore=0
 phishscore=0 malwarescore=0 spamscore=0 adultscore=0 suspectscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2110150000
 definitions=main-2201070112
X-Proofpoint-ORIG-GUID: 4dAnZtV2TB6-tljVcb2WzXvK4Yb1tKNo
X-Proofpoint-GUID: 4dAnZtV2TB6-tljVcb2WzXvK4Yb1tKNo
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Fri, Jan 07, 2022 at 11:18:07AM -0400, Jason Gunthorpe wrote:
> On Thu, Jan 06, 2022 at 10:06:42PM -0500, Daniel Jordan wrote:
> 
> > > At least it seems like it is not an insurmountable problem if it makes
> > > an appreciable difference..
> > 
> > Ok, I can think more about this.
> 
> Unfortunately iommufd is not quite ready yet, otherwise I might
> suggest just focus on that not type 1 optimizations. Depends on your
> timeframe I suppose.

Ok, I see.  Well, sooner the better I guess, we've been carrying changes
for this a while.

> > > After seeing Daniels's patches I've been wondering if the pin step in
> > > iommufd's draft could be parallized on a per-map basis without too
> > > much trouble. It might give Daniel a way to do a quick approach
> > > comparison..
> > 
> > Sorry, comparison between what?  I can take a look at iommufd tomorrow
> > though and see if your comment makes more sense.
> 
> I think it might be easier to change the iommufd locking than the
> type1 locking to allow kernel-side parallel map ioctls. It is already
> almost properly locked for this right now, just the iopt lock covers a
> little bit too much.
> 
> It could give some idea what kind of performance user managed
> concurrency gives vs kernel auto threading.

Aha, I see, thanks!