Date:   Tue, 11 Jan 2022 11:20:27 -0500
From:   Daniel Jordan <daniel.m.jordan@oracle.com>
To:     Jason Gunthorpe <jgg@nvidia.com>
Cc:     Alexander Duyck <alexanderduyck@fb.com>,
        Alex Williamson <alex.williamson@redhat.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        Ben Segall <bsegall@google.com>,
        Cornelia Huck <cohuck@redhat.com>,
        Dan Williams <dan.j.williams@intel.com>,
        Dave Hansen <dave.hansen@linux.intel.com>,
        Dietmar Eggemann <dietmar.eggemann@arm.com>,
        Herbert Xu <herbert@gondor.apana.org.au>,
        Ingo Molnar <mingo@redhat.com>,
        Johannes Weiner <hannes@cmpxchg.org>,
        Josh Triplett <josh@joshtriplett.org>,
        Michal Hocko <mhocko@suse.com>, Nico Pache <npache@redhat.com>,
        Pasha Tatashin <pasha.tatashin@soleen.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Steffen Klassert <steffen.klassert@secunet.com>,
        Steve Sistare <steven.sistare@oracle.com>,
        Tejun Heo <tj@kernel.org>,
        Tim Chen <tim.c.chen@linux.intel.com>,
        Vincent Guittot <vincent.guittot@linaro.org>,
        linux-mm@kvack.org, kvm@vger.kernel.org,
        linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org
Subject: Re: [RFC 00/16] padata, vfio, sched: Multithreaded VFIO page pinning
Message-ID: <20220111162027.3brb7ga3vgtvv6th@oracle.com>
References: <20220106004656.126790-1-daniel.m.jordan@oracle.com>
 <20220106011306.GY2328285@nvidia.com>
 <20220107030330.2kcpekbtxn7xmsth@oracle.com>
 <20220107171248.GU2328285@nvidia.com>
 <20220110222725.paug7n5oznicceck@oracle.com>
 <20220111001751.GI2328285@nvidia.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20220111001751.GI2328285@nvidia.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?VVcAljDuZ/7H7TiI6EpB0VnYKOBurzF4nZyO8W/3j9pEbt1dRqN3ZQ7kIV9J?=
 =?us-ascii?Q?NCq3L4Nr1QI86yzhD6GmleyFiwprZskCawsjpNZAXlCU6l+iE6yf/NPDQL8V?=
 =?us-ascii?Q?fn4x7Zxw9rL8iwjuelIEjT3rMfYBdJ7hPTkCGpssUCBP0zfJp8Rlp8QYA70z?=
 =?us-ascii?Q?uyYv40S7NnkJ3b7cV4yMHOWrlWvc8ldDM/JK7Okxug+f+Jw8HH4fo2t16bbp?=
 =?us-ascii?Q?CxK9/eSndLldQ1YrFduyGEOGWDwUSeZiaIlLLYxT0RgY7tbKyTtk4PR8HNwJ?=
 =?us-ascii?Q?rQCV+PqbHZYV8ZlfTJZ+ADSLPsTLdVlqLZtB/fGixb65U9G/mJ/se5qt+hSN?=
 =?us-ascii?Q?7XFfqfoBSTzHwh6g0qrXikDcD7v1GKwTd3j+ytJXcfdz/LIjPh5sPAr2Ndwz?=
 =?us-ascii?Q?uzESoy/2Gqft0dEASlyyBHk3i7rOkdqK8WxeGWgBVyDBHD8Bt0Oc7DWfOY62?=
 =?us-ascii?Q?jPaqAN4KhnNLOIq46eo22gmW7Lv93SKFFKQzGpv1/Fp8Aql4eFrw2J9i4pBo?=
 =?us-ascii?Q?5Vjg+65iphiYa6+ibeYHYTAg/Xn//vOJhJsg7BQVm5GDWlAKW1RIB/Yzdm88?=
 =?us-ascii?Q?CtLhEmKg3bx8WtC8vOZSaTMH8r4tH0bKcOQJqNXGWhscYKUJZ6VOZ4p9iDW1?=
 =?us-ascii?Q?LvV8OTxTy53OJH4b7gPVq+SXsCal9xcFXZIpAj7CIgm6AP/VsoTebXuzz65o?=
 =?us-ascii?Q?e4WoLy2KcJxdvQOkUfSY7w0gwRo7a5HsFO13WPmWLMdF/vXB5hiHOOj1lLsl?=
 =?us-ascii?Q?m5f3Zp/djWCAYJwPWKcbPZH7eIHcEAt4HWHQT92s/Hng21SbIS+UzW6Oxnu1?=
 =?us-ascii?Q?1+LpUnI4/I5NWOCOo4nMBWU0PWBD6cJEeUT6Xc8OznerRycOp5TreOwhYjT5?=
 =?us-ascii?Q?ljDOoHJeevVAdRJbQF7KCMykUYc47GuROvp7ItB6LJyqfKGqFS0LW/FauVks?=
 =?us-ascii?Q?RqAwbKwZ3I07/vnNtSgyejITRWMgFxdgD8mBpDeczQ/ZotjWFkrMf2xSBzQ0?=
 =?us-ascii?Q?83BL71FO8ss8h0/vDQqAOYI9h6+UAlqDLMdbTEJjPzq26+UDR9MH2UcR/+yF?=
 =?us-ascii?Q?wTxtP/l8Kc/HjpAURw2khMxe+AYKF6GdnCBaY53TL4oQhQyNubRoTfc3M8hI?=
 =?us-ascii?Q?3C9pWZjgylRtmyzKSlvJpshz0Ndh4U+wibTY9FckJSswS+BNyulVszvTOi1g?=
 =?us-ascii?Q?/gY+3jsmUvKac1X6JZsb72jM5MUNliFIdItCOQIRXRLmQfzkr8ucceT1dMaI?=
 =?us-ascii?Q?a9lErKwzsw/OPXFpH9Kpx+Ub1JhAjJMUPybK1D0N2522Yd286KXTn0TtMj0l?=
 =?us-ascii?Q?jxUJrUjR4+1Lxn6fDdWBNSTL6hwvx0fMRfO78QgLvZwlHMgpXLOHzgri8bxQ?=
 =?us-ascii?Q?1rmQQ2X/hxuZYaRB1OMfmP7RF/x+5/6Jd19es2wO3U0dxn06ZlLIyt8cmnyT?=
 =?us-ascii?Q?b97EhpLO9+3LHDTVPyvRR8tWjFmHGggU3ndwWzpWTlXOGXDv4FxMkgrAwp9p?=
 =?us-ascii?Q?/zqnYMtapAbsYtrFEIGToMeAGaepC+tdRi/nXcdVikDZFAWO0+2A3rR127G4?=
 =?us-ascii?Q?jFOLxA1G4TQXLSXTR/khzNiOjSur2QVGukdGIdWeFKr4PPmYoQRN80BqgrNM?=
 =?us-ascii?Q?pi0xlEX8cMEuY/PTVd6ioOLC+2y10IfGmuREQtYNyIWYOsP6O9b4aHawpiOi?=
 =?us-ascii?Q?ystb5g=3D=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 22d31977-cf8c-4df1-3386-08d9d51e49ef
X-MS-Exchange-CrossTenant-AuthSource: SA1PR10MB5711.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Jan 2022 16:20:31.8705
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: hRleLgZeRxDS4tnIhSEP1JFD2VSfS/6Iy+Wlt2Nc9t96IDDH3Km8WzJcsi3UUJVYwJQxektupS8Jsqgvx1gSZCJ510Dxr95kFO8wzkOKGfk=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN6PR10MB2863
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10223 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=751 bulkscore=0 spamscore=0
 phishscore=0 adultscore=0 suspectscore=0 mlxscore=0 malwarescore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2110150000
 definitions=main-2201110092
X-Proofpoint-ORIG-GUID: FkVyPF2kmNt8m4r5hHe53udYiG7CW0ap
X-Proofpoint-GUID: FkVyPF2kmNt8m4r5hHe53udYiG7CW0ap
Precedence: bulk
List-ID: <linux-crypto.vger.kernel.org>
X-Mailing-List: linux-crypto@vger.kernel.org

On Mon, Jan 10, 2022 at 08:17:51PM -0400, Jason Gunthorpe wrote:
> On Mon, Jan 10, 2022 at 05:27:25PM -0500, Daniel Jordan wrote:
> 
> > > > Pinning itself, the only thing being optimized, improves 8.5x in that
> > > > experiment, bringing the time from 1.8 seconds to .2 seconds.  That's a
> > > > significant savings IMHO
> > > 
> > > And here is where I suspect we'd get similar results from folio's
> > > based on the unpin performance uplift we already saw.
> > > 
> > > As long as PUP doesn't have to COW its work is largely proportional to
> > > the number of struct pages it processes, so we should be expecting an
> > > upper limit of 512x gains on the PUP alone with foliation.
> > >
> > > This is in line with what we saw with the prior unpin work.
> > 
> > "in line with what we saw"  Not following.  The unpin work had two
> > optimizations, I think, 4.5x and 3.5x which together give 16x.  Why is
> > that in line with the potential gains from pup?
> 
> It is the same basic issue, doing extra work, dirtying extra memory..

Ok, gotcha.

> I don't know of other users that use such huge memory sizes this would
> matter, besides a VMM..

Right, all the VMMs out there that use vfio.

> > My assumption going into this series was that multithreading VFIO page
> > pinning in the kernel was a viable way forward given the positive
> > feedback I got from the VFIO maintainer last time I posted this, which
> > was admittedly a while ago, and I've since been focused on the other
> > parts of this series rather than what's been happening in the mm lately.
> > Anyway, your arguments are reasonable, so I'll go take a look at some of
> > these optimizations and see where I get.
> 
> Well, it is not *unreasonable* it just doesn't seem compelling to me
> yet.
> 
> Especially since we are not anywhere close to the limit of single
> threaded performance. Aside from GUP, the whole way we transfer the
> physical pages into the iommu is just begging for optimizations
> eg Matthew's struct phyr needs to be an input and output at the iommu
> layer to make this code really happy.

/nods/  There are other ways forward.  As I say, I'll take a look.