Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Fri, 9 Sep 2022 11:31:49 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Alex Williamson <alex.williamson@redhat.com>
Cc:     Linus Torvalds <torvalds@linux-foundation.org>,
        John Hubbard <jhubbard@nvidia.com>,
        David Hildenbrand <david@redhat.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        linux-kernel@vger.kernel.org
Subject: Re: [GIT PULL] VFIO fix for v6.0-rc5
Message-ID: <YxtOVURw00Dtm3rT@nvidia.com>
References: <20220909045225.3a572a57.alex.williamson@redhat.com>
 <CAHk-=wj3rrkPvPJB_u4qoHK4=PVUuBHKB67f_oZO62EE22pNPQ@mail.gmail.com>
 <20220909060832.17f6607f.alex.williamson@redhat.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20220909060832.17f6607f.alex.williamson@redhat.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?EqeYVG5L2t/i48IMvVakV0PwktWDrtZtoKuAzq3Abvv5SH4rp3gEK+O7t9fv?=
 =?us-ascii?Q?ARNgfyqJpLqxZ1kEaKKP0q035fLvK9h+ygNXkLxxjTONQ9jOB+dTdiQpYdUE?=
 =?us-ascii?Q?ChT9lVHX2pU2YFS03tGqqk5I3wtyKAD1ZdZwM2PEftYH8UjhZu4km5id6QNd?=
 =?us-ascii?Q?TsKvlwqtfeRZs39O82n741jzjMidWTeUNzO7VjPGO38p6plTy+aCO2VElVbe?=
 =?us-ascii?Q?6SQSJM/dxwlcLuzLatCxNBYTr28dRpVBEv2H0Do9IA+Y/FP0LWdbvCu3K6kL?=
 =?us-ascii?Q?IKFvRPnsz044l1td1AA8KnTTMLlG08OXT/hDrEitWtoMnwXJq8VZYGGiRCzB?=
 =?us-ascii?Q?hjZlY+apM8lZdetZ03GUPBM9Dqbce20/PXzkccxPSFPTg+IF1mVwKjBm+5e+?=
 =?us-ascii?Q?u5Wi/gvFJT1TK3PEc1vOEuK1TNfYnl7UG1iKUoGvRR/xIJCnmBVaczOby/ay?=
 =?us-ascii?Q?yjoiDe5YW/MRophtFBqDtQaPIV9UQ0Aw+YuLToectuF2UXgoo4QG5O1AgfCE?=
 =?us-ascii?Q?T4ANKR5TfDxxN3lzIaUrDnzG23l1phGXw5jLVNYDsR5h6JYrxCtdumTvjce/?=
 =?us-ascii?Q?IeXmgwhUIXyv7B5xTAbtP7KH/II67g2/zaQGJEOeITFTvChir8ndKw6Pxcsv?=
 =?us-ascii?Q?LkBiw2xbSvBM493qqtqN3WiotoLVshiuABP9erVi7BN55uIT89SCQB6Ck4eU?=
 =?us-ascii?Q?wFID9HoP3UqBKlIXyTcbNCL+t2zBQQh9vXhswaBrMBaqXOu9Y1zisb/7Ip0L?=
 =?us-ascii?Q?zXLUsNwkoDJEm2SrvkjfNq22B8mPjqurPwVYLGPHn13REjUUvHlhYS1VRMQV?=
 =?us-ascii?Q?08cAa2XCf2x4Dd5qoVGSi3ZuquBRpoaMwX+2GFhatQLNw/4JlX09juxb40Rc?=
 =?us-ascii?Q?lr/te32/Rz+NctPkk8xy5Vw1eAVOryyCN6zR95BX5UYFm6LLsSCdt8KrzPl8?=
 =?us-ascii?Q?Mq3S8x8+6Kf62STm/vSi2rJKed7m0EEt3A/lnQeGD7X+jk6RmFpEbOiSfIVB?=
 =?us-ascii?Q?a5Mz/lw+s7L5oozWqnVTclb5y4pAaMn3Lkw8MFAYuwON55qfloxz2W5oepxE?=
 =?us-ascii?Q?NagjXPmSB4AOuXqZYHBXyjnjwBvueB+WkCkMd3grDmwl5wNXT54zjazpQV0L?=
 =?us-ascii?Q?eDsi0HfzdmiguPwPUbr3zrB6OscRPFcGR/gkY8yNA/zZV8zOJrOVXlW9qlA9?=
 =?us-ascii?Q?vr0CLLTUbGdHTqSiUeZPYCRo3PbGg9WOJy4SSXWllhOsLiQTmzZA10CmqeHT?=
 =?us-ascii?Q?VIct7HUMY/p/SWmx4UQ4fVLiwurU6+RJu16iUs+4O7O09QbE6k6U3aUQQOXw?=
 =?us-ascii?Q?7hGq2FdGIAMIZG8DHv//Xzm8WfIcl9teEqBxCy/1WRWMK0UMBezejIjl72Rx?=
 =?us-ascii?Q?Svw3Pf7tuiyPN6KYg5HLr3uVx2P12nU+qXMNcEY739n+SL2GPerCyjdTOL26?=
 =?us-ascii?Q?6sXZ7/ZaxZmUEgqjQrTLFHq0C90dIP3cJDH23x6yAUXYfEpf1ch/Dqf/oQYw?=
 =?us-ascii?Q?kDpwhWRRfsXSNVBy9hJrNkVVc0k0XgamsTjGyxQoyLHC5cfH1R7pjXPIzc1Y?=
 =?us-ascii?Q?rzrjT3z6YV6NOFRcIT0L9nDK2zWklu1OVWyqKH+u?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 1ff4144d-dc07-46c4-66d9-08da927008d9
X-MS-Exchange-CrossTenant-AuthSource: MN2PR12MB4192.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Sep 2022 14:31:51.1004
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: TdKMDBu5fXxsfbSOtdYz57HryYAyoD34ntGl/fgZ4ZEiQf5y0Rx+cixDB7RIaAbL
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB6277
X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Fri, Sep 09, 2022 at 06:08:32AM -0600, Alex Williamson wrote:
> On Fri, 9 Sep 2022 07:53:17 -0400
> Linus Torvalds <torvalds@linux-foundation.org> wrote:
> 
> > On Fri, Sep 9, 2022 at 6:52 AM Alex Williamson
> > <alex.williamson@redhat.com> wrote:
> > >
> > > VFIO fix for v6.0-rc5
> > >
> > >  - Fix zero page refcount leak (Alex Williamson)  
> > 
> > Ugh. This is disgusting.
> > 
> > Don't get me wrong - I've pulled this, but I think there's some deeper
> > problem that made this patch required.

The basic issue is that VFIO is using the reserved page as an
indication that pin_user_pages_remote() didn't process the page and
instead VFIO's bad follow_pfn() path was used.

It really shouldn't be working like this, which path was used to
obtain the PFN should be recorded independently. VFIO just doesn't
have a datastructure to accommodate it.

Aside from that, most longterm users don't ever even see the zero page
because that typically creates that GUP incoherence we've talked about
alot. The ugly FOLL_FORCE prevents it from happening. VFIO doesn't do
this either so it suffers the incoherence bug along with exposure to
the zero pages :(

David will fix the FOLL_FORCE issue and this will almost go away.

We had a long thread about this and discussed fixing it by simply
setting FOLL_FORCE as other pin_user_page users do. This reached an
interesting point that I'm curious on your view about ABI stability.

It is related to mlock accounting. When vfio pins pages it accounts
for them in mlock. Alex says there are users that set a mlock
limit. VFIO has this historical bug where it doesn't account for every
page it is asked to pin in mlock, eg it ignores the zero page. So, if
we change how GUP or VFIO does page pinning in a way that increases
its mlock charge is this an ABI break?

On one hand users with an very strict limit may find their environment
requires a limit increase after a kernel upgrade. So it fails the
'everything keeps working after kernel upgrade' test.

On the other hand, treating these sorts of limits as ABI means that a
lot of internal stuff is suddenly ABI - eg with a memory cgroup is
increasing the charge on GFP_KERNEL_ACCOUNT also ABI breaking? That
seems unworkable.

Thanks,
Jason