Date:   Tue, 21 Dec 2021 19:54:02 -0400
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     David Hildenbrand <david@redhat.com>
Cc:     Linus Torvalds <torvalds@linux-foundation.org>,
        Nadav Amit <namit@vmware.com>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        Andrew Morton <akpm@linux-foundation.org>,
        Hugh Dickins <hughd@google.com>,
        David Rientjes <rientjes@google.com>,
        Shakeel Butt <shakeelb@google.com>,
        John Hubbard <jhubbard@nvidia.com>,
        Mike Kravetz <mike.kravetz@oracle.com>,
        Mike Rapoport <rppt@linux.ibm.com>,
        Yang Shi <shy828301@gmail.com>,
        "Kirill A . Shutemov" <kirill.shutemov@linux.intel.com>,
        Matthew Wilcox <willy@infradead.org>,
        Vlastimil Babka <vbabka@suse.cz>, Jann Horn <jannh@google.com>,
        Michal Hocko <mhocko@kernel.org>,
        Rik van Riel <riel@surriel.com>,
        Roman Gushchin <guro@fb.com>,
        Andrea Arcangeli <aarcange@redhat.com>,
        Peter Xu <peterx@redhat.com>,
        Donald Dutile <ddutile@redhat.com>,
        Christoph Hellwig <hch@lst.de>,
        Oleg Nesterov <oleg@redhat.com>, Jan Kara <jack@suse.cz>,
        Linux-MM <linux-mm@kvack.org>,
        "open list:KERNEL SELFTEST FRAMEWORK" 
        <linux-kselftest@vger.kernel.org>,
        "open list:DOCUMENTATION" <linux-doc@vger.kernel.org>
Subject: Re: [PATCH v1 06/11] mm: support GUP-triggered unsharing via
 FAULT_FLAG_UNSHARE (!hugetlb)
Message-ID: <20211221235402.GH1432915@nvidia.com>
References: <4D97206A-3B32-4818-9980-8F24BC57E289@vmware.com>
 <CAHk-=whxvVQReBqZeaV41=sAWfT4xTfn6sMSWDfkHKVS3zX85w@mail.gmail.com>
 <5A7D771C-FF95-465E-95F6-CD249FE28381@vmware.com>
 <CAHk-=wgMuSkumYxeaaxbKFoAbw_gjYo1eRXXSFcBHzNG2xauTA@mail.gmail.com>
 <CAHk-=whYT0Q1F=bxG0yi=LN5gXY64zBwefsbkLoRiP5p598d5A@mail.gmail.com>
 <fca16906-8e7d-5d04-6990-dfa8392bad8b@redhat.com>
 <20211221010312.GC1432915@nvidia.com>
 <fd7e3195-4f36-3804-1793-d453d5bd3e9f@redhat.com>
 <20211221142812.GD1432915@nvidia.com>
 <303f21d3-42b4-2f11-3f22-28f89f819080@redhat.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <303f21d3-42b4-2f11-3f22-28f89f819080@redhat.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?UZtGtGFdRD1opsZntRxR/dRzraq68v7it6jk6cYL1MFHL38v5cf8l+eo5Vfn?=
 =?us-ascii?Q?xz5Fd5+s1URRNXD1TGL7P8BSN4wroSYSNlM7/SdgLH2kiGOnW1LauTkiCqyx?=
 =?us-ascii?Q?hU3waRh4eTBPwn5WNGI1v1zZUR+UlpLb4/mOIYUTpXSTQ+oe5/t/jl/QsY1R?=
 =?us-ascii?Q?2uHwMPUcB3r98JqAFMG5XRNl02M/Jzx09vvbz57/cwQdO4FtFDdspJzfp54a?=
 =?us-ascii?Q?mkeuuqJ/mMks/vEkOH007Nr8f1ZiljyABDuoya90RMpX56RT/bHEewwl9xjs?=
 =?us-ascii?Q?7MhwyWdse91753HjVkYDifZ7sEA9cDlnjjH1SiWj9LvmsDVoX9h5RW8knGEu?=
 =?us-ascii?Q?0pG2MgsNxVE+T1qw4AbjHs9BTOU9pR4LCXYZVLOquInCl8VfRmiTOLFmy9Um?=
 =?us-ascii?Q?JOkxHNBz3gVxIvku/1tWeU9kaZ5TSH3x6mBErNUiIKvc2M3sPNCP35U5Y4E5?=
 =?us-ascii?Q?/YBNVEMnovMt74GKgxUou/0QER2An3IxEWEwqJp/5fwWFZj47DxuBTpbsE2B?=
 =?us-ascii?Q?NGkKHRWmB8swG7ZBD4AOJY27Cdfa2yn62lqdXQEso95VSMdVTUtXzCfAou6b?=
 =?us-ascii?Q?WlPSCYuliRlEvm1jk09J7r4cchcuUXgGPSvCyIJcxk/9ws0T/FPrlpGSl08r?=
 =?us-ascii?Q?wARTNk9XaKSdxyLwbqTimH/OKn6uj5/osD0NHqIa/V9hGUFFlHxE09XIxxhh?=
 =?us-ascii?Q?aAX+CcmWUqwgmf9DSxDwaq/miIdEsFt8pFxD5PkqxV5gf5NXTTRdtEIK5OBY?=
 =?us-ascii?Q?4SS60jXicHtLxT1IJcsXMQ2H7Kbx4LnzXMgQaKzqLbYYROvJwVpvScpxRBf0?=
 =?us-ascii?Q?+gcOdgZgXy3C+vdTpShTbbBeiydAMkfBk2LQp8ghfJxLUVzxuWWPRDV5otba?=
 =?us-ascii?Q?ym8WW4ecIK1+Afe1Y7HbvLIBQarBXiAwX4/GgGnpi07Pe4GHOCa3gP25Aiht?=
 =?us-ascii?Q?kvyO8iv8NYlddKnB9wYmY1E0goOiq2/IkRXBUNC+KsvyAnQkyqtYzxKLo7RZ?=
 =?us-ascii?Q?JMYhKHMNIIdBV++SaMeCpjKgH6IJsXiPddfEF+hm2appYr5GVOiQ9LMjv0Tc?=
 =?us-ascii?Q?72hK8BGyj6dRhmjQyEfKnFf8+CluW9IHF79gc7/qN8MB9gY65CCoY1IuPVUn?=
 =?us-ascii?Q?lbKdDoXYtgCEqImUDXP5a2RWKP4j9Hw80doH8UOKUpHI063bnfMDIxXjY/MW?=
 =?us-ascii?Q?o9TZSBgYPPLNIN7TsHaPk6kto47dEE2mHJIkynIT4zXi7Wb54HrhYvUoc5U+?=
 =?us-ascii?Q?j+lVscmsW1z4+qZBHgcdrYgSZLB1dUkvgivgMHhSQgl5U6m6h8+6MsIBoPdJ?=
 =?us-ascii?Q?ZLYrvcKYuyO7lD6qejMMlzRWEG1mxc7MhEX7f12BW26t94sIEaPJOti/RE81?=
 =?us-ascii?Q?LiDEQrLagIgFDhZYCcunetU2bd/T9gLOWa3h76clLGotpYflyhh8yHe6Z5H1?=
 =?us-ascii?Q?dB1iOX0uCNYcdVs36sIUzOUBCn6GDLHjF3LRxeIRqSgmqpYGk+VhMGVjXb7B?=
 =?us-ascii?Q?twg41VtEtCCn3zUsAbER13F0Gxy+ujZ53pqBZeOIi/ENTa9iut+4fPEzuYA5?=
 =?us-ascii?Q?BBrCq/U5PJO2dwzC2i0=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: c4a7129c-bb8d-4d26-be8a-08d9c4dd2b06
X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB5506.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Dec 2021 23:54:04.3796
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: tHTDgmv6epXip+BUB+b/K17BQ2e2KqLvhvM+WbpP2trGYj3ONoWZutCEZKVAruTT
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5317
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Tue, Dec 21, 2021 at 04:19:33PM +0100, David Hildenbrand wrote:

> >> Note that I am trying to make also any kind of R/O pins on an anonymous
> >> page work as expected as well, to fix any kind of GUP after fork() and
> >> GUP before fork(). So taking a R/O pin on an !PageAnonExclusive() page
> >> similarly has to make sure that the page is exclusive -- even if it's
> >> mapped R/O (!).
> > 
> > Why? AFAIK we don't have bugs here. If the page is RO and has an
> > elevated refcount it cannot be 'PageAnonExclusive' and so any place
> > that wants to drop the WP just cannot. What is the issue?

> But what I think you actually mean is if we want to get R/O pins
> right.

What I ment was a page that is GUP'd RO, is not PageAnonExclusive and
has an elevated refcount. Those cannot be transformed to
PageAnonExclusive, or re-used during COW, but also they don't have
problems today. Either places are like O_DIRECT read and are tolerant
of a false COW, or they are broken like VFIO and should be using
FOLL_FORCE|FOLL_WRITE, which turns them into a WRITE and then we know
they get PageAnonExclusive.

So, the swap issue is fixed directly with PageAnonExclusive and no
change to READ GUP is required, at least in your #1 scenario, AFAICT..

> There are 2 models, leaving FOLL_FAULT_UNSHARE out of the picture for now:
> 
> 1) Whenever mapping an anonymous page R/W (after COW, during ordinary
> fault, on swapin), we mark the page exclusive. We must never lose the
> PageAnonExclusive bit, not during migration, not during swapout.

I prefer this one as well.

It allows us to keep Linus's simple logic that refcount == 1 means
always safe to re-use, no matter what.

And refcount != 1 goes on to consider the additional bit to decide
what to do. The simple bit really means 'we know this page has one PTE
so ignore the refcount for COW reuse decisions'.

> fork() will process the bit for each and every process, even if there
> was no GUP, and will copy if there are additional references.

Yes, just like it does today already for mapcount.

> 2) Whenever GUP wants to pin/ref a page, we try marking it exclusive. We
> can lose the PageAnonExclusive bit during migration and swapout, because
> that can only happen when there are no additional references.

I haven't thought of a way this is achievable.

At least not without destroying GUP fast..

Idea #2 is really a "this page is GUP'd" flag with some sneaky logic
to clear it.  That comes along with all the races too because as an
idea it is fundamentally about GUP which runs without locks.

Jason