Received-SPF: pass (google.com: domain of linux-ext4-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Message-ID: <48eaf3d9-b9b4-09b7-379f-edc48d06820d@oracle.com>
Date:   Wed, 20 Oct 2021 19:51:26 +0100
Subject: Re: [PATCH v1 2/2] mm: remove extra ZONE_DEVICE struct page refcount
Content-Language: en-US
To:     Dan Williams <dan.j.williams@intel.com>
Cc:     Jason Gunthorpe <jgg@ziepe.ca>,
        Matthew Wilcox <willy@infradead.org>,
        Alex Sierra <alex.sierra@amd.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        "Kuehling, Felix" <Felix.Kuehling@amd.com>,
        Linux MM <linux-mm@kvack.org>,
        Ralph Campbell <rcampbell@nvidia.com>,
        linux-ext4 <linux-ext4@vger.kernel.org>,
        linux-xfs <linux-xfs@vger.kernel.org>,
        amd-gfx list <amd-gfx@lists.freedesktop.org>,
        Maling list - DRI developers 
        <dri-devel@lists.freedesktop.org>, Christoph Hellwig <hch@lst.de>,
        =?UTF-8?B?SsOpcsO0bWUgR2xpc3Nl?= <jglisse@redhat.com>,
        Alistair Popple <apopple@nvidia.com>,
        Vishal Verma <vishal.l.verma@intel.com>,
        Dave Jiang <dave.jiang@intel.com>,
        Linux NVDIMM <nvdimm@lists.linux.dev>,
        David Hildenbrand <david@redhat.com>
References: <YWh6PL7nvh4DqXCI@casper.infradead.org>
 <CAPcyv4hBdSwdtG6Hnx9mDsRXiPMyhNH=4hDuv8JZ+U+Jj4RUWg@mail.gmail.com>
 <20211014230606.GZ2744544@nvidia.com>
 <CAPcyv4hC4qxbO46hp=XBpDaVbeh=qdY6TgvacXRprQ55Qwe-Dg@mail.gmail.com>
 <20211016154450.GJ2744544@nvidia.com>
 <CAPcyv4j0kHREAOG6_07E2foz6e4FP8D72mZXH6ivsiUBu_8c6g@mail.gmail.com>
 <20211018182559.GC3686969@ziepe.ca>
 <CAPcyv4jvZjeMcKLVuOEQ_gXRd87i3NUX5D=MmsJ++rWafnK-NQ@mail.gmail.com>
 <20211018230614.GF3686969@ziepe.ca>
 <499043a0-b3d8-7a42-4aee-84b81f5b633f@oracle.com>
 <20211019160136.GH3686969@ziepe.ca>
 <CAPcyv4gmvxi5tpT+xgxPLMPGZiLqKsft_5PzpMQZ-aCvwpbCvw@mail.gmail.com>
 <a82a1307-938b-eaf1-cf3d-b9dc76215636@oracle.com>
 <CAPcyv4g7+FTNi-vFvx_3qzrosTSET6nPc=ozdxs=p1dm0hBJtg@mail.gmail.com>
From:   Joao Martins <joao.m.martins@oracle.com>
In-Reply-To: <CAPcyv4g7+FTNi-vFvx_3qzrosTSET6nPc=ozdxs=p1dm0hBJtg@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?Z0MyMWFYYVJPVEliSTNaVm94Ry9mNWRyamFIR3FjbVFLejRibW5vcnB0cFhi?=
 =?utf-8?B?YW51VHBGKzJDM1B6eFNnWENqanBZdjlXQ0VJOWVqQy8xQmlJaXF3WHUycUpD?=
 =?utf-8?B?Z05xMlNRMTJ5b01PV0pJUCtaQWY3SHd6clpYMnBPSDJISnlCa1M5VFhRS2p1?=
 =?utf-8?B?Y2ttY2xBb2pWQ1FVbjBiRHd0cFNRTCtxZlkvRnN1Mmo0Zk8yV3BTREpVTmM5?=
 =?utf-8?B?YUh4TERFd3NpTnIxLzdGbWZNT0xId0xZK25UMThZbUVLSVdZT3cxQkwxc0la?=
 =?utf-8?B?SE1MclF3YU5FVHRrYkJNRjdoaUFMYjlQZjRINk85b0VSSzdFWDFveXNCeGht?=
 =?utf-8?B?TzlFcFI1MVFuNDhqWDRXd2dxdnNSQXhVZjhsbDY3cGNGUW42bGFOTXQzSDNh?=
 =?utf-8?B?S09lMGVOVmhuQ1FaN2FITzRDQTZWcUFyY01Pc3ZGM2JsQ0JacHJqZk1Fb1NY?=
 =?utf-8?B?KzRJY2VjVmZhZWZBS1RWNEJONzdaa0FqVVVoVUF3QkdpVjFjdy9MMU1xYm9o?=
 =?utf-8?B?MWlXYTF2VEkwMFpkTGovekZGb29NN2lmYkpzYkdEYjBBekNMSm5uVW9yZHdx?=
 =?utf-8?B?cVpjdHNtOXhiTy93bmdielpNSXBEejNkQ0MxK1lZZ0tGc3JDMFZVcnMwcXpQ?=
 =?utf-8?B?SkRrN2UwTmphQWNqMnd0eW4rRWZEQlNIdnZOQVZjNlpwT2twU016RTJrVS9k?=
 =?utf-8?B?VkpiTzVrVnluTno1LzZJUmdldUhucTdVT1l1blNIOWo1TkN4Z0FQNERiYUhK?=
 =?utf-8?B?ZUhmaU5ySlNFa1ByMzRrQjRBZTdRazZkZUZPb0FRTll0VnBuamtsLzdvL01z?=
 =?utf-8?B?VGxFZ2hUTlpZK3lRV1IzSDJNWnRoUjFTMnpTcW43cHRIeDZxdWw0RXJjdm1U?=
 =?utf-8?B?ODQ0YVpRNTR6SkJLU1FFMHNoR1VkeTA2NkY1WlNYRU04dXByVndyKzRZSzZj?=
 =?utf-8?B?a21JWkp3RUVoU0ZWbG1XTDViVUVVVkZTWHFhbnhZQzhmT1JXcWRoVi96SjhZ?=
 =?utf-8?B?TVl4Q0lJNisrU29SVUxYR0NicGd6UWgxUUY5a3g5cE5BdHc5aitVQkJRWWRR?=
 =?utf-8?B?NHJTK3VmNVFoNDh0YiszQ3VTZXE1emlTMDRYOE81cUZnQUZtajY0Zm1IM1RI?=
 =?utf-8?B?TWsvMkhvcG0xU2tuVFliMWFuRUtQODl2cVNweWtUN3Y1OVZWblNJc3ZzV29z?=
 =?utf-8?B?STVlSXVPTmJxWFVQZk5LYk84cDhucDh1NC93VXhXcDVyWDdWcTBRS2IzVXRW?=
 =?utf-8?B?N1g1aUFzaGVhOENZTHVHSG43dDRyTFF3Wkt5VkhSVVN2TEpTbkt6WW5TTVZZ?=
 =?utf-8?B?RmFNWTJYUngrdUpIM3B1ZjhaelZySDVqZmNCRDBEaEFqNDZ4UEE3VzgrWTMr?=
 =?utf-8?B?SXJ5Rmg3RTYvdW5oQU1DbnhDTys2bU9tTXphcE13N3JaOUpoOUswb2tvY29w?=
 =?utf-8?B?eDFLZHlVZzIrM0FpN1VTUkhFalQvWUhLWFpqd2ltWDJxQ3J0K3FrZ1NNMi9G?=
 =?utf-8?B?MVRSNjQwUVVjMU1kZklIWEVIcGpaOW5wZGNCOG1KaXQ2bkJ1TUFzVkk5dUVY?=
 =?utf-8?B?aTdUQU4rb0tqMkhDUmpLd1dXejVEdG1WOVlCZURvVjNNaVc4Vm53c1ZJd1BP?=
 =?utf-8?B?SWUwYnNiS1kxbEdESUgrTjgxbENtZjBxSFVBdmk0TTI5bjhhQlg0c2dkRWth?=
 =?utf-8?B?bUptQXl5QXg0ek4vRGh1TlhZaFNEbW1QZ0YyYkh1VUw4M25DVHNpWDh3cW9w?=
 =?utf-8?B?T296bTRZdU5ra21DbjRRV3V4TFJUb3V5YTRGWXgvZWpFQWZMVE83N3RXOFJQ?=
 =?utf-8?B?OHE0NmNlNmNtaXFmVHA4QmFJRVBIZ2RiTkFNenhoTzVrYTZ0T1UzeEp3VVBB?=
 =?utf-8?B?aDBLRTdTVkF4Q1ArOVJpNnM3SW82U0xLaFFsOGFWalZ0M1VFbGR6b0RqdjBJ?=
 =?utf-8?Q?SIFwyyzyv4iPc7JjiPsose1DwJIymFS/?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: ab6f14fd-e4a6-45a3-7eda-08d993faa3e6
X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB4835.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Oct 2021 18:51:35.3382
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: joao.m.martins@oracle.com
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLAPR10MB5106
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10143 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 mlxscore=0 adultscore=0
 spamscore=0 phishscore=0 bulkscore=0 suspectscore=0 malwarescore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2109230001
 definitions=main-2110200107
X-Proofpoint-ORIG-GUID: qqe3Wl8z2pyRoXB-JoJ2TtjEBM9OeUxz
X-Proofpoint-GUID: qqe3Wl8z2pyRoXB-JoJ2TtjEBM9OeUxz
Precedence: bulk
List-ID: <linux-ext4.vger.kernel.org>
X-Mailing-List: linux-ext4@vger.kernel.org

On 10/20/21 18:12, Dan Williams wrote:
> On Wed, Oct 20, 2021 at 10:09 AM Joao Martins <joao.m.martins@oracle.com> wrote:
>> On 10/19/21 20:21, Dan Williams wrote:
>>> On Tue, Oct 19, 2021 at 9:02 AM Jason Gunthorpe <jgg@ziepe.ca> wrote:
>>>> On Tue, Oct 19, 2021 at 04:13:34PM +0100, Joao Martins wrote:
>>>>> On 10/19/21 00:06, Jason Gunthorpe wrote:
>>>>>> On Mon, Oct 18, 2021 at 12:37:30PM -0700, Dan Williams wrote:
>>>>> Whats the benefit between preventing longterm at start
>>>>> versus only after mounting the filesystem? Or is the intended future purpose
>>>>> to pass more context into an holder potential future callback e.g. nack longterm
>>>>> pins on a page basis?
>>>>
>>>> I understood Dan's remark that the device-dax path allows
>>>> FOLL_LONGTERM and the FSDAX path does not ?
>>>>
>>>> Which, IIRC, today is signaled basd on vma properties and in all cases
>>>> fast-gup is denied.
>>>
>>> Yeah, I forgot that 7af75561e171 eliminated any possibility of
>>> longterm-gup-fast for device-dax, let's not disturb that status quo.
>>>
>> I am slightly confused by this comment -- the status quo is what we are
>> questioning here -- And we talked about changing that in the past too
>> (thread below), that longterm-gup-fast was an oversight that that commit
>> was only applicable to fsdax. [Maybe this is just my english confusion]
> 
> No, you have it correct. However that "regression" has gone unnoticed,
> so unless there is data showing that gup-fast on device-dax is
> critical for longterm page pinning workflows I'm ok for longterm to
> continue to trigger gup-slow.
> 
To be fair, it's not surprising that nobody noticed -- my general intent
was just to special-case less for device-dax. Not many places use
pin_user_pages_fast(FOLL_LONGTERM). This is only exposed on those
few cases that do try to use it (e.g. RDMA/IB, vDPA), and specifically
when the page fault occurs (that requires fallback-ing to gup-slow) at a
higher level than the amount you're pinning e.g. pinning in 2M extents on a
device-dax of 1G pagesize. Pin size is limited to a 2M extent at a time by the
users I mentioned above -- regardless of the total size of the extent you will
be pinning (i.e. 512 struct pages pointers fit one page). But even with all this,
this [FOLL_LONGTERM on pin-fast] would still go unnoticed because gup-fast
on devmap is just as slow as gup :)