Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <217b4a17-1355-06c5-291e-7980c0d3cea6@nvidia.com>
Date:   Sun, 28 Aug 2022 21:59:49 -0700
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.12.0
Subject: Re: [PATCH 5/6] NFS: direct-io: convert to FOLL_PIN pages
Content-Language: en-US
To:     Al Viro <viro@zeniv.linux.org.uk>
Cc:     Andrew Morton <akpm@linux-foundation.org>,
        Jens Axboe <axboe@kernel.dk>,
        Miklos Szeredi <miklos@szeredi.hu>,
        Christoph Hellwig <hch@infradead.org>,
        "Darrick J . Wong" <djwong@kernel.org>,
        Trond Myklebust <trond.myklebust@hammerspace.com>,
        Anna Schumaker <anna@kernel.org>, Jan Kara <jack@suse.cz>,
        Logan Gunthorpe <logang@deltatee.com>,
        linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org,
        linux-xfs@vger.kernel.org, linux-nfs@vger.kernel.org,
        linux-mm@kvack.org, LKML <linux-kernel@vger.kernel.org>
References: <20220827083607.2345453-1-jhubbard@nvidia.com>
 <20220827083607.2345453-6-jhubbard@nvidia.com> <YwqfWoAE2Awp4YvT@ZenIV>
 <353f18ac-0792-2cb7-6675-868d0bd41d3d@nvidia.com> <Ywq5ILRNxsbWvFQe@ZenIV>
 <Ywq5VrSrY341UVpL@ZenIV>
From:   John Hubbard <jhubbard@nvidia.com>
In-Reply-To: <Ywq5VrSrY341UVpL@ZenIV>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?RU9KQlNEejY5RzNySjdrUmJoU3piS3ZVNVhKR0IzZXJMbEFNSlkwalpBOHQw?=
 =?utf-8?B?OVVENDdrYk9mMllMblRtdXpWSlQzMjdnTU40SllrUVhzbHFQaEFzMkRDencr?=
 =?utf-8?B?U0Jzb0haQTZqY0RzVHN2TndsS1pVeEtwRDJoWkVQa2dYRlBoRWh0VnNrVHlk?=
 =?utf-8?B?T3pxREtCUzJaL1pLc2dwTHZLVW1zeGdxT2gxTCtPbkcraTY0cldwOWtxMndX?=
 =?utf-8?B?bnpNMldBNXpKUGZGS2JLNEx1YVpjRnBDYTBQUWQ2NlRrc3dzLzZzenhrcDFu?=
 =?utf-8?B?dUJtVmdOeFI2SW91L1h5Rm9TSWxPbkhQbnJPOVQ4ZUxjTG03TXpSOTU2UTRL?=
 =?utf-8?B?em5EVFk3NUhQaUZocjZ4TmcvajhXN0Y2U2xhNTBTVUJpN0FZS01KWFkrTmx2?=
 =?utf-8?B?NXlqcnltdUN3SzhxTjhPYUFOaVV6VnVOa0o0Ujd2M2d1Z1hqM0hkVkhvWnl4?=
 =?utf-8?B?VUY1Yjh6T2xkK0N3ZkZmYlhUTW5sc3dxWklaVUZaRng1R0gvcDNpVkFDQkdw?=
 =?utf-8?B?MzN0MUEvc2NYQS95ckticDczM2wyVk5GVEE5bjVaQXNpR1hvZjRMR1VqczJ4?=
 =?utf-8?B?Sm1tR0JBQ2hub2xIQzV3NUN3WTh6MFVDQVR1ME8yZHZST21jcXpqS01iSjA4?=
 =?utf-8?B?QU1rUFJpaGo1WlcvZ0hIOCtVUElKNVdCcGc4YTBvM1l0cWRqVkZwRmpCb2NZ?=
 =?utf-8?B?eWdHNWREWTVMWDd0SW0xbUkwRVlTL2p6RHN3OTJmRC9GLzM4bkp6dDc0eHFE?=
 =?utf-8?B?Vnh4azJzVG51TWdubC9JM1pCeG5wYWVIWFhLZTB0UVFOZThDamFCK2xMTVJX?=
 =?utf-8?B?MEZ2d3VZQ1JUY290Kzc0NzFBRWtMSG9kcUVoY3FMcDA3eGo3MG13cHZFNlJE?=
 =?utf-8?B?SmF4WTVYVEdXa2JjNjdLWCsxTjcxR3NBZ1laVjlpRnE2cHlwSnEybU5aOHpR?=
 =?utf-8?B?WTFDV2dSU3BJNVl6bWlBY3BqdjRjOGU0V253TUoxQ0tzZ0hLOFg4Y29RMUFk?=
 =?utf-8?B?bGNsS2lDOFdvY2lFbDFWUFBTTGsyQ0E5cCs5QkZuYlBBVkZuRXRRRmxTZ0Zn?=
 =?utf-8?B?VVdPSFFmSWRxZ0duQUtkSG56emxXZy8rcHBuRWhxc2txVHZ3dUNJRmJsSmRD?=
 =?utf-8?B?S2lkaWY0U0lUbjBpUlY5N0JQNG16L292ak1mRGEzK2x5TGM5ZkFWeEc4Vkpq?=
 =?utf-8?B?UkpXR2lvdkcvYXQ3a0dybXlkYjBVVlVVd21xVlRVWHBYMVk4WDgrdHV2MXRF?=
 =?utf-8?B?VEhIa1FLWkZiRGsvaGIyRXk1Rm5WZ2dZaTZsMnU0U3JZTTV1ZTZjUWRYWThs?=
 =?utf-8?B?S2lhaHVHWld5NVpqcXpTYTA1ajkzbi9yZ3V3dURkY0s2b1VxaHl3d1BjOUlw?=
 =?utf-8?B?alVvMWFjL0Evdit3SlM4Nmp5YWtTTXp0OGdCbVhtZzJmU29iUTJEVWw0NUJB?=
 =?utf-8?B?Qm1naTk5VXFlcmpXRjhvdklFS2gwcXhHQ3RmQkhKNFNBTlozYnZvek05eCt1?=
 =?utf-8?B?YWZqd2FFbzB0UTkyOTJoSjNGK2NOZXNMOHByTHBWdWdESWR0cGs5SFh4SXBX?=
 =?utf-8?B?MzI4eE9VQ2l0czV3M3lhQ2xsdWRya3dSR00wSWEvWlB4TkdyS2hJNjlyTklz?=
 =?utf-8?B?YnhreUt2djNsdVFHNGtyd2NtVi9PYlRqaEQrWmJzaHBCc1g1R0FmaitCbndK?=
 =?utf-8?B?WUdPc0hnclZQMzBTMFc3MHVXQW95M1lxdjN4OC9peUl5S0pRenloWTFuQnhi?=
 =?utf-8?B?aUNZL21zbjViS1A0RElmUmRJTHlKaU1ONXV2R1Jpb3Jpck9HNHhUeEdDTmpw?=
 =?utf-8?B?aG50bnB2cnVHQ0FqZ1NtRVp0a0JwMHBrSDBXYnVwVkdOK3JDYUQ5N1Z2OVpi?=
 =?utf-8?B?bDZKY0ZRdWc1b3N4SXcwMGNPbkY2dXFIVEo3NEVJQjVWQnA4cWxLTHgxQlll?=
 =?utf-8?B?MHNodi9TMklTTHlVMDY1Vk9FeUh1amc5Uk5QYWh0eks0YzJWU081Y1hxdm9B?=
 =?utf-8?B?VjBSNXN5cFgzSU9pK2NuTTNIRE9MMC9tTTZKRU8vQkUyTWtwYi9LVzBGWTFV?=
 =?utf-8?B?azdNaTMxZGVZM1paOXZ4N01NNjFvWW5NZnBiTlUzYlZsNmsyNkxUTUpZaXJs?=
 =?utf-8?Q?mXBMFVXU5AMCchdqR7et+yHJV?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 0bce0aeb-c394-4d9c-b6fb-08da897b4e1e
X-MS-Exchange-CrossTenant-AuthSource: BY5PR12MB4130.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Aug 2022 04:59:51.2191
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: EnQy0gUtWEXMgZlNDJYSpibI3OTFEVZCQWPZFQ/UqzTOprTKekcobYuZIdurVnB5OlWgnvkm2pqhBtt+kE3VfQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN9PR12MB5289
X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        NICE_REPLY_A,RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,
        SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org

On 8/27/22 17:39, Al Viro wrote:
> On Sun, Aug 28, 2022 at 01:38:57AM +0100, Al Viro wrote:
>> On Sat, Aug 27, 2022 at 04:55:18PM -0700, John Hubbard wrote:
>>> On 8/27/22 15:48, Al Viro wrote:
>>>> On Sat, Aug 27, 2022 at 01:36:06AM -0700, John Hubbard wrote:
>>>>> Convert the NFS Direct IO layer to use pin_user_pages_fast() and
>>>>> unpin_user_page(), instead of get_user_pages_fast() and put_page().
>>>>
>>>> Again, this stuff can be hit with ITER_BVEC iterators
>>>>
>>>>> -		result = iov_iter_get_pages_alloc2(iter, &pagevec,
>>>>> +		result = dio_w_iov_iter_pin_pages_alloc(iter, &pagevec,
>>>>>  						  rsize, &pgbase);
>>>>
>>>> and this will break on those.
>>>
>>> If anyone has an example handy, of a user space program that leads
>>> to this situation (O_DIRECT with ITER_BVEC), it would really help
>>> me reach enlightenment a lot quicker in this area. :)
>>
>> Er...  splice(2) to O_DIRECT-opened file on e.g. ext4?  Or
>> sendfile(2) to the same, for that matter...
> 
> s/ext4/nfs/ to hit this particular codepath, obviously.

OK, I have a solution to this that's pretty easy:

1) Get rid of the user_backed_iter(i) check in
dio_w_iov_iter_pin_pages() and dio_w_iov_iter_pin_pages_alloc(), and

2) At the call sites, match up the unpin calls appropriately.

...and apply a similar fix for the fuse conversion patch.

However, the core block/bio conversion in patch 4 still does depend upon
a key assumption, which I got from a 2019 email discussion with
Christoph Hellwig and others here [1], which says:

    "All pages released by bio_release_pages should come from
     get_get_user_pages...".

I really hope that still holds true. Otherwise this whole thing is in
trouble.


[1] https://lore.kernel.org/kvm/20190724053053.GA18330@infradead.org/

thanks,

-- 
John Hubbard
NVIDIA