Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Wed, 24 Mar 2021 10:48:33 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Thomas =?utf-8?B?SGVsbHN0csO2bSAoSW50ZWwp?= 
        <thomas_os@shipmail.org>
Cc:     David Airlie <airlied@linux.ie>, linux-kernel@vger.kernel.org,
        dri-devel@lists.freedesktop.org, linux-mm@kvack.org,
        Andrew Morton <akpm@linux-foundation.org>,
        Christian Koenig <christian.koenig@amd.com>
Subject: Re: [RFC PATCH 1/2] mm,drm/ttm: Block fast GUP to TTM huge pages
Message-ID: <20210324134833.GE2356281@nvidia.com>
References: <20210321184529.59006-2-thomas_os@shipmail.org>
 <YFnST5VLcEgv9q+s@phenom.ffwll.local>
 <314fc020-d243-dbf0-acb3-ecfcc9c2443c@shipmail.org>
 <20210323163715.GJ2356281@nvidia.com>
 <5824b731-ca6a-92fd-e314-d986b6a7b101@shipmail.org>
 <YFsM23t2niJwhpM/@phenom.ffwll.local>
 <20210324122430.GW2356281@nvidia.com>
 <e12e2c49-afaf-dbac-b18c-272c93c83e06@shipmail.org>
 <20210324124127.GY2356281@nvidia.com>
 <6c9acb90-8e91-d8af-7abd-e762d9a901aa@shipmail.org>
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <6c9acb90-8e91-d8af-7abd-e762d9a901aa@shipmail.org>
MIME-Version: 1.0
X-MS-Exchange-MessageSentRepresentingType: 1
Received: from mlx.ziepe.ca (206.223.160.26) by YT1PR01CA0085.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:2d::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24 via Frontend Transport; Wed, 24 Mar 2021 13:48:35 +0000
Received: from jgg by mlx with local (Exim 4.94)        (envelope-from <jgg@nvidia.com>)        id 1lP3s9-002125-5r; Wed, 24 Mar 2021 10:48:33 -0300
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: b77e31cc-ac60-4992-f5b8-08d8eecb8520
X-MS-TrafficTypeDiagnostic: DM6PR12MB4299:
X-Microsoft-Antispam-PRVS: <DM6PR12MB42993F63F5E284A5B470938CC2639@DM6PR12MB4299.namprd12.prod.outlook.com>
X-MS-Oob-TLC-OOBClassifiers: OLM:8273;
X-MS-Exchange-SenderADCheck: 1
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info: fkrmE1OcA+0wwqaCCOc0voeiF985lUHTn05MiSETfEJEaHAd1NRba1Pc94sOckl3DbtUAfBmsb/wqRH1RIhj5YZE0qivCTSO/pc95PPivoP4XHgR39g0ihBvIpHDaZ1NpPcMigx4L0NAHI5Nde2Nxu8AUm1KtHyGAq6hO5SxCW9yrNDiv4F/zZSWIPLDXss8GDDHHGuofpRqbCTg/kntTYtFEJQSGHb0jTY3cx57CIkhGly04UXDHrIOfwjHd9oxHuSlfmu+zrRh7OUkqcXjZqO4yMlli1BKERBfR+dvUWEGzW93X1sd/tsA2eY2FZd3UmaOkON4qR8oBtY8jSICeqtv48OMHwBE2hyXQTFDgx2d48yCaRdGNcdswfzO7AsSEUwtnVA+P7afbjijClPnzgDUsSDNV1ZRTb0o2kxWojZ+AS5MgcvdP4H07bCtMpHp+tn63NXa9zJjYNkx4cCCfw6GMHRWGPs5oA071+vlilR6yVWDXkarMCzwQYfU6hOV7HVOknS8jH2gvY9GZys1BSfT6OV8dYruZ8xTm+YsgCJrnf99nLbdIFFO0EF0V0iyIXtGhW8JqZFhJepR5FJI6d49CnvL0HfbehQhTdkn9QV/7ExqFTNqUchUIdmoYilDBBrvlYM9EpF7k7BfKfkGf13iwxQx8r7F0mdGj05Wl3I/xKw4wMQAkkO3TjbtYGJp
X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3834.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(39860400002)(346002)(366004)(396003)(376002)(136003)(2906002)(36756003)(33656002)(6916009)(8676002)(8936002)(38100700001)(9746002)(5660300002)(1076003)(9786002)(316002)(54906003)(4326008)(83380400001)(66946007)(86362001)(66556008)(2616005)(26005)(66476007)(426003)(186003)(478600001)(14583001);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData: =?utf-8?B?WDQ1UVBMSWRKMkl5SEwzWUNuWFdJd3lTbGVJTFFIWDZNTHpCVWpWZ0FJdDhv?=
 =?utf-8?B?ZHpzZVBUdlordko2Z1FsS1UycTBtVEdQSWlBK3JDRzcxWVlaQ2hIMFNZbHE3?=
 =?utf-8?B?ak1kVEdzcmh5Q3BLNWV2RTlmOEpPRlFzUnFIdWFpaWpQNWZ4VjdsOC9DQk5N?=
 =?utf-8?B?WWdCQkVwWUlOQlljMm5DQXZXdVZOZURMVkUxY21qK1h4NDlUUkZpaEd6c0p4?=
 =?utf-8?B?MnFhbTZiSWR0S3dsNllTMnNBdWhQS0R3bzU3NDNxZk5Gc0MrbGsraWQrNW9p?=
 =?utf-8?B?M1UwU3E2VysyNkd6VW4rTEFQVXNkZ1RoSGtsckgzYmJ6Ry9tMXpmeFVuMC9r?=
 =?utf-8?B?blJWSUlEY1hSN2RXeVdtTnZ4bmR3c2tmbG02cTVsZUwxRmJ4ZjlNcE5Zb01R?=
 =?utf-8?B?OVhLdnBzc0hOM2dLQUt2UVppZkRaUDdyT3dTSUcvY3VDMUlyazhPRTc2SmE3?=
 =?utf-8?B?amt5NVlIWStsK0NlN3lPaU1VTkh2dFg3YUhsV2MyalhLU1pETkZTQUFmSS96?=
 =?utf-8?B?cFpaYkhrd1R3RTNERTVqTURzemdRejZOeHlkNmNrRFQ4OCt4d3hSZWZJOVYv?=
 =?utf-8?B?V0Z5WHBSMHpBTUkvRXhZSHdFSXp6clVnNDVZVXRpUDdLSmNvSmpjTlA5ekE4?=
 =?utf-8?B?Tm1NSFI0TDJMV21ubWJDd241UDRqaFdmQko4SG41V0lRRkRFdTdhVGFqOWRX?=
 =?utf-8?B?Njg5cHI5ZjN5N1p2MDRIWCtFeWhOVUw5NGdzZHhkWTgwSVJST1FmVWRpSVhO?=
 =?utf-8?B?SmNYdlBIZWZVNGF3Vk1rM1JTcEp1TTlZd0lXS2FwSlFkclpwSUtQd0RoalFS?=
 =?utf-8?B?N05FZDJucC9wbE5rSEtHdm0wYS9nblduN0NZZXcyME5wSGhxcS92Q3lEbWtl?=
 =?utf-8?B?anJBS3IrUW9kSFBOcU5QTUYwQUhYcEo1dVZVRlNHa2g4YzhPejlrMm5Sa1Rw?=
 =?utf-8?B?blJQNzRlS25HSStadVJjZ3pCdHRIU3BYRzgrRXIrWm9EZkF1ck5XUFNoMGg3?=
 =?utf-8?B?cmh5TTdlZlV4ZXpqYW1hWVN6bk9iNm92d0xVaFAxYWhpdTlrRFU2ZEpDL0ZC?=
 =?utf-8?B?d2hOOVBwRGN5WG5LOUl6MHdoT1NvMmlNRHVLOGduNTJ3VXUvK0JkM0FnTzh0?=
 =?utf-8?B?SmQ3Znh3RFpLWGFmZW4yaFZqcFp1cFNEcnpRbGoxNUFyZTNUYUxvSjZPWlR0?=
 =?utf-8?B?QlRKL3QwaUtxM1FuYkVqdW01Mko0M2pVSTlUMHNMZllWYVFsbHBiTlU0NTVj?=
 =?utf-8?B?b3dMU3RjM3dhWWQ1WFR2YU9JQkdoSGFTL3VSS0N2bHp2WUhUUE8rVk15dGg5?=
 =?utf-8?B?Q1UyTnZSNEllWTV6T2NPY2tqSDY2ODBqVS90cWR4M0VkZG9Dczc0QWRQNlYx?=
 =?utf-8?B?aUlFSTNNN2VBZmV5NmJONm9vOHZPd0dSZHBwQ2I1VkFLVE9VYk1jeGQrQlpO?=
 =?utf-8?B?Ky9EUXlTZTcwWlN3ZDBKejJTS0tUSXFGdyt6MGIvd0RadlE4WCtlaVl4SUxM?=
 =?utf-8?B?R3FvY0JzOUlaOUZjN1REUTlLSHpGbytaZ29lMzJYU1A2dis5T3Q4Z2NKdUho?=
 =?utf-8?B?V0RZNWFVVkluVVFXT2o5czFmVG5XUTMzR2dLV3JjdXFDUFhkUlZZNUxDelFi?=
 =?utf-8?B?TDh4RHJydm40SkZKck91ZlNlL0Q1blh0Y2hjNDVNYVUxQ1doWmlzaFRXYlpV?=
 =?utf-8?B?YjRQZ0xVUXBieFB0MkdjRGpiVWRBNUE3cjN1bnU0NjNxclZNY3ZyRXd6NG5O?=
 =?utf-8?Q?WMH7BvK8UF+Y5Tm9EvcfpdFspj8Mo7ilkDe5xne?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: b77e31cc-ac60-4992-f5b8-08d8eecb8520
X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3834.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Mar 2021 13:48:35.3536
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: yMYrrBwsm7N3caJncL/tgMgeorWdfhnSv5tJcYlP1Sat7kiukKslvrtMqv7jCwwN
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4299
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Wed, Mar 24, 2021 at 02:35:38PM +0100, Thomas Hellström (Intel) wrote:

> > In an ideal world the creation/destruction of page table levels would
> > by dynamic at this point, like THP.
> 
> Hmm, but I'm not sure what problem we're trying to solve by changing the
> interface in this way?

We are trying to make a sensible driver API to deal with huge pages.
 
> Currently if the core vm requests a huge pud, we give it one, and if we
> can't or don't want to (because of dirty-tracking, for example, which is
> always done on 4K page-level) we just return VM_FAULT_FALLBACK, and the
> fault is retried at a lower level.

Well, my thought would be to move the pte related stuff into
vmf_insert_range instead of recursing back via VM_FAULT_FALLBACK.

I don't know if the locking works out, but it feels cleaner that the
driver tells the vmf how big a page it can stuff in, not the vm
telling the driver to stuff in a certain size page which it might not
want to do.

Some devices want to work on a in-between page size like 64k so they
can't form 2M pages but they can stuff 64k of 4K pages in a batch on
every fault.

That idea doesn't fit naturally if the VM is driving the size.

Jason