Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <594f1013-b925-3c75-be61-2d649f5ca54e@amd.com>
Date:   Mon, 15 Aug 2022 15:06:22 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.11.0
Subject: Re: [PATCH v1] drm/ttm: Refcount allocated tail pages
Content-Language: en-US
To:     Dmitry Osipenko <dmitry.osipenko@collabora.com>,
        David Airlie <airlied@linux.ie>, Huang Rui <ray.huang@amd.com>,
        Daniel Vetter <daniel@ffwll.ch>,
        Trigger Huang <Trigger.Huang@gmail.com>,
        Gert Wollny <gert.wollny@collabora.com>,
        Antonio Caggiano <antonio.caggiano@collabora.com>
Cc:     dri-devel@lists.freedesktop.org, linux-kernel@vger.kernel.org,
        Dmitry Osipenko <digetx@gmail.com>, kvm@vger.kernel.org,
        kernel@collabora.com, virtualization@lists.linux-foundation.org
References: <20220815095423.11131-1-dmitry.osipenko@collabora.com>
 <8230a356-be38-f228-4a8e-95124e8e8db6@amd.com>
 <134bce02-58d6-8553-bb73-42dfda18a595@collabora.com>
 <8caf3008-dcf3-985a-631e-e019b277c6f0@amd.com>
 <4fcc4739-2da9-1b89-209c-876129604d7d@amd.com>
 <14be3b22-1d60-732b-c695-ddacc6b21055@collabora.com>
 <2df57a30-2afb-23dc-c7f5-f61c113dd5b4@collabora.com>
 <57562db8-bacf-e82d-8417-ab6343c1d2fa@amd.com>
 <86a87de8-24a9-3c53-3ac7-612ca97e41df@collabora.com>
 <8f749cd0-9a04-7c72-6a4f-a42d501e1489@amd.com>
 <5340d876-62b8-8a64-aa6d-7736c2c8710f@collabora.com>
From:   =?UTF-8?Q?Christian_K=c3=b6nig?= <christian.koenig@amd.com>
In-Reply-To: <5340d876-62b8-8a64-aa6d-7736c2c8710f@collabora.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?S2swYTNIQXJSblNETnJSMlFIdGlUYUV5MFZXMEllR28xNENxdG9KU2tHOTFG?=
 =?utf-8?B?K3E5MXgvS2xFYkE2RHVsYTVZVE02V3pNMkJDMVhmYTR2SHJTd3hZZmpFYWg0?=
 =?utf-8?B?UmhpVXN0TVVFWXp2N1BjbjQ1RmRLNDYrUmJNelYzVk90UWhnNlpLcldoa0lS?=
 =?utf-8?B?eFBwMDZyRU81VzhWNXJ3WEkzRFZLL25VTTEyQVhuODhDdWdsUWloS1paSHd6?=
 =?utf-8?B?OXNRYyszZXV5VkdLMjMxcWFpRUlEVGJvU0hwTXUwKzBkN3BQWnByWlBsKzZj?=
 =?utf-8?B?YVRPV3UxY0VXcU1ad0owRjR2VWg5UFJUMzc2MjhQVTZlNzJGd21GL1kwMkVm?=
 =?utf-8?B?eFcveDN6bURZSE5nT1RPNUdiQXBIV2dXL0VHSmNYMjM2OG5GSzNUUEkyWTQv?=
 =?utf-8?B?LzNRUC82akdLRllMVlNab3RFMXpLdXFDT0d6RWI1aDAveHdLYm5uMHRYRDIz?=
 =?utf-8?B?YnhOSG9LK256UlNCdGhrVDdCZk1hWVFUN2tNNEptOW1oYy80ZlB1V2tvL1do?=
 =?utf-8?B?SFB6WXZ1M1hlR3E2RXFtekcwS0ljWjIvb2ZIQ0pxb0w0N1hqOTcvSnRSOFA4?=
 =?utf-8?B?WW5VZG5LaVAxTEdrK0orMTB2Y1J1ZE5CcG1EWFZjclJIWUdyMkZRdmxKRXAv?=
 =?utf-8?B?SFJlMms0bVJld0ZFQ0hUa2Nkd0JsY0N2VFFmVDFTclQ0N2Q4eTR0RjN3M0ZB?=
 =?utf-8?B?emxJNEJFR0R5T2t2V3FNZkE0RnhzTlB4dVhmcHRad2x0TTNyTUFTbFh6d2hk?=
 =?utf-8?B?c1BRWE9XOEFlenVCVk52YTRUdkM5Z1oyeEoyVUdZYWt1QWpZQ281dHFMeCs2?=
 =?utf-8?B?ckdzS0JpZnlUbkNvYXN4NU12SHFDQzZkU3RWcDlvc2t1YnFNODJjMllMNUNH?=
 =?utf-8?B?cFBCZGRQamJZSVdaTjBlNEl3RzlZNUFobnZ5ekNWckhka1VqTTFQTWoyT01C?=
 =?utf-8?B?YmRidS8yQldtUThNTGNzSkJkNEpOUU1jOUVyd2NVTlpsQlRuNWRXYmdUTkZU?=
 =?utf-8?B?YTdCQVMzZHVZMEVRblpDNUV3L0pqZDNmVUxORkRnLzVHYkRQZHBiSk53YTNW?=
 =?utf-8?B?R3VzazI0TXkxZmZjRXRkM212UUtkNWpzVnJGajhJWGNOdVZ1amdCZ0ozUmdP?=
 =?utf-8?B?ZVRmL0d6MWhXQTdCS2RzSmFVaDZYT1ZWaHd5QWxVeE1oSm16RmhnVzFaYm5E?=
 =?utf-8?B?V3U4ZmZGVjZnUTk3aDVHV0FIeEF4eDlraXdXSy9Ya1JCZGkwQVVBR1ZmemJP?=
 =?utf-8?B?NkovOUNFelgyQ1V2bmxvWW5LUitXOFdrdm02UU1JU1k0eXVKamUyNlZvTVRk?=
 =?utf-8?B?SUtRTzJDOVlOOTdKRzkyTTB5YlpTSHA5ZzNrdVpyYmg1Z2pBUExtVGxKZXFs?=
 =?utf-8?B?YUhYR3loMDc3NE5yOGQ5TVpGWWQ4SFFJOE9ncE1OdldaN0lQSVBoOER5ampl?=
 =?utf-8?B?SUZQSDk0NE0zWFRLNWNMN0o1SS9mdHNBSDhwNWNGZGpRVEN1Vk5QYXhlaEZn?=
 =?utf-8?B?UDB6UFlwMVVvVEN3cFNlbi9PTnBXQXZwcEs0QThiZHBCcGVnUlkxdU1FWkds?=
 =?utf-8?B?VktrVDNidW9PZEdva0R3OVJjaW0zUDVkQ09jSEErNnZ2YkFTN1hidUZ3dFBi?=
 =?utf-8?B?SzVuRlA1a3FQcWU0N2YvaUY4SWxRbExnQWlUN0VsbVhJdW1weXlUWmdUZG1w?=
 =?utf-8?B?WnMvVE5BZS9PbzVVU0Y1emFDdXJFY0hRUkltbkExdVF2SlJ2NDdBaEtLdkt0?=
 =?utf-8?B?citBdi9janc1Z1FkcDBnMjRuOUVnRFZFdzY4QnhhM3Q0Z1BLdG5iRlRMZUF2?=
 =?utf-8?B?RGh2aWZOVVNRZitrb2R4cG1veEQzM2phdEgwelR5di85MWVRMUNpU055TTBu?=
 =?utf-8?B?SDlvaDlTV2Uzd1M5L2xaWWh3eFF4ZC9NMDZLOEl5ZTVoMi9nN24xTTE0ajVO?=
 =?utf-8?B?SWZyQU93bGRaenRBVi9QVC8rV3k1V2IrcmJGVjd2YktBZlFyd0M2T0hsRmZG?=
 =?utf-8?B?MkxwQ0g3YkhzYXY2YzBLeitpelFRalVGQmFMSDNRSnFua3E2SmQzUlo5bmdR?=
 =?utf-8?B?WUtDcUptdWtETk9tTWc3V1h0OWJYSnNWd1BlQnFmS1d6KzE2Y2FnU3JmRmlV?=
 =?utf-8?B?YS9TNVdQYkxQNlkya3UrZ2k4RTRoN0k0VFFmSitNek9LZ0pMMGwxVTAyOTND?=
 =?utf-8?Q?kCPbFAflYwMI76lhRUtn8mGZb43iGnUYe5kx/hu9DSkc?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 7984a87b-a0af-4998-ee1a-08da7ebef6ea
X-MS-Exchange-CrossTenant-AuthSource: BN8PR12MB3587.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Aug 2022 13:06:28.0323
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: mswQRxQ96Rs0EA6QHO2IYBmvTxJX3cT0/rBDyNzoQXqOhgG6CYPCuuw56b67QAw1
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB5070
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_NONE,
        RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS,T_SCC_BODY_TEXT_LINE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Am 15.08.22 um 13:50 schrieb Dmitry Osipenko:
> On 8/15/22 14:28, Christian König wrote:
>>>>> Maybe it was discussed privately? In this case I will be happy to get
>>>>> more info from you about the root of the problem so I could start to
>>>>> look at how to fix it properly. It's not apparent where the problem is
>>>>> to a TTM newbie like me.
>>>>>
>>>> Well this is completely unfixable. See the whole purpose of TTM is to
>>>> allow tracing where what is mapped of a buffer object.
>>>>
>>>> If you circumvent that and increase the page reference yourself than
>>>> that whole functionality can't work correctly any more.
>>> Are you suggesting that the problem is that TTM doesn't see the KVM page
>>> faults/mappings?
>> Yes, and no. It's one of the issues, but there is more behind that (e.g.
>> what happens when TTM switches from pages to local memory for backing a
>> BO).
> If KVM page fault could reach TTM, then it should be able to relocate
> BO. I see now where is the problem, thanks. Although, I'm wondering
> whether it already works somehow.. I'll try to play with the the AMDGPU
> shrinker and see what will happen on guest mapping of a relocated BO.

Well the page fault already somehow reaches TTM, otherwise the pfn 
couldn't be filled in in the first place.

The issues is more that KVM should never ever grab a page reference to 
pages mapped with VM_IO or VM_PFNMAP.

Essentially we need to apply the same restriction as with 
get_user_pages() here.

>> Another question is why is KVM accessing the page structure in the first
>> place? The VMA is mapped with VM_PFNMAP and VM_IO, KVM should never ever
>> touch any of those pages.
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Felixir.bootlin.com%2Flinux%2Fv5.19%2Fsource%2Fvirt%2Fkvm%2Fkvm_main.c%23L2549&amp;data=05%7C01%7Cchristian.koenig%40amd.com%7C2f38c27f20f842fc582a08da7eb4580d%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637961610314049167%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=Pu5F1EF9UvDPdOQ7sjJ1WDRt5XpFZmAMXdkexnDpEmU%3D&amp;reserved=0

Well that comment sounds like KVM is doing the right thing, so I'm 
wondering what exactly is going on here.

Regards,
Christian.

>