Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <41dc3e5a-9e90-70df-74df-ccdf8fa5ae86@amd.com>
Date:   Thu, 9 Jun 2022 17:22:14 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.9.1
Subject: Re: [PATCH 03/13] mm: shmem: provide oom badness for shmem files
Content-Language: en-US
To:     Felix Kuehling <felix.kuehling@amd.com>,
        Michal Hocko <mhocko@suse.com>,
        =?UTF-8?Q?Christian_K=c3=b6nig?= <ckoenig.leichtzumerken@gmail.com>
Cc:     linux-media@vger.kernel.org, linux-kernel@vger.kernel.org,
        intel-gfx@lists.freedesktop.org, amd-gfx@lists.freedesktop.org,
        nouveau@lists.freedesktop.org, linux-tegra@vger.kernel.org,
        linux-fsdevel@vger.kernel.org, linux-mm@kvack.org,
        alexander.deucher@amd.com, daniel@ffwll.ch,
        viro@zeniv.linux.org.uk, akpm@linux-foundation.org,
        hughd@google.com, andrey.grodzovsky@amd.com
References: <20220531100007.174649-1-christian.koenig@amd.com>
 <20220531100007.174649-4-christian.koenig@amd.com>
 <YqG67sox6L64E6wV@dhcp22.suse.cz>
 <77b99722-fc13-e5c5-c9be-7d4f3830859c@amd.com>
 <YqHuH5brYFQUfW8l@dhcp22.suse.cz>
 <26d3e1c7-d73c-cc95-54ef-58b2c9055f0c@gmail.com>
 <YqIB0bavUeU8Abwl@dhcp22.suse.cz>
 <d841c1ab-c0d1-5130-11fc-c8ea04cc9511@amd.com>
From:   =?UTF-8?Q?Christian_K=c3=b6nig?= <christian.koenig@amd.com>
In-Reply-To: <d841c1ab-c0d1-5130-11fc-c8ea04cc9511@amd.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 2
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?cHRIMXo5MWlPL3FpbEhXV0hMRTAzcGJkM2RzWUFid2VFbldCa0lZQy81RGJE?=
 =?utf-8?B?aG9xTDJ0UFFGN3NMa0pKZlVaYTBHM0RTWVV4eTdQR0F5M0pkclpGYUVxUWNM?=
 =?utf-8?B?UWljTXZSNm9aZXdmZlZMWHRIcldWYVM3MWU2cWFWSUFpRkpiNmdPSklSYzRm?=
 =?utf-8?B?SFRhMkxSTUpDTlA0c2JpSVh3bVA4VUovZEpCTTJCdXRvYXlkdGF6TEpPQUtC?=
 =?utf-8?B?a2tOSndiRDQvYUFIVHNDRmJ5WUpxcy9aMTJFaXRWdXZ4SzhaTnNkcDVLeUVx?=
 =?utf-8?B?RU5EckJwRVR5Zjl5QUNSalNUSy9EU1BLMnNmeXN6MHp3L1hJN3FYUElWVnZU?=
 =?utf-8?B?SWoyaXpETHc4Ukhnd05JcWwrN0x1bzUvZkdnYzEzUHlaWS9mZ1k4aVErMm9U?=
 =?utf-8?B?NlY1RlpOcEVwcE5BZnRwSzVzdURPS0N6ZlN4dWNvN1M3M0Y1bGJPYU1jRzda?=
 =?utf-8?B?U2hDNk96N1ZiY2tKZnc5RENwejlySERVUmJiYVdjNTFpclQ5TmdDQjdnekJs?=
 =?utf-8?B?R0tkSm05Nk44QlFKWHo4eTR6NFd5dmtoUkpxUWRReGdqOHkxWXIwWXRpeXQ0?=
 =?utf-8?B?b25ZRTZRV1ZKR3c5cWhtbDNwaSswYkpITitCZE9nSEEzSXhzWnpHSEFlekov?=
 =?utf-8?B?Y3F6UzExUS9GTWp6S2hxZXAxQnpjZWR5RWFkRVVXeHJkNnBWZHNuRXl4YVlx?=
 =?utf-8?B?TGE1MkpTK0tjdm9BWmtVZldlNWVPUTlRWDhjd1FmYS9ZZ1lpUUd5Z21PWCtw?=
 =?utf-8?B?Z29FQVlNVUNzSTR6V1hqRFlNalVkWGpabWd0Y1Q3NFludDZyaVhCMTdmRnhx?=
 =?utf-8?B?MHU2MEV0VVYzd1MyaTV0aGZUQ3VPREFQNVRYZi82ZXJQRVdDRnVlcWR6dFFs?=
 =?utf-8?B?Q0F3U2w0WU9SNmlyTEJ4Q1N5TkZxVDlscEhUdS9hMnBQS0dCRnVkdWsyQUFw?=
 =?utf-8?B?UTZRMHE5T3ovUkJQT0hFYTB6L21RdWZ6WDhnSmswbTZ0UWVDaE9FdEVNU2Ur?=
 =?utf-8?B?Z2lhUkVFM2wrRUNnUmZzWVNzYk51U3AyY2hUc01UcGtDdW1CVVl3Skl0MFIx?=
 =?utf-8?B?U2c2M1VmNGV1OTRXMWZERlVCWVVuaVJybXQrdlp1QzFUT3lvZTQ1dGI1TFJK?=
 =?utf-8?B?RStucVhnNUJDUndCdkRCYmkzTkFUSmptZFZaMUIrSUtXZlNQTHd3c1o0eFkv?=
 =?utf-8?B?MHptc3J0Z1lPMStGRWoyYTM5ZnBDMUhhdFJ4Uzd0MDlJbHp4eTNlaXhkY25s?=
 =?utf-8?B?a0VCNi9MNkE4ZHFqNjhwRHR0MUJMc2NWTkdPeElHY20xaHVMTWZML0lWTGw2?=
 =?utf-8?B?YXN0MjVVRDhDVVh5MTVsN04rdGc2QlY5Mk5BT2c4aXhvMXlwK3ZzSTFTUkV3?=
 =?utf-8?B?QkNidkdBdUVMMXRXYWZLWFJ2dk9KaVh3MEwremV5ZXlaKzVaRHQrNE54RUsy?=
 =?utf-8?B?NnkxVTN5UmVtQlVhSTZUR20wc05MWDhoTmdoSFZTM1VjZmx0ZEJxVzIvZmZx?=
 =?utf-8?B?L21vemF6OUlNbzNDYjdYS0I0R3Nnd2E4TzJjb0g4MnA4V2planRxVkpvanlk?=
 =?utf-8?B?K1E1eDIvVStOUlk2NkQ2VENmUFhPTGtldTFYTjg3RUlHR0U2MVE3dnVXOWho?=
 =?utf-8?B?bjdTeU5VWlpYOGc1dGpMQWlzcVRkVVFWbXhVTy9jejdhcnRCZWdqck5LcVd0?=
 =?utf-8?B?dmord0FCaUo5TXUyMDhPR0FDQURNaHlSZmdMY3VoUWVLazN6M3ZNMFJRSEM1?=
 =?utf-8?B?ZG56clcvbmdDNUdCNVhwOUt2VHBjYnlHSmx6RTYzcERwaFVYWUdlRUsrWVdy?=
 =?utf-8?B?bnJWK3RwWnJUNjhGa1ZFYUJyOG83cDNXaFFkRis3dWlySXVKSDZBVThlMDQ0?=
 =?utf-8?B?cG5XV0NhdnpYTEs1QkZsNzNaR2llYldCRTU3NTVEM2xnTXhjV1BYUHlBZWpl?=
 =?utf-8?B?c3NwN2wwelgzYnZYV3hzUW5TYVEvdjNLMGthdEt5MEVqQ3A1ajJVcW5BNVd0?=
 =?utf-8?B?NHZRRWRqcUdkRVhxTGJuYmQrVWpPZ1lEY2p1c0tWQzNKWi9idWxQbW9OYXNY?=
 =?utf-8?B?aFJXWmFiN2JHdGlIdjJSZWMrcGJVY1N2RzNWbThMY1YwVnJ5MmpMaVpNZ2FS?=
 =?utf-8?B?YW0xajZPVTl2ejVMRHpKUCtZVnRCck5KYXpyOVJIV2ZOeXRoWXRKby82eWJy?=
 =?utf-8?B?K25CWExQQlM4a3dYSm5yV3pVbXNhbmlTVDNKMXFyZHF4anZnbHcva3BDMUU3?=
 =?utf-8?B?alNzR1NWTkdGRStmQ3pHRXU3ZEpLS0JoNnZLQStid2ZmbWt5RlRXSkkyNUpt?=
 =?utf-8?B?S2dyS09LSVdxMjhjV0h6c2MwODAyY3RBclZMVlY2QXdOcWFLa1IzYkl1UW9z?=
 =?utf-8?Q?41tY+jf6EjeE00RTCQqZaXN8gELe6CQqfPEzLWIY8/SfH?=
X-MS-Exchange-AntiSpam-MessageData-1: VjTGjkvaAuFE8A==
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: bd18c7e9-8d72-4ffc-f341-08da4a2bda0d
X-MS-Exchange-CrossTenant-AuthSource: BN8PR12MB3587.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Jun 2022 15:22:23.0149
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: +0jO5R68gSJaQQd+QO6mBlSpmZD4Ol9zhjsTBEjzP3D5RcHJXLC/+Bfljsb21NkS
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR12MB2354
X-Spam-Status: No, score=-3.3 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_NONE,
        RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS,T_SCC_BODY_TEXT_LINE,
        URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Am 09.06.22 um 17:19 schrieb Felix Kuehling:
>
> Am 2022-06-09 um 10:21 schrieb Michal Hocko:
>> On Thu 09-06-22 16:10:33, Christian König wrote:
>>> Am 09.06.22 um 14:57 schrieb Michal Hocko:
>>>> On Thu 09-06-22 14:16:56, Christian König wrote:
>>>>> Am 09.06.22 um 11:18 schrieb Michal Hocko:
>>>>>> On Tue 31-05-22 11:59:57, Christian König wrote:
>>>>>>> This gives the OOM killer an additional hint which processes are
>>>>>>> referencing shmem files with potentially no other accounting for 
>>>>>>> them.
>>>>>>>
>>>>>>> Signed-off-by: Christian König <christian.koenig@amd.com>
>>>>>>> ---
>>>>>>>     mm/shmem.c | 6 ++++++
>>>>>>>     1 file changed, 6 insertions(+)
>>>>>>>
>>>>>>> diff --git a/mm/shmem.c b/mm/shmem.c
>>>>>>> index 4b2fea33158e..a4ad92a16968 100644
>>>>>>> --- a/mm/shmem.c
>>>>>>> +++ b/mm/shmem.c
>>>>>>> @@ -2179,6 +2179,11 @@ unsigned long 
>>>>>>> shmem_get_unmapped_area(struct file *file,
>>>>>>>         return inflated_addr;
>>>>>>>     }
>>>>>>> +static long shmem_oom_badness(struct file *file)
>>>>>>> +{
>>>>>>> +    return i_size_read(file_inode(file)) >> PAGE_SHIFT;
>>>>>>> +}
>>>>>> This doesn't really represent the in memory size of the file, 
>>>>>> does it?
>>>>> Well the file could be partially or fully swapped out as anonymous 
>>>>> memory or
>>>>> the address space only sparse populated, but even then just using 
>>>>> the file
>>>>> size as OOM badness sounded like the most straightforward approach 
>>>>> to me.
>>>> It covers hole as well, right?
>>> Yes, exactly.
>> So let's say I have a huge sparse shmem file. I will get killed because
>> the oom_badness of such a file would be large as well...
>
> Would killing processes free shmem files, though? Aren't those 
> persistent anyway? In that case, shmem files should not contribute to 
> oom_badness at all.

At least for the memfd_create() case they do, yes.

Those files were never part of any filesystem in the first place, so by 
killing all the process referencing them you can indeed free the memory 
locked by them.

Regards,
Christian.

>
> I guess a special case would be files that were removed from the 
> filesystem but are still open in some processes.
>
> Regards,
>   Felix
>
>
>>
>>>>> What could happen is that the file is also mmaped and we double 
>>>>> account.
>>>>>
>>>>>> Also the memcg oom handling could be considerably skewed if the 
>>>>>> file was
>>>>>> shared between more memcgs.
>>>>> Yes, and that's one of the reasons why I didn't touched the memcg 
>>>>> by this
>>>>> and only affected the classic OOM killer.
>>>> oom_badness is for all oom handlers, including memcg. Maybe I have
>>>> misread an earlier patch but I do not see anything specific to global
>>>> oom handling.
>>> As far as I can see the oom_badness() function is only used in
>>> oom_kill.c and in procfs to return the oom score. Did I missed
>>> something?
>> oom_kill.c implements most of the oom killer functionality. Memcg oom
>> killing is a part of that. Have a look at select_bad_process.
>>