Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <d55338c4-d15f-14ec-c057-806a5d5aa618@virtuozzo.com>
Date:   Fri, 7 Oct 2022 13:58:49 +0300
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0)
 Gecko/20100101 Thunderbird/102.3.1
Subject: Re: RFC [PATCH v4 2/7] Enable balloon drivers to report inflated
 memory
To:     Nadav Amit <nadav.amit@gmail.com>
Cc:     "Michael S. Tsirkin" <mst@redhat.com>,
        David Hildenbrand <david@redhat.com>,
        Andrew Morton <akpm@linux-foundation.org>, kernel@openvz.org,
        Linux Virtualization <virtualization@lists.linux-foundation.org>,
        LKML <linux-kernel@vger.kernel.org>, linux-mm@kvack.org
References: <20221005090158.2801592-1-alexander.atanasov@virtuozzo.com>
 <20221005090158.2801592-3-alexander.atanasov@virtuozzo.com>
 <88EDC41D-408F-4ADF-A933-0A6F36E5F262@gmail.com>
 <a8ce5c48-3efc-5ea3-6f5c-53b9e33f65c7@virtuozzo.com>
 <42C75E59-696B-41D5-BD77-68EFF0B075C6@gmail.com>
From:   Alexander Atanasov <alexander.atanasov@virtuozzo.com>
In-Reply-To: <42C75E59-696B-41D5-BD77-68EFF0B075C6@gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?Zk00dm8weUNEQW1zclJxNDZoK1FuS2dKSDFldjlla09kSnZBUEIvZG9LMjBj?=
 =?utf-8?B?N3NtaW5aT0N4NG84dTZFUzh2aDhiWUo4azBsallmVjRLMzdYQ2I1RTFXVUVw?=
 =?utf-8?B?VFI1QTJ5MmJjSnRvejN1Y3M4WktuTG5kWTVGZWFDUTEzZ3RjRE5xbUZOQ081?=
 =?utf-8?B?Qis0VldKQkl2U2ordDBFUDVSaHpkazBmZ1RpbXNBWGw0c0QycUhIWkpMc0FQ?=
 =?utf-8?B?MjBkZ3pOa3BCM3VLdWwrbEhxS1Fnb3pBU2txK1B3MWwyT3NCQUVpVUoxZWVU?=
 =?utf-8?B?SDJ6NHNjQlB5Y1QvNFN6R29TeTYrRzQwZ3VFSHIwRHJ0NEpOMzE4NTNzeUFB?=
 =?utf-8?B?VXJvajVyM1JsSkEwM1BWNko4TXhPYnQwc1MxOExLTmpObS9CNjJCMk5MOGt0?=
 =?utf-8?B?eVFva0hGd1BBc3dKVS9hY3RoUm5MZm5pMTlNL3FrNGR2WEtNTTR2OTNGdUhW?=
 =?utf-8?B?Sy93aDRRT2dNOGZqRUs3dllCK0cwbTIvM3BBTE9ibkwyUnJId1hMS3grb1BM?=
 =?utf-8?B?dXhDZlF4Z09xRXhKeis4N2lkYXNLZGtKdkIzTElWV2JuR0lnN2ZvT1RxbkJq?=
 =?utf-8?B?OUMwdHg5MjhmUzVVemdKTWQya1lTNCtQVlV2WHRnbjBYQmV3L2crOFozaGpJ?=
 =?utf-8?B?TU1oZG5aMzl2b0o1OFllR3BmWlRKcnJlUXh6cEFTOFBoelNzTW9TVlBjRVk4?=
 =?utf-8?B?TDVxVXlrS0RtOE1JQ2oyelpqcjZZa0ovSFNkMEdvanhZYkFMR1VFSlptTTdY?=
 =?utf-8?B?T2pDZ29IZWs4THB5Ulc3UUxYaVVMSGRaYkhKdWZxeDNQUEJjdlJRaVRkQkpB?=
 =?utf-8?B?bEd6S2loSXRIVkk0U3c5S2VMKzBBbis1WnFrb08zcVRuM2NaU2xzWFkwZnQv?=
 =?utf-8?B?OU5DTTlDaUJ4K3RmZENoQkR0OGVzM3NTdVJBR25lUXRZQWwyOXpSRHkxckNj?=
 =?utf-8?B?RnJmbWFYOW5DTy9NVnVFRGM4bThuVWlFclBOWS9hVmNBaXA1Q21XL1pnbS94?=
 =?utf-8?B?c01ITUpMQXM3RnZoeUFNTENKamk1amR0eEJKdm5ZM2cwbmNmY1IrbXA0VUFz?=
 =?utf-8?B?eldQNm0yRHBBNFVxSWNsWGcvRU9hZU91Z3NJV1Y0cWFIdHNBQllSOGF6VmxZ?=
 =?utf-8?B?azNEc2twU1JROW4yOFFpdEtkcjhRZUIvMkV5ckwxN28rSnVqbFdPWUxyZHhC?=
 =?utf-8?B?U1JJcWQrU1RVRklGMUkzb2tNU29nbEtwMFZLRU1uc29tQ0VoOXRnZDNCK3lG?=
 =?utf-8?B?MCtpMGhIYkVYQ1pYdkVJUldvNFFDd1F2OTFOZ1B1Q3A5UHlZNUdQN2hzNVFF?=
 =?utf-8?B?YnZuemZBZnd1QlpsenFJTjhaNzNlQjlGTjZKQnBSdUdRTzV4K3IvYU5SNG14?=
 =?utf-8?B?bWVGTWE5cXg3NWlzdlU1cHlTcHRaZjM3WEcxeThTbDllQndxWmgvOTJJajhH?=
 =?utf-8?B?dS9QcE9KSkc2SE1UL3VyaWwvVWZ4SHlpVXRsdFdzYWFadDRNaGw0a1gxQ0Nx?=
 =?utf-8?B?THB4UEFUeFRVMGU3Q29EdGF0SzBIbEpRSUh5aWpUWVZSd3BnNFZUcE9pUzlj?=
 =?utf-8?B?SWdMdzQ0TGtMc2hwbHhHZVptVVd6Wk9TMFVjQ1YxaGlSb2ZWTUZ4Rm1uTXo3?=
 =?utf-8?B?dEI1c3diYnUzZHNaODFiWmJUZzloamlMQnduU0hONy9lRm81TGNQWGU5YkZJ?=
 =?utf-8?B?MS9lNjRvUjdNSU53aEs5OTJHSlFzYTA5OWdJbDFid1NqbFgyNlBCbk5vdUFm?=
 =?utf-8?B?bjNWZGlqV2x0Zmp3eHcxMTl5YmYrcWkvM0UwVXA0VU1FTzdRODR4MGF6VUxE?=
 =?utf-8?B?K2t2dnF4VEFaKzdZZWVvZG1BYktPRmthYlg1MFlrOFRBWVF5c1BaRkVOQ014?=
 =?utf-8?B?Z3gzVWhxQVBBbEJOVGE5Y2FVbWRRLzBESWpWOGIyWjBORlVxZHhLVVJ3Skla?=
 =?utf-8?B?dHBIR25VUytPK1BCMERrUkUzSHp3SldSOWJLdTZDSm91OUVHQzFjeWRNYWxO?=
 =?utf-8?B?L3pVb0NWV2xwL2NyeXU5cmIrNmlJSTJPcjh5eVN6dG9Ya2ovZ0JVcEpBYWJT?=
 =?utf-8?B?ZVRpNDhicDRJWHI5REVjSUIrYzNyMFlibjdiNVI1K2c1Z0EwNUs5UEt4WkNB?=
 =?utf-8?B?UUhKT0VYZ0VleW1SSFlVdVBhNjRTdUJvZGxXRnpUREN1TWJPM3RFdmlKSDZK?=
 =?utf-8?B?cVE9PQ==?=
X-OriginatorOrg: virtuozzo.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 82f9c133-2b9e-4bfb-ec89-08daa852eaac
X-MS-Exchange-CrossTenant-AuthSource: VE1PR08MB4765.eurprd08.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Oct 2022 10:58:50.6051
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: jry+9978ypsBV8p0oeR4bpRcZ3fZBTphVMjRI+OMvPOvSVVsAW2xdMW9w8PIt5QIUC7tAC2poUDp5spf/zAd3YdvzBJfkyzqV+vpCs9WtAr5wr1uWOJMXG+zxW7k4oDO
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB9PR08MB7676
X-Spam-Status: No, score=-3.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,NICE_REPLY_A,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 7.10.22 0:07, Nadav Amit wrote:
> On Oct 6, 2022, at 12:34 AM, Alexander Atanasov <alexander.atanasov@virtuozzo.com> wrote:
> 
>> Hello,
>>
>>
>> On 5.10.22 20:25, Nadav Amit wrote:
>>> On Oct 5, 2022, at 2:01 AM, Alexander Atanasov <alexander.atanasov@virtuozzo.com> wrote:
>>>> Add counters to be updated by the balloon drivers.
>>>> Create balloon notifier to propagate changes.
>>> I missed the other patches before (including this one). Sorry, but next
>>> time, please cc me.
>>
>> You are CCed in the cover letter since the version. I will add CC to you
>> in the individual patches if you want so.
> 
> Thanks.
> 
> Just to clarify - I am not attacking you. It’s more of me making an excuse
> for not addressing some issues in earlier versions.

ok, i am glad that you did it now but i do not think you need to excuse.

>>> I was looking through the series and I did not see actual users of the
>>> notifier. Usually, it is not great to build an API without users.
>>
>>
>> You are right. I hope to get some feedback/interest from potential users that i mentioned in the cover letter. I will probably split the notifier
>> in separate series. To make it usefull it will require more changes.
>> See bellow more about them.
> 
> Fair, but this is something that is more suitable for RFC. Otherwise, more
> likely than not - your patches would go in as is.

Yes, i will remove the notifier and resend both as RFC. I think that 
every patch is an RFC and RFC tag is used for more general changes that 
could affect unexpected areas, change functionality, change design and 
in general can lead to bigger impact. In the case with this it adds 
functionality that is missing and it could hardly affect anything else.
In essence it provides information that you can not get without it.
But i will take your advice and push everything thru RFC from now on.

>>> [snip]
>>>> +
>>>> +static int balloon_notify(unsigned long val)
>>>> +{
>>>> +	return srcu_notifier_call_chain(&balloon_chain, val, NULL);
>>> Since you know the inflated_kb value here, why not to use it as an argument
>>> to the callback? I think casting to (void *) and back is best. But you can
>>> also provide pointer to the value. Doesn’t it sound better than having
>>> potentially different notifiers reading different values?
>>
>> My current idea is to have a struct with current and previous value,
>> may be change in percents. The actual value does not matter to anyone
>> but the size of change does. When a user gets notified it can act upon
>> the change - if it is small it can ignore it , if it is above some threshold it can act - if it makes sense for some receiver  is can accumulate changes from several notification. Other option/addition is to have si_meminfo_current(..) and totalram_pages_current(..) that return values adjusted with the balloon values.
>>
>> Going further - there are few places that calculate something based on available memory that do not have sysfs/proc interface for setting limits. Most of them work in percents so they can be converted to do calculations when they get notification.
>>
>> The one that have interface for configuration but use memory values can be handled in two ways - convert to use percents of what is available or extend the notifier to notify userspace which in turn to do calculations and update configuration.
> 
> I really need to see code to fully understand what you have in mind.

Sure - you can check some of the users with git grep totalram_pages - 
shows self explanatory results of usage like:
fs/f2fs/node.c:bool f2fs_available_free_memory(struct f2fs_sb_info *sbi, 
int type) - calculations in percents - one good example
fs/ceph/super.h:        congestion_kb = (16*int_sqrt(totalram_pages())) 
<< (PAGE_SHIFT-10);
fs/fuse/inode.c:                *limit = ((totalram_pages() << 
PAGE_SHIFT) >> 13) / 392;
fs/nfs/write.c: nfs_congestion_kb = (16*int_sqrt(totalram_pages())) << 
(PAGE_SHIFT-10);
fs/nfsd/nfscache.c:     unsigned long low_pages = totalram_pages() - 
totalhigh_pages()
mm/oom_kill.c:  oc->totalpages = totalram_pages() + total_swap_pages;


So all balloon drivers give large amount of RAM on boot , then inflate 
the balloon. But this places have already been initiallized and they 
know that the system have given amount of totalram which is not true the 
moment they start to operate. the result is that too much space gets 
used and it degrades the userspace performance.
example - fs/eventpoll.c:static int __init eventpoll_init(void) - 4% of 
ram for eventpool - when you inflate half of the ram it becomes 8% of 
the ram - do you really need 8% of your ram to be used for eventpool?

To solve this you need to register and when notified update - cache 
size, limits and for whatever is the calculated amount of memory used.

The difference is here:

mm/zswap.c:     return totalram_pages() * zswap_max_pool_percent / 100 <
mm/zswap.c:     return totalram_pages() * zswap_accept_thr_percent / 100
uses percents and you can recalculate easy with

+static inline unsigned long totalram_pages_current(void)
+{
+       unsigned long inflated = 0;
+#ifdef CONFIG_MEMORY_BALLOON
+       extern atomic_long_t mem_balloon_inflated_free_kb;
+       inflated = atomic_long_read(&mem_balloon_inflated_free_kb);
+       inflated >>= (PAGE_SHIFT - 10);
+#endif
+       return (unsigned long)atomic_long_read(&_totalram_pages) - inflated;
+}

And you are good when you switch to _current version - 
si_meminfo_current is alike .

On init (probably) all use some kind of fractions to calculate but when 
there is a set value via /proc/sys/net/ipv4/tcp_wmem for example it is 
just a value and you can not recalculate it. And here, please, share 
your ideas how to solve this.


> Division, as you know, is not something that we really want to do very
> frequently.

Yes, thats true but in the actual implementation there are a lot of ways 
to avoid it. But it is easier to explain with division.

Even if you do have to do a division you can limit the recalculations
struct balloon_notify {
    unsigned long last_inflated_free_kb;
    unsigned long last_inflated_free_kb;
    unsigned long inflated_free_kb;
    unsigned long inflated_used_kb;
}

So you can do it only if change is more then 1GB and you do nothing if 
change is 1MB.

> 
>>> Anyhow, without users (actual notifiers) it’s kind of hard to know how
>>> reasonable it all is. For instance, is it balloon_notify() supposed to
>>> prevent further balloon inflating/deflating until the notifier completes?
>>
>> No, we must avoid that at any cost.
>>
>>> Accordingly, are callers to balloon_notify() expected to relinquish locks
>>> before calling balloon_notify() to prevent deadlocks and high latency?
>>
>> My goal is to avoid any possible impact on performance. Drivers are free to delay notifications if they get in the way. (I see that i need to move the notification after the semaphore in the vmw driver - i missed that - will fix in the next iterration.)
>> Deadlocks - depends on the users but a few to none will possibly have to deal with common locks.
> 
> I will need to see the next version to give better feedback. One more thing
> that comes to mind though is whether saving the balloon size in multiple
> places (both mem_balloon_inflated_total_kb and each balloon’s accounting) is
> the right way. It does not sounds very clean.
> 
> Two other options is to move *all* the accounting to your new
> mem_balloon_inflated_total_kb-like interface or expose some per-balloon
> function to get the balloon size (indirect-function-call would likely have
> some overhead though).
> 
> Anyhow, I am not crazy about having the same data replicated. Even from
> reading the code stand-of-view it is not intuitive.

If such interface existed before the drivers it would ideally be like 
that all in one place. But keeping the internal (for the driver) 
representation which may differ from the system and the external 
(system) representation separate is a a good option. if a driver can 
convert and use only the system counters they can do so.

-- 
Regards,
Alexander Atanasov