Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <CAOf5uwk=Tx+W-JuJBXUYjt2BLmSvr9Q153D1RTyJG_cmeO4AUw@mail.gmail.com>
 <826cd775-14d2-12ae-2e96-cf0766aa1502@redhat.com> <CAOf5uwmKfcC0OiiuN82tUzcE1XkPuA3N3u+o3Ue_ZPNJqeSM+g@mail.gmail.com>
 <4475783a-73c1-94f1-804e-507abeb84ab1@redhat.com> <CAOf5uwk8RLRrMa3vM-1+k0oi8XfnWVZH6_uc_UtagWYFVrMYKQ@mail.gmail.com>
 <c2efeb24-0da0-ee25-7cb9-2b9b05523f25@redhat.com> <CAOf5uwkq0aLg8KQB1HFRqPafXpk0_YohW_U_O5=8oQWcui-avQ@mail.gmail.com>
 <04907907-8eab-01ef-8341-e2fecb10b601@redhat.com> <CAOf5uwnd85N-dvBXtZunYgx8Bd58JNDVNx3TezHKFHzpV42WtA@mail.gmail.com>
 <f87c0619-684c-9c32-fbc0-15ed6191d342@redhat.com>
In-Reply-To: <f87c0619-684c-9c32-fbc0-15ed6191d342@redhat.com>
From:   Michael Nazzareno Trimarchi <michael@amarulasolutions.com>
Date:   Mon, 19 Sep 2022 14:11:36 +0200
Message-ID: <CAOf5uw=fw9Zi5UTKO_HGmg66eaCxrLoNyk3psRU6ZrJOn-CkAg@mail.gmail.com>
Subject: Re: Correlation CMA size and FORCE_MAX_ZONEORDER
To:     David Hildenbrand <david@redhat.com>
Cc:     Mike Rapoport <rppt@kernel.org>,
        LKML <linux-kernel@vger.kernel.org>,
        Andrew Morton <akpm@linux-foundation.org>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

Hi

On Mon, Sep 19, 2022 at 2:08 PM David Hildenbrand <david@redhat.com> wrote:
>
> On 19.09.22 13:59, Michael Nazzareno Trimarchi wrote:
> > Hi David
> >
> >
> > On Mon, Sep 19, 2022 at 1:28 PM David Hildenbrand <david@redhat.com> wrote:
> >>
> >> On 19.09.22 13:17, Michael Nazzareno Trimarchi wrote:
> >>> Hi
> >>>
> >>> On Mon, Sep 19, 2022 at 1:03 PM David Hildenbrand <david@redhat.com> wrote:
> >>>>
> >>>> On 19.09.22 11:57, Michael Nazzareno Trimarchi wrote:
> >>>>> Hi
> >>>>>
> >>>>> On Mon, Sep 19, 2022 at 11:31 AM David Hildenbrand <david@redhat.com> wrote:
> >>>>>>
> >>>>>> On 19.09.22 11:17, Michael Nazzareno Trimarchi wrote:
> >>>>>>> Hi David
> >>>>>>>
> >>>>>>> On Mon, Sep 19, 2022 at 10:38 AM David Hildenbrand <david@redhat.com> wrote:
> >>>>>>>>
> >>>>>>>> On 15.09.22 23:36, Michael Nazzareno Trimarchi wrote:
> >>>>>>>>> Hi all
> >>>>>>>>
> >>>>>>>> Hi,
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Working on a small device with 128MB of memory and using imx_v6_v7
> >>>>>>>>> defconfig I found that CMA_SIZE_MBYTES, CMA_SIZE_PERCENTAGE
> >>>>>>>>> are not respected. The calculation done does not allow the requested
> >>>>>>>>> size. I think that this should be somehow documented and described but
> >>>>>>>>> I did not
> >>>>>>>>> find the documentation. Does it work this way?
> >>>>>>>>>
> >>>>>>>>> With CMA_SIZE of 8MB I need to have FORCE_MAX_ZONEORDER=12 if I have
> >>>>>>>>> the default FORCE_MAX_ZONEORDER=14 the min size is 32Mb
> >>>>>>>>
> >>>>>>>> The underlying constraint is that CMA regions require a certain minimum
> >>>>>>>> alignment+size. They cannot be arbitrarily in size.
> >>>>>>>>
> >>>>>>>> CMA_MIN_ALIGNMENT_BYTES expresses that, and corresponds in upstream
> >>>>>>>> kernels to the size of a single pageblock.
> >>>>>>>>
> >>>>>>>> In previous kernels, it used to be the size of the largest buddy
> >>>>>>>> allocation granularity (derived from MAX_ORDER, derived from
> >>>>>>>> FORCE_MAX_ZONEORDER).
> >>>>>>>>
> >>>>>>>> On upstream kernels, the FORCE_MAX_ZONEORDER constraint should no longer
> >>>>>>>> apply. On most archs, the minimum alignment+size should be 2 MiB
> >>>>>>>> (x86-64, aarch64 with 4k base pages) -- the size of a single pageblock.
> >>>>>>>>
> >>>>>>>> So far the theory. Are you still running into this limitation on
> >>>>>>>> upstream kernels?
> >>>>>>>>
> >>>>>>>
> >>>>>>> I can run 6-rc2 on my board. I test again but according to it, if I
> >>>>>>> put 4M as CMA in cma=4M in boot
> >>>>>>> parameters, the result is 32Mb of CMA. Apart of that seems that
> >>>>>>> process lime tiny membench can not even start
> >>>>>>> to mblock memory
> >>>>>>>
> >>>>>>
> >>>>>> The CMA alignemnt change went into v5.19. If "cma=4M" still gives you >
> >>>>>> 4M, can you post /proc/meminfo and the early console output?
> >>>>>>
> >>>>>
> >>>>> cat /proc/cmdline
> >>>>> cma=4M mtdparts=gpmi-nand:4m(nandboot),1m(env),24m(kernel),1m(nanddtb),-(rootfs)
> >>>>> root=ubi0:root rw ubi.mtd=ro
> >>>>> otfs rootfstype=ubifs rootwait=1
> >>>>> # cat /proc/meminfo
> >>>>> MemTotal:         109560 kB
> >>>>> MemFree:           56084 kB
> >>>>> MemAvailable:      56820 kB
> >>>>> Buffers:               0 kB
> >>>>> Cached:            39680 kB
> >>>>> SwapCached:            0 kB
> >>>>> Active:               44 kB
> >>>>> Inactive:            644 kB
> >>>>> Active(anon):         44 kB
> >>>>> Inactive(anon):      644 kB
> >>>>> Active(file):          0 kB
> >>>>> Inactive(file):        0 kB
> >>>>> Unevictable:       39596 kB
> >>>>> Mlocked:               0 kB
> >>>>> HighTotal:             0 kB
> >>>>> HighFree:              0 kB
> >>>>> LowTotal:         109560 kB
> >>>>> LowFree:           56084 kB
> >>>>> SwapTotal:             0 kB
> >>>>> SwapFree:              0 kB
> >>>>> Dirty:                 0 kB
> >>>>> Writeback:             0 kB
> >>>>> AnonPages:           628 kB
> >>>>> Mapped:             1480 kB
> >>>>> Shmem:                84 kB
> >>>>> KReclaimable:       4268 kB
> >>>>> Slab:               8456 kB
> >>>>> SReclaimable:       4268 kB
> >>>>> SUnreclaim:         4188 kB
> >>>>> KernelStack:         392 kB
> >>>>> PageTables:           88 kB
> >>>>> NFS_Unstable:          0 kB
> >>>>> Bounce:                0 kB
> >>>>> WritebackTmp:          0 kB
> >>>>> CommitLimit:       54780 kB
> >>>>> Committed_AS:       1876 kB
> >>>>> VmallocTotal:     901120 kB
> >>>>> VmallocUsed:        2776 kB
> >>>>> VmallocChunk:          0 kB
> >>>>> Percpu:               72 kB
> >>>>> CmaTotal:          32768 kB
> >>>>> CmaFree:           32484 kB
> >>>>> # uname -a
> >>>>> Linux buildroot 6.0.0-rc5 #20 SMP Mon Sep 19 11:51:26 CEST 2022 armv7l GNU/Linux
> >>>>> #
> >>>>>
> >>>>> Then here https://pastebin.com/6MUB2VBM dmesg
> >>>>>
> >>>>> CONFIG_ARM_MODULE_PLTS=y
> >>>>> CONFIG_FORCE_MAX_ZONEORDER=14
> >>>>> CONFIG_ALIGNMENT_TRAP=y
> >>>>> ...
> >>>>> CONFIG_CMA
> >>>>> CONFIG_CMA_AREAS=7
> >>>>> ...
> >>>>>
> >>>>> CONFIG_CMA_SIZE_MBYTES=8
> >>>>> CONFIG_CMA_SIZE_SEL_MBYTES=y
> >>>>>
> >>>>
> >>>> Thanks!
> >>>>
> >>>> I assume that in your setup, the pageblock size depends on MAX_ORDER
> >>>> and, therefore, FORCE_MAX_ZONEORDER.
> >>>>
> >>>> This should be the case especially if CONFIG_HUGETLB_PAGE is not defined
> >>>> (include/linux/pageblock-flags.h).
> >>>>
> >>>> In contrast to what I remember, the pageblock size does not seem to
> >>>> depend on the THP size (weird) as well.
> >>>>
> >>>>
> >>>> So, yes, that limitation is still in effect for some kernel configs.
> >>>>
> >>>> One could make the pageblock size configurable (similar to
> >>>> CONFIG_HUGETLB_PAGE_SIZE_VARIABLE) or simply default to a smaller
> >>>> pageblock size as default with CONFIG_CMA and !CONFIG_HUGETLB_PAGE.
> >>>>
> >>>> I imagine something reasonable might be to set the pageblock size to
> >>>> 2MiB without CONFIG_HUGETLB_PAGE but with CONFIG_CMA.
> >>>>
> >>>
> >>> I don't think making more configuration options makes things clear.
> >>
> >> Yes, in an ideal case it should be automatic.
> >>
> >>> When we enable some configuration
> >>> we can force down the configuration. You need to explain clearly how
> >>> you envision it. FORCE_MAX_ZONEORDER
> >>> for me is the largest allocation that you can get from a zone (ex CMA
> >>> one). Any request allocation that is align to the
> >>
> >> FORCE_MAX_ZONEORDER is just a way to increase/decrease the maximum
> >> allocation size of the buddy in general.
> >>
> >>> CMA align and can fit inside a region should be allowed
> >>>
> >>> What am I missing?
> >>
> >> I think that the issue is that the CMA alignments nowadays depend on the
> >> pageblock size. And the pageblock size depends on *some* configurations
> >> on the maximum allocation size of the buddy.
> >>
> >> Documenting the interaction between FORCE_MAX_ZONEORDER and CMA size
> >> alignment is not trivial.
> >>
> >> For example, with CONFIG_HUGETLB_PAGE there might not be such an
> >> interaction. With CONFIG_HUGETLB_PAGE, there clearly is one.
> >>
> >>
> >> Let me phrase it this way: is it good enough in you setup to get 32mb vs
> >> 8mb or do you want/need to reduce it without adjusting
> >> FORCE_MAX_ZONEORDER ?
> >
> > Wait we have:
> > - CMA kconfig alignment that in most config we have not considered
> > natural dma alignment but is put to 1Mb in a lot of embedded
> > - We have CMA_SIZE, CMA_SIZE_PERCENTAGE etc. Those seems that are not
> > effect if ZONEORDER is not reasonable and without
> > HUGETLB_PAGE
> > - etc
> >
> > Changing MAX_ZONEORDER is ok and yes if you have an IOT device that
> > you know about your CMA allocation, it makes no sense to have
> > it 32MB for a 128Mb device. What I point out is that I need to figure
> > it out because in Kconfig there is no mention of it. Should it be
> > added there?
>
> Something like
>
> "Note that in some configurations, the CMA size must be aligned to the
> maximum allocation granularity of the buddy allocator, consequently
> depending on FORCE_MAX_ZONEORDER. The requested CMA size might get
> increased accordingly."
>
> Maybe we'd want some kind of a warning in the kernel as well. If someone
> specifies "cma=2MB" but gets 32MB or more that might be a problem.
>
> Does something like that make sense to you?

Yes we need to warn the user and add to Kconfig. Maybe wait for other
to get a feedback

Michael

>
> --
> Thanks,
>
> David / dhildenb
>


-- 
Michael Nazzareno Trimarchi
Co-Founder & Chief Executive Officer
M. +39 347 913 2170
michael@amarulasolutions.com
__________________________________

Amarula Solutions BV
Joop Geesinkweg 125, 1114 AB, Amsterdam, NL
T. +31 (0)85 111 9172
info@amarulasolutions.com
www.amarulasolutions.com