Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Thu, 04 Aug 2022 19:07:20 +0900
From:   Li Chen <me@linux.beauty>
To:     "Arnd Bergmann" <arnd@arndb.de>
Cc:     "Catalin Marinas" <catalin.marinas@arm.com>,
        "Will Deacon" <will@kernel.org>,
        "Rob Herring" <robh+dt@kernel.org>,
        "Frank Rowand" <frowand.list@gmail.com>,
        "Andrew Morton" <akpm@linux-foundation.org>,
        "Li Chen" <lchen@ambarella.com>,
        "Linux ARM" <linux-arm-kernel@lists.infradead.org>,
        "Linux Kernel Mailing List" <linux-kernel@vger.kernel.org>,
        "DTML" <devicetree@vger.kernel.org>,
        "Linux-MM" <linux-mm@kvack.org>
Message-ID: <1826852ea3a.10a917cc826728.6077661125986568031@linux.beauty>
In-Reply-To: <CAK8P3a0zSGqj3YEi+i9yfSLk8-aJtyiY6Bj069cxCdErk81+cw@mail.gmail.com>
References: <20220711122459.13773-1-me@linux.beauty> <20220711122459.13773-5-me@linux.beauty>
 <CAK8P3a2Mr0ZMXGDx6htYEbBBtm4mubk-meSASJjPRK1j1O-hEA@mail.gmail.com>
 <181efcca6ae.de84203d522625.7740936811073442334@linux.beauty>
 <CAK8P3a30o1RLifV1TMqDJ26vLhVdOzz3wP6yPrayLV2GPxUtwQ@mail.gmail.com> <18267b7a61f.12b26bd91245310.4476663913461696630@linux.beauty> <CAK8P3a0zSGqj3YEi+i9yfSLk8-aJtyiY6Bj069cxCdErk81+cw@mail.gmail.com>
Subject: Re: [PATCH 4/4] sample/reserved_mem: Introduce a sample of struct
 page and dio support to no-map rmem
MIME-Version: 1.0
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
Importance: Medium
User-Agent: Zoho Mail
Precedence: bulk


 ---- On Thu, 04 Aug 2022 17:24:20 +0900  Arnd Bergmann  wrote --- 
 > On Thu, Aug 4, 2022 at 9:17 AM Li Chen me@linux.beauty> wrote:
 > > ---- On Tue, 12 Jul 2022 16:50:46 +0900  Arnd Bergmann  wrote ---
 > >  > Does your hardware require a fixed address for the buffer? If it can be
 > >  > anywhere in memory (or at least within a certain range) but just has to
 > >  > be physically contiguous, the normal way would be to use a CMA area
 > >  > to allocate from, which gives you 'struct page' backed pages.
 > >
 > > CMA does support Direct I/O, but it has its own issue:
 > > It does not guarantee that the memory previously borrowed by the OS will be returned to the device.
 > >
 > > We've been plagued by examples like this in the past:
 > > Many other kernel modules/subsystems have already allocated much memory from both non-CMA and CMA memory,
 > > When our DSP driver got probed then, cma_alloc will fail in that non-CMA system memory is not enough
 > > for CMA memory to migrate.
 > 
 > This part should at least be possible to solve by declaring the amount
 > and location of
 > CMA areas in the right way. It's not great to fine-tune the DT for a
 > particular kernel's
 > use, but if you know which other drivers require CMA type allocations
 > you can find a lower
 > bound that should suffice.

That's the problem, haha. End users(customers) may modprobe many other modules and modprobe our
driver in the end. We cannot decide the probe order for end users.

Apart from our cases, I heard there are some other cases where cma_alloc failed even non-cma system memory has
enough memory because pages in CMA memory are pinned and cannot move out of CMA. There are some fixes like
1. move these memory out of CMA before pinned
2. only allow non-long-time pinned memory allocation from CMA.

But these two solutions are not merged into the mainline yet.

 > 
 > Most coherent allocations tend to be long-lived and only for very
 > small memory regions.
 > If you have another driver that uses large or periodic
 > dma_alloc_coherent() type allocations,
 > you can consider either giving that device its own CMA area, or fixing
 > it to use streaming
 > mappings.

Device-wise CMA also suffers from the problems I mentioned, other modules/subsystems may have already
alloc from this CMA area and refuse to return it back.

Regards,
Li