Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <20230406040515.383238-1-jhubbard@nvidia.com> <CAMj1kXHxyntweiq76CdW=ov2_CkEQUbdPekGNDtFp7rBCJJE2w@mail.gmail.com>
 <a421b96a-ed4b-ae7d-2fe9-ed5f5f8deacf@nvidia.com> <CAMj1kXGtFyugzi9MZW=4_oVTy==eAF6283fwvX9fdZhO98ZA3g@mail.gmail.com>
 <8dd0e252-8d8b-a62d-8836-f9f26bc12bc7@nvidia.com> <90505ef2-9250-d791-e05d-dbcb7672e4c4@nvidia.com>
In-Reply-To: <90505ef2-9250-d791-e05d-dbcb7672e4c4@nvidia.com>
From:   Ard Biesheuvel <ardb@kernel.org>
Date:   Fri, 12 May 2023 16:42:11 +0200
Message-ID: <CAMj1kXFZ=4hLL1w6iCV5O5uVoVLHAJbc0rr40j24ObenAjXe9w@mail.gmail.com>
Subject: Re: [PATCH] arm64/mm: don't WARN when alloc/free-ing device private pages
To:     John Hubbard <jhubbard@nvidia.com>
Cc:     Andrew Morton <akpm@linux-foundation.org>,
        Catalin Marinas <catalin.marinas@arm.com>,
        Will Deacon <will@kernel.org>,
        Anshuman Khandual <anshuman.khandual@arm.com>,
        Mark Rutland <mark.rutland@arm.com>,
        Kefeng Wang <wangkefeng.wang@huawei.com>,
        Feiyang Chen <chenfeiyang@loongson.cn>,
        Alistair Popple <apopple@nvidia.com>,
        Ralph Campbell <rcampbell@nvidia.com>,
        linux-arm-kernel@lists.infradead.org,
        LKML <linux-kernel@vger.kernel.org>, linux-mm@kvack.org,
        stable@vger.kernel.org
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

On Tue, 11 Apr 2023 at 04:48, John Hubbard <jhubbard@nvidia.com> wrote:
>
> On 4/10/23 00:39, John Hubbard wrote:
> >> pfn_to_page(x) for values 0xc00_0000 < x < 0x1000_0000 will produce a
> >> kernel VA that points outside the region set aside for the vmemmap.
> >> This region is currently unused, but that will likely change soon.
> >>
> >
> > I tentatively think I'm in this case right now. Because there is no wrap
> > around happening in my particular config, which is CONFIG_ARM64_VA_BITS
> > == 48, and PAGE_SIZE == 4KB and sizeof(struct page) == 64 (details
> > below).
> >
>
> Correction, actually it *is* wrapping around, and ending up as a bogus
> user space address, as you said it would when being above the range:
>
> page_to_pfn(0xffffffffaee00000):  0x0000000ffec38000
>

Interesting.

>
> > It occurs to me that ZONE_DEVICE and (within that category) device
> > private page support need only support rather large setups. On x86, it
> > requires 64-bit. And on arm64, from what I'm learning after a day or so
> > of looking around and comparing, I think we must require at least 48 bit
> > VA support. Otherwise there's just no room for things.
>
> I'm still not sure of how to make room, but working on it.
>

The assumption that only the linear map needs to be covered by struct
pages is rather fundamental to the arm64 mm code, as far as I am
aware.

Given that these device memory regions have no direct correspondence
with the linear map at all, couldn't we simply vmalloc() a range of
memory for struct pages for such a region and wire that up in the
existing code? That seems far more maintainable to me than
reorganizing the entire kernel VA space, and only for some choices for
the dimensions.