Received-SPF: pass (google.com: domain of linux-kernel+bounces-55156-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Message-ID: <888072e30cc003dbed3f41675242b877246e2f0a.camel@gmail.com>
Subject: Re: [RFC 00/18] Pkernfs: Support persistence for live update
From: Luca Boccassi <luca.boccassi@gmail.com>
To: jgowans@amazon.com
Cc: akpm@linux-foundation.org, anthony.yznaga@oracle.com,
 brauner@kernel.org,  dwmw@amazon.co.uk, ebiederm@xmission.com,
 graf@amazon.com, iommu@lists.linux.dev,  joro@8bytes.org,
 jschoenh@amazon.de, kexec@lists.infradead.org,  kvm@vger.kernel.org,
 linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, 
 linux-mm@kvack.org, madvenka@linux.microsoft.com, pbonzini@redhat.com, 
 seanjc@google.com, skinsburskii@linux.microsoft.com,
 steven.sistare@oracle.com,  usama.arif@bytedance.com,
 viro@zeniv.linux.org.uk, will@kernel.org,  yuleixzhang@tencent.com
Date: Tue, 06 Feb 2024 14:55:42 +0000
In-Reply-To: <20240205120203.60312-1-jgowans@amazon.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
User-Agent: Evolution 3.46.4-2 
Precedence: bulk
MIME-Version: 1.0


> Also, the question of a hard separation between
> persistent memory and ephemeral memory, compared to allowing
> arbitrary pages to
> be persisted. Pkernfs does it via a hard separation defined at boot
> time, other
> approaches could make the carving out of persistent pages dynamic.

Speaking from experience here - in Azure (Boost) we have been using
hard-carved out memory areas (DAX devices with ranges configured via
DTB) for persisting state across kexec for ~5 years or so. In a
nutshell: don't, it's a mistake.

It's a constant and consistence source of problems, headaches, issues
and workarounds piled upon workarounds, held together with duct tape
and prayers. It's just not flexible enough for any modern system. For
example, unless _all_ the machines are ridicolously overprovisioned in
terms of memory capacity (and guaranteed to remain so, forever), you
end up wasting enormous amounts of memory.

In Azure we are very much interested in a nice, well-abstracted, first-
class replacement for that setup that allows persisting data across
kexec, and in systemd userspace we'd very much want to use it as well,
but it really, really needs to be dynamic, and avoid the pitfall of
hard-configured carved out chunk.

--=20
Kind regards,
Luca Boccassi