Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
MIME-Version: 1.0
References: <CAA25o9TABY=3C+FQEg8FDyF1rim315G2hmeB1DBWJLn-wG1j0g@mail.gmail.com>
 <CAJZ5v0gJWxLJTi7TjaRP-3aR3f4VnX1n9dRE_jxdS6e3SM46LQ@mail.gmail.com> <CAA25o9TiaaKGH4ZkOa=FhZW7PdXQ592YZ4q52o-QNx=yFsR4Pw@mail.gmail.com>
In-Reply-To: <CAA25o9TiaaKGH4ZkOa=FhZW7PdXQ592YZ4q52o-QNx=yFsR4Pw@mail.gmail.com>
From:   "Rafael J. Wysocki" <rafael@kernel.org>
Date:   Wed, 23 Oct 2019 00:13:54 +0200
Message-ID: <CAJZ5v0j7N=o0S1tSwn5DP10oAmb5oN5SsM3jWhs+ZJ2YxMEvnw@mail.gmail.com>
Subject: Re: is hibernation usable?
To:     Luigi Semenzato <semenzato@google.com>
Cc:     "Rafael J. Wysocki" <rafael@kernel.org>,
        linux-kernel <linux-kernel@vger.kernel.org>,
        Linux PM <linux-pm@vger.kernel.org>,
        Andrew Morton <akpm@linux-foundation.org>,
        Geoff Pike <gpike@google.com>, Bas Nowaira <bassem@google.com>,
        Sonny Rao <sonnyrao@google.com>,
        Brian Geffon <bgeffon@google.com>
Content-Type: text/plain; charset="UTF-8"
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Tue, Oct 22, 2019 at 11:26 PM Luigi Semenzato <semenzato@google.com> wrote:
>
> Thank you for the quick reply!
>
> On Tue, Oct 22, 2019 at 1:57 PM Rafael J. Wysocki <rafael@kernel.org> wrote:
> >
> > On Tue, Oct 22, 2019 at 10:09 PM Luigi Semenzato <semenzato@google.com> wrote:
> > >
> > > Following a thread in linux-pm
> > > (https://marc.info/?l=linux-mm&m=157012300901871) I have some issues
> > > that may be of general interest.
> > >
> > > 1. To the best of my knowledge, Linux hibernation is guaranteed to
> > > fail if more than 1/2 of total RAM is in use (for instance, by
> > > anonymous pages).  My knowledge is based on evidence, experiments,
> > > code inspection, the thread above, and a comment in
> > > Documentation/swsusp.txt, copied here:
> >
> > So I use it on a regular basis (i.e. every day) on a system that often
> > has over 50% or RAM in use and it all works.
> >
> > I also know about other people using it on a regular basis.
> >
> > For all of these users, it is usable.
> >
> > >  "Instead, we load the image into unused memory and then atomically
> > > copy it back to it original location. This implies, of course, a
> > > maximum image size of half the amount of memory."
> >
> > That isn't right any more.  An image that is loaded during resume can,
> > in fact, be larger than 50% of RAM.  An image that is created during
> > hibernation, however, cannot.
>
> Sorry, I don't understand this.  Are you saying that, for instance,
> you can resume a 30 GB image on a 32 GB device, but that image could
> only have been created on a 64 GB device?

Had it been possible to create images larger than 50% of memory during
hibernation, it would have been possible to load them during resume as
well.

The resume code doesn't have a 50% of RAM limitation, the image
creation code does.

> > > 2. There's no simple/general workaround.  Rafael suggested on the
> > > thread "Whatever doesn't fit into 50% of RAM needs to be swapped out
> > > before hibernation".  This is a good suggestion: I am actually close
> > > to achieving this using memcgroups, but it's a fair amount of work,
> > > and a fairly special case.  Not everybody uses memcgroups, and I don't
> > > know of other reliable ways of forcing swap from user level.
> >
> > I don't need to do anything like that.
>
> Again, I don't understand.  Why did you make that suggestion then?
>
> > hibernate_preallocate_memory() manages to free a sufficient amount of
> > memory on my system every time.
>
> Unfortunately this doesn't work for me.  I may have described a simple
> experiment: on a 4GB device, create two large processes like this:
>
> dd if=/dev/zero bs=1100M count=1 | sleep infinity &
> dd if=/dev/zero bs=1100M count=1 | sleep infinity &
>
> so that more than 50% of TotalMem is used for anonymous pages.  Then
> echo disk > /sys/power/state fails with ENOMEM.

I guess hibernate_preallocate_memory() is not able to free enough
memory for itself in that case.

> Is this supposed to work?

Yes, it is, in general.

> Maybe I am doing something wrong?
> Hibernation works before I create the dd processes.  After I force
> some of those pages to a separate swap device, hibernation works too,
> so those pages aren't mlocked or anything.

It looks like you are doing something that is not covered by
hibernate_preallocate_memory().

> > > 3. A feature that works only when 1/2 of total RAM can be allocated
> > > is, in my opinion, not usable, except possibly under special
> > > circumstances, such as mine. Most of the available articles and
> > > documentation do not mention this important fact (but for the excerpt
> > > I mentioned, which is not in a prominent position).
> >
> > It can be used with over 1/2 of RAM allocated and that is quite easy
> > to demonstrate.
> >
> > Honestly, I'm not sure what your problem is really.
>
> I apologize if I am doing something stupid and I should know better
> before I waste other people's time.  I have been trying to explain
> these issues as best as I can.  I have a reproducible failure.  I'll
> be happy to provide any additional detail.

Simply put, hibernation, as implemented today, needs to allocate over
50% of RAM (or at least as much as to be able to copy all of the
non-free pages) for image creation.  If it cannot do that, it will
fail and you know how to prevent it from allocating enough memory in a
reproducible way.  AFAICS that's a situation in which every attempt to
allocate 50% of memory for any other purpose will fail as well.

Frankly, you are first to report this problem, so it arguably is not
common.  It looks like hibernate_preallocate_memory() may be improved
to cover that case, but then the question is how much more complicated
it will have to become for this purpose and whether or not that's
worth pursuing.

> >
> > > Two questions then:
> > >
> > > A. Should the documentation be changed to reflect this fact more
> > > clearly?  I feel that the current situation is a disservice to the
> > > user community.
> >
> > Propose changes.
>
> Sure, after we resolve the above questions.
>
> > > B. Would it be worthwhile to improve the hibernation code to remove
> > > this limitation?  Is this of interest to anybody (other than me)?
> >
> > Again, propose specific changes.