LinuxLists.cc - [PATCH -mm] kexec jump -v9

[permalink] [raw]

Subject: Re: [PATCH -mm] kexec jump -v9

On Thu, Mar 06, 2008 at 11:13:08AM +0800, Huang, Ying wrote:
> This is a minimal patch with only the essential features. All
> additional features are split out and can be discussed later. I think
> it may be easier to get consensus on this minimal patch.
>

Hi Huang,

This patchset is slowly getting better. True that first we need to come
up with minimal infrastructure patch and then think of building more
functionality on top of it.

> This patch provides an enhancement to kexec/kdump. It implements
> the following features:
>
> - Jumping between the original kernel and the kexeced kernel.
>
> - Backup/restore memory used by both the original kernel and the
> kexeced kernel.
>
> - Save/restore CPU and devices state before after kexec.
>
>
> The features of this patch can be used for as follow:
>
> - A simple hibernation implementation without ACPI support. You can
> kexec a hibernating kernel, save the memory image of original system
> and shutdown the system. When resuming, you restore the memory image
> of original system via ordinary kexec load then jump back.
>

The main usage of this functionality is for hibernation. I am not sure
what has been the conclusion of previous discussions.

Rafael/Pavel, does the approach of doing hibernation using a separate
kernel holds promise?

[..]
> Usage example of simple hibernation:
>
> 1. Compile and install patched kernel with following options selected:
>
> CONFIG_X86_32=y
> CONFIG_RELOCATABLE=y
> CONFIG_KEXEC=y
> CONFIG_CRASH_DUMP=y
> CONFIG_PM=y
>
> 2. Build an initramfs image contains kexec-tool and makedumpfile, or
> download the pre-built initramfs image, called rootfs.gz in
> following text.
>
> 3. Prepare a partition to save memory image of original kernel, called
> hibernating partition in following text.
>
> 3. Boot kernel compiled in step 1 (kernel A).
>
> 4. In the kernel A, load kernel compiled in step 1 (kernel B) with
> /sbin/kexec. The shell command line can be as follow:
>
> /sbin/kexec --load-preserve-context /boot/bzImage --mem-min=0x100000
> --mem-max=0xffffff --initrd=rootfs.gz
>
> 5. Boot the kernel B with following shell command line:
>
> /sbin/kexec -e
>
> 6. The kernel B will boot as normal kexec. In kernel B the memory
> image of kernel A can be saved into hibernating partition as
> follow:
>
> jump_back_entry=`cat /proc/cmdline | tr ' ' '\n' | grep kexec_jump_back_entry | cut -d '='`
> echo $jump_back_entry > kexec_jump_back_entry
> cp /proc/vmcore dump.elf
>

Why not store the entry point in dump.elf itself, instead of storing it
in a separate file?

I think this is more like a resumable core file. Something similar to
functionality what qemu does for resuming an already booted kernel image.
So we might have to introduce an ELF_NOTE to mark an image as resumable
core.

> Then you can shutdown the machine as normal.
>
> 7. Boot kernel compiled in step 1 (kernel C). Use the rootfs.gz as
> root file system.
>
> 8. In kernel C, load the memory image of kernel A as follow:
>
> /sbin/kexec -l --args-none --entry=`cat kexec_jump_back_entry` dump.elf
>

How the memory segments of dump.elf loaded? Normal kexec way? Memory
segments of dump.elf are first stored somewhere and then moved to
destination at "kexec -e" time?

Does this really work? If we have 4G RAM, what will be the size of
dump.elf? And when we load it back for resuming, do we have sufficient
memory left?

May be we can have a separate load flag (--load-resume-image) to mark
that we are resuming an hibernated image and kexec does not have to
prepare commandline, does not have to prepare zero page/setup page etc.

I have thought through it again and try to put together some of the
new kexec options we can introduce to make the whole thing work. I am
considering a simple case where a user boots the kernel A and then
launches kernel B using "kexec --load-preseve-context". Now a user
might save the hibernated image or might want to come back to A.

- kexec -l <kernel-image>
Normal kexec functionality. Boot a new kernel, without preserving
existing kernel's context.

- kexec --load-preserve-context <kernel-image>
Boot a new kernel while preserving existing kernel's context.

Will be used for booting kernel B for the first time.

- kexec --load-resume-image <resumable-core>
Resumes an hibernated image. Load a ELF64 hibernated image.

Context of first kernel/boot-loader will not be preserved.

First kernel will not save cpu states. Will put devices into
suspended state though so that these can be resumed by resumable
core

This option can be used by kboot or kernel C to resume an hibernated
image.

- kexec --load-resume-entry <entry-point>
Image is already loaded. Just prepare the entry point so that one
can enter back to previous image. cpu states will be saved and devices
will be put to suspended states.

will be used for A --> B and B ---> A transitions. Both A and B are
booted. This is just for switching back and forth between A and B.

- kexec -e
Transition into the new kernel

This patch looks in pretty decent shape. Once there is some sort of
understanding that this approach is promising for hibernation and we
have consensus on high level interface, then we can get into line by
line review of the patch set.

Thanks
Vivek

2008-03-11 21:59:29

by Nigel Cunningham

[permalink] [raw]

Subject: Re: [PATCH -mm] kexec jump -v9

Hi all.

I hope kexec turns out to be a good, usable solution. Unfortunately,
however, I still have some areas where I'm not convinced that kexec is
going to work or work well:

1. Reliability.

It's being sold as a replacement for freezing processes, yet AFAICS it's
still going to require the freezer in order to be reliable. In the
normal case, there isn't much of an issue with freeing memory or
allocating swap, and so these steps can be expected to progress without
pain. Imagine, however, the situation where another process or processes
are trying to allocate large amounts of memory at the same time, or the
system is swapping heavily. Although such situations will not be common,
they are entirely conceivable, and any implementation ought to be able
to handle such a situation efficiently. If the freezer is removed, any
hibernation implementation - not just kexec - is going to have a much
harder job of being reliable in all circumstances. AFAICS, the only way
a kexec based solution is going to be able to get around this will be to
not have to allocate memory, but that will require permanent allocation
of memory for the kexec kernel and it's work area as well as the
permanent, exclusive allocation of storage for the kexec hibernation
implementation that's currently in place (making the LCA complaint about
not being able to hibernate to swap on NTFS on fuse equally relevant).

While this might be feasible on machines with larger amounts of memory
(you might validly be able to argue that a user won't miss 10MB of RAM),
it does make hibernation less viable or unviable for systems with less
memory (embedded!). It also means that there are 10MB of RAM (or
whatever amount) that the user has paid good money for, but which are
probably only used for 30s at a time a couple of times a day.

Any attempt to start to use storage available to the hibernating kernel
is also going to have these race issues.

2. Lack of ACPI support.

At the moment, noone is going to want to use kexec based hibernation if
they have an ACPI system. This needs to be addressed before it can be
considered a serious contender.

3. Usability.

Right now, kexec based hibernation looks quite complicated to configure,
and the user is apparently going to have to remember to boot a different
kernel or at least a different bootloader entry in order to resume. Not
a plus. It would be good if you could find a way to use one bootloader
entry, resuming if there's an image, booting normally if there's not.

Nigel

2008-03-11 22:19:37

[permalink] [raw]

Subject: Re: [PATCH -mm] kexec jump -v9

On Tuesday, 11 of March 2008, Vivek Goyal wrote:
> On Thu, Mar 06, 2008 at 11:13:08AM +0800, Huang, Ying wrote:
> > This is a minimal patch with only the essential features. All
> > additional features are split out and can be discussed later. I think
> > it may be easier to get consensus on this minimal patch.
> >
>
> Hi Huang,
>
> This patchset is slowly getting better. True that first we need to come
> up with minimal infrastructure patch and then think of building more
> functionality on top of it.
>
>
> > This patch provides an enhancement to kexec/kdump. It implements
> > the following features:
> >
> > - Jumping between the original kernel and the kexeced kernel.
> >
> > - Backup/restore memory used by both the original kernel and the
> > kexeced kernel.
> >
> > - Save/restore CPU and devices state before after kexec.
> >
> >
> > The features of this patch can be used for as follow:
> >
> > - A simple hibernation implementation without ACPI support. You can
> > kexec a hibernating kernel, save the memory image of original system
> > and shutdown the system. When resuming, you restore the memory image
> > of original system via ordinary kexec load then jump back.
> >
>
> The main usage of this functionality is for hibernation. I am not sure
> what has been the conclusion of previous discussions.
>
> Rafael/Pavel, does the approach of doing hibernation using a separate
> kernel holds promise?

Well, what can I say?

I haven't been a big fan of doing hibernation this way since the very beginning
and I still have the same reservations. Namely, my opinion is that the
hibernation-related problems we have are not just solvable this way. For one
example, in order to stop using the freezer for suspend/hibernation we first
need to revamp the suspending/resuming of devices (uder way) and the
kexec-based approach doesn't help us here. I wouldn't like to start another
discussion about it though.

That said, I can imagine some applications of the $subject functionality
not directly related to hibernation. For example, one can use it for kernel
debgging (jump to a new kernel, change something in the old kernel's
data, jump back and see what happens etc.). Also, in principle it may be used
for such things as live migration of VMs.

Thanks,
Rafael

2008-03-11 23:23:51

by Pavel Machek

[permalink] [raw]

Subject: Re: [PATCH -mm] kexec jump -v9

Hi!

> > This is a minimal patch with only the essential features. All
> > additional features are split out and can be discussed later. I think
> > it may be easier to get consensus on this minimal patch.
> >
>
> Hi Huang,
>
> This patchset is slowly getting better. True that first we need to come
> up with minimal infrastructure patch and then think of building more
> functionality on top of it.
>
...
> > The features of this patch can be used for as follow:
> >
> > - A simple hibernation implementation without ACPI support. You can
> > kexec a hibernating kernel, save the memory image of original system
> > and shutdown the system. When resuming, you restore the memory image
> > of original system via ordinary kexec load then jump back.
> >
>
> The main usage of this functionality is for hibernation. I am not sure
> what has been the conclusion of previous discussions.
>
> Rafael/Pavel, does the approach of doing hibernation using a separate
> kernel holds promise?

Its certainly "more traditional" method of doing hibernation than
tricks swsusp currently plays. Yes, I'd like these patches to go in,
being able to switch kernels seems like useful tool.

Now, I guess they are some difficulties, like ACPI integration, and
some basic drawbacks, like few seconds needed to boot second kernel
during suspend.

...OTOH this is probably only chance to eliminate freezer from
swsusp...

Yes, I'd like to see this to go ahead.

No, this does not make swsusp obsolete just yet.
Pavel

--
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

2008-03-11 23:50:09

[permalink] [raw]

Subject: Re: [PATCH -mm] kexec jump -v9

On Wednesday, 12 of March 2008, Pavel Machek wrote:
> Hi!

Hi,

> > > This is a minimal patch with only the essential features. All
> > > additional features are split out and can be discussed later. I think
> > > it may be easier to get consensus on this minimal patch.
> > >
> >
> > Hi Huang,
> >
> > This patchset is slowly getting better. True that first we need to come
> > up with minimal infrastructure patch and then think of building more
> > functionality on top of it.
> >
> ...
> > > The features of this patch can be used for as follow:
> > >
> > > - A simple hibernation implementation without ACPI support. You can
> > > kexec a hibernating kernel, save the memory image of original system
> > > and shutdown the system. When resuming, you restore the memory image
> > > of original system via ordinary kexec load then jump back.
> > >
> >
> > The main usage of this functionality is for hibernation. I am not sure
> > what has been the conclusion of previous discussions.
> >
> > Rafael/Pavel, does the approach of doing hibernation using a separate
> > kernel holds promise?
>
> Its certainly "more traditional" method of doing hibernation than
> tricks swsusp currently plays.

What exactly are you referring to?

> Yes, I'd like these patches to go in, being able to switch kernels seems like
> useful tool.

No objection from me.

> Now, I guess they are some difficulties, like ACPI integration, and
> some basic drawbacks, like few seconds needed to boot second kernel
> during suspend.
>
> ...OTOH this is probably only chance to eliminate freezer from
> swsusp...

Some facts:

* In order to be able to do suspend (STR) without the freezer, we need to make
device drivers block access to devices from applications during suspend.
* There's no reason to think that we can't use this same mechanism for
hibernation (the only difficulty seems to be the handling of devices used for
saving the image).
* We need the drivers to quiesce devices to be able to do the kexec jump in the
first place (and to avoid races, we'll need them to block applications'
access to devices just like for STR, which is the sufficient condition for
removing the freezer).

So, I don't really think that the "freezer removal" argument is valid here.

Moreover, if this had been the _only_ argument for the $subject functionality,
I'd have been against it.

Thanks,
Rafael

2008-03-12 00:01:11

[permalink] [raw]

Subject: Re: [PATCH -mm] kexec jump -v9

Nigel Cunningham <[email protected]> writes:

> Hi all.
>
> I hope kexec turns out to be a good, usable solution. Unfortunately,
> however, I still have some areas where I'm not convinced that kexec is
> going to work or work well:
>
> 1. Reliability.
>
> It's being sold as a replacement for freezing processes, yet AFAICS it's
> still going to require the freezer in order to be reliable. In the
> normal case, there isn't much of an issue with freeing memory or
> allocating swap, and so these steps can be expected to progress without
> pain. Imagine, however, the situation where another process or processes
> are trying to allocate large amounts of memory at the same time, or the
> system is swapping heavily. Although such situations will not be common,
> they are entirely conceivable, and any implementation ought to be able
> to handle such a situation efficiently. If the freezer is removed, any
> hibernation implementation - not just kexec - is going to have a much
> harder job of being reliable in all circumstances. AFAICS, the only way
> a kexec based solution is going to be able to get around this will be to
> not have to allocate memory, but that will require permanent allocation
> of memory for the kexec kernel and it's work area as well as the
> permanent, exclusive allocation of storage for the kexec hibernation
> implementation that's currently in place (making the LCA complaint about
> not being able to hibernate to swap on NTFS on fuse equally relevant).
>
> While this might be feasible on machines with larger amounts of memory
> (you might validly be able to argue that a user won't miss 10MB of RAM),
> it does make hibernation less viable or unviable for systems with less
> memory (embedded!). It also means that there are 10MB of RAM (or
> whatever amount) that the user has paid good money for, but which are
> probably only used for 30s at a time a couple of times a day.

Right. I can address the memory concerns with a kexec based approach
as they are core to kexec and completely orthogonal to the rest.

A kexec in done in two passes. The first to load the target kernel
and do whatever memory allocation is needed. The second to actually
switch which kernel is running.

Using a linux kernel to save off the image or in any other way be the
target is not required it is simply the sane thing to do in a general
implementation. An embedded developer could likely implement a save
to disk routing in a couple of hundred lines of C and a couple of K
RAM if it was an important feature.

> Any attempt to start to use storage available to the hibernating kernel
> is also going to have these race issues.

Yep. Although disk storage is frequently less expensive, and more
readily available, so this is less of an issue. Still it does suggest
that a dedicated partition likely will be required.

> 2. Lack of ACPI support.
>
> At the moment, noone is going to want to use kexec based hibernation if
> they have an ACPI system. This needs to be addressed before it can be
> considered a serious contender.

Yes.

> 3. Usability.
>
> Right now, kexec based hibernation looks quite complicated to configure,
> and the user is apparently going to have to remember to boot a different
> kernel or at least a different bootloader entry in order to resume. Not
> a plus. It would be good if you could find a way to use one bootloader
> entry, resuming if there's an image, booting normally if there's
> not.

I completely agree here.

Eric

2008-03-12 00:09:37

by Nigel Cunningham

[permalink] [raw]

Subject: Re: [PATCH -mm] kexec jump -v9

Hi.

On Wed, 2008-03-12 at 00:24 +0100, Pavel Machek wrote:
> ...OTOH this is probably only chance to eliminate freezer from
> swsusp...

I think eliminating the freezer and having reliable hibernation under
load look like incompatible goals at the moment. Do you see that as 'not
a problem' or have some idea on how that issue can be addressed?

Regards,

Nigel

2008-03-12 00:09:49

by David Lang

[permalink] [raw]

Subject: Re: [PATCH -mm] kexec jump -v9

On Wed, 12 Mar 2008, Nigel Cunningham wrote:

> 3. Usability.
>
> Right now, kexec based hibernation looks quite complicated to configure,
> and the user is apparently going to have to remember to boot a different
> kernel or at least a different bootloader entry in order to resume. Not
> a plus. It would be good if you could find a way to use one bootloader
> entry, resuming if there's an image, booting normally if there's not.

I don't see any reason why this couldn't be done with an initrd to decide
if you are doing a normal boot or a restore.

David Lang

2008-03-12 01:44:36

[permalink] [raw]

Subject: Re: [PATCH -mm] kexec jump -v9

On Tue, 2008-03-11 at 17:10 -0400, Vivek Goyal wrote:
> On Thu, Mar 06, 2008 at 11:13:08AM +0800, Huang, Ying wrote:
> > This is a minimal patch with only the essential features. All
> > additional features are split out and can be discussed later. I think
> > it may be easier to get consensus on this minimal patch.
> >
>
> Hi Huang,
>
> This patchset is slowly getting better. True that first we need to come
> up with minimal infrastructure patch and then think of building more
> functionality on top of it.
>
>
> > This patch provides an enhancement to kexec/kdump. It implements
> > the following features:
> >
> > - Jumping between the original kernel and the kexeced kernel.
> >
> > - Backup/restore memory used by both the original kernel and the
> > kexeced kernel.
> >
> > - Save/restore CPU and devices state before after kexec.
> >
> >
> > The features of this patch can be used for as follow:
> >
> > - A simple hibernation implementation without ACPI support. You can
> > kexec a hibernating kernel, save the memory image of original system
> > and shutdown the system. When resuming, you restore the memory image
> > of original system via ordinary kexec load then jump back.
> >
>
> The main usage of this functionality is for hibernation. I am not sure
> what has been the conclusion of previous discussions.
>
> Rafael/Pavel, does the approach of doing hibernation using a separate
> kernel holds promise?
>
> [..]
> > Usage example of simple hibernation:
> >
> > 1. Compile and install patched kernel with following options selected:
> >
> > CONFIG_X86_32=y
> > CONFIG_RELOCATABLE=y
> > CONFIG_KEXEC=y
> > CONFIG_CRASH_DUMP=y
> > CONFIG_PM=y
> >
> > 2. Build an initramfs image contains kexec-tool and makedumpfile, or
> > download the pre-built initramfs image, called rootfs.gz in
> > following text.
> >
> > 3. Prepare a partition to save memory image of original kernel, called
> > hibernating partition in following text.
> >
> > 3. Boot kernel compiled in step 1 (kernel A).
> >
> > 4. In the kernel A, load kernel compiled in step 1 (kernel B) with
> > /sbin/kexec. The shell command line can be as follow:
> >
> > /sbin/kexec --load-preserve-context /boot/bzImage --mem-min=0x100000
> > --mem-max=0xffffff --initrd=rootfs.gz
> >
> > 5. Boot the kernel B with following shell command line:
> >
> > /sbin/kexec -e
> >
> > 6. The kernel B will boot as normal kexec. In kernel B the memory
> > image of kernel A can be saved into hibernating partition as
> > follow:
> >
> > jump_back_entry=`cat /proc/cmdline | tr ' ' '\n' | grep kexec_jump_back_entry | cut -d '='`
> > echo $jump_back_entry > kexec_jump_back_entry
> > cp /proc/vmcore dump.elf
> >
>
> Why not store the entry point in dump.elf itself, instead of storing it
> in a separate file?
>
> I think this is more like a resumable core file. Something similar to
> functionality what qemu does for resuming an already booted kernel image.
> So we might have to introduce an ELF_NOTE to mark an image as resumable
> core.

Yes. The entry point should be saved in dump.elf itself, this can be
done via a user-space tool such as "makedumpfile". Because
"makedumpfile" is also used to exclude free pages from disk image, it
needs a communication method between two kernels (to get backup pages
map or something like that from kernel A). We have talked about this
before.

- Your opinion is to communicate via the purgatory. (But I don't know
how to communicate between kernel A and purgatory).
- Eric's opinion is to communicate between the user space in kernel A
and user space in kernel B.
- My opinion is to communicate between two kernel directly.

I think as a minimal infrastructure patch, we can communicate minimal
information between user space of two kernels. When we have consensus on
this topic, we can use makedumpfile for both excluding free pages and
saving the entry point. Now, we can save the entry point in a separate
file or I can write a simple tool to do this.

> > Then you can shutdown the machine as normal.
> >
> > 7. Boot kernel compiled in step 1 (kernel C). Use the rootfs.gz as
> > root file system.
> >
> > 8. In kernel C, load the memory image of kernel A as follow:
> >
> > /sbin/kexec -l --args-none --entry=`cat kexec_jump_back_entry` dump.elf
> >
>
> How the memory segments of dump.elf loaded? Normal kexec way? Memory
> segments of dump.elf are first stored somewhere and then moved to
> destination at "kexec -e" time?

Yes. Exactly. But during kexec loading, if the source page is same as
destination page, we need just one page.

> Does this really work? If we have 4G RAM, what will be the size of
> dump.elf? And when we load it back for resuming, do we have sufficient
> memory left?

Yes. It really works. If we have 4G RAM, the size of dump.elf is 4G -
(memory area used by second kernel), in this example, it is 4G - 16M.
The loading kernel will live in 16M memory, and load dump.elf into all
other memory area.

> May be we can have a separate load flag (--load-resume-image) to mark
> that we are resuming an hibernated image and kexec does not have to
> prepare commandline, does not have to prepare zero page/setup page etc.

There is already similar flag in original kexec-tools implementation:
"--args-none". If it is specified, kexec-tools does not prepare command
line and zero page/setup page etc. I think we can just re-use this flag.
And If it is desired an alias is good for me too.

> I have thought through it again and try to put together some of the
> new kexec options we can introduce to make the whole thing work. I am
> considering a simple case where a user boots the kernel A and then
> launches kernel B using "kexec --load-preseve-context". Now a user
> might save the hibernated image or might want to come back to A.
>
> - kexec -l <kernel-image>
> Normal kexec functionality. Boot a new kernel, without preserving
> existing kernel's context.
>
> - kexec --load-preserve-context <kernel-image>
> Boot a new kernel while preserving existing kernel's context.
>
> Will be used for booting kernel B for the first time.
>
> - kexec --load-resume-image <resumable-core>

In original kexec-tools, this can be done through:
kexec -l --args-none <resumable-core>

Do you need to define an alias for it?

> Resumes an hibernated image. Load a ELF64 hibernated image.
>
> Context of first kernel/boot-loader will not be preserved.
>
> First kernel will not save cpu states. Will put devices into
> suspended state though so that these can be resumed by resumable
> core
>
> This option can be used by kboot or kernel C to resume an hibernated
> image.
>
> - kexec --load-resume-entry <entry-point>

In current implementation, this can be done through:
kexec --load-jump-back-helper --entry <entry-point>.

I think the new name is good.

> Image is already loaded. Just prepare the entry point so that one
> can enter back to previous image. cpu states will be saved and devices
> will be put to suspended states.
>
> will be used for A --> B and B ---> A transitions. Both A and B are
> booted. This is just for switching back and forth between A and B.
>
> - kexec -e
> Transition into the new kernel
>
> This patch looks in pretty decent shape. Once there is some sort of
> understanding that this approach is promising for hibernation and we
> have consensus on high level interface, then we can get into line by
> line review of the patch set.
>

Best Regards,
Huang Ying

2008-03-12 01:55:19

[permalink] [raw]

Subject: Re: [PATCH -mm] kexec jump -v9

On Wed, 2008-03-12 at 00:49 +0100, Rafael J. Wysocki wrote:
> On Wednesday, 12 of March 2008, Pavel Machek wrote:
[...]
> >
> > Its certainly "more traditional" method of doing hibernation than
> > tricks swsusp currently plays.
>
> What exactly are you referring to?

Long long ago, the hibernation is not done by Linux kernel itself but
BIOS (APM). Those days, kernel just does some preparation and jump to
BIOS to do the hibernation. Imagine kernel B is the hibernation BIOS,
kernel A does some prepare and jump to the BIOS (kernel B) just like the
old days.

> > Yes, I'd like these patches to go in, being able to switch kernels seems like
> > useful tool.
>
> No objection from me.
>
> > Now, I guess they are some difficulties, like ACPI integration, and
> > some basic drawbacks, like few seconds needed to boot second kernel
> > during suspend.
> >
> > ...OTOH this is probably only chance to eliminate freezer from
> > swsusp...
>
> Some facts:
>
> * In order to be able to do suspend (STR) without the freezer, we need to make
> device drivers block access to devices from applications during suspend.
> * There's no reason to think that we can't use this same mechanism for
> hibernation (the only difficulty seems to be the handling of devices used for
> saving the image).

I think "kexec based hibernation" is the only currently available
possible method to write out image without freezer (after driver works
are done). If other process is running, how to prevent them from writing
to disk without freezing them in current implementation?

> * We need the drivers to quiesce devices to be able to do the kexec jump in the
> first place (and to avoid races, we'll need them to block applications'
> access to devices just like for STR, which is the sufficient condition for
> removing the freezer).
>
> So, I don't really think that the "freezer removal" argument is valid here.
>
> Moreover, if this had been the _only_ argument for the $subject functionality,
> I'd have been against it.

Best Regards,
Huang Ying

2008-03-12 02:08:10

[permalink] [raw]

Subject: Re: [PATCH -mm] kexec jump -v9

"Rafael J. Wysocki" <[email protected]> writes:

> Well, what can I say?
>
> I haven't been a big fan of doing hibernation this way since the very beginning
> and I still have the same reservations. Namely, my opinion is that the
> hibernation-related problems we have are not just solvable this way. For one
> example, in order to stop using the freezer for suspend/hibernation we first
> need to revamp the suspending/resuming of devices (uder way) and the
> kexec-based approach doesn't help us here. I wouldn't like to start another
> discussion about it though.

Agreed. At best all this does is moving the policy on how to save the kernel
image from the kernel itself out to user space, and it not a cure all.

> That said, I can imagine some applications of the $subject functionality
> not directly related to hibernation. For example, one can use it for kernel
> debgging (jump to a new kernel, change something in the old kernel's
> data, jump back and see what happens etc.). Also, in principle it may be used
> for such things as live migration of VMs.

Also such things as calling BIOS services or EFI services on x86_64. Where
vm86 is not useful.

So in principle I think a kexec with return is a logical extension to
the current kexec functionality.

That said it looks like next month before I will have time to do a reasonable
job of reviewing the current patches.

Eric

2008-03-12 02:13:31

[permalink] [raw]

Subject: Re: [PATCH -mm] kexec jump -v9

On Wed, 2008-03-12 at 08:59 +1100, Nigel Cunningham wrote:
> Hi all.
>
> I hope kexec turns out to be a good, usable solution. Unfortunately,
> however, I still have some areas where I'm not convinced that kexec is
> going to work or work well:
>
> 1. Reliability.
>
> It's being sold as a replacement for freezing processes, yet AFAICS it's
> still going to require the freezer in order to be reliable. In the
> normal case, there isn't much of an issue with freeing memory or
> allocating swap, and so these steps can be expected to progress without
> pain. Imagine, however, the situation where another process or processes
> are trying to allocate large amounts of memory at the same time, or the
> system is swapping heavily. Although such situations will not be common,
> they are entirely conceivable, and any implementation ought to be able
> to handle such a situation efficiently. If the freezer is removed, any
> hibernation implementation - not just kexec - is going to have a much
> harder job of being reliable in all circumstances. AFAICS, the only way
> a kexec based solution is going to be able to get around this will be to
> not have to allocate memory, but that will require permanent allocation
> of memory for the kexec kernel and it's work area as well as the
> permanent, exclusive allocation of storage for the kexec hibernation
> implementation that's currently in place (making the LCA complaint about
> not being able to hibernate to swap on NTFS on fuse equally relevant).

As Eric said kexec need only to allocate memory during loading, not
executing.

> While this might be feasible on machines with larger amounts of memory
> (you might validly be able to argue that a user won't miss 10MB of RAM),
> it does make hibernation less viable or unviable for systems with less
> memory (embedded!). It also means that there are 10MB of RAM (or
> whatever amount) that the user has paid good money for, but which are
> probably only used for 30s at a time a couple of times a day.
>
> Any attempt to start to use storage available to the hibernating kernel
> is also going to have these race issues.

I think this can be avoid such as preallocate some hard disk space (such
as a dedicate hibernating file, the block list are loaded by
kexec-tools).

> 2. Lack of ACPI support.
>
> At the moment, noone is going to want to use kexec based hibernation if
> they have an ACPI system. This needs to be addressed before it can be
> considered a serious contender.

ACPI is the biggest challenge for kexec based hibernation. I will try to
deal with it. But for most people, ACPI is not a big issue. This is
hibernation, not suspend to RAM.

> 3. Usability.
>
> Right now, kexec based hibernation looks quite complicated to configure,
> and the user is apparently going to have to remember to boot a different
> kernel or at least a different bootloader entry in order to resume. Not

No, the newest implementation need not to boot a different kernel or
different bootloader entry. You just use one bootloader entry, it will
resume if there's an image, booting normally if there's not. You can
look at the newest hibernation example description.

And the new method can even be used to load hibernation image of uswsusp
too.

> a plus. It would be good if you could find a way to use one bootloader
> entry, resuming if there's an image, booting normally if there's not.

Best Regards,
Huang Ying

2008-03-12 02:22:51

[permalink] [raw]

Subject: Re: [PATCH -mm] kexec jump -v9

"Huang, Ying" <[email protected]> writes:

> Yes. The entry point should be saved in dump.elf itself, this can be
> done via a user-space tool such as "makedumpfile". Because
> "makedumpfile" is also used to exclude free pages from disk image, it
> needs a communication method between two kernels (to get backup pages
> map or something like that from kernel A). We have talked about this
> before.
>
> - Your opinion is to communicate via the purgatory. (But I don't know
> how to communicate between kernel A and purgatory).

How about the return address on the stack?

> - Eric's opinion is to communicate between the user space in kernel A
> and user space in kernel B.

Purgatory is for all intents and purposes user space. Because the
return address falls on the trampoline page we won't know it's
address before we call kexec. But a return address and a stack
on that page should be a perfectly good way to communicate.

> - My opinion is to communicate between two kernel directly.
>
> I think as a minimal infrastructure patch, we can communicate minimal
> information between user space of two kernels. When we have consensus on
> this topic, we can use makedumpfile for both excluding free pages and
> saving the entry point. Now, we can save the entry point in a separate
> file or I can write a simple tool to do this.

We need a fixed protocol so we do not make assumptions about how things
will be implemented, allowing kernels to diverge and kinds of other
good things.

For communicating extra information from the kernel being shut down
we have elf notes.

Direct kernel to kernel communication is forbidden. We must have
a well defined protocol. Allowing the implementations to change
at their different speeds, and still work together.

>> May be we can have a separate load flag (--load-resume-image) to mark
>> that we are resuming an hibernated image and kexec does not have to
>> prepare commandline, does not have to prepare zero page/setup page etc.
>
> There is already similar flag in original kexec-tools implementation:
> "--args-none". If it is specified, kexec-tools does not prepare command
> line and zero page/setup page etc. I think we can just re-use this flag.
> And If it is desired an alias is good for me too.

My gut feel is we look at the image and detect what kind it is, and simply
not enable image processing after we have read the note that says it
is a resumable core or whatever.

>> I have thought through it again and try to put together some of the
>> new kexec options we can introduce to make the whole thing work. I am
>> considering a simple case where a user boots the kernel A and then
>> launches kernel B using "kexec --load-preseve-context". Now a user
>> might save the hibernated image or might want to come back to A.
>>
>> - kexec -l <kernel-image>
>> Normal kexec functionality. Boot a new kernel, without preserving
>> existing kernel's context.
>>
>> - kexec --load-preserve-context <kernel-image>
>> Boot a new kernel while preserving existing kernel's context.
>>
>> Will be used for booting kernel B for the first time.
>>
>> - kexec --load-resume-image <resumable-core>
>
> In original kexec-tools, this can be done through:
> kexec -l --args-none <resumable-core>
>
> Do you need to define an alias for it?

Make common cases fast to use. The UI equivalent of make the
common case fast.

Eric

2008-03-12 02:27:24

[permalink] [raw]

Subject: Re: [PATCH -mm] kexec jump -v9

On Tue, 2008-03-11 at 23:18 +0100, Rafael J. Wysocki wrote:
> On Tuesday, 11 of March 2008, Vivek Goyal wrote:
[...]
> > Rafael/Pavel, does the approach of doing hibernation using a separate
> > kernel holds promise?
>
> Well, what can I say?
>
> I haven't been a big fan of doing hibernation this way since the very beginning
> and I still have the same reservations. Namely, my opinion is that the
> hibernation-related problems we have are not just solvable this way. For one
> example, in order to stop using the freezer for suspend/hibernation we first
> need to revamp the suspending/resuming of devices (uder way) and the
> kexec-based approach doesn't help us here. I wouldn't like to start another
> discussion about it though.

Yes. We need to work on device drivers for all hibernation
implementations. And kexec-based hibernation provides a possible method
to avoid freezer after driver works done.

Best Regards,
Huang Ying

2008-03-12 06:53:17

[permalink] [raw]

Subject: Re: [PATCH -mm] kexec jump -v9

On Tue, 2008-03-11 at 20:17 -0600, Eric W. Biederman wrote:
> "Huang, Ying" <[email protected]> writes:
>
> > Yes. The entry point should be saved in dump.elf itself, this can be
> > done via a user-space tool such as "makedumpfile". Because
> > "makedumpfile" is also used to exclude free pages from disk image, it
> > needs a communication method between two kernels (to get backup pages
> > map or something like that from kernel A). We have talked about this
> > before.
> >
> > - Your opinion is to communicate via the purgatory. (But I don't know
> > how to communicate between kernel A and purgatory).
>
> How about the return address on the stack?
>
> > - Eric's opinion is to communicate between the user space in kernel A
> > and user space in kernel B.
>
> Purgatory is for all intents and purposes user space. Because the
> return address falls on the trampoline page we won't know it's
> address before we call kexec. But a return address and a stack
> on that page should be a perfectly good way to communicate.
>
> > - My opinion is to communicate between two kernel directly.
> >
> > I think as a minimal infrastructure patch, we can communicate minimal
> > information between user space of two kernels. When we have consensus on
> > this topic, we can use makedumpfile for both excluding free pages and
> > saving the entry point. Now, we can save the entry point in a separate
> > file or I can write a simple tool to do this.
>
> We need a fixed protocol so we do not make assumptions about how things
> will be implemented, allowing kernels to diverge and kinds of other
> good things.
>
> For communicating extra information from the kernel being shut down
> we have elf notes.
>
> Direct kernel to kernel communication is forbidden. We must have
> a well defined protocol. Allowing the implementations to change
> at their different speeds, and still work together.

This sounds reasonable. But after some initial trying I found it is
fairly difficult for me to define a communication protocol to be back
compatible with original kexec/kdump, doing work in user space as far as
possible, dealing with some special scenario (such as: A kexec B, then B
kexec C). So I will try my best to work on this, and propose a
communication protocol combining the proposals from you and Vivek in
several days.

> >> May be we can have a separate load flag (--load-resume-image) to mark
> >> that we are resuming an hibernated image and kexec does not have to
> >> prepare commandline, does not have to prepare zero page/setup page etc.
> >
> > There is already similar flag in original kexec-tools implementation:
> > "--args-none". If it is specified, kexec-tools does not prepare command
> > line and zero page/setup page etc. I think we can just re-use this flag.
> > And If it is desired an alias is good for me too.
>
> My gut feel is we look at the image and detect what kind it is, and simply
> not enable image processing after we have read the note that says it
> is a resumable core or whatever.

Yes. This sounds good.

Best Regards,
Huang Ying

2008-03-12 08:57:35

by Pavel Machek

[permalink] [raw]

Subject: Re: [PATCH -mm] kexec jump -v9

Hi!

> > > > The features of this patch can be used for as follow:
> > > >
> > > > - A simple hibernation implementation without ACPI support. You can
> > > > kexec a hibernating kernel, save the memory image of original system
> > > > and shutdown the system. When resuming, you restore the memory image
> > > > of original system via ordinary kexec load then jump back.
> > > >
> > >
> > > The main usage of this functionality is for hibernation. I am not sure
> > > what has been the conclusion of previous discussions.
> > >
> > > Rafael/Pavel, does the approach of doing hibernation using a separate
> > > kernel holds promise?
> >
> > Its certainly "more traditional" method of doing hibernation than
> > tricks swsusp currently plays.
>
> What exactly are you referring to?

Well, traditionaly it is 'A saves B to disk' (like bootloader saves
kernel&userspace). In swsusp we have 'kernel saves itself'... which
works, too, but is pretty different design.

> > Now, I guess they are some difficulties, like ACPI integration, and
> > some basic drawbacks, like few seconds needed to boot second kernel
> > during suspend.
> >
> > ...OTOH this is probably only chance to eliminate freezer from
> > swsusp...
>
> Some facts:
>
> * There's no reason to think that we can't use this same mechanism for
> hibernation (the only difficulty seems to be the handling of devices used for
> saving the image).

Ok, at least kexec makes handling of suspend device easier.

> Moreover, if this had been the _only_ argument for the $subject functionality,
> I'd have been against it.

Fortunately its not the only one :-).

--
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

2008-03-12 15:02:14

by Alan Stern

[permalink] [raw]

Subject: Re: [linux-pm] [PATCH -mm] kexec jump -v9

On Wed, 12 Mar 2008, Huang, Ying wrote:

> I think "kexec based hibernation" is the only currently available
> possible method to write out image without freezer (after driver works
> are done). If other process is running, how to prevent them from writing
> to disk without freezing them in current implementation?

This is a very good question.

It's a matter of managing the block layer's request queues. Somehow
the existing I/O requests must remain blocked while the requests needed
for writing the image must be allowed to proceed.

I don't know what would be needed to make this work, but it ought to be
possible somehow...

Alan Stern

2008-03-12 18:55:45

[permalink] [raw]

Subject: Re: [PATCH -mm] kexec jump -v9

On Wed, Mar 12, 2008 at 10:14:34AM +0800, Huang, Ying wrote:
> On Wed, 2008-03-12 at 08:59 +1100, Nigel Cunningham wrote:
> > Hi all.
> >
> > I hope kexec turns out to be a good, usable solution. Unfortunately,
> > however, I still have some areas where I'm not convinced that kexec is
> > going to work or work well:
> >
> > 1. Reliability.
> >
> > It's being sold as a replacement for freezing processes, yet AFAICS it's
> > still going to require the freezer in order to be reliable. In the
> > normal case, there isn't much of an issue with freeing memory or
> > allocating swap, and so these steps can be expected to progress without
> > pain. Imagine, however, the situation where another process or processes
> > are trying to allocate large amounts of memory at the same time, or the
> > system is swapping heavily. Although such situations will not be common,
> > they are entirely conceivable, and any implementation ought to be able
> > to handle such a situation efficiently. If the freezer is removed, any
> > hibernation implementation - not just kexec - is going to have a much
> > harder job of being reliable in all circumstances. AFAICS, the only way
> > a kexec based solution is going to be able to get around this will be to
> > not have to allocate memory, but that will require permanent allocation
> > of memory for the kexec kernel and it's work area as well as the
> > permanent, exclusive allocation of storage for the kexec hibernation
> > implementation that's currently in place (making the LCA complaint about
> > not being able to hibernate to swap on NTFS on fuse equally relevant).
>
> As Eric said kexec need only to allocate memory during loading, not
> executing.

Yes. But this memory gets reserved at loading time and then this memory
remains unused for the whole duration (except hibernation).

In the example you gave, looks like you are reserving 15MB of memory for
second kernel. In practice, we we finding it difficult to boot a regular
kernel in 16MB of memory in kdump. We are now reserving 128MB of memory
for kdump kernel on x86 arch, otheriwse OOM kill kicks in during init
or while core is being copied.

Kexec based hibernation does not look any different than kdump in terms
of memory requirements. The only difference seems to be that kdump does
the contiguous memory reservation at boot time and kexec based hibernation
does the memory reservation at kernel loading time.

The only difference I can think of is, kdump will generally run on servers
and hibernation will be required on desktops/laptops and run time memory
requirements might be little different. I don't have numbers though.

At the same time carrying a separate kernel binary just for hibernation
purposes does not sound very good.

[..]
> > 3. Usability.
> >
> > Right now, kexec based hibernation looks quite complicated to configure,
> > and the user is apparently going to have to remember to boot a different
> > kernel or at least a different bootloader entry in order to resume. Not
>
> No, the newest implementation need not to boot a different kernel or
> different bootloader entry. You just use one bootloader entry, it will
> resume if there's an image, booting normally if there's not. You can
> look at the newest hibernation example description.
>

Following is the step from new method you have given.

7. Boot kernel compiled in step 1 (kernel C). Use the rootfs.gz as
root file system.

This mentions that use rootfs.gz as initrd. Without modifying the boot
loader entry, how would I switch the initrd dynamically.

Looks like it might be a typo. So basically we can just boot back into
normal kernel and then a user can load the resumable core file and kexec
to it?

I think all this functionality can be packed into normal initrd itself
to make user interface better.

A user can configure the destination for hibernated image at system
installation time and initrd will be modified accordingly to save the
hibernated image as well to check that user specfied location to find out
if a hibernation image is available and needs to be resumed.

Thanks
Vivek

2008-03-12 19:38:38

[permalink] [raw]

Subject: Re: [PATCH -mm] kexec jump -v9

On Tue, Mar 11, 2008 at 08:17:45PM -0600, Eric W. Biederman wrote:
> "Huang, Ying" <[email protected]> writes:
>
> > Yes. The entry point should be saved in dump.elf itself, this can be
> > done via a user-space tool such as "makedumpfile". Because
> > "makedumpfile" is also used to exclude free pages from disk image, it
> > needs a communication method between two kernels (to get backup pages
> > map or something like that from kernel A). We have talked about this
> > before.
> >
> > - Your opinion is to communicate via the purgatory. (But I don't know
> > how to communicate between kernel A and purgatory).
>
> How about the return address on the stack?
>

I think he needs to pass on much more data than just return address.

IIUC, he needs to pass backup pages map to new kernel, so that any
user space tool can use backup pages map to reconstruct/rearrange the
first kernel's memory core and tools like makedumpfile can do filtering
before hibernated images is saved.

This brings me to a random thought. Can we break the process of loading
a hibernation kernel in two steps.

- In first step just do the memory reservation for running second kernel.
(kexec -l <dummpy-file-for-reserving-memory>)

- This memory map of reserved pages is exported to user space.

- Use this memory map and regenerate the hibernation kernel initrd
(rootfs.gz) and put the memory map there. This memory map can be used
by makedumpfile in second kernel for filtering.

This way it will user space to user space communication of information
which gets fixed at kernel loading time.

> > - Eric's opinion is to communicate between the user space in kernel A
> > and user space in kernel B.
>
> Purgatory is for all intents and purposes user space. Because the
> return address falls on the trampoline page we won't know it's
> address before we call kexec. But a return address and a stack
> on that page should be a perfectly good way to communicate.
>
> > - My opinion is to communicate between two kernel directly.
> >
> > I think as a minimal infrastructure patch, we can communicate minimal
> > information between user space of two kernels. When we have consensus on
> > this topic, we can use makedumpfile for both excluding free pages and
> > saving the entry point. Now, we can save the entry point in a separate
> > file or I can write a simple tool to do this.
>
> We need a fixed protocol so we do not make assumptions about how things
> will be implemented, allowing kernels to diverge and kinds of other
> good things.
>
> For communicating extra information from the kernel being shut down
> we have elf notes.
>
> Direct kernel to kernel communication is forbidden. We must have
> a well defined protocol. Allowing the implementations to change
> at their different speeds, and still work together.
>

Agreed. Without a proper protocol, we will often run into issues that
X version of kernel does not work with Y version of hibernation kernel
etc.

> >> May be we can have a separate load flag (--load-resume-image) to mark
> >> that we are resuming an hibernated image and kexec does not have to
> >> prepare commandline, does not have to prepare zero page/setup page etc.
> >
> > There is already similar flag in original kexec-tools implementation:
> > "--args-none". If it is specified, kexec-tools does not prepare command
> > line and zero page/setup page etc. I think we can just re-use this flag.
> > And If it is desired an alias is good for me too.
>
> My gut feel is we look at the image and detect what kind it is, and simply
> not enable image processing after we have read the note that says it
> is a resumable core or whatever.
>

That makes sense. Just that we shall have to put some kind of ELF NOTE
or some other identifier in resumable core file to identify it.

Thanks
Vivek

2008-03-12 19:48:17

[permalink] [raw]

Subject: Re: [PATCH -mm] kexec jump -v9

On Wed, Mar 12, 2008 at 09:45:26AM +0800, Huang, Ying wrote:

[..]
> > I have thought through it again and try to put together some of the
> > new kexec options we can introduce to make the whole thing work. I am
> > considering a simple case where a user boots the kernel A and then
> > launches kernel B using "kexec --load-preseve-context". Now a user
> > might save the hibernated image or might want to come back to A.
> >
> > - kexec -l <kernel-image>
> > Normal kexec functionality. Boot a new kernel, without preserving
> > existing kernel's context.
> >
> > - kexec --load-preserve-context <kernel-image>
> > Boot a new kernel while preserving existing kernel's context.
> >
> > Will be used for booting kernel B for the first time.
> >
> > - kexec --load-resume-image <resumable-core>
>
> In original kexec-tools, this can be done through:
> kexec -l --args-none <resumable-core>
>
> Do you need to define an alias for it?

Ok, we can get rid of --load-resume-image and go by the Eric's idea
of detecting image type and taking action accordingly.

Thanks
Vivek

2008-03-12 21:54:17

[permalink] [raw]

Subject: Re: [linux-pm] [PATCH -mm] kexec jump -v9

On Wednesday, 12 of March 2008, Alan Stern wrote:
> On Wed, 12 Mar 2008, Huang, Ying wrote:
>
> > I think "kexec based hibernation" is the only currently available
> > possible method to write out image without freezer (after driver works
> > are done). If other process is running, how to prevent them from writing
> > to disk without freezing them in current implementation?
>
> This is a very good question.
>
> It's a matter of managing the block layer's request queues. Somehow
> the existing I/O requests must remain blocked while the requests needed
> for writing the image must be allowed to proceed.
>
> I don't know what would be needed to make this work, but it ought to be
> possible somehow...

Yes, it ought to be possible.

Ultimately, IMHO, we should put all devices unnecessary for saving the image
(and doing some eye-candy work) into low power states before the image is
created and keep them in low power states until the system is eventually
powered off.

If this is done, the remaining problem is the handling of the devices that we
need to save the image. I believe that will be achievable without using the
freezer.

Thanks,
Rafael

2008-03-13 00:06:51

[permalink] [raw]

Subject: Re: [PATCH -mm] kexec jump -v9

Vivek Goyal <[email protected]> writes:

> Yes. But this memory gets reserved at loading time and then this memory
> remains unused for the whole duration (except hibernation).
>
> In the example you gave, looks like you are reserving 15MB of memory for
> second kernel. In practice, we we finding it difficult to boot a regular
> kernel in 16MB of memory in kdump. We are now reserving 128MB of memory
> for kdump kernel on x86 arch, otheriwse OOM kill kicks in during init
> or while core is being copied.

Sounds like something we may want to fix. Living at the default kernel
address may alieviate that problem somewhat.

> Kexec based hibernation does not look any different than kdump in terms
> of memory requirements. The only difference seems to be that kdump does
> the contiguous memory reservation at boot time and kexec based hibernation
> does the memory reservation at kernel loading time.
>
> The only difference I can think of is, kdump will generally run on servers
> and hibernation will be required on desktops/laptops and run time memory
> requirements might be little different. I don't have numbers though.
>
> At the same time carrying a separate kernel binary just for hibernation
> purposes does not sound very good.

One difference is you only get the memory penalty just before you hibernate,
instead of continuously. So potentially you could swap out things to
make run for the kernel to save you to disk.

> [..]
>> > 3. Usability.
>> >
>> > Right now, kexec based hibernation looks quite complicated to configure,
>> > and the user is apparently going to have to remember to boot a different
>> > kernel or at least a different bootloader entry in order to resume. Not
>>
>> No, the newest implementation need not to boot a different kernel or
>> different bootloader entry. You just use one bootloader entry, it will
>> resume if there's an image, booting normally if there's not. You can
>> look at the newest hibernation example description.
>>
>
> Following is the step from new method you have given.
>
> 7. Boot kernel compiled in step 1 (kernel C). Use the rootfs.gz as
> root file system.
>
> This mentions that use rootfs.gz as initrd. Without modifying the boot
> loader entry, how would I switch the initrd dynamically.
>
> Looks like it might be a typo. So basically we can just boot back into
> normal kernel and then a user can load the resumable core file and kexec
> to it?
>
> I think all this functionality can be packed into normal initrd itself
> to make user interface better.
>
> A user can configure the destination for hibernated image at system
> installation time and initrd will be modified accordingly to save the
> hibernated image as well to check that user specfied location to find out
> if a hibernation image is available and needs to be resumed.

Yes. And we don't need to load any of this until just before hibernation
time so we should be able to change things right up until the last moment.

Eric

2008-03-13 00:38:53

[permalink] [raw]

Subject: Re: [linux-pm] [PATCH -mm] kexec jump -v9

"Rafael J. Wysocki" <[email protected]> writes:

> Yes, it ought to be possible.
>
> Ultimately, IMHO, we should put all devices unnecessary for saving the image
> (and doing some eye-candy work) into low power states before the image is
> created and keep them in low power states until the system is eventually
> powered off.

Why? I guess I don't see why we care what power state the devices are in.
Especially since we should be able to quickly save the image.

We need to disconnect the drivers from the hardware yes. So filesystems
still work and applications that do direct hardware access still work
and don't need to reopen their connections.

I'm leery of low power states as they don't always work, and bringing
low power states seems to confuse hibernation to disk with suspend to
ram.

> If this is done, the remaining problem is the handling of the devices that we
> need to save the image. I believe that will be achievable without using the
> freezer.

Reasonable. In general the problem is much easier if we don't store
the hibernation image in a filesystem or partition that the rest of
the system is using. That way we avoid inconsistencies.

Eric

2008-03-13 17:04:17

[permalink] [raw]

Subject: Re: [linux-pm] [PATCH -mm] kexec jump -v9

On Thursday, 13 of March 2008, Eric W. Biederman wrote:
> "Rafael J. Wysocki" <[email protected]> writes:
>
> > Yes, it ought to be possible.
> >
> > Ultimately, IMHO, we should put all devices unnecessary for saving the image
> > (and doing some eye-candy work) into low power states before the image is
> > created and keep them in low power states until the system is eventually
> > powered off.
>
> Why? I guess I don't see why we care what power state the devices are in.
> Especially since we should be able to quickly save the image.
>
> We need to disconnect the drivers from the hardware yes. So filesystems
> still work and applications that do direct hardware access still work
> and don't need to reopen their connections.
>
> I'm leery of low power states as they don't always work, and bringing
> low power states seems to confuse hibernation to disk with suspend to
> ram.

>From the ACPI compliance point of view it's better to do it this way. We need
to put the devices into low power states anyway before "powering off" the
system and we won't need to touch them for the second time if we do that
in advance.

Still, it would be sufficient if we disconnected the drivers from the hardware
and thus prevented applications from accessing that hardware.

Thanks,
Rafael

2008-03-13 23:12:37

[permalink] [raw]

Subject: Re: [linux-pm] [PATCH -mm] kexec jump -v9

"Rafael J. Wysocki" <[email protected]> writes:

> From the ACPI compliance point of view it's better to do it this way. We need
> to put the devices into low power states anyway before "powering off" the
> system and we won't need to touch them for the second time if we do that
> in advance.

Interesting. From a kexec jump where we exit the kernel and then return
to it seem all that is required.

I will have to look at the ACPI case. That seems to be the lynch pin
of a couple of arguments for doing strange things: ACPI requires it.

> Still, it would be sufficient if we disconnected the drivers from the hardware
> and thus prevented applications from accessing that hardware.

My gut feeling is that except for a handful of drivers we could even
get away with simply implementing hot unplug and hot replug. Disks
are the big exception here.

Which suggests to me that it is at least possible that the methods we
want for a kexec jump hibernation may be different from an in-kernel
hibernation and quite possibly are easier to implement.

Eric

2008-03-14 01:32:38

[permalink] [raw]

Subject: Re: [linux-pm] [PATCH -mm] kexec jump -v9

On Friday, 14 of March 2008, Eric W. Biederman wrote:
> "Rafael J. Wysocki" <[email protected]> writes:
>
> > From the ACPI compliance point of view it's better to do it this way. We need
> > to put the devices into low power states anyway before "powering off" the
> > system and we won't need to touch them for the second time if we do that
> > in advance.
>
> Interesting. From a kexec jump where we exit the kernel and then return
> to it seem all that is required.
>
> I will have to look at the ACPI case. That seems to be the lynch pin
> of a couple of arguments for doing strange things: ACPI requires it.

Yes and that's because ACPI regards hibernation as a _sleep_ state, something
more like S3 (suspend to RAM) than S5 (power off).

In fact even now we're doing things that are strange from the ACPI standpoint.
For example, we should really execute _PTS once during the entire transition
and we shouldn't call _WAK after we've created the image. We're doing that
now due to some design limitations, but in fact we shouldn't.

> > Still, it would be sufficient if we disconnected the drivers from the hardware
> > and thus prevented applications from accessing that hardware.
>
> My gut feeling is that except for a handful of drivers we could even
> get away with simply implementing hot unplug and hot replug. Disks
> are the big exception here.
>
> Which suggests to me that it is at least possible that the methods we
> want for a kexec jump hibernation may be different from an in-kernel
> hibernation and quite possibly are easier to implement.

I'm not sure about the "easier" part, quite frankly. Also, with our current
ordering of code the in-kernel hibernation will need the same callbacks
as the kexec-based thing. However, with the in-kernel approach we can
attempt (in the future) to be more ACPI compliant, so to speak, but with the
kexec-based approach that won't be possible.

Whether it's a good idea to follow ACPI, as far as hibernation is concerned, is
a separate question, but IMO we won't be able to answer it without _lots_ of
testing on vaious BIOS/firmware configurations. Our experience so far
indicates that at least some BIOSes expect us to follow ACPI and misbehave
otherwise, so for those systems there should be an "ACPI way" available.
[Others just don't work well if we try to follow ACPI and those may be handled
using the kexec-based approach, but that doesn't mean that we can just ignore
the ACPI compliance issue, at least for now.]

Thanks,
Rafael

2008-03-14 08:05:46

[permalink] [raw]

Subject: Re: [PATCH -mm] kexec jump -v9

On Wed, 2008-03-12 at 15:37 -0400, Vivek Goyal wrote:
> On Tue, Mar 11, 2008 at 08:17:45PM -0600, Eric W. Biederman wrote:
> > "Huang, Ying" <[email protected]> writes:
> >
> > > Yes. The entry point should be saved in dump.elf itself, this can be
> > > done via a user-space tool such as "makedumpfile". Because
> > > "makedumpfile" is also used to exclude free pages from disk image, it
> > > needs a communication method between two kernels (to get backup pages
> > > map or something like that from kernel A). We have talked about this
> > > before.
> > >
> > > - Your opinion is to communicate via the purgatory. (But I don't know
> > > how to communicate between kernel A and purgatory).
> >
> > How about the return address on the stack?
> >
>
> I think he needs to pass on much more data than just return address.
>
> IIUC, he needs to pass backup pages map to new kernel, so that any
> user space tool can use backup pages map to reconstruct/rearrange the
> first kernel's memory core and tools like makedumpfile can do filtering
> before hibernated images is saved.
>
> This brings me to a random thought. Can we break the process of loading
> a hibernation kernel in two steps.
>
> - In first step just do the memory reservation for running second kernel.
> (kexec -l <dummpy-file-for-reserving-memory>)
>
> - This memory map of reserved pages is exported to user space.
>
> - Use this memory map and regenerate the hibernation kernel initrd
> (rootfs.gz) and put the memory map there. This memory map can be used
> by makedumpfile in second kernel for filtering.
>
> This way it will user space to user space communication of information
> which gets fixed at kernel loading time.

Doing kexec load in two steps is a possible solution. Although this is a
little complex, we can wrap the two steps into one /sbin/kexec invoking.
That is, When do /sbin/kexec --load-preserve-context
<kernel-image>, /sbin/kexec first call sys_kexec_load() to load the
kernel image and reserving memory, then amend the memory image of loaded
kernel (B) according to the new information available such as return
address and backup pages map. For this solution, something still need to
be solved is how to pass some information back from kernel B
(hibernating kernel) to kernel A (original kernel) and how to pass some
information from kernel C (resuming kernel) to kernel A (original
kernel).

-----------------------------------------------------------------

Another possible solution to pass information between kernels (in user
space): needed information from kernel are passed in stack, and a
special ELF_NOTES is used to access the information in peer kernel.
Details is as follow:

1. Possible information need to be passed:

1.1 From user space (known before sys_kexec_load):

a. ELF core header
b. vmcoreinfo (pointer only)

1.2 From kernel space (known after sys_kexec_load):

a. jump back entry (return address)
b. backup pages map

2. When jumping from kernel A to kernel B:

2.1 In /sbin/kexec --load-preserve-context <kernel-image>, /sbin/kexec
allocate a special ELF_NOTES (ELF NOTES kernel) for information from
kernel space.

2.2 When doing sys_reboot(REBOOT_CMD_KEXEC), kernel put needed
information and physical address of ELF core header onto stack just
before jump to purgatory.

2.3 After jumping to purgatory, purgatory fills "ELF NOTES kernel" with
corresponding address in stack.

2.4 When kernel B is booted, /proc/vmcore is created and the information
form ELF NOTES kernel is available too.

3. When jumping back from kernel B to kernel A and jumping from kernel C
to kernel A:

3.1 Same as 2.1

3.2 Same as 2.2, but there is no purgatory in kernel A, so when
information are put on stack, jump to "jump back entry" of kernel A
directly.

3.3 The code on jump back entry of kernel A will work as a purgatory to
fill "ELF NOTES kernel" with corresponding address in stack.
Then /proc/vmcore reset code is called again to (re-)construct
the /proc/vmcore with new information.

Best Regards,
Huang Ying

2008-03-21 19:13:26

[permalink] [raw]

Subject: Re: [PATCH -mm] kexec jump -v9

On Fri, Mar 14, 2008 at 04:03:28PM +0800, Huang, Ying wrote:
> On Wed, 2008-03-12 at 15:37 -0400, Vivek Goyal wrote:
> > On Tue, Mar 11, 2008 at 08:17:45PM -0600, Eric W. Biederman wrote:
> > > "Huang, Ying" <[email protected]> writes:
> > >
> > > > Yes. The entry point should be saved in dump.elf itself, this can be
> > > > done via a user-space tool such as "makedumpfile". Because
> > > > "makedumpfile" is also used to exclude free pages from disk image, it
> > > > needs a communication method between two kernels (to get backup pages
> > > > map or something like that from kernel A). We have talked about this
> > > > before.
> > > >
> > > > - Your opinion is to communicate via the purgatory. (But I don't know
> > > > how to communicate between kernel A and purgatory).
> > >
> > > How about the return address on the stack?
> > >
> >
> > I think he needs to pass on much more data than just return address.
> >
> > IIUC, he needs to pass backup pages map to new kernel, so that any
> > user space tool can use backup pages map to reconstruct/rearrange the
> > first kernel's memory core and tools like makedumpfile can do filtering
> > before hibernated images is saved.
> >
> > This brings me to a random thought. Can we break the process of loading
> > a hibernation kernel in two steps.
> >
> > - In first step just do the memory reservation for running second kernel.
> > (kexec -l <dummpy-file-for-reserving-memory>)
> >
> > - This memory map of reserved pages is exported to user space.
> >
> > - Use this memory map and regenerate the hibernation kernel initrd
> > (rootfs.gz) and put the memory map there. This memory map can be used
> > by makedumpfile in second kernel for filtering.
> >
> > This way it will user space to user space communication of information
> > which gets fixed at kernel loading time.
>
> Doing kexec load in two steps is a possible solution. Although this is a
> little complex, we can wrap the two steps into one /sbin/kexec invoking.
> That is, When do /sbin/kexec --load-preserve-context
> <kernel-image>, /sbin/kexec first call sys_kexec_load() to load the
> kernel image and reserving memory, then amend the memory image of loaded
> kernel (B) according to the new information available such as return
> address and backup pages map. For this solution, something still need to
> be solved is how to pass some information back from kernel B
> (hibernating kernel) to kernel A (original kernel) and how to pass some
> information from kernel C (resuming kernel) to kernel A (original
> kernel).
>

Hi Huang,

I am kind of ok with both the methods.

- Communicate information between two kernels using an ELF NOTE
prepared by kernel.

- Communicate information between user space tools using initrd.

But which method to use will depend on what information we want to
exchange between two kernels.

For example, re-entry points can be on stack or in ELF NOTE.

Backup page map probably can be communicated using initrd as only user
space need to access that (ELF Core headers can be put in a memory area
which is not swapped during transition from kernel A to B. This way
kernel B never needs to know that kernel A had done some swapping of
pages?).

So far I have understood only following.

1. We need to pass around entry/re-entry points between kernels.

2. We need to pass backup pages map from kernel A to kernel B, so that user
space tool can do filtering.

3. We need to pass address of ELF core headers from kernel A to kernel B so
that a valid vmcore of kernel A can be exported.

- For first time boot of kernel B, address of ELF core header is
passed through command line.

- For re-entry into B, ELF core header address can be passed
using some register, or on stack or using kernel ELF NOTE.

What else? What information do we need to communicate from kernel B to
kernel A or from kernel C to kernel A?

I am sure that you have told it in the past. Just that I don't recollect
it.

Thanks
Vivek

2008-03-25 07:25:42