Hi Kees, WeiXiong,
I am trying to make use of pstore_blk which is BTW exactly what I had
been looking for to store panic/console logs onto an eMMC partition.
Using the 5.10 kernel plus:
7e2e92e9861b Revert "mark pstore-blk as broken"
01c28bc8f389 pstore/blk: Use the normal block device I/O path
2a7507999638 pstore/blk: remove {un,}register_pstore_blk
fef0b337cd25 pstore/zone: cap the maximum device size
or the android13-5.15 (at Merge 5.15.40 into android13-5.15) kernel with
no changes and using:
mount -t pstore pstore /sys/fs/pstore
modprobe pstore_blk blkdev=/dev/mmcblk1p9 best_effort=yes
upon triggering a crash with:
echo c > /proc/sysrq-trigger
and rebooting and remounting the pstore filesystem and loading
pstore_blk, I only have:
# ls /sys/fs/pstore/
console-pstore_blk-0
which contains the entire console log up to, but excluding the crash.
The kernel does show that pstore_blk was used for all 3 types of kmsg,
pmsg and console:
[ 28.649514] pstore_zone: capping size to 128MiB
[ 28.712894] pstore_zone: registered pstore_blk as backend for
kmsg(Oops) pmsg console
[ 28.721145] pstore: Using crash dump compression: deflate
[ 28.906253] printk: console [pstore_blk-1] enabled
[ 28.911229] pstore: Registered pstore_blk as persistent store backend
[ 28.917735] pstore_blk: attached pstore_blk:/dev/mmcblk1p9
(134217728) (no dedicated panic_write!)
there is no automatic reboot upon panic, so I just tend to reboot after
2-3 seconds manually. The kernel is configured with the default
CONFIG_PSTORE_* options.
Is the observed behavior a limitation of the best_effort mode? If so, do
we have any plans to implementing a non-best effort mode for eMMC devices?
Thanks!
--
Florian
Hi Kees, WeiXiong,
On 7/14/22 20:49, Florian Fainelli wrote:
> Hi Kees, WeiXiong,
>
> I am trying to make use of pstore_blk which is BTW exactly what I had
> been looking for to store panic/console logs onto an eMMC partition.
>
> Using the 5.10 kernel plus:
>
> 7e2e92e9861b Revert "mark pstore-blk as broken"
> 01c28bc8f389 pstore/blk: Use the normal block device I/O path
> 2a7507999638 pstore/blk: remove {un,}register_pstore_blk
> fef0b337cd25 pstore/zone: cap the maximum device size
>
> or the android13-5.15 (at Merge 5.15.40 into android13-5.15) kernel with
> no changes and using:
>
> mount -t pstore pstore /sys/fs/pstore
> modprobe pstore_blk blkdev=/dev/mmcblk1p9 best_effort=yes
>
> upon triggering a crash with:
>
> echo c > /proc/sysrq-trigger
>
> and rebooting and remounting the pstore filesystem and loading
> pstore_blk, I only have:
>
> # ls /sys/fs/pstore/
> console-pstore_blk-0
>
> which contains the entire console log up to, but excluding the crash.
> The kernel does show that pstore_blk was used for all 3 types of kmsg,
> pmsg and console:
>
> [ 28.649514] pstore_zone: capping size to 128MiB
> [ 28.712894] pstore_zone: registered pstore_blk as backend for
> kmsg(Oops) pmsg console
> [ 28.721145] pstore: Using crash dump compression: deflate
> [ 28.906253] printk: console [pstore_blk-1] enabled
> [ 28.911229] pstore: Registered pstore_blk as persistent store backend
> [ 28.917735] pstore_blk: attached pstore_blk:/dev/mmcblk1p9
> (134217728) (no dedicated panic_write!)
>
> there is no automatic reboot upon panic, so I just tend to reboot after
> 2-3 seconds manually. The kernel is configured with the default
> CONFIG_PSTORE_* options.
>
> Is the observed behavior a limitation of the best_effort mode? If so, do
> we have any plans to implementing a non-best effort mode for eMMC devices?
Any feedback on my email? I did try to get kernel panics to be dumped
out to a dedicated /dev/mtdblock* partition for which there ought to be
support for mtd->panic_write, but it still did not work any better. Is
there something obvious that I am missing which prevents kernel panics
from being logged?
Thank you!
--
Florian
On Tue, Aug 09, 2022 at 11:35:08AM -0700, Florian Fainelli wrote:
> Hi Kees, WeiXiong,
>
> On 7/14/22 20:49, Florian Fainelli wrote:
> > Hi Kees, WeiXiong,
> >
> > I am trying to make use of pstore_blk which is BTW exactly what I had
> > been looking for to store panic/console logs onto an eMMC partition.
> >
> > Using the 5.10 kernel plus:
> >
> > 7e2e92e9861b Revert "mark pstore-blk as broken"
> > 01c28bc8f389 pstore/blk: Use the normal block device I/O path
> > 2a7507999638 pstore/blk: remove {un,}register_pstore_blk
> > fef0b337cd25 pstore/zone: cap the maximum device size
> >
> > or the android13-5.15 (at Merge 5.15.40 into android13-5.15) kernel with
> > no changes and using:
> >
> > mount -t pstore pstore /sys/fs/pstore
> > modprobe pstore_blk blkdev=/dev/mmcblk1p9 best_effort=yes
> >
> > upon triggering a crash with:
> >
> > echo c > /proc/sysrq-trigger
> >
> > and rebooting and remounting the pstore filesystem and loading
> > pstore_blk, I only have:
> >
> > # ls /sys/fs/pstore/
> > console-pstore_blk-0
> >
> > which contains the entire console log up to, but excluding the crash.
> > The kernel does show that pstore_blk was used for all 3 types of kmsg,
> > pmsg and console:
> >
> > [?? 28.649514] pstore_zone: capping size to 128MiB
> > [?? 28.712894] pstore_zone: registered pstore_blk as backend for
> > kmsg(Oops) pmsg console
> > [?? 28.721145] pstore: Using crash dump compression: deflate
> > [?? 28.906253] printk: console [pstore_blk-1] enabled
> > [?? 28.911229] pstore: Registered pstore_blk as persistent store backend
> > [?? 28.917735] pstore_blk: attached pstore_blk:/dev/mmcblk1p9
> > (134217728) (no dedicated panic_write!)
> >
> > there is no automatic reboot upon panic, so I just tend to reboot after
> > 2-3 seconds manually. The kernel is configured with the default
> > CONFIG_PSTORE_* options.
> >
> > Is the observed behavior a limitation of the best_effort mode? If so, do
> > we have any plans to implementing a non-best effort mode for eMMC
> > devices?
>
> Any feedback on my email? I did try to get kernel panics to be dumped out to
Hi! Sorry I lost this email originally. :)
> a dedicated /dev/mtdblock* partition for which there ought to be support for
> mtd->panic_write, but it still did not work any better. Is there something
With the mtdblock driver, do you still see:
pstore_blk: attached pstore_blk:/dev/... (no dedicated panic_write!)
^^^^^^^^^^^^^^^^^^^^^^^^^
> obvious that I am missing which prevents kernel panics from being logged?
Unfortunately it really depends on how the drivers are built. If the
block layer is shut down during a panic, pstore_blk won't catch the
panic. :(
--
Kees Cook
Kees,
I am in the process of implementing mmcpstore backend for mmc based
on the mtdpstore driver
This is what is registered with register_pstore_device(&cxt->dev);
cxt->dev.flags = PSTORE_FLAGS_DMESG;
cxt->dev.zone.read = mmcpstore_read;
cxt->dev.zone.write = mmcpstore_write;
cxt->dev.zone.erase = mmcpstore_erase;
cxt->dev.zone.panic_write = mmcpstore_panic_write;
# dmesg | grep pstor
[ 0.000000] Kernel command line: pstore_blk.blkdev=/dev/mmcblk1p8
crash_kexec_post_notifiers printk.always_kmsg_dump
[ 1.993986] pstore_zone: registered pstore_blk as backend for
kmsg(Oops,panic_write) pmsg
[ 2.002582] pstore: Using crash dump compression: deflate
[ 2.008133] pstore: Registered pstore_blk as persistent store backend
[ 2.020907] mmcpstore: /dev/mmcblk1p8 size 131072 start sector
34468 registered as psblk backend
[ 17.868753] psz_kmsg_recover_meta: pstore_zone: no valid data in
kmsg dump zone 0
[ 18.298933] psz_recover_zone: pstore_zone: no valid data in zone pmsg
[ 18.305398] psz_recovery: pstore_zone: recover end!
The driver is successfully registered and the read path works when
/sys/fs/pstor is mounted , however mmc_pstore_panic_write is not
called.
Need help in understanding what could be missing. I am using the
latest upstream kernel for testing.
Kamal
On Tue, Aug 9, 2022 at 7:06 PM Kees Cook <[email protected]> wrote:
>
> On Tue, Aug 09, 2022 at 11:35:08AM -0700, Florian Fainelli wrote:
> > Hi Kees, WeiXiong,
> >
> > On 7/14/22 20:49, Florian Fainelli wrote:
> > > Hi Kees, WeiXiong,
> > >
> > > I am trying to make use of pstore_blk which is BTW exactly what I had
> > > been looking for to store panic/console logs onto an eMMC partition.
> > >
> > > Using the 5.10 kernel plus:
> > >
> > > 7e2e92e9861b Revert "mark pstore-blk as broken"
> > > 01c28bc8f389 pstore/blk: Use the normal block device I/O path
> > > 2a7507999638 pstore/blk: remove {un,}register_pstore_blk
> > > fef0b337cd25 pstore/zone: cap the maximum device size
> > >
> > > or the android13-5.15 (at Merge 5.15.40 into android13-5.15) kernel with
> > > no changes and using:
> > >
> > > mount -t pstore pstore /sys/fs/pstore
> > > modprobe pstore_blk blkdev=/dev/mmcblk1p9 best_effort=yes
> > >
> > > upon triggering a crash with:
> > >
> > > echo c > /proc/sysrq-trigger
> > >
> > > and rebooting and remounting the pstore filesystem and loading
> > > pstore_blk, I only have:
> > >
> > > # ls /sys/fs/pstore/
> > > console-pstore_blk-0
> > >
> > > which contains the entire console log up to, but excluding the crash.
> > > The kernel does show that pstore_blk was used for all 3 types of kmsg,
> > > pmsg and console:
> > >
> > > [ 28.649514] pstore_zone: capping size to 128MiB
> > > [ 28.712894] pstore_zone: registered pstore_blk as backend for
> > > kmsg(Oops) pmsg console
> > > [ 28.721145] pstore: Using crash dump compression: deflate
> > > [ 28.906253] printk: console [pstore_blk-1] enabled
> > > [ 28.911229] pstore: Registered pstore_blk as persistent store backend
> > > [ 28.917735] pstore_blk: attached pstore_blk:/dev/mmcblk1p9
> > > (134217728) (no dedicated panic_write!)
> > >
> > > there is no automatic reboot upon panic, so I just tend to reboot after
> > > 2-3 seconds manually. The kernel is configured with the default
> > > CONFIG_PSTORE_* options.
> > >
> > > Is the observed behavior a limitation of the best_effort mode? If so, do
> > > we have any plans to implementing a non-best effort mode for eMMC
> > > devices?
> >
> > Any feedback on my email? I did try to get kernel panics to be dumped out to
>
> Hi! Sorry I lost this email originally. :)
>
> > a dedicated /dev/mtdblock* partition for which there ought to be support for
> > mtd->panic_write, but it still did not work any better. Is there something
>
> With the mtdblock driver, do you still see:
>
> pstore_blk: attached pstore_blk:/dev/... (no dedicated panic_write!)
> ^^^^^^^^^^^^^^^^^^^^^^^^^
>
> > obvious that I am missing which prevents kernel panics from being logged?
>
> Unfortunately it really depends on how the drivers are built. If the
> block layer is shut down during a panic, pstore_blk won't catch the
> panic. :(
>
> --
> Kees Cook
+ Adrain Hunter
+ Ulf Hansoon
On Fri, Nov 18, 2022 at 3:41 PM Kamal Dasu <[email protected]> wrote:
>
> Kees,
>
> I am in the process of implementing mmcpstore backend for mmc based
> on the mtdpstore driver
>
> This is what is registered with register_pstore_device(&cxt->dev);
> cxt->dev.flags = PSTORE_FLAGS_DMESG;
> cxt->dev.zone.read = mmcpstore_read;
> cxt->dev.zone.write = mmcpstore_write;
> cxt->dev.zone.erase = mmcpstore_erase;
> cxt->dev.zone.panic_write = mmcpstore_panic_write;
>
> # dmesg | grep pstor
> [ 0.000000] Kernel command line: pstore_blk.blkdev=/dev/mmcblk1p8
> crash_kexec_post_notifiers printk.always_kmsg_dump
> [ 1.993986] pstore_zone: registered pstore_blk as backend for
> kmsg(Oops,panic_write) pmsg
> [ 2.002582] pstore: Using crash dump compression: deflate
> [ 2.008133] pstore: Registered pstore_blk as persistent store backend
> [ 2.020907] mmcpstore: /dev/mmcblk1p8 size 131072 start sector
> 34468 registered as psblk backend
> [ 17.868753] psz_kmsg_recover_meta: pstore_zone: no valid data in
> kmsg dump zone 0
> [ 18.298933] psz_recover_zone: pstore_zone: no valid data in zone pmsg
> [ 18.305398] psz_recovery: pstore_zone: recover end!
>
> The driver is successfully registered and the read path works when
> /sys/fs/pstor is mounted , however mmc_pstore_panic_write is not
> called.
> Need help in understanding what could be missing. I am using the
> latest upstream kernel for testing.
>
> Kamal
>
> On Tue, Aug 9, 2022 at 7:06 PM Kees Cook <[email protected]> wrote:
> >
> > On Tue, Aug 09, 2022 at 11:35:08AM -0700, Florian Fainelli wrote:
> > > Hi Kees, WeiXiong,
> > >
> > > On 7/14/22 20:49, Florian Fainelli wrote:
> > > > Hi Kees, WeiXiong,
> > > >
> > > > I am trying to make use of pstore_blk which is BTW exactly what I had
> > > > been looking for to store panic/console logs onto an eMMC partition.
> > > >
> > > > Using the 5.10 kernel plus:
> > > >
> > > > 7e2e92e9861b Revert "mark pstore-blk as broken"
> > > > 01c28bc8f389 pstore/blk: Use the normal block device I/O path
> > > > 2a7507999638 pstore/blk: remove {un,}register_pstore_blk
> > > > fef0b337cd25 pstore/zone: cap the maximum device size
> > > >
> > > > or the android13-5.15 (at Merge 5.15.40 into android13-5.15) kernel with
> > > > no changes and using:
> > > >
> > > > mount -t pstore pstore /sys/fs/pstore
> > > > modprobe pstore_blk blkdev=/dev/mmcblk1p9 best_effort=yes
> > > >
> > > > upon triggering a crash with:
> > > >
> > > > echo c > /proc/sysrq-trigger
> > > >
> > > > and rebooting and remounting the pstore filesystem and loading
> > > > pstore_blk, I only have:
> > > >
> > > > # ls /sys/fs/pstore/
> > > > console-pstore_blk-0
> > > >
> > > > which contains the entire console log up to, but excluding the crash.
> > > > The kernel does show that pstore_blk was used for all 3 types of kmsg,
> > > > pmsg and console:
> > > >
> > > > [ 28.649514] pstore_zone: capping size to 128MiB
> > > > [ 28.712894] pstore_zone: registered pstore_blk as backend for
> > > > kmsg(Oops) pmsg console
> > > > [ 28.721145] pstore: Using crash dump compression: deflate
> > > > [ 28.906253] printk: console [pstore_blk-1] enabled
> > > > [ 28.911229] pstore: Registered pstore_blk as persistent store backend
> > > > [ 28.917735] pstore_blk: attached pstore_blk:/dev/mmcblk1p9
> > > > (134217728) (no dedicated panic_write!)
> > > >
> > > > there is no automatic reboot upon panic, so I just tend to reboot after
> > > > 2-3 seconds manually. The kernel is configured with the default
> > > > CONFIG_PSTORE_* options.
> > > >
> > > > Is the observed behavior a limitation of the best_effort mode? If so, do
> > > > we have any plans to implementing a non-best effort mode for eMMC
> > > > devices?
> > >
> > > Any feedback on my email? I did try to get kernel panics to be dumped out to
> >
> > Hi! Sorry I lost this email originally. :)
> >
> > > a dedicated /dev/mtdblock* partition for which there ought to be support for
> > > mtd->panic_write, but it still did not work any better. Is there something
> >
> > With the mtdblock driver, do you still see:
> >
> > pstore_blk: attached pstore_blk:/dev/... (no dedicated panic_write!)
> > ^^^^^^^^^^^^^^^^^^^^^^^^^
> >
> > > obvious that I am missing which prevents kernel panics from being logged?
> >
> > Unfortunately it really depends on how the drivers are built. If the
> > block layer is shut down during a panic, pstore_blk won't catch the
> > panic. :(
> >
> > --
> > Kees Cook
On Fri, Nov 18, 2022 at 03:41:44PM -0500, Kamal Dasu wrote:
> Kees,
>
> I am in the process of implementing mmcpstore backend for mmc based
> on the mtdpstore driver
>
> This is what is registered with register_pstore_device(&cxt->dev);
> cxt->dev.flags = PSTORE_FLAGS_DMESG;
> cxt->dev.zone.read = mmcpstore_read;
> cxt->dev.zone.write = mmcpstore_write;
> cxt->dev.zone.erase = mmcpstore_erase;
> cxt->dev.zone.panic_write = mmcpstore_panic_write;
>
> # dmesg | grep pstor
> [ 0.000000] Kernel command line: pstore_blk.blkdev=/dev/mmcblk1p8
> crash_kexec_post_notifiers printk.always_kmsg_dump
> [ 1.993986] pstore_zone: registered pstore_blk as backend for
> kmsg(Oops,panic_write) pmsg
> [ 2.002582] pstore: Using crash dump compression: deflate
> [ 2.008133] pstore: Registered pstore_blk as persistent store backend
> [ 2.020907] mmcpstore: /dev/mmcblk1p8 size 131072 start sector
> 34468 registered as psblk backend
> [ 17.868753] psz_kmsg_recover_meta: pstore_zone: no valid data in
> kmsg dump zone 0
> [ 18.298933] psz_recover_zone: pstore_zone: no valid data in zone pmsg
> [ 18.305398] psz_recovery: pstore_zone: recover end!
>
> The driver is successfully registered and the read path works when
> /sys/fs/pstor is mounted , however mmc_pstore_panic_write is not
> called.
> Need help in understanding what could be missing. I am using the
> latest upstream kernel for testing.
Hi!
Can you send an RFC patch? I'd expect this to work as you've currently
described it, but without code I'd just be guessing. :)
-Kees
--
Kees Cook
Kees,
Thanks for your reply. I will send the RFC patch.
Kamal
On Wed, Dec 7, 2022 at 5:13 PM Kees Cook <[email protected]> wrote:
>
> On Fri, Nov 18, 2022 at 03:41:44PM -0500, Kamal Dasu wrote:
> > Kees,
> >
> > I am in the process of implementing mmcpstore backend for mmc based
> > on the mtdpstore driver
> >
> > This is what is registered with register_pstore_device(&cxt->dev);
> > cxt->dev.flags = PSTORE_FLAGS_DMESG;
> > cxt->dev.zone.read = mmcpstore_read;
> > cxt->dev.zone.write = mmcpstore_write;
> > cxt->dev.zone.erase = mmcpstore_erase;
> > cxt->dev.zone.panic_write = mmcpstore_panic_write;
> >
> > # dmesg | grep pstor
> > [ 0.000000] Kernel command line: pstore_blk.blkdev=/dev/mmcblk1p8
> > crash_kexec_post_notifiers printk.always_kmsg_dump
> > [ 1.993986] pstore_zone: registered pstore_blk as backend for
> > kmsg(Oops,panic_write) pmsg
> > [ 2.002582] pstore: Using crash dump compression: deflate
> > [ 2.008133] pstore: Registered pstore_blk as persistent store backend
> > [ 2.020907] mmcpstore: /dev/mmcblk1p8 size 131072 start sector
> > 34468 registered as psblk backend
> > [ 17.868753] psz_kmsg_recover_meta: pstore_zone: no valid data in
> > kmsg dump zone 0
> > [ 18.298933] psz_recover_zone: pstore_zone: no valid data in zone pmsg
> > [ 18.305398] psz_recovery: pstore_zone: recover end!
> >
> > The driver is successfully registered and the read path works when
> > /sys/fs/pstor is mounted , however mmc_pstore_panic_write is not
> > called.
> > Need help in understanding what could be missing. I am using the
> > latest upstream kernel for testing.
>
> Hi!
>
> Can you send an RFC patch? I'd expect this to work as you've currently
> described it, but without code I'd just be guessing. :)
>
> -Kees
>
> --
> Kees Cook