Hi,
This is what I get, if I remove mmc card while system is suspended:
<4>[15241.041945] Call Trace:
<4>[15241.042047] [<ffffffff8106620a>] ? prepare_to_wait+0x2a/0x90
<4>[15241.042159] [<ffffffff810790bd>] ? trace_hardirqs_on+0xd/0x10
<4>[15241.042271] [<ffffffff8140db12>] ? _raw_spin_unlock_irqrestore+0x42/0x80
<4>[15241.042386] [<ffffffff8112a390>] ? bdi_sched_wait+0x0/0x20
<4>[15241.042496] [<ffffffff8112a39e>] bdi_sched_wait+0xe/0x20
<4>[15241.042606] [<ffffffff8140af6f>] __wait_on_bit+0x5f/0x90
<4>[15241.042714] [<ffffffff8112a390>] ? bdi_sched_wait+0x0/0x20
<4>[15241.042824] [<ffffffff8140b018>] out_of_line_wait_on_bit+0x78/0x90
<4>[15241.042935] [<ffffffff81065fd0>] ? wake_bit_function+0x0/0x40
<4>[15241.043045] [<ffffffff8112a2d3>] ? bdi_queue_work+0xa3/0xe0
<4>[15241.043155] [<ffffffff8112a37f>] bdi_sync_writeback+0x6f/0x80
<4>[15241.043265] [<ffffffff8112a3d2>] sync_inodes_sb+0x22/0x120
<4>[15241.043375] [<ffffffff8112f1d2>] __sync_filesystem+0x82/0x90
<4>[15241.043485] [<ffffffff8112f3db>] sync_filesystem+0x4b/0x70
<4>[15241.043594] [<ffffffff811391de>] fsync_bdev+0x2e/0x60
<4>[15241.043704] [<ffffffff812226be>] invalidate_partition+0x2e/0x50
<4>[15241.043816] [<ffffffff8116b92f>] del_gendisk+0x3f/0x140
<4>[15241.043926] [<ffffffffa00c0233>] mmc_blk_remove+0x33/0x60 [mmc_block]
<4>[15241.044043] [<ffffffff81338977>] mmc_bus_remove+0x17/0x20
<4>[15241.044152] [<ffffffff812ce746>] __device_release_driver+0x66/0xc0
<4>[15241.044264] [<ffffffff812ce89d>] device_release_driver+0x2d/0x40
<4>[15241.044375] [<ffffffff812cd9b5>] bus_remove_device+0xb5/0x120
<4>[15241.044486] [<ffffffff812cb46f>] device_del+0x12f/0x1a0
<4>[15241.044593] [<ffffffff81338a5b>] mmc_remove_card+0x5b/0x90
<4>[15241.044702] [<ffffffff8133ac27>] mmc_sd_remove+0x27/0x50
<4>[15241.044811] [<ffffffff81337d8c>] mmc_resume_host+0x10c/0x140
<4>[15241.044929] [<ffffffffa00850e9>] sdhci_resume_host+0x69/0xa0 [sdhci]
<4>[15241.045044] [<ffffffffa0bdc39e>] sdhci_pci_resume+0x8e/0xb0 [sdhci_pci]
<4>[15241.045159] [<ffffffff8124b0a2>] pci_legacy_resume+0x42/0x60
<4>[15241.045268] [<ffffffff8124b148>] pci_pm_restore+0x88/0xb0
<4>[15241.045378] [<ffffffff812d3942>] pm_op+0x1a2/0x1c0
<4>[15241.045483] [<ffffffff812d44cd>] dpm_resume_end+0x14d/0x520
<4>[15241.045593] [<ffffffff8108c0f1>] hibernation_snapshot+0xd1/0x290
<4>[15241.045704] [<ffffffff8108c3ad>] hibernate+0xfd/0x200
<4>[15241.045811] [<ffffffff8108ac5c>] state_store+0xec/0x100
<4>[15241.045919] [<ffffffff81172e17>] ? sysfs_get_active_two+0x27/0x60
<4>[15241.046032] [<ffffffff8122db07>] kobj_attr_store+0x17/0x20
<4>[15241.046141] [<ffffffff811710a6>] sysfs_write_file+0xe6/0x170
<4>[15241.046253] [<ffffffff811087f8>] vfs_write+0xb8/0x1a0
<4>[15241.046361] [<ffffffff811089d1>] sys_write+0x51/0x90
<4>[15241.046470] [<ffffffff8100305b>] system_call_fastpath+0x16/0x1b
<4>[15241.046579] INFO: lockdep is turned off.
It seems that del_disk can't be called from .resume methods.
It sleeps for threads that are frozen at that point.
Since I wrote my own driver (for xD cards) I have seen same problem.
I solved this (it is just very nice that way anyway) by a freezable
kernel thread that polls for card state changes,
and thus calls del_disk (indirectly) after system got fully resumed.
What do you think?
Best regards,
Maxim Levitsky
Currently removal of the card leads to del_disk called indirectly by mmc core.
This function expects userspace to be running, which isn't when .resume is called
Fix that by removing the code that did that in mmc_resume_host. It is possible
because card detection logic will kick it later and remove the card.
Also make mtd workqueue freezeable, so it won't attempt to add/remove the card
while userspace is frozen.
Signed-off-by: Maxim Levitsky <[email protected]>
---
drivers/mmc/core/core.c | 9 ++-------
1 files changed, 2 insertions(+), 7 deletions(-)
diff --git a/drivers/mmc/core/core.c b/drivers/mmc/core/core.c
index 30acd52..879d48d 100644
--- a/drivers/mmc/core/core.c
+++ b/drivers/mmc/core/core.c
@@ -1257,7 +1257,6 @@ int mmc_suspend_host(struct mmc_host *host, pm_message_t state)
if (host->caps & MMC_CAP_DISABLE)
cancel_delayed_work(&host->disable);
cancel_delayed_work(&host->detect);
- mmc_flush_scheduled_work();
mmc_bus_get(host);
if (host->bus_ops && !host->bus_dead) {
@@ -1300,15 +1299,11 @@ int mmc_resume_host(struct mmc_host *host)
mmc_select_voltage(host, host->ocr);
BUG_ON(!host->bus_ops->resume);
err = host->bus_ops->resume(host);
+
if (err) {
printk(KERN_WARNING "%s: error %d during resume "
"(card was removed?)\n",
mmc_hostname(host), err);
- if (host->bus_ops->remove)
- host->bus_ops->remove(host);
- mmc_claim_host(host);
- mmc_detach_bus(host);
- mmc_release_host(host);
/* no need to bother upper layers */
err = 0;
}
@@ -1332,7 +1327,7 @@ static int __init mmc_init(void)
{
int ret;
- workqueue = create_singlethread_workqueue("kmmcd");
+ workqueue = create_freezeable_workqueue("kmmcd");
if (!workqueue)
return -ENOMEM;
--
1.6.3.3
On Fri, 5 Feb 2010 01:18:15 +0200 Maxim Levitsky <[email protected]> wrote:
> Currently removal of the card leads to del_disk called indirectly by mmc core.
> This function expects userspace to be running, which isn't when .resume is called
>
> Fix that by removing the code that did that in mmc_resume_host. It is possible
> because card detection logic will kick it later and remove the card.
I don't really understand. The above implies that to trigger this bug,
one needs to physically remove the card during a resume operation. ie:
a human-vs-computer race. Sounds unlikely?
So... exactly what steps does the user need to take to trigger this
bug?
> Also make mtd workqueue freezeable, so it won't attempt to add/remove the card
> while userspace is frozen.
>
>
> diff --git a/drivers/mmc/core/core.c b/drivers/mmc/core/core.c
> index 30acd52..879d48d 100644
> --- a/drivers/mmc/core/core.c
> +++ b/drivers/mmc/core/core.c
> @@ -1257,7 +1257,6 @@ int mmc_suspend_host(struct mmc_host *host, pm_message_t state)
> if (host->caps & MMC_CAP_DISABLE)
> cancel_delayed_work(&host->disable);
> cancel_delayed_work(&host->detect);
> - mmc_flush_scheduled_work();
>
> mmc_bus_get(host);
> if (host->bus_ops && !host->bus_dead) {
> @@ -1300,15 +1299,11 @@ int mmc_resume_host(struct mmc_host *host)
> mmc_select_voltage(host, host->ocr);
> BUG_ON(!host->bus_ops->resume);
> err = host->bus_ops->resume(host);
> +
> if (err) {
> printk(KERN_WARNING "%s: error %d during resume "
> "(card was removed?)\n",
> mmc_hostname(host), err);
> - if (host->bus_ops->remove)
> - host->bus_ops->remove(host);
> - mmc_claim_host(host);
> - mmc_detach_bus(host);
> - mmc_release_host(host);
afacit that code's been there since March 2009. I'd have thought that
someone would have noticed "kernel hangs on resume" before now.
Do you think the patch should be backported into 2.6.32.x and eariler?
> /* no need to bother upper layers */
> err = 0;
> }
> @@ -1332,7 +1327,7 @@ static int __init mmc_init(void)
> {
> int ret;
>
> - workqueue = create_singlethread_workqueue("kmmcd");
> + workqueue = create_freezeable_workqueue("kmmcd");
> if (!workqueue)
> return -ENOMEM;
On Thu, 2010-02-04 at 16:09 -0800, Andrew Morton wrote:
> On Fri, 5 Feb 2010 01:18:15 +0200 Maxim Levitsky <[email protected]> wrote:
>
> > Currently removal of the card leads to del_disk called indirectly by mmc core.
> > This function expects userspace to be running, which isn't when .resume is called
> >
> > Fix that by removing the code that did that in mmc_resume_host. It is possible
> > because card detection logic will kick it later and remove the card.
>
> I don't really understand. The above implies that to trigger this bug,
> one needs to physically remove the card during a resume operation. ie:
> a human-vs-computer race. Sounds unlikely?
>
> So... exactly what steps does the user need to take to trigger this
Sorry for describing this poorly.
The steps are:
-> Have a kernel with CONFIG_MMC_UNSAFE_RESUME
-> Insert MMC/SD card
-> Suspend/hibernate the system
-> While system is hibernated/suspended pull the card off
-> Resume the system
-> Hang
if CONFIG_MMC_UNSAFE_RESUME is set, mmc core allows the user to
suspend/resume the card normally assuming he won't change the card or
modify it in another system. The former case is actually handled quite
well.
if CONFIG_MMC_UNSAFE_RESUME isn't set, it removes the card during
suspend, and I now think (and will test) that this will still hang the
system this time on suspend.
Maybe we can make del_disk behave well if called with userspace frozen?
After all if user calls it, very likely that hardware is absent thus
there is no point in syncing (which I think triggers the hang)....
Best regards,
Maxim Levitsky
ext Andrew Morton wrote:
> On Fri, 5 Feb 2010 01:18:15 +0200 Maxim Levitsky <[email protected]> wrote:
>
>> Currently removal of the card leads to del_disk called indirectly by mmc core.
>> This function expects userspace to be running, which isn't when .resume is called
>>
>> Fix that by removing the code that did that in mmc_resume_host. It is possible
>> because card detection logic will kick it later and remove the card.
>
> I don't really understand. The above implies that to trigger this bug,
> one needs to physically remove the card during a resume operation. ie:
> a human-vs-computer race. Sounds unlikely?
>
> So... exactly what steps does the user need to take to trigger this
> bug?
>
>> Also make mtd workqueue freezeable, so it won't attempt to add/remove the card
>> while userspace is frozen.
>>
>>
>> diff --git a/drivers/mmc/core/core.c b/drivers/mmc/core/core.c
>> index 30acd52..879d48d 100644
>> --- a/drivers/mmc/core/core.c
>> +++ b/drivers/mmc/core/core.c
>> @@ -1257,7 +1257,6 @@ int mmc_suspend_host(struct mmc_host *host, pm_message_t state)
>> if (host->caps & MMC_CAP_DISABLE)
>> cancel_delayed_work(&host->disable);
>> cancel_delayed_work(&host->detect);
>> - mmc_flush_scheduled_work();
>>
>> mmc_bus_get(host);
>> if (host->bus_ops && !host->bus_dead) {
>> @@ -1300,15 +1299,11 @@ int mmc_resume_host(struct mmc_host *host)
>> mmc_select_voltage(host, host->ocr);
>> BUG_ON(!host->bus_ops->resume);
>> err = host->bus_ops->resume(host);
>> +
>> if (err) {
>> printk(KERN_WARNING "%s: error %d during resume "
>> "(card was removed?)\n",
>> mmc_hostname(host), err);
>> - if (host->bus_ops->remove)
>> - host->bus_ops->remove(host);
>> - mmc_claim_host(host);
>> - mmc_detach_bus(host);
>> - mmc_release_host(host);
>
> afacit that code's been there since March 2009. I'd have thought that
> someone would have noticed "kernel hangs on resume" before now.
>
> Do you think the patch should be backported into 2.6.32.x and eariler?
It looks like the code was introduced in 2.6.32.x by commit
95cdfb72b9bc568803f395c266152c71b034b461
cc'ing the author Nicolas Pitre
>
>> /* no need to bother upper layers */
>> err = 0;
>> }
>> @@ -1332,7 +1327,7 @@ static int __init mmc_init(void)
>> {
>> int ret;
>>
>> - workqueue = create_singlethread_workqueue("kmmcd");
>> + workqueue = create_freezeable_workqueue("kmmcd");
>> if (!workqueue)
>> return -ENOMEM;
> --
> To unsubscribe from this list: send the line "unsubscribe linux-mmc" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
On Fri, 2010-02-05 at 12:17 +0200, Adrian Hunter wrote:
> ext Andrew Morton wrote:
> > On Fri, 5 Feb 2010 01:18:15 +0200 Maxim Levitsky <[email protected]> wrote:
> >
> >> Currently removal of the card leads to del_disk called indirectly by mmc core.
> >> This function expects userspace to be running, which isn't when .resume is called
> >>
> >> Fix that by removing the code that did that in mmc_resume_host. It is possible
> >> because card detection logic will kick it later and remove the card.
> >
> > I don't really understand. The above implies that to trigger this bug,
> > one needs to physically remove the card during a resume operation. ie:
> > a human-vs-computer race. Sounds unlikely?
> >
> > So... exactly what steps does the user need to take to trigger this
> > bug?
> >
> >> Also make mtd workqueue freezeable, so it won't attempt to add/remove the card
> >> while userspace is frozen.
> >>
> >>
> >> diff --git a/drivers/mmc/core/core.c b/drivers/mmc/core/core.c
> >> index 30acd52..879d48d 100644
> >> --- a/drivers/mmc/core/core.c
> >> +++ b/drivers/mmc/core/core.c
> >> @@ -1257,7 +1257,6 @@ int mmc_suspend_host(struct mmc_host *host, pm_message_t state)
> >> if (host->caps & MMC_CAP_DISABLE)
> >> cancel_delayed_work(&host->disable);
> >> cancel_delayed_work(&host->detect);
> >> - mmc_flush_scheduled_work();
> >>
> >> mmc_bus_get(host);
> >> if (host->bus_ops && !host->bus_dead) {
> >> @@ -1300,15 +1299,11 @@ int mmc_resume_host(struct mmc_host *host)
> >> mmc_select_voltage(host, host->ocr);
> >> BUG_ON(!host->bus_ops->resume);
> >> err = host->bus_ops->resume(host);
> >> +
> >> if (err) {
> >> printk(KERN_WARNING "%s: error %d during resume "
> >> "(card was removed?)\n",
> >> mmc_hostname(host), err);
> >> - if (host->bus_ops->remove)
> >> - host->bus_ops->remove(host);
> >> - mmc_claim_host(host);
> >> - mmc_detach_bus(host);
> >> - mmc_release_host(host);
> >
> > afacit that code's been there since March 2009. I'd have thought that
> > someone would have noticed "kernel hangs on resume" before now.
> >
> > Do you think the patch should be backported into 2.6.32.x and eariler?
>
> It looks like the code was introduced in 2.6.32.x by commit
>
> 95cdfb72b9bc568803f395c266152c71b034b461
>
> cc'ing the author Nicolas Pitre
I don't think this is this commit fault.
The problem lies somewhere in block layer.
del_disk hangs if called while usrspace is frozen.
Because I assume that this code was tested, I guess that it was possible
to call del_disk in this way once.
Fixing CONFIG_MMC_UNSAFE_RESUME=n not to do del_disk, won't be easy...
Best regards,
Maxim Levitsky
On Fri, 05 Feb 2010 10:31:42 +0200 Maxim Levitsky <[email protected]> wrote:
> On Thu, 2010-02-04 at 16:09 -0800, Andrew Morton wrote:
> > On Fri, 5 Feb 2010 01:18:15 +0200 Maxim Levitsky <[email protected]> wrote:
> >
> > > Currently removal of the card leads to del_disk called indirectly by mmc core.
> > > This function expects userspace to be running, which isn't when .resume is called
> > >
> > > Fix that by removing the code that did that in mmc_resume_host. It is possible
> > > because card detection logic will kick it later and remove the card.
> >
> > I don't really understand. The above implies that to trigger this bug,
> > one needs to physically remove the card during a resume operation. ie:
> > a human-vs-computer race. Sounds unlikely?
> >
> > So... exactly what steps does the user need to take to trigger this
>
> Sorry for describing this poorly.
> The steps are:
>
> -> Have a kernel with CONFIG_MMC_UNSAFE_RESUME
> -> Insert MMC/SD card
> -> Suspend/hibernate the system
> -> While system is hibernated/suspended pull the card off
> -> Resume the system
> -> Hang
>
>
> if CONFIG_MMC_UNSAFE_RESUME is set, mmc core allows the user to
> suspend/resume the card normally assuming he won't change the card or
> modify it in another system. The former case is actually handled quite
> well.
>
> if CONFIG_MMC_UNSAFE_RESUME isn't set, it removes the card during
> suspend, and I now think (and will test) that this will still hang the
> system this time on suspend.
>
> Maybe we can make del_disk behave well if called with userspace frozen?
> After all if user calls it, very likely that hardware is absent thus
> there is no point in syncing (which I think triggers the hang)....
>
There is no del_disk in the kernel. Let's be more specific (and
accurate!) about the hang. I assume it's
mmc_remove_card->device_del->kobject_uevent?
Yes, I'd have thought that it would be a good idea for the
kobject_uevent code (or lower, in call_usermodehelper) to take avoiding
action if userspace is frozen. However such action would probably
involve doing a WARN_ON() too, so we'd still need MMC changes to avoid
that.
On Fri, 2010-02-05 at 06:13 -0800, Andrew Morton wrote:
> On Fri, 05 Feb 2010 10:31:42 +0200 Maxim Levitsky <[email protected]> wrote:
>
> > On Thu, 2010-02-04 at 16:09 -0800, Andrew Morton wrote:
> > > On Fri, 5 Feb 2010 01:18:15 +0200 Maxim Levitsky <[email protected]> wrote:
> > >
> > > > Currently removal of the card leads to del_disk called indirectly by mmc core.
> > > > This function expects userspace to be running, which isn't when .resume is called
> > > >
> > > > Fix that by removing the code that did that in mmc_resume_host. It is possible
> > > > because card detection logic will kick it later and remove the card.
> > >
> > > I don't really understand. The above implies that to trigger this bug,
> > > one needs to physically remove the card during a resume operation. ie:
> > > a human-vs-computer race. Sounds unlikely?
> > >
> > > So... exactly what steps does the user need to take to trigger this
> >
> > Sorry for describing this poorly.
> > The steps are:
> >
> > -> Have a kernel with CONFIG_MMC_UNSAFE_RESUME
> > -> Insert MMC/SD card
> > -> Suspend/hibernate the system
> > -> While system is hibernated/suspended pull the card off
> > -> Resume the system
> > -> Hang
> >
> >
> > if CONFIG_MMC_UNSAFE_RESUME is set, mmc core allows the user to
> > suspend/resume the card normally assuming he won't change the card or
> > modify it in another system. The former case is actually handled quite
> > well.
> >
> > if CONFIG_MMC_UNSAFE_RESUME isn't set, it removes the card during
> > suspend, and I now think (and will test) that this will still hang the
> > system this time on suspend.
> >
> > Maybe we can make del_disk behave well if called with userspace frozen?
> > After all if user calls it, very likely that hardware is absent thus
> > there is no point in syncing (which I think triggers the hang)....
> >
>
> There is no del_disk in the kernel. Let's be more specific (and
> accurate!) about the hang. I assume it's
> mmc_remove_card->device_del->kobject_uevent?
Sorry!
I was referring to del_gendisk.
<4>[15241.042047] [<ffffffff8106620a>] ? prepare_to_wait+0x2a/0x90
<4>[15241.042159] [<ffffffff810790bd>] ? trace_hardirqs_on+0xd/0x10
<4>[15241.042271] [<ffffffff8140db12>] ? _raw_spin_unlock_irqrestore+0x42/0x80
<4>[15241.042386] [<ffffffff8112a390>] ? bdi_sched_wait+0x0/0x20
<4>[15241.042496] [<ffffffff8112a39e>] bdi_sched_wait+0xe/0x20
<4>[15241.042606] [<ffffffff8140af6f>] __wait_on_bit+0x5f/0x90
<4>[15241.042714] [<ffffffff8112a390>] ? bdi_sched_wait+0x0/0x20
<4>[15241.042824] [<ffffffff8140b018>] out_of_line_wait_on_bit+0x78/0x90
<4>[15241.042935] [<ffffffff81065fd0>] ? wake_bit_function+0x0/0x40
<4>[15241.043045] [<ffffffff8112a2d3>] ? bdi_queue_work+0xa3/0xe0
<4>[15241.043155] [<ffffffff8112a37f>] bdi_sync_writeback+0x6f/0x80
<4>[15241.043265] [<ffffffff8112a3d2>] sync_inodes_sb+0x22/0x120
<4>[15241.043375] [<ffffffff8112f1d2>] __sync_filesystem+0x82/0x90
<4>[15241.043485] [<ffffffff8112f3db>] sync_filesystem+0x4b/0x70
<4>[15241.043594] [<ffffffff811391de>] fsync_bdev+0x2e/0x60
<4>[15241.043704] [<ffffffff812226be>] invalidate_partition+0x2e/0x50
<4>[15241.043816] [<ffffffff8116b92f>] del_gendisk+0x3f/0x140
<4>[15241.043926] [<ffffffffa00c0233>] mmc_blk_remove+0x33/0x60 [mmc_block]
<4>[15241.044043] [<ffffffff81338977>] mmc_bus_remove+0x17/0x20
<4>[15241.044152] [<ffffffff812ce746>] __device_release_driver+0x66/0xc0
<4>[15241.044264] [<ffffffff812ce89d>] device_release_driver+0x2d/0x40
<4>[15241.044375] [<ffffffff812cd9b5>] bus_remove_device+0xb5/0x120
<4>[15241.044486] [<ffffffff812cb46f>] device_del+0x12f/0x1a0
<4>[15241.044593] [<ffffffff81338a5b>] mmc_remove_card+0x5b/0x90
<4>[15241.044702] [<ffffffff8133ac27>] mmc_sd_remove+0x27/0x50
<4>[15241.044811] [<ffffffff81337d8c>] mmc_resume_host+0x10c/0x140
<4>[15241.044929] [<ffffffffa00850e9>] sdhci_resume_host+0x69/0xa0 [sdhci]
<4>[15241.045044] [<ffffffffa0bdc39e>] sdhci_pci_resume+0x8e/0xb0 [sdhci_pci]
>
> Yes, I'd have thought that it would be a good idea for the
> kobject_uevent code (or lower, in call_usermodehelper) to take avoiding
> action if userspace is frozen. However such action would probably
> involve doing a WARN_ON() too, so we'd still need MMC changes to avoid
> that.
>
>
Best regards,
Maxim Levitsky
On Fri, 05 Feb 2010 16:19:20 +0200 Maxim Levitsky <[email protected]> wrote:
> >
> > There is no del_disk in the kernel. Let's be more specific (and
> > accurate!) about the hang. I assume it's
> > mmc_remove_card->device_del->kobject_uevent?
> Sorry!
> I was referring to del_gendisk.
>
> <4>[15241.042047] [<ffffffff8106620a>] ? prepare_to_wait+0x2a/0x90
> <4>[15241.042159] [<ffffffff810790bd>] ? trace_hardirqs_on+0xd/0x10
> <4>[15241.042271] [<ffffffff8140db12>] ? _raw_spin_unlock_irqrestore+0x42/0x80
> <4>[15241.042386] [<ffffffff8112a390>] ? bdi_sched_wait+0x0/0x20
> <4>[15241.042496] [<ffffffff8112a39e>] bdi_sched_wait+0xe/0x20
> <4>[15241.042606] [<ffffffff8140af6f>] __wait_on_bit+0x5f/0x90
> <4>[15241.042714] [<ffffffff8112a390>] ? bdi_sched_wait+0x0/0x20
> <4>[15241.042824] [<ffffffff8140b018>] out_of_line_wait_on_bit+0x78/0x90
> <4>[15241.042935] [<ffffffff81065fd0>] ? wake_bit_function+0x0/0x40
> <4>[15241.043045] [<ffffffff8112a2d3>] ? bdi_queue_work+0xa3/0xe0
> <4>[15241.043155] [<ffffffff8112a37f>] bdi_sync_writeback+0x6f/0x80
> <4>[15241.043265] [<ffffffff8112a3d2>] sync_inodes_sb+0x22/0x120
> <4>[15241.043375] [<ffffffff8112f1d2>] __sync_filesystem+0x82/0x90
> <4>[15241.043485] [<ffffffff8112f3db>] sync_filesystem+0x4b/0x70
> <4>[15241.043594] [<ffffffff811391de>] fsync_bdev+0x2e/0x60
> <4>[15241.043704] [<ffffffff812226be>] invalidate_partition+0x2e/0x50
> <4>[15241.043816] [<ffffffff8116b92f>] del_gendisk+0x3f/0x140
> <4>[15241.043926] [<ffffffffa00c0233>] mmc_blk_remove+0x33/0x60 [mmc_block]
> <4>[15241.044043] [<ffffffff81338977>] mmc_bus_remove+0x17/0x20
> <4>[15241.044152] [<ffffffff812ce746>] __device_release_driver+0x66/0xc0
> <4>[15241.044264] [<ffffffff812ce89d>] device_release_driver+0x2d/0x40
> <4>[15241.044375] [<ffffffff812cd9b5>] bus_remove_device+0xb5/0x120
> <4>[15241.044486] [<ffffffff812cb46f>] device_del+0x12f/0x1a0
> <4>[15241.044593] [<ffffffff81338a5b>] mmc_remove_card+0x5b/0x90
> <4>[15241.044702] [<ffffffff8133ac27>] mmc_sd_remove+0x27/0x50
> <4>[15241.044811] [<ffffffff81337d8c>] mmc_resume_host+0x10c/0x140
> <4>[15241.044929] [<ffffffffa00850e9>] sdhci_resume_host+0x69/0xa0 [sdhci]
> <4>[15241.045044] [<ffffffffa0bdc39e>] sdhci_pci_resume+0x8e/0xb0 [sdhci_pci]
So what's the hang? del_gendisk is doing IO? I'd assumed that it was
because it was calling kobject_uevent, but userspace is frozen.
Why is it this hard :(
On Fri, 2010-02-05 at 06:39 -0800, Andrew Morton wrote:
> On Fri, 05 Feb 2010 16:19:20 +0200 Maxim Levitsky <[email protected]> wrote:
>
> > >
> > > There is no del_disk in the kernel. Let's be more specific (and
> > > accurate!) about the hang. I assume it's
> > > mmc_remove_card->device_del->kobject_uevent?
> > Sorry!
> > I was referring to del_gendisk.
> >
> > <4>[15241.042047] [<ffffffff8106620a>] ? prepare_to_wait+0x2a/0x90
> > <4>[15241.042159] [<ffffffff810790bd>] ? trace_hardirqs_on+0xd/0x10
> > <4>[15241.042271] [<ffffffff8140db12>] ? _raw_spin_unlock_irqrestore+0x42/0x80
> > <4>[15241.042386] [<ffffffff8112a390>] ? bdi_sched_wait+0x0/0x20
> > <4>[15241.042496] [<ffffffff8112a39e>] bdi_sched_wait+0xe/0x20
> > <4>[15241.042606] [<ffffffff8140af6f>] __wait_on_bit+0x5f/0x90
> > <4>[15241.042714] [<ffffffff8112a390>] ? bdi_sched_wait+0x0/0x20
> > <4>[15241.042824] [<ffffffff8140b018>] out_of_line_wait_on_bit+0x78/0x90
> > <4>[15241.042935] [<ffffffff81065fd0>] ? wake_bit_function+0x0/0x40
> > <4>[15241.043045] [<ffffffff8112a2d3>] ? bdi_queue_work+0xa3/0xe0
> > <4>[15241.043155] [<ffffffff8112a37f>] bdi_sync_writeback+0x6f/0x80
> > <4>[15241.043265] [<ffffffff8112a3d2>] sync_inodes_sb+0x22/0x120
> > <4>[15241.043375] [<ffffffff8112f1d2>] __sync_filesystem+0x82/0x90
> > <4>[15241.043485] [<ffffffff8112f3db>] sync_filesystem+0x4b/0x70
> > <4>[15241.043594] [<ffffffff811391de>] fsync_bdev+0x2e/0x60
> > <4>[15241.043704] [<ffffffff812226be>] invalidate_partition+0x2e/0x50
> > <4>[15241.043816] [<ffffffff8116b92f>] del_gendisk+0x3f/0x140
> > <4>[15241.043926] [<ffffffffa00c0233>] mmc_blk_remove+0x33/0x60 [mmc_block]
> > <4>[15241.044043] [<ffffffff81338977>] mmc_bus_remove+0x17/0x20
> > <4>[15241.044152] [<ffffffff812ce746>] __device_release_driver+0x66/0xc0
> > <4>[15241.044264] [<ffffffff812ce89d>] device_release_driver+0x2d/0x40
> > <4>[15241.044375] [<ffffffff812cd9b5>] bus_remove_device+0xb5/0x120
> > <4>[15241.044486] [<ffffffff812cb46f>] device_del+0x12f/0x1a0
> > <4>[15241.044593] [<ffffffff81338a5b>] mmc_remove_card+0x5b/0x90
> > <4>[15241.044702] [<ffffffff8133ac27>] mmc_sd_remove+0x27/0x50
> > <4>[15241.044811] [<ffffffff81337d8c>] mmc_resume_host+0x10c/0x140
> > <4>[15241.044929] [<ffffffffa00850e9>] sdhci_resume_host+0x69/0xa0 [sdhci]
> > <4>[15241.045044] [<ffffffffa0bdc39e>] sdhci_pci_resume+0x8e/0xb0 [sdhci_pci]
>
> So what's the hang? del_gendisk is doing IO? I'd assumed that it was
> because it was calling kobject_uevent, but userspace is frozen.
This is a backtrace of a hang.
The patch I sent tries not to remove the card during suspend, by letting
card presence logic to run after system is fully resumed.
However if CONFIG_MMC_UNSAFE_RESUME is not set, card would be removed at
suspend time, again with userspace frozen.
I will need to add some knobs to remove and add (if present) the card
when system is fully resumed. It is possible, but I will need to dig the
mmc code a bit deeper.
What I suggested is make del_gendisk suspend safe somehow.
Best regards,
Maxim Levitsky
> -----Original Message-----
> From: [email protected] [mailto:linux-mmc-
> [email protected]] On Behalf Of Maxim Levitsky
> Sent: Friday, February 05, 2010 9:52 AM
> To: Andrew Morton
> Cc: [email protected]; Philip Langdale; linux-kernel; Jorg
> Schummer; linux-pm
> Subject: Re: [PATCH] MMC: fix hang if card was removed during suspend and
> unsafe resume was enabled
>
> On Fri, 2010-02-05 at 06:39 -0800, Andrew Morton wrote:
> > On Fri, 05 Feb 2010 16:19:20 +0200 Maxim Levitsky
> <[email protected]> wrote:
> >
> > > >
> > > > There is no del_disk in the kernel. Let's be more specific (and
> > > > accurate!) about the hang. I assume it's
> > > > mmc_remove_card->device_del->kobject_uevent?
> > > Sorry!
> > > I was referring to del_gendisk.
> > >
> > > <4>[15241.042047] [<ffffffff8106620a>] ? prepare_to_wait+0x2a/0x90
> > > <4>[15241.042159] [<ffffffff810790bd>] ? trace_hardirqs_on+0xd/0x10
> > > <4>[15241.042271] [<ffffffff8140db12>] ?
> _raw_spin_unlock_irqrestore+0x42/0x80
> > > <4>[15241.042386] [<ffffffff8112a390>] ? bdi_sched_wait+0x0/0x20
> > > <4>[15241.042496] [<ffffffff8112a39e>] bdi_sched_wait+0xe/0x20
> > > <4>[15241.042606] [<ffffffff8140af6f>] __wait_on_bit+0x5f/0x90
> > > <4>[15241.042714] [<ffffffff8112a390>] ? bdi_sched_wait+0x0/0x20
> > > <4>[15241.042824] [<ffffffff8140b018>]
> out_of_line_wait_on_bit+0x78/0x90
> > > <4>[15241.042935] [<ffffffff81065fd0>] ? wake_bit_function+0x0/0x40
> > > <4>[15241.043045] [<ffffffff8112a2d3>] ? bdi_queue_work+0xa3/0xe0
> > > <4>[15241.043155] [<ffffffff8112a37f>] bdi_sync_writeback+0x6f/0x80
> > > <4>[15241.043265] [<ffffffff8112a3d2>] sync_inodes_sb+0x22/0x120
> > > <4>[15241.043375] [<ffffffff8112f1d2>] __sync_filesystem+0x82/0x90
> > > <4>[15241.043485] [<ffffffff8112f3db>] sync_filesystem+0x4b/0x70
> > > <4>[15241.043594] [<ffffffff811391de>] fsync_bdev+0x2e/0x60
> > > <4>[15241.043704] [<ffffffff812226be>] invalidate_partition+0x2e/0x50
> > > <4>[15241.043816] [<ffffffff8116b92f>] del_gendisk+0x3f/0x140
> > > <4>[15241.043926] [<ffffffffa00c0233>] mmc_blk_remove+0x33/0x60
> [mmc_block]
> > > <4>[15241.044043] [<ffffffff81338977>] mmc_bus_remove+0x17/0x20
> > > <4>[15241.044152] [<ffffffff812ce746>]
> __device_release_driver+0x66/0xc0
> > > <4>[15241.044264] [<ffffffff812ce89d>]
> device_release_driver+0x2d/0x40
> > > <4>[15241.044375] [<ffffffff812cd9b5>] bus_remove_device+0xb5/0x120
> > > <4>[15241.044486] [<ffffffff812cb46f>] device_del+0x12f/0x1a0
> > > <4>[15241.044593] [<ffffffff81338a5b>] mmc_remove_card+0x5b/0x90
> > > <4>[15241.044702] [<ffffffff8133ac27>] mmc_sd_remove+0x27/0x50
> > > <4>[15241.044811] [<ffffffff81337d8c>] mmc_resume_host+0x10c/0x140
> > > <4>[15241.044929] [<ffffffffa00850e9>] sdhci_resume_host+0x69/0xa0
> [sdhci]
> > > <4>[15241.045044] [<ffffffffa0bdc39e>] sdhci_pci_resume+0x8e/0xb0
> [sdhci_pci]
> >
> > So what's the hang? del_gendisk is doing IO? I'd assumed that it was
> > because it was calling kobject_uevent, but userspace is frozen.
>
> This is a backtrace of a hang.
>
> The patch I sent tries not to remove the card during suspend, by letting
> card presence logic to run after system is fully resumed.
>
The assumption with CONFIG_MMC_UNSAFE_RESUME is that the card remains in the
slot during suspend. The controller driver can simply ignore a card removal
event if received in the suspended state. Wouldn't that solve your problem?
The system would resume because of the event and reinsertion of the card
would enumerate it again.
Regards,
Madhu
> However if CONFIG_MMC_UNSAFE_RESUME is not set, card would be removed at
> suspend time, again with userspace frozen.
> I will need to add some knobs to remove and add (if present) the card
> when system is fully resumed. It is possible, but I will need to dig the
> mmc code a bit deeper.
>
> What I suggested is make del_gendisk suspend safe somehow.
>
> Best regards,
> Maxim Levitsky
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-mmc" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
On Fri, 2010-02-05 at 10:19 -0600, Madhusudhan wrote:
>
> > -----Original Message-----
> > From: [email protected] [mailto:linux-mmc-
> > [email protected]] On Behalf Of Maxim Levitsky
> > Sent: Friday, February 05, 2010 9:52 AM
> > To: Andrew Morton
> > Cc: [email protected]; Philip Langdale; linux-kernel; Jorg
> > Schummer; linux-pm
> > Subject: Re: [PATCH] MMC: fix hang if card was removed during suspend and
> > unsafe resume was enabled
> >
> > On Fri, 2010-02-05 at 06:39 -0800, Andrew Morton wrote:
> > > On Fri, 05 Feb 2010 16:19:20 +0200 Maxim Levitsky
> > <[email protected]> wrote:
> > >
> > > > >
> > > > > There is no del_disk in the kernel. Let's be more specific (and
> > > > > accurate!) about the hang. I assume it's
> > > > > mmc_remove_card->device_del->kobject_uevent?
> > > > Sorry!
> > > > I was referring to del_gendisk.
> > > >
> > > > <4>[15241.042047] [<ffffffff8106620a>] ? prepare_to_wait+0x2a/0x90
> > > > <4>[15241.042159] [<ffffffff810790bd>] ? trace_hardirqs_on+0xd/0x10
> > > > <4>[15241.042271] [<ffffffff8140db12>] ?
> > _raw_spin_unlock_irqrestore+0x42/0x80
> > > > <4>[15241.042386] [<ffffffff8112a390>] ? bdi_sched_wait+0x0/0x20
> > > > <4>[15241.042496] [<ffffffff8112a39e>] bdi_sched_wait+0xe/0x20
> > > > <4>[15241.042606] [<ffffffff8140af6f>] __wait_on_bit+0x5f/0x90
> > > > <4>[15241.042714] [<ffffffff8112a390>] ? bdi_sched_wait+0x0/0x20
> > > > <4>[15241.042824] [<ffffffff8140b018>]
> > out_of_line_wait_on_bit+0x78/0x90
> > > > <4>[15241.042935] [<ffffffff81065fd0>] ? wake_bit_function+0x0/0x40
> > > > <4>[15241.043045] [<ffffffff8112a2d3>] ? bdi_queue_work+0xa3/0xe0
> > > > <4>[15241.043155] [<ffffffff8112a37f>] bdi_sync_writeback+0x6f/0x80
> > > > <4>[15241.043265] [<ffffffff8112a3d2>] sync_inodes_sb+0x22/0x120
> > > > <4>[15241.043375] [<ffffffff8112f1d2>] __sync_filesystem+0x82/0x90
> > > > <4>[15241.043485] [<ffffffff8112f3db>] sync_filesystem+0x4b/0x70
> > > > <4>[15241.043594] [<ffffffff811391de>] fsync_bdev+0x2e/0x60
> > > > <4>[15241.043704] [<ffffffff812226be>] invalidate_partition+0x2e/0x50
> > > > <4>[15241.043816] [<ffffffff8116b92f>] del_gendisk+0x3f/0x140
> > > > <4>[15241.043926] [<ffffffffa00c0233>] mmc_blk_remove+0x33/0x60
> > [mmc_block]
> > > > <4>[15241.044043] [<ffffffff81338977>] mmc_bus_remove+0x17/0x20
> > > > <4>[15241.044152] [<ffffffff812ce746>]
> > __device_release_driver+0x66/0xc0
> > > > <4>[15241.044264] [<ffffffff812ce89d>]
> > device_release_driver+0x2d/0x40
> > > > <4>[15241.044375] [<ffffffff812cd9b5>] bus_remove_device+0xb5/0x120
> > > > <4>[15241.044486] [<ffffffff812cb46f>] device_del+0x12f/0x1a0
> > > > <4>[15241.044593] [<ffffffff81338a5b>] mmc_remove_card+0x5b/0x90
> > > > <4>[15241.044702] [<ffffffff8133ac27>] mmc_sd_remove+0x27/0x50
> > > > <4>[15241.044811] [<ffffffff81337d8c>] mmc_resume_host+0x10c/0x140
> > > > <4>[15241.044929] [<ffffffffa00850e9>] sdhci_resume_host+0x69/0xa0
> > [sdhci]
> > > > <4>[15241.045044] [<ffffffffa0bdc39e>] sdhci_pci_resume+0x8e/0xb0
> > [sdhci_pci]
> > >
> > > So what's the hang? del_gendisk is doing IO? I'd assumed that it was
> > > because it was calling kobject_uevent, but userspace is frozen.
> >
> > This is a backtrace of a hang.
> >
> > The patch I sent tries not to remove the card during suspend, by letting
> > card presence logic to run after system is fully resumed.
> >
>
> The assumption with CONFIG_MMC_UNSAFE_RESUME is that the card remains in the
> slot during suspend. The controller driver can simply ignore a card removal
> event if received in the suspended state. Wouldn't that solve your problem?
The assumption is that user doesn't play games with the card while
system is in low power state. Why should my computer hang if I remove
the card and then resume the system?
>
> The system would resume because of the event and reinsertion of the card
> would enumerate it again.
Don't know about resume on removal, bacause its not supported by most
controllers, and it very very dangerous (Think about closing the lid,
removing the card, and putting the system in a bag....
Best regards,
Maxim Levitsky
On Fri, 05 Feb 2010 17:52:00 +0200
Maxim Levitsky <[email protected]> wrote:
> > > <4>[15241.042047] [<ffffffff8106620a>] ? prepare_to_wait+0x2a/0x90
> > > <4>[15241.042159] [<ffffffff810790bd>] ? trace_hardirqs_on+0xd/0x10
> > > <4>[15241.042271] [<ffffffff8140db12>] ? _raw_spin_unlock_irqrestore+0x42/0x80
> > > <4>[15241.042386] [<ffffffff8112a390>] ? bdi_sched_wait+0x0/0x20
> > > <4>[15241.042496] [<ffffffff8112a39e>] bdi_sched_wait+0xe/0x20
> > > <4>[15241.042606] [<ffffffff8140af6f>] __wait_on_bit+0x5f/0x90
> > > <4>[15241.042714] [<ffffffff8112a390>] ? bdi_sched_wait+0x0/0x20
> > > <4>[15241.042824] [<ffffffff8140b018>] out_of_line_wait_on_bit+0x78/0x90
> > > <4>[15241.042935] [<ffffffff81065fd0>] ? wake_bit_function+0x0/0x40
> > > <4>[15241.043045] [<ffffffff8112a2d3>] ? bdi_queue_work+0xa3/0xe0
> > > <4>[15241.043155] [<ffffffff8112a37f>] bdi_sync_writeback+0x6f/0x80
> > > <4>[15241.043265] [<ffffffff8112a3d2>] sync_inodes_sb+0x22/0x120
> > > <4>[15241.043375] [<ffffffff8112f1d2>] __sync_filesystem+0x82/0x90
> > > <4>[15241.043485] [<ffffffff8112f3db>] sync_filesystem+0x4b/0x70
> > > <4>[15241.043594] [<ffffffff811391de>] fsync_bdev+0x2e/0x60
> > > <4>[15241.043704] [<ffffffff812226be>] invalidate_partition+0x2e/0x50
> > > <4>[15241.043816] [<ffffffff8116b92f>] del_gendisk+0x3f/0x140
> > > <4>[15241.043926] [<ffffffffa00c0233>] mmc_blk_remove+0x33/0x60 [mmc_block]
> > > <4>[15241.044043] [<ffffffff81338977>] mmc_bus_remove+0x17/0x20
> > > <4>[15241.044152] [<ffffffff812ce746>] __device_release_driver+0x66/0xc0
> > > <4>[15241.044264] [<ffffffff812ce89d>] device_release_driver+0x2d/0x40
> > > <4>[15241.044375] [<ffffffff812cd9b5>] bus_remove_device+0xb5/0x120
> > > <4>[15241.044486] [<ffffffff812cb46f>] device_del+0x12f/0x1a0
> > > <4>[15241.044593] [<ffffffff81338a5b>] mmc_remove_card+0x5b/0x90
> > > <4>[15241.044702] [<ffffffff8133ac27>] mmc_sd_remove+0x27/0x50
> > > <4>[15241.044811] [<ffffffff81337d8c>] mmc_resume_host+0x10c/0x140
> > > <4>[15241.044929] [<ffffffffa00850e9>] sdhci_resume_host+0x69/0xa0 [sdhci]
> > > <4>[15241.045044] [<ffffffffa0bdc39e>] sdhci_pci_resume+0x8e/0xb0 [sdhci_pci]
> >
> > So what's the hang? del_gendisk is doing IO? I'd assumed that it was
> > because it was calling kobject_uevent, but userspace is frozen.
>
> This is a backtrace of a hang.
But why did it hang? Because the BDI worker threads are trying to
perform IO through a suspended device?
On Fri, 2010-02-05 at 10:26 -0800, Andrew Morton wrote:
> On Fri, 05 Feb 2010 17:52:00 +0200
> Maxim Levitsky <[email protected]> wrote:
>
> > > > <4>[15241.042047] [<ffffffff8106620a>] ? prepare_to_wait+0x2a/0x90
> > > > <4>[15241.042159] [<ffffffff810790bd>] ? trace_hardirqs_on+0xd/0x10
> > > > <4>[15241.042271] [<ffffffff8140db12>] ? _raw_spin_unlock_irqrestore+0x42/0x80
> > > > <4>[15241.042386] [<ffffffff8112a390>] ? bdi_sched_wait+0x0/0x20
> > > > <4>[15241.042496] [<ffffffff8112a39e>] bdi_sched_wait+0xe/0x20
> > > > <4>[15241.042606] [<ffffffff8140af6f>] __wait_on_bit+0x5f/0x90
> > > > <4>[15241.042714] [<ffffffff8112a390>] ? bdi_sched_wait+0x0/0x20
> > > > <4>[15241.042824] [<ffffffff8140b018>] out_of_line_wait_on_bit+0x78/0x90
> > > > <4>[15241.042935] [<ffffffff81065fd0>] ? wake_bit_function+0x0/0x40
> > > > <4>[15241.043045] [<ffffffff8112a2d3>] ? bdi_queue_work+0xa3/0xe0
> > > > <4>[15241.043155] [<ffffffff8112a37f>] bdi_sync_writeback+0x6f/0x80
> > > > <4>[15241.043265] [<ffffffff8112a3d2>] sync_inodes_sb+0x22/0x120
> > > > <4>[15241.043375] [<ffffffff8112f1d2>] __sync_filesystem+0x82/0x90
> > > > <4>[15241.043485] [<ffffffff8112f3db>] sync_filesystem+0x4b/0x70
> > > > <4>[15241.043594] [<ffffffff811391de>] fsync_bdev+0x2e/0x60
> > > > <4>[15241.043704] [<ffffffff812226be>] invalidate_partition+0x2e/0x50
> > > > <4>[15241.043816] [<ffffffff8116b92f>] del_gendisk+0x3f/0x140
> > > > <4>[15241.043926] [<ffffffffa00c0233>] mmc_blk_remove+0x33/0x60 [mmc_block]
> > > > <4>[15241.044043] [<ffffffff81338977>] mmc_bus_remove+0x17/0x20
> > > > <4>[15241.044152] [<ffffffff812ce746>] __device_release_driver+0x66/0xc0
> > > > <4>[15241.044264] [<ffffffff812ce89d>] device_release_driver+0x2d/0x40
> > > > <4>[15241.044375] [<ffffffff812cd9b5>] bus_remove_device+0xb5/0x120
> > > > <4>[15241.044486] [<ffffffff812cb46f>] device_del+0x12f/0x1a0
> > > > <4>[15241.044593] [<ffffffff81338a5b>] mmc_remove_card+0x5b/0x90
> > > > <4>[15241.044702] [<ffffffff8133ac27>] mmc_sd_remove+0x27/0x50
> > > > <4>[15241.044811] [<ffffffff81337d8c>] mmc_resume_host+0x10c/0x140
> > > > <4>[15241.044929] [<ffffffffa00850e9>] sdhci_resume_host+0x69/0xa0 [sdhci]
> > > > <4>[15241.045044] [<ffffffffa0bdc39e>] sdhci_pci_resume+0x8e/0xb0 [sdhci_pci]
> > >
> > > So what's the hang? del_gendisk is doing IO? I'd assumed that it was
> > > because it was calling kobject_uevent, but userspace is frozen.
> >
> > This is a backtrace of a hang.
>
> But why did it hang? Because the BDI worker threads are trying to
> perform IO through a suspended device?
>
Something like that I guess.
Also this is 100% reproducible, and I can reproduce this with my own
driver too (by making the card detection workqueue be non freezable)
Best regards,
Maxim Levitsky