Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S966456AbbLQPnS (ORCPT ); Thu, 17 Dec 2015 10:43:18 -0500 Received: from mail-wm0-f41.google.com ([74.125.82.41]:38532 "EHLO mail-wm0-f41.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755761AbbLQPnR (ORCPT ); Thu, 17 Dec 2015 10:43:17 -0500 Subject: Re: corruption causing crash in __queue_work To: Tejun Heo References: <20151209162744.GN30240@mtj.duckdns.org> <566945A2.1050208@kyup.com> <20151210152901.GR30240@mtj.duckdns.org> <566AF262.8050009@kyup.com> <20151211170805.GT30240@mtj.duckdns.org> <566E80AE.7020502@kyup.com> <20151214153147.GA14957@redhat.com> <20151214203138.GA2871@redhat.com> <56729272.2010803@kyup.com> <20151217153327.GI4026@mtj.duckdns.org> Cc: Mike Snitzer , "Linux-Kernel@Vger. Kernel. Org" , SiteGround Operations , Alasdair Kergon , device-mapper development From: Nikolay Borisov Message-ID: <5672D810.2010404@kyup.com> Date: Thu, 17 Dec 2015 17:43:12 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.1.0 MIME-Version: 1.0 In-Reply-To: <20151217153327.GI4026@mtj.duckdns.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 2208 Lines: 58 On 12/17/2015 05:33 PM, Tejun Heo wrote: > Hello, Nikolay. > > On Thu, Dec 17, 2015 at 12:46:10PM +0200, Nikolay Borisov wrote: >> diff --git a/drivers/md/dm-thin.c b/drivers/md/dm-thin.c >> index 493c38e08bd2..ccbbf7823cf3 100644 >> --- a/drivers/md/dm-thin.c >> +++ b/drivers/md/dm-thin.c >> @@ -3506,8 +3506,8 @@ static void pool_postsuspend(struct dm_target *ti) >> struct pool_c *pt = ti->private; >> struct pool *pool = pt->pool; >> >> - cancel_delayed_work(&pool->waker); >> - cancel_delayed_work(&pool->no_space_timeout); >> + cancel_delayed_work_sync(&pool->waker); >> + cancel_delayed_work_sync(&pool->no_space_timeout); >> flush_workqueue(pool->wq); >> (void) commit(pool); >> } >> >> And this seems to have resolved the crashes. For the past 24 hours I >> haven't seen a single server crash whereas before at least 3-5 servers >> would crash. > > So, that's an obvious bug on dm-thin side. Mike if you are ok with this I will submit a proper patch ? > >> Given that, it seems like a race condition between destroying the >> workqueue from dm-thin and cancelling all the delayed work. >> >> Tejun, I've looked at cancel_delayed_work/cancel_delayed_work_sync and >> they both call try_to_grab_pending and then their function diverges. Is >> it possible that there is a latent race condition between canceling the >> delayed work and the subsequent re-scheduling of the work item? > > It's just the wrong variant being used. cancel_delayed_work() doesn't > guarantee that the work item isn't running on return. If the work > item was running and the workqueue is destroyed afterwards, it may end > up trying to requeue itself on a destroyed workqueue. Right, but my initial understanding was that when canceling the delayed work and then issuing flush_workqueue would act the same way as if cancel_delayed_work_sync is called wrt to this particular delayed item, no? > > Thanks. > -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/