Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S967123AbbLQPdb (ORCPT ); Thu, 17 Dec 2015 10:33:31 -0500 Received: from mail-yk0-f173.google.com ([209.85.160.173]:36131 "EHLO mail-yk0-f173.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751151AbbLQPd3 (ORCPT ); Thu, 17 Dec 2015 10:33:29 -0500 Date: Thu, 17 Dec 2015 10:33:27 -0500 From: Tejun Heo To: Nikolay Borisov Cc: Mike Snitzer , "Linux-Kernel@Vger. Kernel. Org" , SiteGround Operations , Alasdair Kergon , device-mapper development Subject: Re: corruption causing crash in __queue_work Message-ID: <20151217153327.GI4026@mtj.duckdns.org> References: <20151209162744.GN30240@mtj.duckdns.org> <566945A2.1050208@kyup.com> <20151210152901.GR30240@mtj.duckdns.org> <566AF262.8050009@kyup.com> <20151211170805.GT30240@mtj.duckdns.org> <566E80AE.7020502@kyup.com> <20151214153147.GA14957@redhat.com> <20151214203138.GA2871@redhat.com> <56729272.2010803@kyup.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <56729272.2010803@kyup.com> User-Agent: Mutt/1.5.24 (2015-08-30) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 1843 Lines: 47 Hello, Nikolay. On Thu, Dec 17, 2015 at 12:46:10PM +0200, Nikolay Borisov wrote: > diff --git a/drivers/md/dm-thin.c b/drivers/md/dm-thin.c > index 493c38e08bd2..ccbbf7823cf3 100644 > --- a/drivers/md/dm-thin.c > +++ b/drivers/md/dm-thin.c > @@ -3506,8 +3506,8 @@ static void pool_postsuspend(struct dm_target *ti) > struct pool_c *pt = ti->private; > struct pool *pool = pt->pool; > > - cancel_delayed_work(&pool->waker); > - cancel_delayed_work(&pool->no_space_timeout); > + cancel_delayed_work_sync(&pool->waker); > + cancel_delayed_work_sync(&pool->no_space_timeout); > flush_workqueue(pool->wq); > (void) commit(pool); > } > > And this seems to have resolved the crashes. For the past 24 hours I > haven't seen a single server crash whereas before at least 3-5 servers > would crash. So, that's an obvious bug on dm-thin side. > Given that, it seems like a race condition between destroying the > workqueue from dm-thin and cancelling all the delayed work. > > Tejun, I've looked at cancel_delayed_work/cancel_delayed_work_sync and > they both call try_to_grab_pending and then their function diverges. Is > it possible that there is a latent race condition between canceling the > delayed work and the subsequent re-scheduling of the work item? It's just the wrong variant being used. cancel_delayed_work() doesn't guarantee that the work item isn't running on return. If the work item was running and the workqueue is destroyed afterwards, it may end up trying to requeue itself on a destroyed workqueue. Thanks. -- tejun -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/