Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752287AbbLLLt4 (ORCPT ); Sat, 12 Dec 2015 06:49:56 -0500 Received: from mail-wm0-f52.google.com ([74.125.82.52]:36107 "EHLO mail-wm0-f52.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751369AbbLLLty (ORCPT ); Sat, 12 Dec 2015 06:49:54 -0500 Subject: Re: corruption causing crash in __queue_work To: Mike Snitzer References: <566819D8.5090804@kyup.com> <20151209160803.GK30240@mtj.duckdns.org> <56685573.1020805@kyup.com> <20151209162744.GN30240@mtj.duckdns.org> <566945A2.1050208@kyup.com> <20151210152901.GR30240@mtj.duckdns.org> <566AF262.8050009@kyup.com> <20151211170805.GT30240@mtj.duckdns.org> <20151211191400.GA24229@redhat.com> Cc: Tejun Heo , Nikolay Borisov , "Linux-Kernel@Vger. Kernel. Org" , SiteGround Operations , Alasdair Kergon , device-mapper development From: Nikolay Borisov Message-ID: <566C09DE.9020802@siteground.com> Date: Sat, 12 Dec 2015 13:49:50 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.1.0 MIME-Version: 1.0 In-Reply-To: <20151211191400.GA24229@redhat.com> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 2256 Lines: 53 On 12/11/2015 09:14 PM, Mike Snitzer wrote: > On Fri, Dec 11 2015 at 1:00pm -0500, > Nikolay Borisov wrote: > >> On Fri, Dec 11, 2015 at 7:08 PM, Tejun Heo wrote: >>> >>> Hmmm... No idea why it didn't show up in the debug log but the only >>> way a workqueue could be in the above state is either it got >>> explicitly destroyed or somehow pwq refcnting is messed up, in both >>> cases it should have shown up in the log. >>> >>> cc'ing dm people. Is there any chance dm-thinp could be using >>> workqueue after destroying it? > > Not that I'm aware of. But never say never? > > Plus I'd think we'd see other dm-thinp specific use-after-free issues > aside from the thin-pool's workqueue. > >> In __pool_destroy in dm-thin.c I don't see a call to >> cancel_delayed_work before destroying the workqueue. Is it possible >> that this is the causeI > > Cannot see how, __pool_destroy()'s destroy_workqueue() would spew a > bunch of WARN_ONs (and the wq wouldn't be destroyed) if the workqueue > had outstanding work. > > __pool_destroy() is called once the thin-pool's ref count drops to 0 > (see __pool_dec which is called when the thin-pool is removed -- > e.g. with 'dmsetup remove'). This code is only reachable when nothing > else is using the thin-pool. > > And the thin-pool is only able to be removed if all thin devices that > depend on it have first been removed. And each individual thin device > waits for all outstanding IO before they can be removed. Ok, I had a look at the code closer now and it indeed seems that when the pool is suspended in its postsuspend callback the delay work is indeed canceled and the workqueue is being flushed. But given that I see those failures on at least 2-3 servers perday I doubt it it is a hardware/machine-specific issue. Furthermore, the fact that it is always a dm-thin queue that's being referenced points to the direction of dm-thin, even though the code looks solid in that regard. Regards, Nikolay -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/