Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755762AbXFPTyo (ORCPT ); Sat, 16 Jun 2007 15:54:44 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1752222AbXFPTyf (ORCPT ); Sat, 16 Jun 2007 15:54:35 -0400 Received: from pentafluge.infradead.org ([213.146.154.40]:36182 "EHLO pentafluge.infradead.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751619AbXFPTyf (ORCPT ); Sat, 16 Jun 2007 15:54:35 -0400 Date: Sat, 16 Jun 2007 20:54:01 +0100 From: Christoph Hellwig To: Jens Axboe Cc: Tejun Heo , David Greaves , "Rafael J. Wysocki" , Linus Torvalds , David Chinner , xfs@oss.sgi.com, "'linux-kernel@vger.kernel.org'" , linux-pm , Neil Brown , Jeff Garzik Subject: Re: [PATCH] block: always requeue !fs requests at the front Message-ID: <20070616195401.GA6929@infradead.org> Mail-Followup-To: Christoph Hellwig , Jens Axboe , Tejun Heo , David Greaves , "Rafael J. Wysocki" , Linus Torvalds , David Chinner , xfs@oss.sgi.com, "'linux-kernel@vger.kernel.org'" , linux-pm , Neil Brown , Jeff Garzik References: <200706020122.49989.rjw@sisk.pl> <46706968.7000703@dgreaves.com> <200706140115.58733.rjw@sisk.pl> <46714ECF.8080203@gmail.com> <46715A66.8030806@suse.de> <20070615094246.GN29122@htj.dyndns.org> <20070615110544.GR6149@kernel.dk> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20070615110544.GR6149@kernel.dk> User-Agent: Mutt/1.4.2.3i X-SRS-Rewrite: SMTP reverse-path rewritten from by pentafluge.infradead.org See http://www.infradead.org/rpr.html Sender: linux-kernel-owner@vger.kernel.org X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 3074 Lines: 67 On Fri, Jun 15, 2007 at 01:05:44PM +0200, Jens Axboe wrote: > On Fri, Jun 15 2007, Tejun Heo wrote: > > SCSI marks internal commands with REQ_PREEMPT and push it at the front > > of the request queue using blk_execute_rq(). When entering suspended > > or frozen state, SCSI devices are quiesced using > > scsi_device_quiesce(). In quiesced state, only REQ_PREEMPT requests > > are processed. This is how SCSI blocks other requests out while > > suspending and resuming. As all internal commands are pushed at the > > front of the queue, this usually works. > > > > Unfortunately, this interacts badly with ordered requeueing. To > > preserve request order on requeueing (due to busy device, active EH or > > other failures), requests are sorted according to ordered sequence on > > requeue if IO barrier is in progress. > > > > The following sequence deadlocks. > > > > 1. IO barrier sequence issues. > > > > 2. Suspend requested. Queue is quiesced with part of all of IO > > barrier sequence at the front. > > > > 3. During suspending or resuming, SCSI issues internal command which > > gets deferred and requeued for some reason. As the command is > > issued after the IO barrier in #1, ordered requeueing code puts the > > request after IO barrier sequence. > > > > 4. The device is ready to process requests again but still is in > > quiesced state and the first request of the queue isn't > > REQ_PREEMPT, so command processing is deadlocked - > > suspending/resuming waits for the issued request to complete while > > the request can't be processed till device is put back into > > running state by resuming. > > > > This can be fixed by always putting !fs requests at the front when > > requeueing. > > > > The following thread reports this deadlock. > > > > http://thread.gmane.org/gmane.linux.kernel/537473 > > > > Signed-off-by: Tejun Heo > > Cc: Jenn Axboe > > Cc: David Greaves > > --- > > Okay, it took a lot of hours of debugging but boiled down to two liner > > fix. I feel so empty. :-) RAID6 triggers this reliably because it > > uses BIO_BARRIER heavily to update its superblock. The recent ATA > > suspend/resume rewrite is hit by this because it uses SCSI internal > > commands to spin down and up the drives for suspending and resuming. > > > > David, please test this. Jens, does it look okay? > > Yep looks good, except for the bad multi-line comment style, but that's > minor stuff ;-) > > Acked-by: Jens Axboe I'd much much prefer having a description of the problem in the actual comment then a hyperlink. There's just too much chance of the latter breaking over time, and it's impossible to update it when things change that should be reflected in the comment. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/