Hi,
I'm adding cache flush support to the PS3 disk driver and have a few questions
related to barriers.
In my driver, I do:
blk_queue_issue_flush_fn(queue, ps3disk_issue_flush);
blk_queue_ordered(queue, QUEUE_ORDERED_DRAIN_FLUSH, ps3disk_prepare_flush);
but I can't find a way to actually trigger the calling of ps3disk_issue_flush()
and ps3disk_prepare_flush().
1. My prepare_flush_fn() routine should be called from queue_flush(), which is
in turn called by start_ordered().
start_ordered() is called by blk_do_ordered(), but only if there's no
barrier (REQ_HARDBARRIER is not set).
Apart from drivers/block/pktcdvd.c and init_request_from_bio()
(BIO_RW_BARRIER is set by drivers/md/md.c only?), the only other way
REQ_HARDBARRIER can be set is in queue_flush(), which is not possible.
2. My issue_flush_fn() should be called from blkdev_issue_flush() (ignoring
drivers/md). But blkdev_issue_flush() is called by ReiserFS and XFS only.
I read the Block Device chapter in Linux Device Drivers 3rd edition, which says
to check blk_barrier_rq(), but the barrier part seems to be obsolete, as we now
have the prepare_flush_fn parameter of blk_queue_ordered().
Am I missing something? Should I care about barriers?
Thanks!
BTW, Documentation/block/barrier.txt seems to be out-of-date. Patch to update
it is below.
---
Documentation/block/barrier.txt is not in sync with the actual code:
- blk_queue_ordered() no longer has a gfp_mask parameter
- blk_queue_ordered_locked() no longer exists
- sd_prepare_flush() looks slightly different
Signed-off-by: Geert Uytterhoeven <[email protected]>
--- a/Documentation/block/barrier.txt
+++ b/Documentation/block/barrier.txt
@@ -82,23 +82,12 @@ including draining and flushing.
typedef void (prepare_flush_fn)(request_queue_t *q, struct request *rq);
int blk_queue_ordered(request_queue_t *q, unsigned ordered,
- prepare_flush_fn *prepare_flush_fn,
- unsigned gfp_mask);
-
-int blk_queue_ordered_locked(request_queue_t *q, unsigned ordered,
- prepare_flush_fn *prepare_flush_fn,
- unsigned gfp_mask);
-
-The only difference between the two functions is whether or not the
-caller is holding q->queue_lock on entry. The latter expects the
-caller is holding the lock.
+ prepare_flush_fn *prepare_flush_fn);
@q : the queue in question
@ordered : the ordered mode the driver/device supports
@prepare_flush_fn : this function should prepare @rq such that it
flushes cache to physical medium when executed
-@gfp_mask : gfp_mask used when allocating data structures
- for ordered processing
For example, SCSI disk driver's prepare_flush_fn looks like the
following.
@@ -106,9 +95,10 @@ following.
static void sd_prepare_flush(request_queue_t *q, struct request *rq)
{
memset(rq->cmd, 0, sizeof(rq->cmd));
- rq->flags |= REQ_BLOCK_PC;
+ rq->cmd_type = REQ_TYPE_BLOCK_PC;
rq->timeout = SD_TIMEOUT;
rq->cmd[0] = SYNCHRONIZE_CACHE;
+ rq->cmd_len = 10;
}
The following seven ordered modes are supported. The following table
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- Sony Network and Software Technology Center Europe (NSCE)
[email protected] ------- The Corporate Village, Da Vincilaan 7-D1
Voice +32-2-7008453 Fax +32-2-7008622 ---------------- B-1935 Zaventem, Belgium
On Monday May 21, [email protected] wrote:
> Hi,
>
> I'm adding cache flush support to the PS3 disk driver and have a few questions
> related to barriers.
>
> In my driver, I do:
>
> blk_queue_issue_flush_fn(queue, ps3disk_issue_flush);
> blk_queue_ordered(queue, QUEUE_ORDERED_DRAIN_FLUSH, ps3disk_prepare_flush);
>
> but I can't find a way to actually trigger the calling of ps3disk_issue_flush()
> and ps3disk_prepare_flush().
>
> 1. My prepare_flush_fn() routine should be called from queue_flush(), which is
> in turn called by start_ordered().
> start_ordered() is called by blk_do_ordered(), but only if there's no
> barrier (REQ_HARDBARRIER is not set).
> Apart from drivers/block/pktcdvd.c and init_request_from_bio()
> (BIO_RW_BARRIER is set by drivers/md/md.c only?), the only other way
> REQ_HARDBARRIER can be set is in queue_flush(), which is not possible.
BIO_RW_BARRIER is set by various filesystems when mounted with
-o barrier
or
-o barrier=1 (ext3)
See calls to set_buffer_ordered in fs/jbd/commit.c
>
> 2. My issue_flush_fn() should be called from blkdev_issue_flush() (ignoring
> drivers/md). But blkdev_issue_flush() is called by ReiserFS and XFS only.
Yeh, it isn't widely used at the moment.
>
> I read the Block Device chapter in Linux Device Drivers 3rd edition, which says
> to check blk_barrier_rq(), but the barrier part seems to be obsolete, as we now
> have the prepare_flush_fn parameter of blk_queue_ordered().
>
> Am I missing something? Should I care about barriers?
Yes, you should care about barriers, though it is true that there is
some confusion around them and it might be difficult.
>
> Thanks!
>
> BTW, Documentation/block/barrier.txt seems to be out-of-date. Patch to update
> it is below.
You might like to post this directly to
Tejun Heo <[email protected]>
NeilBrown
>
> ---
>
> Documentation/block/barrier.txt is not in sync with the actual code:
> - blk_queue_ordered() no longer has a gfp_mask parameter
> - blk_queue_ordered_locked() no longer exists
> - sd_prepare_flush() looks slightly different
>
> Signed-off-by: Geert Uytterhoeven <[email protected]>
>
> --- a/Documentation/block/barrier.txt
> +++ b/Documentation/block/barrier.txt
> @@ -82,23 +82,12 @@ including draining and flushing.
> typedef void (prepare_flush_fn)(request_queue_t *q, struct request *rq);
>
> int blk_queue_ordered(request_queue_t *q, unsigned ordered,
> - prepare_flush_fn *prepare_flush_fn,
> - unsigned gfp_mask);
> -
> -int blk_queue_ordered_locked(request_queue_t *q, unsigned ordered,
> - prepare_flush_fn *prepare_flush_fn,
> - unsigned gfp_mask);
> -
> -The only difference between the two functions is whether or not the
> -caller is holding q->queue_lock on entry. The latter expects the
> -caller is holding the lock.
> + prepare_flush_fn *prepare_flush_fn);
>
> @q : the queue in question
> @ordered : the ordered mode the driver/device supports
> @prepare_flush_fn : this function should prepare @rq such that it
> flushes cache to physical medium when executed
> -@gfp_mask : gfp_mask used when allocating data structures
> - for ordered processing
>
> For example, SCSI disk driver's prepare_flush_fn looks like the
> following.
> @@ -106,9 +95,10 @@ following.
> static void sd_prepare_flush(request_queue_t *q, struct request *rq)
> {
> memset(rq->cmd, 0, sizeof(rq->cmd));
> - rq->flags |= REQ_BLOCK_PC;
> + rq->cmd_type = REQ_TYPE_BLOCK_PC;
> rq->timeout = SD_TIMEOUT;
> rq->cmd[0] = SYNCHRONIZE_CACHE;
> + rq->cmd_len = 10;
> }
>
> The following seven ordered modes are supported. The following table
>
> Gr{oetje,eeting}s,
>
> Geert
>
> --
> Geert Uytterhoeven -- Sony Network and Software Technology Center Europe (NSCE)
> [email protected] ------- The Corporate Village, Da Vincilaan 7-D1
> Voice +32-2-7008453 Fax +32-2-7008622 ---------------- B-1935 Zaventem, Belgium
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
Hello,
Neil Brown wrote:
>> but I can't find a way to actually trigger the calling of ps3disk_issue_flush()
>> and ps3disk_prepare_flush().
>>
>> 1. My prepare_flush_fn() routine should be called from queue_flush(), which is
>> in turn called by start_ordered().
>> start_ordered() is called by blk_do_ordered(), but only if there's no
>> barrier (REQ_HARDBARRIER is not set).
>> Apart from drivers/block/pktcdvd.c and init_request_from_bio()
>> (BIO_RW_BARRIER is set by drivers/md/md.c only?), the only other way
>> REQ_HARDBARRIER can be set is in queue_flush(), which is not possible.
>
> BIO_RW_BARRIER is set by various filesystems when mounted with
> -o barrier
> or
> -o barrier=1 (ext3)
>
> See calls to set_buffer_ordered in fs/jbd/commit.c
You driver first needs to call blk_queue_ordered() to set ordered mode
(probably QUEUE_ORDERED_DRAIN_FLUSH) and the filesystem needs to be
mounted with barrier enabled as Neil explained. Then, the FS will issue
barriers and the block layer will interpret them as draining + cache flush.
>> 2. My issue_flush_fn() should be called from blkdev_issue_flush() (ignoring
>> drivers/md). But blkdev_issue_flush() is called by ReiserFS and XFS only.
>
> Yeh, it isn't widely used at the moment.
I thought about re-implementing it in terms of barrier callbacks but
wasn't sure whether it would be better than completely removing
blkdev_issue_flush() && a bit lazy. :-)
I'll look into it.
>> I read the Block Device chapter in Linux Device Drivers 3rd edition, which says
>> to check blk_barrier_rq(), but the barrier part seems to be obsolete, as we now
>> have the prepare_flush_fn parameter of blk_queue_ordered().
>>
>> Am I missing something? Should I care about barriers?
>
> Yes, you should care about barriers, though it is true that there is
> some confusion around them and it might be difficult.
Yeap, basically a block driver needs two callbacks - issue_flush and
prepare_flush plus a call to blk_queue_ordered().
>> BTW, Documentation/block/barrier.txt seems to be out-of-date. Patch to update
>> it is below.
>
> You might like to post this directly to
> Tejun Heo <[email protected]>
Received, acked && forwarded to Jens.
Thanks.
--
tejun