Hi Linus,
This is the big round of updates for the IO bits for 2.6.36-rc1.
Not sure what happened on the excessive number of merges listed
as:
Merge branch 'for-2.6.36' into for-next
as these should 1) just have been regular fast forwards, and 2)
not be listed under 2.6.36 when the pull was in the other
direction. I tend to run -git git versions, so perhaps there
was a bad version in there. I was planning on reshuffling
the branch, but it's 3 months of churn and fixups in there and
that would take me a while. Let me know if you absolutely
hate it and I'll do it.
Anyway, summary of changes:
- The block parts of the BKL removal from Arnd.
- The race fixing and wakeup appeasing writeback thread
cleanups/fixes from Artem. Originally done to be easier
on the power consumption, but Artem also found and fixed
some races on the exiting writeback task part.
- The writeback cleanups from Christoph.
- The writeback tracing bits from David Chinner, which
allow us to trace and inspect whether things are doing
as we inspect.
- Also from Christoph is the unification of the bio
and request flags and removal of the flag wrappers,
which makes things more readable and maintainable.
- Fixes from Tejun so that the read-ahead and sync WRITE
bits match up again. Also puts the bio read/write bits
into a separate header file.
- Two fixes for splice, alsoing going into stable.
- Series update of the blkfront driver.
- Series update of the CCISS driver.
- Removal of the ancient ISA_DMA_THRESHOLD in block/scsi
from Tomo. This is the arch/ churn in this patch series.
- Also from Tomo is the nice cleanup of the empty barrier
code and the conversion of barriers to be regular file
system request instead of block PC requests.
- A fix for the discard handling, making sure that we
don't traverse back into the file system on allocation.
- drbd gets a feature reverted that apparently wasn't
well enough tested. This is also going into stable. A
few other fixes for drbd as well.
- The floppy cleanup series from Stephen.
- And the usual string of little cleanups/fixes from a
bunch of other people.
Please pull!
git://git.kernel.dk/linux-2.6-block.git for-2.6.36
Andi Kleen (2):
gcc-4.6: block: fix unused but set variables in blk-merge
gcc-4.6: fs: fix unused but set warnings
Arnd Bergmann (8):
scsi/i2o_block: cleanup ioctl handling
block: push down BKL into .locked_ioctl
block: push down BKL into .open and .release
block: push BKL into blktrace ioctls
block: remove BKL from BLKROSET and BLKFLSBUF
block: remove BKL from partition ioctls
scsi/sd: remove big kernel lock
scsi/i2o: restore ioctl changes
Artem Bityutskiy (15):
writeback: harmonize writeback threads naming
writeback: fix possible race when creating bdi threads
writeback: do not lose wake-ups in the forker thread - 1
writeback: do not lose wake-ups in the forker thread - 2
writeback: do not lose wake-ups in bdi threads
writeback: simplify bdi code a little
writeback: do not remove bdi from bdi_list
writeback: move last_active to bdi
writeback: restructure bdi forker loop a little
writeback: move bdi threads exiting logic to the forker thread
writeback: prevent unnecessary bdi threads wakeups
writeback: optimize periodic bdi thread wakeups
writeback: remove unnecessary init_timer call
writeback: add new tracepoints
writeback: cleanup bdi_register
Changli Gao (2):
splice: direct_splice_actor() should not use pos in sd
splice: check f_mode for seekable file
Christoph Hellwig (11):
writeback: remove writeback_inodes_wbc
writeback: split writeback_inodes_wb
block: BARRIER request should imply SYNC
block: remove wrappers for request type/flags
block: unify flags for struct bio and struct request
block: fix some more cmd_type cleanup fallout
virtio_blk: remove overzealous warning
writeback: simplify the write back thread queue
writeback: remove wb_list
writeback: merge bdi_writeback_task and bdi_start_fn
block: don't allocate a payload for discard request
Daniel Stodden (10):
xenbus: Make xenbus_switch_state transactional
blkfront: Fix backtrace in del_gendisk
blkfront: Fix gendisk leak
blkfront: Clean up vbd release
blkfront: Lock blkfront_info when closing
blkfront: Fix blkfront backend switch race (bdev open)
blkfront: Fix blkfront backend switch race (bdev release)
blkfront: Lock blockfront_info during xbdev removal
blkfront: Remove obsolete info->users
blkfront: Klog the unclean release path
Dave Chinner (4):
writeback: Initial tracing support
writeback: Add tracing to balance_dirty_pages
writeback: Add tracing to write_cache_pages
blkdev: check for valid request queue before issuing flush
FUJITA Tomonori (21):
aha1532: remove ISA_DMA_THRESHOLD usage
block: kill ISA_DMA_THRESHOLD usage
remove needless ISA_DMA_THRESHOLD
scsi: add sd_unprep_fn to free discard page
scsi: remove unused free discard page in sd_done
block: introduce REQ_FLUSH flag
block: permit PREFLUSH and POSTFLUSH without prepare_flush_fn
scsi: stop using q->prepare_flush_fn
osdblk: stop using q->prepare_flush_fn
ps3disk: stop using q->prepare_flush_fn
dm: stop using q->prepare_flush_fn
virtio_blk: stop using q->prepare_flush_fn
ide: stop using q->prepare_flush_fn
block: remove q->prepare_flush_fn completely
scsi: need to reset unprep_rq_fn in sd_remove
block: remove unused REQ_TYPE_LINUX_BLOCK
scsi: fix discard page leak
scsi: convert discard to REQ_TYPE_FS from REQ_TYPE_BLOCK_PC
block: set REQ_TYPE_FS on flush requests
block: set up rq->rq_disk properly for flush requests
scsi: use REQ_TYPE_FS for flush request
Ian Campbell (1):
xen: use less generic names in blkfront driver.
James Bottomley (1):
block: implement an unprep function corresponding directly to prep
Jan Beulich (2):
blkfront: fixes for 'xm block-detach ... --force'
blkfront: don't access freed struct xenbus_device
Jens Axboe (29):
block: add sysfs knob for turning off disk entropy contributions
block: add helpers for the trivial queue flag sysfs show/store entries
block: fix reversed set/clear of bits in helpers
Merge branch 'for-2.6.36' into for-next
Merge branch 'writeback-2.6.36' into for-next
Merge branch 'for-2.6.36' into for-next
Merge branch 'for-linus' into for-next
Merge branch 'for-linus' into for-next
Merge branch 'for-2.6.36' into for-next
Merge branch 'for-2.6.36' into for-next
Merge branch 'for-linus' into for-next
Merge branch 'for-2.6.36' into for-next
Merge branch 'for-linus' into for-next
virtio_blk: add default case to cmd type switch
Merge branch 'for-linus' into for-next
Merge branch 'writeback-2.6.36' into for-2.6.36
Merge branch 'for-2.6.36' into for-next
Merge branch 'for-2.6.36' into for-next
Merge branch 'for-2.6.36' into for-next
Merge branch 'for-2.6.36' into for-next
Merge branch 'for-linus' into for-2.6.36
block: fixup missing conversion from BIO_RW_DISCARD to REQ_DISCARD
Merge branch 'master' into for-2.6.36
block: fix problem with sending down discard that isn't of correct granularity
coda: fixup clash with block layer REQ_* defines
writeback: fix bad _bh spinlock nesting
block: fix missing export of blk_types.h
Merge branch 'master' into for-2.6.36
Merge branch 'master' into for-2.6.36
Jeremy Fitzhardinge (4):
xen/blkfront: avoid compiler warning from missing cases
Merge remote branch 'jens-linux-2.6-block/for-2.6.36' into upstream/blkfront
xen/blkfront: use tagged queuing for barriers
xen/blkfront: Use QUEUE_ORDERED_DRAIN for old backends
Joe Perches (1):
drivers/cdrom: use pr_<level>
Julia Lawall (1):
drivers/block: use memdup_user
K. Y. Srinivasan (2):
xen/front: Propagate changed size of VBDs
xen/blkfront: revalidate after setting capacity
Kulikov Vasiliy (1):
cpqarray: check put_user() result
Lars Ellenberg (1):
drbd: revert "delay probes", feature is being re-implemented differently
Mike Miller (11):
cciss: enqueue and submit io
cciss: clean up interrupt handler
cciss: check for msi in interrupt_not_for_us
cciss: make interrupt access methods return type bool
cciss: add performant mode support for Stars/Sirius
cciss: new controller support and bump driver version
cciss: make sure we request the performant mode irq
cciss: fix call to put_controller_in_performant_mode
cciss: move next_command function from ifdef
cciss: remove errant debug code
cciss: change pad value from 32 to 0
Mike Snitzer (1):
block: disallow FS recursion from sb_issue_discard allocation
Miklos Szeredi (1):
splice: fix misuse of SPLICE_F_NONBLOCK
Minchan Kim (1):
writeback: remove wb in get_next_work_item
Philipp Reisner (2):
drbd: Disable delay probes for the upcomming release
drbd: Initialize all members of sync_conf to their defaults [Bugz 315]
Randy Dunlap (2):
block/xd.c: fix brace typo
writeback.h: needs linux/device.h
Stephen Hemminger (9):
floppy: initialize debug jiffies offset
floppy: remove unnecessary inlines
floppy: silence warning during disk test
floppy: use atomic type for usage_count
floppy: cmos attribute should be static
floppy: fix signed/unsigned warnings
floppy: use wait_event_interruptible
floppy: use warning macros
floppy: make controller const
Stephen M. Cameron (26):
cciss: Set the performant mode bit in the scsi half of the driver
cciss: save pdev pointer in per hba structure early to avoid passing it around so much.
cciss: factor out cciss_lookup_board_id
cciss: factor out cciss_board_disabled
cciss: remove board_id parameter from cciss_interrupt_mode()
cciss: factor out cciss_find_memory_BAR()
cciss: factor out cciss_wait_for_board_ready()
cciss: factor out cciss_find_cfgtables
cciss: fix leak of ioremapped memory
cciss: factor out cciss_find_board_params
cciss: factor out CISS_signature_present()
cciss: factor out cciss_enable_scsi_prefetch()
cciss: factor out cciss_p600_dma_prefetch_quirk()
cciss: cleanup some debug ifdefs
cciss: make cciss_put_controller_into_performant_mode as __devinit
cciss: factor out cciss_wait_for_mode_change_ack()
cciss: factor out cciss_enter_performant_mode
cciss: factor out cciss_find_cfg_addrs.
cciss: factor out cciss_reset_devices()
cciss: fix hard reset code.
cciss: sanitize max commands
cciss: forbid hard reset of 640x boards
cciss: use consistent variable names
cciss: separate cmd_alloc() and cmd_special_alloc()
cciss: change printks to dev_warn, etc.
cciss: cleanup interrupt_not_for_us
Stephen Rothwell (1):
block: fix for block tracing build error
Tejun Heo (2):
bio, fs: update RWA_MASK, READA and SWRITE to match the corresponding BIO_RW_* bits
bio, fs: separate out bio_types.h and define READ/WRITE constants in terms of BIO_RW_* flags
arch/alpha/include/asm/scatterlist.h | 2 -
arch/avr32/include/asm/scatterlist.h | 2 -
arch/blackfin/include/asm/scatterlist.h | 2 -
arch/cris/include/asm/scatterlist.h | 2 -
arch/frv/include/asm/scatterlist.h | 2 -
arch/h8300/include/asm/scatterlist.h | 2 -
arch/ia64/include/asm/scatterlist.h | 9 -
arch/m32r/include/asm/scatterlist.h | 2 -
arch/m68k/include/asm/scatterlist.h | 3 -
arch/microblaze/include/asm/scatterlist.h | 2 -
arch/mips/include/asm/scatterlist.h | 2 -
arch/mn10300/include/asm/scatterlist.h | 2 -
arch/parisc/include/asm/scatterlist.h | 1 -
arch/powerpc/include/asm/scatterlist.h | 3 -
arch/s390/include/asm/scatterlist.h | 2 -
arch/score/include/asm/scatterlist.h | 2 -
arch/sh/include/asm/scatterlist.h | 2 -
arch/sparc/include/asm/scatterlist.h | 1 -
arch/um/drivers/ubd_kern.c | 7 +-
arch/x86/include/asm/scatterlist.h | 1 -
arch/xtensa/include/asm/scatterlist.h | 2 -
block/blk-barrier.c | 35 +-
block/blk-core.c | 112 +-
block/blk-exec.c | 2 +-
block/blk-lib.c | 48 +-
block/blk-map.c | 2 +-
block/blk-merge.c | 9 +-
block/blk-settings.c | 17 +
block/blk-sysfs.c | 82 +-
block/blk.h | 6 +-
block/cfq-iosched.c | 21 +-
block/compat_ioctl.c | 56 -
block/elevator.c | 19 +-
block/ioctl.c | 21 +-
drivers/ata/libata-scsi.c | 4 +-
drivers/block/DAC960.c | 13 +-
drivers/block/amiflop.c | 29 +-
drivers/block/aoe/aoeblk.c | 6 +-
drivers/block/ataflop.c | 32 +-
drivers/block/brd.c | 9 +-
drivers/block/cciss.c | 2165 +++++++++++++++++------------
drivers/block/cciss.h | 135 ++-
drivers/block/cciss_cmd.h | 36 +-
drivers/block/cciss_scsi.c | 670 +++++-----
drivers/block/cpqarray.c | 78 +-
drivers/block/drbd/drbd_actlog.c | 8 +-
drivers/block/drbd/drbd_int.h | 16 +-
drivers/block/drbd/drbd_main.c | 102 +--
drivers/block/drbd/drbd_nl.c | 4 -
drivers/block/drbd/drbd_proc.c | 19 +-
drivers/block/drbd/drbd_receiver.c | 135 +--
drivers/block/drbd/drbd_req.c | 2 +-
drivers/block/drbd/drbd_worker.c | 15 +-
drivers/block/floppy.c | 182 +--
drivers/block/hd.c | 2 +-
drivers/block/loop.c | 9 +-
drivers/block/mg_disk.c | 4 +-
drivers/block/nbd.c | 7 +-
drivers/block/osdblk.c | 15 +-
drivers/block/paride/pcd.c | 21 +-
drivers/block/paride/pd.c | 11 +-
drivers/block/paride/pf.c | 26 +-
drivers/block/pktcdvd.c | 20 +-
drivers/block/ps3disk.c | 25 +-
drivers/block/swim.c | 20 +-
drivers/block/swim3.c | 32 +-
drivers/block/ub.c | 35 +-
drivers/block/umem.c | 2 +-
drivers/block/viodasd.c | 21 +-
drivers/block/virtio_blk.c | 88 +-
drivers/block/xd.c | 19 +-
drivers/block/xen-blkfront.c | 393 ++++--
drivers/block/xsysace.c | 8 +-
drivers/block/z2ram.c | 13 +-
drivers/cdrom/cdrom.c | 46 +-
drivers/cdrom/gdrom.c | 48 +-
drivers/cdrom/viocd.c | 106 +-
drivers/ide/ide-atapi.c | 17 +-
drivers/ide/ide-cd.c | 98 +-
drivers/ide/ide-cd_ioctl.c | 2 +-
drivers/ide/ide-disk.c | 18 +-
drivers/ide/ide-disk_ioctl.c | 9 +-
drivers/ide/ide-eh.c | 5 +-
drivers/ide/ide-floppy.c | 27 +-
drivers/ide/ide-floppy_ioctl.c | 12 +-
drivers/ide/ide-gd.c | 19 +-
drivers/ide/ide-io.c | 8 +-
drivers/ide/ide-pm.c | 8 +-
drivers/ide/ide-tape.c | 22 +-
drivers/md/dm-io.c | 12 +-
drivers/md/dm-kcopyd.c | 2 +-
drivers/md/dm-raid1.c | 2 +-
drivers/md/dm-stripe.c | 2 +-
drivers/md/dm.c | 47 +-
drivers/md/linear.c | 2 +-
drivers/md/md.c | 16 +-
drivers/md/md.h | 4 +-
drivers/md/multipath.c | 8 +-
drivers/md/raid0.c | 2 +-
drivers/md/raid1.c | 22 +-
drivers/md/raid10.c | 12 +-
drivers/md/raid5.c | 2 +-
drivers/memstick/core/mspro_block.c | 12 +-
drivers/message/i2o/i2o_block.c | 30 +-
drivers/mmc/card/block.c | 5 +
drivers/mmc/card/queue.c | 4 +-
drivers/mtd/mtd_blkdevs.c | 15 +-
drivers/s390/block/dasd.c | 8 +-
drivers/s390/block/dcssblk.c | 5 +
drivers/s390/char/tape_block.c | 8 +-
drivers/scsi/aha1542.c | 25 -
drivers/scsi/osd/osd_initiator.c | 8 +-
drivers/scsi/scsi_error.c | 12 +-
drivers/scsi/scsi_lib.c | 14 +-
drivers/scsi/sd.c | 126 ++-
drivers/scsi/sd.h | 2 +-
drivers/scsi/sr.c | 25 +-
drivers/scsi/sun3_NCR5380.c | 2 +-
drivers/scsi/sun3_scsi.c | 2 +-
drivers/scsi/sun3_scsi_vme.c | 2 +-
drivers/staging/hv/blkvsc_drv.c | 13 +-
drivers/xen/xenbus/xenbus_client.c | 90 +-
fs/bio.c | 5 +-
fs/block_dev.c | 10 +-
fs/btrfs/disk-io.c | 8 +-
fs/btrfs/inode.c | 6 +-
fs/btrfs/volumes.c | 18 +-
fs/coda/psdev.c | 12 +-
fs/coda/upcall.c | 12 +-
fs/exofs/ios.c | 2 +-
fs/fs-writeback.c | 161 ++-
fs/gfs2/log.c | 4 +-
fs/gfs2/meta_io.c | 8 +-
fs/gfs2/ops_fstype.c | 2 +-
fs/nilfs2/segbuf.c | 2 +-
fs/splice.c | 14 +-
include/linux/Kbuild | 1 +
include/linux/audit.h | 2 +-
include/linux/backing-dev.h | 23 +-
include/linux/bio.h | 158 +--
include/linux/blk_types.h | 193 +++
include/linux/blkdev.h | 142 +--
include/linux/blktrace_api.h | 18 +-
include/linux/coda_psdev.h | 8 +-
include/linux/drbd.h | 2 +-
include/linux/drbd_nl.h | 9 +-
include/linux/fs.h | 45 +-
include/trace/events/block.h | 15 +-
include/trace/events/writeback.h | 159 +++
kernel/power/block_io.c | 2 +-
kernel/trace/blktrace.c | 80 +-
mm/backing-dev.c | 446 +++----
mm/page-writeback.c | 5 +
mm/page_io.c | 2 +-
154 files changed, 4264 insertions(+), 3210 deletions(-)
create mode 100644 include/linux/blk_types.h
create mode 100644 include/trace/events/writeback.h
--
Jens Axboe
Confidentiality Notice: This e-mail message, its contents and any attachments to it are confidential to the intended recipient, and may contain information that is privileged and/or exempt from disclosure under applicable law. If you are not the intended recipient, please immediately notify the sender and destroy the original e-mail message and any attachments (and any copies that may have been made) from your system or otherwise. Any unauthorized use, copying, disclosure or distribution of this information is strictly prohibited.
On 2010-08-06 12:38, Jens Axboe wrote:
> Hi Linus,
>
> This is the big round of updates for the IO bits for 2.6.36-rc1.
> Not sure what happened on the excessive number of merges listed
> as:
>
> Merge branch 'for-2.6.36' into for-next
>
> as these should 1) just have been regular fast forwards, and 2)
> not be listed under 2.6.36 when the pull was in the other
> direction. I tend to run -git git versions, so perhaps there
> was a bad version in there. I was planning on reshuffling
> the branch, but it's 3 months of churn and fixups in there and
> that would take me a while. Let me know if you absolutely
> hate it and I'll do it.
Oh, and the 'master' merges into for-2.6.36 were all done to
resolve conflicts. I had to do two just this week after you
starting pulling in changes from others, as they were beyond
git to automatically resolve. So there are no excessive merges,
just the weird listings for for-2.6.36 -> for-next.
--
Jens Axboe
Confidentiality Notice: This e-mail message, its contents and any attachments to it are confidential to the intended recipient, and may contain information that is privileged and/or exempt from disclosure under applicable law. If you are not the intended recipient, please immediately notify the sender and destroy the original e-mail message and any attachments (and any copies that may have been made) from your system or otherwise. Any unauthorized use, copying, disclosure or distribution of this information is strictly prohibited.
On Fri, Aug 6, 2010 at 3:38 AM, Jens Axboe <[email protected]> wrote:
>
> Not sure what happened on the excessive number of merges listed
> as:
>
> ? ? ?Merge branch 'for-2.6.36' into for-next
Yeah, I'm not taking this. This has all the signs of the things people
used to do a year or two ago - daily merges into random points of my
tree. Some of them are more than daily.
There is no excuse. None. Your "I thought they were all going to be
fast-forwards" excuse is pointless. You shouldn't have done it in the
first place, and once you did it, you should have noticed, instead of
doing _another_ merge on the same effing day!
I accept a few merges over many weeks. I accept things like merging
tags (preferably the _release_ tags, rather than -rc's). But you have
two merges of my tree ON THE SAME DAY, and you have three merges - of
just totally trivial and uninteresting individual patches - from your
own branch within a couple of days. For no reason I can fathom. Why
didn't you do those commits on the right branch to begin with?
That's just f*cked up. And quite frankly, I have no other way to fix
crap like this than to say "I won't merge it". If I do merge it, not
only do I get the crap, but I just know I'll _continue_ to get crap
because things don't get fixed. Which is why I've spent so much time
talking about clean history over the years.
I thought we were over this.
I would suggest you:
(a) think about what the hell you do wrong. I can point to at least a
couple of things:
- you use two branches for the same thing, for no reason. You
apparently have "for-next" and "for-2.6.36", and you seem to have
thought they were different (why?) and then merged between them
sometimes daily.
If they are separate, you shouldn't be merging them (and you should
_never_ merge daily - if it's under development, it just messes things
up). And if they aren't separate, you shouldn't mess things up and use
two different branches for the same thing, and then just make things
messy that way.
- You merged from my tree at random points. Why? Until you ask me to
merge from you, there is NO REASON to try to resolve conflicts. And
once you _do_ ask me to merge from you, I still don't want you to do
the merge, because I want to know about what conflicts. That's where
the bugs almost always are.
And finally: if you do merge from me, don't merge some random
"master" branch. Merge the v2.6.35 release tag.
(b) Once you are sure the current mess doesn't ever happen again,
rebase the mess on top of 2.6.35. NOT on top of whatever random
unstable tree-of-the-day-during-the-merge-window. Rebasing on top of
something unstable is crazy work, and should absolutely never be done.
And then test the hell out of it, and walk through it all making sure
it's ok. And do that for a couple of _days_.
And note: one of the reasons I want the block tree to be clean is that
the biggest problems we had last release cycle was with the writeback
code. Quite frankly, if you can't get it sorted out and tested well by
the end of the merge window, I will absolutely _happily_ not pull your
tree this release at all. Because last release was annoying and
clearly not tested enough. If the work gets delayed by a release (or
two), I'll be perfectly ok with it.
So if we hadn't had problems last release, I might have let this
slide. But I am of the very strong opinion that the block tree has not
been careful enough. The mess with ugly history is just a symptom of
that, I think.
Linus
On Fri, Aug 6, 2010 at 8:44 AM, Linus Torvalds
<[email protected]> wrote:
>
> ... And
> once you _do_ ask me to merge from you, I still don't want you to do
> the merge, because I want to know about what conflicts. That's where
> the bugs almost always are.
Actually, let me rephrase that.
Almost all bugs are just individual commits. But the "oh, we had
conflicts between trees" is where the subtle bugs that are due to
interactions between two different development projects tend to be..
So "almost always" is not really true - almost always bugs are just
simply bugs: incorrect code. I wish we didn't have that, but hey,
reality clearly hates me. But the reason I want to see the merge
problems (even if I then occasionally end up having to send it back
and say "ok, I see the merge problem and I can't handle it, you do it
for me") is because that way I _see_ when people step on each others
feet. Because when it happens, it's ripe for nasty issues, including
simply ones that are due to bad development habits, or due to bad
source tree organization.
Linus
On 08/06/2010 05:44 PM, Linus Torvalds wrote:
> On Fri, Aug 6, 2010 at 3:38 AM, Jens Axboe <[email protected]> wrote:
>>
>> Not sure what happened on the excessive number of merges listed
>> as:
>>
>> Merge branch 'for-2.6.36' into for-next
>
> Yeah, I'm not taking this. This has all the signs of the things people
> used to do a year or two ago - daily merges into random points of my
> tree. Some of them are more than daily.
>
> There is no excuse. None. Your "I thought they were all going to be
> fast-forwards" excuse is pointless. You shouldn't have done it in the
> first place, and once you did it, you should have noticed, instead of
> doing _another_ merge on the same effing day!
Yep I should have noticed, but to be honest this is how I've done
for-next updates for quite a while and I didn't expect issues. In most
cases, for-next is the same as for-2.6.36. Sometimes other things are
pulled in there as well. I just don't see why the merges are listed in
the for-2.6.36 'main' branch, for-next is just a throwaway that can and
is rebased if need be while for-2.6.36 should remain stable on the same
base.
> I accept a few merges over many weeks. I accept things like merging
> tags (preferably the _release_ tags, rather than -rc's). But you have
> two merges of my tree ON THE SAME DAY, and you have three merges - of
> just totally trivial and uninteresting individual patches - from your
> own branch within a couple of days. For no reason I can fathom. Why
> didn't you do those commits on the right branch to begin with?
Sometimes urgent patches will sit in for-linus and then I will merge
those in to for-2.6.36 to save Stephen the hassle of carrying conflict
update patches.
> That's just f*cked up. And quite frankly, I have no other way to fix
> crap like this than to say "I won't merge it". If I do merge it, not
> only do I get the crap, but I just know I'll _continue_ to get crap
> because things don't get fixed. Which is why I've spent so much time
> talking about clean history over the years.
>
> I thought we were over this.
>
> I would suggest you:
>
> (a) think about what the hell you do wrong. I can point to at least a
> couple of things:
>
> - you use two branches for the same thing, for no reason. You
> apparently have "for-next" and "for-2.6.36", and you seem to have
> thought they were different (why?) and then merged between them
> sometimes daily.
As mentioned above, for-next is just a throw away for linux-next where
things are merged together. See below.
> If they are separate, you shouldn't be merging them (and you should
> _never_ merge daily - if it's under development, it just messes things
> up). And if they aren't separate, you shouldn't mess things up and use
> two different branches for the same thing, and then just make things
> messy that way.
>
> - You merged from my tree at random points. Why? Until you ask me to
> merge from you, there is NO REASON to try to resolve conflicts. And
> once you _do_ ask me to merge from you, I still don't want you to do
> the merge, because I want to know about what conflicts. That's where
> the bugs almost always are.
I tend to merge in your tree when I _know_ there's a conflict there,
since it's much easier to fix things up when they happen and the change
is fresh, than wait potentially 2-3 months and then have to resolve
everything in one go. The latter carries a much higher risk of screw ups
when merging - and as you note, this is usually where the bugs creap in.
It also helps linux-next since then the tree will merge cleanly with
your -git variant at least and Stephen doesn't have to fiddle around
with doing patches for lots of trees to fix things up.
But if you don't like that way of doing things, I will stop and just
keep for-2.6.X+1 patches in a branch that is based off 2.6.X and never
updated. I will try that for the next release and see how that goes.
> And finally: if you do merge from me, don't merge some random
> "master" branch. Merge the v2.6.35 release tag.
>
> (b) Once you are sure the current mess doesn't ever happen again,
> rebase the mess on top of 2.6.35. NOT on top of whatever random
> unstable tree-of-the-day-during-the-merge-window. Rebasing on top of
> something unstable is crazy work, and should absolutely never be done.
> And then test the hell out of it, and walk through it all making sure
> it's ok. And do that for a couple of _days_.
Agree, that is stupid and I should not be doing that...
> And note: one of the reasons I want the block tree to be clean is that
> the biggest problems we had last release cycle was with the writeback
> code. Quite frankly, if you can't get it sorted out and tested well by
> the end of the merge window, I will absolutely _happily_ not pull your
> tree this release at all. Because last release was annoying and
> clearly not tested enough. If the work gets delayed by a release (or
> two), I'll be perfectly ok with it.
Completely agree, and I understand why you are cautious around things
that have the potentially to screw up users data. The writeback and
block IO changes are both such candidates.
> So if we hadn't had problems last release, I might have let this
> slide. But I am of the very strong opinion that the block tree has not
> been careful enough. The mess with ugly history is just a symptom of
> that, I think.
OK, not a problem, that's why I asked! I'm travelling to Boston right
now and will do the rebase and reshuffle in the air. That should leave
me time to do some testing before asking you to pull it again.
I'm not very proud of how this series ended up looking, that's a given.
Another mistake is carrying too many things in one branch, I will switch
to topic branches for the next release and ensure that the process is a
lot saner history wise.
--
Jens Axboe
Confidentiality Notice: This e-mail message, its contents and any attachments to it are confidential to the intended recipient, and may contain information that is privileged and/or exempt from disclosure under applicable law. If you are not the intended recipient, please immediately notify the sender and destroy the original e-mail message and any attachments (and any copies that may have been made) from your system or otherwise. Any unauthorized use, copying, disclosure or distribution of this information is strictly prohibited.
On 08/06/2010 05:51 PM, Linus Torvalds wrote:
> On Fri, Aug 6, 2010 at 8:44 AM, Linus Torvalds
> <[email protected]> wrote:
>>
>> ... And
>> once you _do_ ask me to merge from you, I still don't want you to do
>> the merge, because I want to know about what conflicts. That's where
>> the bugs almost always are.
>
> Actually, let me rephrase that.
>
> Almost all bugs are just individual commits. But the "oh, we had
> conflicts between trees" is where the subtle bugs that are due to
> interactions between two different development projects tend to be..
>
> So "almost always" is not really true - almost always bugs are just
> simply bugs: incorrect code. I wish we didn't have that, but hey,
> reality clearly hates me. But the reason I want to see the merge
> problems (even if I then occasionally end up having to send it back
> and say "ok, I see the merge problem and I can't handle it, you do it
> for me") is because that way I _see_ when people step on each others
> feet. Because when it happens, it's ripe for nasty issues, including
> simply ones that are due to bad development habits, or due to bad
> source tree organization.
OK, so a question on this. Say a bug surfaces in the middle of the
release and we push in a change to fix that at 2.6.36-rc3 time. This
same patch will not apply directly to the branch holding 2.6.37 patches
due to code reshuffling or whatnot. How do you want that handled? I
can't pull in your branch and resolve it. The merge conflict may not be
visible to you until 2.6.36 is released and I want to offload the
patches to you, but it will be visible in linux-next pretty much
immediately.
This puts a lot of extra work on Stephen.
--
Jens Axboe
Confidentiality Notice: This e-mail message, its contents and any attachments to it are confidential to the intended recipient, and may contain information that is privileged and/or exempt from disclosure under applicable law. If you are not the intended recipient, please immediately notify the sender and destroy the original e-mail message and any attachments (and any copies that may have been made) from your system or otherwise. Any unauthorized use, copying, disclosure or distribution of this information is strictly prohibited.
On Sat, Aug 7, 2010 at 12:31 AM, Jens Axboe <[email protected]> wrote:
>
> Sometimes urgent patches will sit in for-linus and then I will merge
> those in to for-2.6.36 to save Stephen the hassle of carrying conflict
> update patches.
The merges I saw weren't even urgent, because you had never actually
pushed them to me as such for the previous release. So both sides of
the merge was "for-2.3.36" as far as I can tell. That's what makes me
think there's some odd duplication of branches with no point (except
to make the history look ugly).
Now, don't get me wrong - I _like_ branches. I love it when people
maintain different branches for different features, and then merge
them together in order for me to pull them (or just ask me to pull
multiple branches). See for example merge commit 415cb47998 that
Pekka did to create one single thing for me to pull when he had
maintained four different topic branches. Or see the x86 "tip" pulls
from Ingo, Thomas and Peter as an example of just asking me to pull
multiple branches separately.
So branches (and the merges that go with them) are a wonderful thing
when they _clarify_ history. I don't mind merges at all in that case.
When the merge has a reason for existing, it's actually a good thing,
and one guiding model for that is to ask yourself "Does this merge
message make sense and tell people something interesting"?
So just to take a look at that merge by Pekka:
commit 415cb47998c5
Merge: 9fe6206f4006 78b435368fcd d602dabaeba7 2bce64858442 bc6488e91078
Author: Pekka Enberg <[email protected]>
Date: Wed Aug 4 22:04:43 2010 +0300
Merge branches 'slab/fixes', 'slob/fixes', 'slub/cleanups' and
'slub/fixes' into for-linus
and even outside of the history itself, I'd argue that the merge
message already tells you _why_ the merge was done. Despite it being
just the trivial automatic merge message that git generates on its
own. The merge actually clarifies history, I think.
So one guiding light for doing merges is really just to ask yourself
whether the merge message will actually tell anybody anything. And in
particular, merge messages like
Merge branch 'master' into for-2.6.36
don't do that. Think about it as an outsider: what does the above tell
anybody? It looks like it has no relevant information in it. That's a
hint that the merge simply shouldn't have been done.
It's just a _hint_ though. I'm not saying that the merge message
always has to be a revelation. I'm just saying that a merge message
like "Merge branch 'master' ..." should raise red flags.
Now, the "into for-2.6.36" part is still informative. But if there are
multiple merges of the same branch into that same thing, then that is
another red flag, bringing up the question "why did he merge something
that wasn't ready yet".
What I'm trying to say is that if you start thinking about what the
merge messages tell outsiders, then you probably are thinking about
merges the right way. And if the automatic git messages don't seem to
tell the reason, one thing you can actually do is to do
git pull --no-commit ....
git commit
and edit the merge message manually to explain what/why the merge does
(or you could do "git commit --amend" after the pull, but I would
encourage people to do the commit separately if only because it
actually shows that you thought about the issue _before_ you did the
pull rather than as a "oops, that merge message was uninformative, let
me fix it up" after-the-fact)
> I tend to merge in your tree when I _know_ there's a conflict there,
> since it's much easier to fix things up when they happen and the change
> is fresh, than wait potentially 2-3 months and then have to resolve
> everything in one go.
Now, this is a good reason for a merge. But:
- make that reason known
and
- DON'T DO THIS WHILE THE CODE IS STILL UNDER DEVELOPMENT
I'm shouting, because the second thing is what really gets me: daily
merges. If you are doing daily merges for conflict resolution, that's
a big big red flag. That means that you're merging stuff while it's
still being developed.
So the "let's do a merge to avoid hard merges much later" is a good
reason to merge, but in that case do point it out, and do it after the
development has died down, and after you know that you aren't going to
have another thing that will also clash.
In other words, wait a few days. Or even a week or two. Don't think
"Oh, I just applied something that I know will clash, so let's merge
it now". Let the code calm down, and make sure it's all done. And
never _ever_ merge a random point. If the merge window is months away,
you'll know that there is going to be a few -rc releases still, so
there's ample time to just wait a week or so, and then do a merge like
# I know this is going to have conflicts, so I'm not even doing --no-commit
git merge v2.6.35-rc5
# edit the message to say _why_ you're merging an -rc release,
talking about resolving the conflicts early
see? If you do this, your history will suddenly make sense. There
won't be the daily merges, and the merges that _are_ there are
actually explanatory. A bad merge has turned into a good merge, and
history doesn't look ugly.
So again, I'm not saying that merges are bad. I'm saying that random
and unexplained merges are bad.
> But if you don't like that way of doing things, I will stop and just
> keep for-2.6.X+1 patches in a branch that is based off 2.6.X and never
> updated. I will try that for the next release and see how that goes.
It's worth trying. In fact, it might be worth trying having a few
different branches, especially since there has been several clearly
different development threads for the block layer. It would be
_beautiful_ if you were to send me a couple of different pull requests
for things like "fix writeback" and "update cfs", and they'd be
independent. Because I think they really _are_ independent things.
But see above. Merges per se are not evil or bad. But thoughtless
merges are bad (and quite frankly, I don't mind a _couple_ of
unnecessary merges. It's when I see the daily kind that I go "no,
there's something seriously wrong here". So none of this is about hard
rules that have to be followed 100%, it's more flexible than that).
Give it a try,
Linus
On Sat, Aug 7, 2010 at 12:34 AM, Jens Axboe <[email protected]> wrote:
>
> OK, so a question on this. Say a bug surfaces in the middle of the
> release and we push in a change to fix that at 2.6.36-rc3 time. This
> same patch will not apply directly to the branch holding 2.6.37 patches
> due to code reshuffling or whatnot. How do you want that handled? I
> can't pull in your branch and resolve it. The merge conflict may not be
> visible to you until 2.6.36 is released and I want to offload the
> patches to you, but it will be visible in linux-next pretty much
> immediately.
So I think there's a few possible answers to that.
One is the one I outlined in my previous email: merge the next -rc
tag, and explain why you merged it in the commit message. That not
only makes the merge commit message be way more informative ("Merge
commit v2.6.3x-rcy" rather than "Merge branch 'master'"), but it also
automatically acts as a "rate limiter" for the merges.
Now, that may cause problems for linux-next for a few days too, since
I think linux-next always starts from some random tree-of-the-day of
mine. That itself may be more indicative of a linux-next problem,
though. It might well make sense to base linux-next itself on the
latest tagged release rather than on some random daily thing (and if
the things that get merged _into_ linux-next then are based on a
random daily thing and bring linux-next forward, then that's a problem
with the trees getting merged - they shouldn't be doing that either).
The other possibility is for you to do throw-away merges just for
linux-next. That way _you_ do the merge (not Stephen or one of the
linux-next helpers), but the merge is going only into for-next, not
into your for-2.6.36 branch. "git rerere" will help you re-do the same
merge for future for-next trees - the same way linux-next already
generally only needs to do the merge resolution once.
Then, when you actually want to send it to me, at that point (if it's
a really complicated merge and you know it's too complex for me), you
can do one final merge into 'for-linus' before you send me the pull
request. Again, git rerere will help you re-use your previous merge
resolutions.
Or don't merge at all when you send it to me, and only do the merge if
I then reply with "ok, that's too complicated for me".
I will _never_ complain about you sending me something I can't merge.
I may throw it back at you, but I won't complain about you trying to
give me merge work. I really do like knowing about the conflicts.
Of course, if I do the merge conflict resolution I may then see
something odd and complain about it. Something I might not have even
noticed if it hadn't been pointed out to me by the conflict ;)
Linus
On 08/07/2010 05:00 PM, Linus Torvalds wrote:
> On Sat, Aug 7, 2010 at 12:31 AM, Jens Axboe <[email protected]> wrote:
>>
>> Sometimes urgent patches will sit in for-linus and then I will merge
>> those in to for-2.6.36 to save Stephen the hassle of carrying conflict
>> update patches.
>
> The merges I saw weren't even urgent, because you had never actually
> pushed them to me as such for the previous release. So both sides of
> the merge was "for-2.3.36" as far as I can tell. That's what makes me
> think there's some odd duplication of branches with no point (except
> to make the history look ugly).
No, the for-2.6.36 -> for-next was nothing urgent, just me being too
eager updating for-next. Those should never have been merges.
> Now, don't get me wrong - I _like_ branches. I love it when people
> maintain different branches for different features, and then merge
> them together in order for me to pull them (or just ask me to pull
> multiple branches). See for example merge commit 415cb47998 that
> Pekka did to create one single thing for me to pull when he had
> maintained four different topic branches. Or see the x86 "tip" pulls
> from Ingo, Thomas and Peter as an example of just asking me to pull
> multiple branches separately.
I will likely take the same approach that Pekka did, at least if there
are merge issues before sending it out.
[snip lots of good info]
>> But if you don't like that way of doing things, I will stop and just
>> keep for-2.6.X+1 patches in a branch that is based off 2.6.X and never
>> updated. I will try that for the next release and see how that goes.
>
> It's worth trying. In fact, it might be worth trying having a few
> different branches, especially since there has been several clearly
> different development threads for the block layer. It would be
> _beautiful_ if you were to send me a couple of different pull requests
> for things like "fix writeback" and "update cfs", and they'd be
> independent. Because I think they really _are_ independent things.
Yes, there tends to be several indepent topics of development in there.
Core block bits, io scheduler, writeback, splice, driver updates, etc. I
will clearly separate them. I think the biggest part of the problem is
that it earlier was just core block and IO scheduler, but it evolved
into something more. And my current approach just doesn't work for that,
since it's gotten progressively worse over the last 2-3 releases.
> But see above. Merges per se are not evil or bad. But thoughtless
> merges are bad (and quite frankly, I don't mind a _couple_ of
> unnecessary merges. It's when I see the daily kind that I go "no,
> there's something seriously wrong here". So none of this is about hard
> rules that have to be followed 100%, it's more flexible than that).
>
> Give it a try,
Will do :-)
--
Jens Axboe
Confidentiality Notice: This e-mail message, its contents and any attachments to it are confidential to the intended recipient, and may contain information that is privileged and/or exempt from disclosure under applicable law. If you are not the intended recipient, please immediately notify the sender and destroy the original e-mail message and any attachments (and any copies that may have been made) from your system or otherwise. Any unauthorized use, copying, disclosure or distribution of this information is strictly prohibited.
On 08/07/2010 05:15 PM, Linus Torvalds wrote:
> On Sat, Aug 7, 2010 at 12:34 AM, Jens Axboe <[email protected]> wrote:
>>
>> OK, so a question on this. Say a bug surfaces in the middle of the
>> release and we push in a change to fix that at 2.6.36-rc3 time. This
>> same patch will not apply directly to the branch holding 2.6.37 patches
>> due to code reshuffling or whatnot. How do you want that handled? I
>> can't pull in your branch and resolve it. The merge conflict may not be
>> visible to you until 2.6.36 is released and I want to offload the
>> patches to you, but it will be visible in linux-next pretty much
>> immediately.
>
> So I think there's a few possible answers to that.
>
> One is the one I outlined in my previous email: merge the next -rc
> tag, and explain why you merged it in the commit message. That not
> only makes the merge commit message be way more informative ("Merge
> commit v2.6.3x-rcy" rather than "Merge branch 'master'"), but it also
> automatically acts as a "rate limiter" for the merges.
>
> Now, that may cause problems for linux-next for a few days too, since
> I think linux-next always starts from some random tree-of-the-day of
> mine. That itself may be more indicative of a linux-next problem,
> though. It might well make sense to base linux-next itself on the
> latest tagged release rather than on some random daily thing (and if
> the things that get merged _into_ linux-next then are based on a
> random daily thing and bring linux-next forward, then that's a problem
> with the trees getting merged - they shouldn't be doing that either).
>
> The other possibility is for you to do throw-away merges just for
> linux-next. That way _you_ do the merge (not Stephen or one of the
> linux-next helpers), but the merge is going only into for-next, not
> into your for-2.6.36 branch. "git rerere" will help you re-do the same
> merge for future for-next trees - the same way linux-next already
> generally only needs to do the merge resolution once.
I can easily do it for for-next, merge things together there, test, and
then push it out. That will save Stephen the hassle. I think that basing
linux-next off the latest -rc is a good idea, instead of it being random
snap of the day. That should make his job easier as well.
I like the idea of just keeping the for-2.6.X branch based off 2.6.X-1
the whole cycle, since it means that downstream folks never get anything
when pulling my tree that wasn't done or merged there.
> Then, when you actually want to send it to me, at that point (if it's
> a really complicated merge and you know it's too complex for me), you
> can do one final merge into 'for-linus' before you send me the pull
> request. Again, git rerere will help you re-use your previous merge
> resolutions.
>
> Or don't merge at all when you send it to me, and only do the merge if
> I then reply with "ok, that's too complicated for me".
>
> I will _never_ complain about you sending me something I can't merge.
> I may throw it back at you, but I won't complain about you trying to
> give me merge work. I really do like knowing about the conflicts.
>
> Of course, if I do the merge conflict resolution I may then see
> something odd and complain about it. Something I might not have even
> noticed if it hadn't been pointed out to me by the conflict ;)
On the merges, I just prefer to do them myself since I think I'm in the
best possible position to do it. Plus the non-easy merges tend to be
interesting to do, at least it's a less mundane part of assembling
peoples bits and pieces together.
--
Jens Axboe
Confidentiality Notice: This e-mail message, its contents and any attachments to it are confidential to the intended recipient, and may contain information that is privileged and/or exempt from disclosure under applicable law. If you are not the intended recipient, please immediately notify the sender and destroy the original e-mail message and any attachments (and any copies that may have been made) from your system or otherwise. Any unauthorized use, copying, disclosure or distribution of this information is strictly prohibited.
On Sat, 2010-08-07 at 14:15 -0700, Linus Torvalds wrote:
>
> It might well make sense to base linux-next itself on the
> latest tagged release rather than on some random daily thing (and if
> the things that get merged _into_ linux-next then are based on a
> random daily thing and bring linux-next forward, then that's a problem
> with the trees getting merged - they shouldn't be doing that either).
Hm? I do that whenever you pull from me -- I'll pull the merge commit
into my own tree. So the mtd-2.6.git tree you pulled last night, for
example, was based on some "random daily thing" half-way between 2.6.35
and 2.6.26-rc1, which happened to be entitled 'Merge git://...mtd-2.6'.
I try to avoid having to *merge* such a thing, but it certainly does end
up in linux-next as the *base* for maintainers' tree. It's difficult to
see how that could be avoided.
--
David Woodhouse Open Source Technology Centre
[email protected] Intel Corporation