Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) client-ip=23.128.96.34;
From:   John Garry <john.g.garry@oracle.com>
To:     axboe@kernel.dk, kbusch@kernel.org, hch@lst.de, sagi@grimberg.me,
        jejb@linux.ibm.com, martin.petersen@oracle.com, djwong@kernel.org,
        viro@zeniv.linux.org.uk, brauner@kernel.org, dchinner@redhat.com,
        jack@suse.cz
Cc:     linux-block@vger.kernel.org, linux-kernel@vger.kernel.org,
        linux-nvme@lists.infradead.org, linux-xfs@vger.kernel.org,
        linux-fsdevel@vger.kernel.org, tytso@mit.edu, jbongio@google.com,
        linux-scsi@vger.kernel.org, ming.lei@redhat.com,
        jaswin@linux.ibm.com, bvanassche@acm.org,
        John Garry <john.g.garry@oracle.com>
Subject: [PATCH v2 00/16] block atomic writes
Date:   Tue, 12 Dec 2023 11:08:28 +0000
Message-Id: <20231212110844.19698-1-john.g.garry@oracle.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?0QZVgZrw9rBjdsidp+VWKHx0Z0mnzfBvEGCfLhCV01iCa/S0sd0UR/QkhvoW?=
 =?us-ascii?Q?62QFYRrD0tmGaUy4WjXLN83o4GYL/1iAcDHw2NOKHW6/cImgQkN6YEDspP6c?=
 =?us-ascii?Q?m0ELdk/qx4dt0TJnOnBmgH1cGf2U0/wz58wtJ2fvcB5RFdXAYMXtWO9zjs0K?=
 =?us-ascii?Q?xJ5kzBNae1uTD7XAx/R1T0gvDOJesj6RpjPfXyEp09bPt+b50tPHd0mg7+9W?=
 =?us-ascii?Q?j6rXTITfXyCpW6WOiMf38KWtZiox1/YyTibtobN2Kcf+t2tM012prNgZ8XQw?=
 =?us-ascii?Q?5gmMDF+Hv7yyTBuuPX70JZQQl4jLFmuPsV7gRHnilC0oALZ1etZ7yoKMNmdN?=
 =?us-ascii?Q?jA6DscLkHfNTPw+Gi4yzmg0q6RnJP1i/JQd8e3XXbIhCSI07ax2/E+bUgVLq?=
 =?us-ascii?Q?80rUclW0CJVw3H1l/cS+5hva5FMMNxKjTl6PxNrdrDld3z2Rf2JMqUTawSee?=
 =?us-ascii?Q?2Uw6ckAM0mFfHWBbtBRIqlq/VZvrmIKVDzEROnzOhaSBwMg5y95x4JT6rmIq?=
 =?us-ascii?Q?oBMfnr3ONXPaWRTOFWLD2jhgEXLPuNztznrycxxdULYTCwUfZ0g3p6SY0LrL?=
 =?us-ascii?Q?OhpagmAcIAPVOLGKunPiIlvugqeBAnSjbC57oCVuICNej+zQHl3jcHwKIZOy?=
 =?us-ascii?Q?fPU5nig/adzRwH7ArlfsKzJQp5MXa2DlNrEeM8GDRs23boIduUeHi+DJH167?=
 =?us-ascii?Q?rMkUwi32o9Oh6SWFLZzOkbhv8tHTB1Yso/B7S3OXFaKV+7RzM/J/CIaTFPaZ?=
 =?us-ascii?Q?qPA5JMoA7dvQcDe9JKtxIX47gJExyIqQUZNXiDxHlKdlQwQSuTIewRwcg2QA?=
 =?us-ascii?Q?B+2MZEgibzEhq4/oSSfq9wQ+zgtYfKVOK4Jf7LnvvhWzVfFPEUSjS5ejI0b6?=
 =?us-ascii?Q?6km8ahUItOLkz39QAQtDrv74EyGgOAxrWmMgicSqmhx6lEHkOROBDe7d1C95?=
 =?us-ascii?Q?ccWY+2kHqu9Hw4r7XLv1QSgnHcMgy9Bc9Og1qcTs7B4QMdFqDWnv8OhQ7WLg?=
 =?us-ascii?Q?uM//0HMhou6htbF4F+dUeaaYlRXjFMEqg0rjDWPmc2K/hnYrAuHxVV5KJBY1?=
 =?us-ascii?Q?ByhMtlFiOd9VTFQi9Q4kO5Amfrtz/vL1vNnYirJYPDGBo4rglwQft3ZKCuKk?=
 =?us-ascii?Q?D5snCOZAkvcVndEha2HfZUhYSmYZRivo6Nq22nk7v+u/Ib0VTTnXyUzk8s9f?=
 =?us-ascii?Q?ZDgWo8h4YWcSq82eFld2IeJ1trKAKTUSVKyGA49QXY4dBI81taPdq+HQUOLz?=
 =?us-ascii?Q?Y2EYRHe4eOFUwtzCIzmEB/nJCkllNosVLpq5l/K4BrQ12i7r4YcAsTg5WqFi?=
 =?us-ascii?Q?Uv2GZ8Yul//4SiENvAfxdWo/3Le35vVtEr4MZsr4vYhPq5YloUhmtF006Ql3?=
 =?us-ascii?Q?BSUahR7XFET7RAdACcqmXD9V+HEHU6F6tL09zLvtni0peoprEwZYiiH4nWLW?=
 =?us-ascii?Q?ao6BUY4dlBRqhXLtyb1pqby+gzbQ7akzMPRNyyEE4kq9KUB63I/QhY44l7Fw?=
 =?us-ascii?Q?9J4ImHzHxN32XQvZhe/O6/PYhRfv0olAPjxSfbEHIOS6QYdlWNoULv3Epgk8?=
 =?us-ascii?Q?zpBVF6g9U7nBxcpsDBrDy8h7voG51RzBPcz/mpizcLrQHLxSPaQ7pmBHszu+?=
 =?us-ascii?Q?lA=3D=3D?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: rRXR/+UJJfhSLy2aatk12HvFltrRSa41L0Q9FQADCRkOoABpVfchJb7z8LNlTV1m7wz8xGnzPA4FDsELLXBdDIu17FZh3zTFS9FLf26+8CU7lBMS0ZN8K71EbYtoTmf4qsaxMLUgsHGBdUO39alYszLqEFBRhmhmoaKdiuV7UTFJlPEUihsCtLCUBAKCwzyQvKo4+2KHd1yb0eCrQCc/RunR/wjQcHaxYZMV3uST8TLsxuRsnWdLMcmo+wOpu/vUKlGMUXBKgUxcaMRBLexN3STw9VWXmHJOKNVMTzlXOO0LGZFgiYSC/Nok1Sp0MKq2yDALujxHZWf1w1cyUkCjZMnV7PT8Sxv7EazIoVQz8AzDEYwAgBWbdvI+bosOkWQNhxSTm+/c05467vxvGyY4y2/tzobbzzyvK9b6OqL2cWbVrnvB2IHOrggDIKL1r/qCGrHIUt+LIE7c+uLOXBxVDSEpTUC3XgE00sXIvYXJJn0o0dTKcr/872+R6xxZthQ/APe4EoaBhp5x04xZ85qUr6ZwtFYwX44DoPWntpc9z7v8n5mkzqQgo99qyYA4NvA+Xl6czM2x52XiZBYXn2rPWvnr7w7nYWk+Flp9oKwaiMc=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 2b644e75-064f-48d2-3a6a-08dbfb02c699
X-MS-Exchange-CrossTenant-AuthSource: DM6PR10MB4313.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Dec 2023 11:09:14.4726
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: fk7wU95Ct1MUKPgDBNRrPpHnf7x9qwdW5k+jKHrIISjQPzcW5G4zBzUChbQhmdcbtmUd8xG3xAvhE0KHrLUuLA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR10MB6663
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2023-12-12_04,2023-12-12_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 bulkscore=0 spamscore=0
 malwarescore=0 mlxscore=0 mlxlogscore=999 adultscore=0 suspectscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000
 definitions=main-2312120089
X-Proofpoint-GUID: jAd-7shv0zqSgIEqreW_MhyxQ00SoQGU
X-Proofpoint-ORIG-GUID: jAd-7shv0zqSgIEqreW_MhyxQ00SoQGU
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID,
	DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,
	SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Tue, 12 Dec 2023 03:10:22 -0800 (PST)

This series introduces a proposal to implementing atomic writes in the
kernel for torn-write protection.

This series takes the approach of adding a new "atomic" flag to each of
pwritev2() and iocb->ki_flags - RWF_ATOMIC and IOCB_ATOMIC, respectively.
When set, these indicate that we want the write issued "atomically".

Only direct IO is supported and for block devices here. For this, atomic
write HW is required, like SCSI ATOMIC WRITE (16). For now, XFS support
from earlier versions is sidelined. That is until the interface is agreed
which does not fully rely on HW offload support.

man pages update has been posted at:
https://lore.kernel.org/linux-api/20230929093717.2972367-1-john.g.garry@oracle.com/T/#t
(not updated since I posted v1 kernel series)

The goal here is to provide an interface that allows applications use
application-specific block sizes larger than logical block size
reported by the storage device or larger than filesystem block size as
reported by stat().

With this new interface, application blocks will never be torn or
fractured when written. For a power fail, for each individual application
block, all or none of the data to be written. A racing atomic write and
read will mean that the read sees all the old data or all the new data,
but never a mix of old and new.

Two new fields are added to struct statx - atomic_write_unit_min and
atomic_write_unit_max. For each atomic individual write, the total length
of a write must be a between atomic_write_unit_min and
atomic_write_unit_max, inclusive, and a power-of-2. The write must also be
at a natural offset in the file wrt the write length.

SCSI sd.c and scsi_debug and NVMe kernel support is added.

Some open questions:
- How to make API extensible for when we have no HW support? In that case,
  we would prob not have to follow rule of power-of-2 length et al.
  As a possible solution, maybe we can say that atomic writes are
  supported for the file via statx, but not set unit_min and max values,
  and this means that writes need to be just FS block aligned there.
- For block layer, should atomic_write_unit_max be limited by
  max_sectors_kb? Currently it is not.
- How to improve requirement that iovecs are PAGE-aligned.
  There are 2x issues:
  a. We impose this rule to not split BIOs due to virt boundary for
     NVMe, but there virt boundary is 4K (and not PAGE size, so broken for
     16K/64K pages). Easy solution is to impose requirement that iovecs
     are 4K-aligned.
  b. We don't enforce this rule for virt boundary == 0, i.e. SCSI
- Since debugging torn-writes due to unwanted kernel BIO splitting/merging
  would be horrible, should we add some kernel storage stack software
  integrity checks?

This series is based on v6.7-rc5.

Changes since v1:
- Drop XFS support for now
- Tidy NVMe changes and also add checks for atomic write violating max
  AW PF length and boundary (if any)
- Reject - instead of ignoring - RWF_ATOMIC for files which do not
  support atomic writes
- Update block sysfs documentation
- Various tidy-ups

Alan Adamson (2):
  nvme: Support atomic writes
  nvme: Ensure atomic writes will be executed atomically

Himanshu Madhani (2):
  block: Add atomic write operations to request_queue limits
  block: Add REQ_ATOMIC flag

John Garry (10):
  block: Limit atomic writes according to bio and queue limits
  fs: Increase fmode_t size
  block: Pass blk_queue_get_max_sectors() a request pointer
  block: Limit atomic write IO size according to
    atomic_write_max_sectors
  block: Error an attempt to split an atomic write bio
  block: Add checks to merging of atomic writes
  block: Add fops atomic write support
  scsi: sd: Support reading atomic write properties from block limits
    VPD
  scsi: sd: Add WRITE_ATOMIC_16 support
  scsi: scsi_debug: Atomic write support

Prasad Singamsetty (2):
  fs/bdev: Add atomic write support info to statx
  fs: Add RWF_ATOMIC and IOCB_ATOMIC flags for atomic write support

 Documentation/ABI/stable/sysfs-block |  47 +++
 block/bdev.c                         |  31 +-
 block/blk-merge.c                    |  95 ++++-
 block/blk-mq.c                       |   2 +-
 block/blk-settings.c                 |  84 ++++
 block/blk-sysfs.c                    |  33 ++
 block/blk.h                          |   9 +-
 block/fops.c                         |  40 +-
 drivers/dma-buf/dma-buf.c            |   2 +-
 drivers/nvme/host/core.c             | 108 ++++-
 drivers/nvme/host/nvme.h             |   2 +
 drivers/scsi/scsi_debug.c            | 590 +++++++++++++++++++++------
 drivers/scsi/scsi_trace.c            |  22 +
 drivers/scsi/sd.c                    |  93 ++++-
 drivers/scsi/sd.h                    |   8 +
 fs/stat.c                            |  44 +-
 include/linux/blk_types.h            |   2 +
 include/linux/blkdev.h               |  41 +-
 include/linux/fs.h                   |  11 +
 include/linux/stat.h                 |   2 +
 include/linux/types.h                |   2 +-
 include/scsi/scsi_proto.h            |   1 +
 include/trace/events/scsi.h          |   1 +
 include/uapi/linux/fs.h              |   5 +-
 include/uapi/linux/stat.h            |   7 +-
 25 files changed, 1098 insertions(+), 184 deletions(-)

-- 
2.35.3