2022-11-10 02:05:33

by Zhang Yi

Subject: [PATCH v2 00/12] ext4: enhance simulate fail facility

Changes since v1:
- Fix format error in ext4_fault_ops_write().

Now we can test ext4's reliability by simulating fail facility introduced
in commit 46f870d690fe ("ext4: simulate various I/O and checksum errors
when reading metadata"), it can simulate checksum error or I/O error
when reading metadata from disk. But it is functional limited, it cannot
set failure times, probability, filters, etc. Fortunately, we already
have common fault-injection frame in Linux, so above limitation could be
easily supplied by using it in ext4. This patch set add ext4
fault-injection facility to replace the old frame, supply some kinds of
checksum error and I/O error, and also add group, inode, physical
block and inode logical block filters. After this patch set, we could
inject failure more precisly. The facility could be used to do fuzz
stress test include random errors, and it also could be used to
reprodece issues more conveniently.

Patch 1: add debugfs for preparing.
Patch 2: introduce the fault-injection frame for ext4.
Patch 3-11: add various kinds of faults and also do some cleanup.
Patch 12: remove the old simulating facility.

It provides a debugfs interface in ext4/<disk>/fault_inject, besides the
common config interfaces, we give 6 more.
- available_faults: present available faults we can inject.
- inject_faults: set faults, can set multiple at a time.
- inject_inode: set the inode filter, matches all inodes if not set.
- inject_group: set the block group filter, similar to inject_inode.
- inject_logical_block: set the logical block filter for one inode.
- inject_physical_block: set the physical block filter.

Current we add 20 available faults list below, include 8 kinds of
metadata checksum error, 7 metadata I/O error and 5 journal error.
After we have this facility, more other faults could be added easily
in the future.
- group_desc_checksum
- inode_bitmap_checksum
- block_bitmap_checksum
- inode_checksum
- extent_block_checksum
- dir_block_checksum
- dir_index_block_checksum
- xattr_block_checksum
- inode_bitmap_eio
- block_bitmap_eio
- inode_eio
- extent_block_eio
- dir_block_eio
- xattr_block_eio
- symlink_block_eio
- journal_start
- journal_start_sb
- journal_get_create_access
- journal_get_write_access
- journal_dirty_metadata

For example: inject inode metadata checksum error on file 'foo'.

$ mkfs.ext4 -F /dev/pmem0
$ mount /dev/pmem0 /mnt
$ mkdir /mnt/dir
$ touch /mnt/dir/foo
$ ls -i /mnt/dir/foo
262146 /mnt/foo

$ echo 100 > /sys/kernel/debug/ext4/pmem0/fault_inject/probability
$ echo 1 > /sys/kernel/debug/ext4/pmem0/fault_inject/times
$ echo 262146 > /sys/kernel/debug/ext4/pmem0/fault_inject/inject_inode
$ echo inode_checksum > /sys/kernel/debug/ext4/pmem0/fault_inject/inject_faults
$ echo 1 > /sys/kernel/debug/ext4/pmem0/fault_inject/enable
$ echo 3 > /proc/sys/vm/drop_caches ##drop cache
$ stat /mnt/dir/foo
stat: cannot statx '/mnt/dir/foo': Bad message

The kmesg print the injection location.

[ 461.433817] FAULT_INJECTION: forcing a failure.
[ 461.433817] name fault_inject, interval 1, probability 100, space 0, times 1
...
[ 461.438609] Call Trace:
[ 461.438875] <TASK>
[ 461.439116] ? dump_stack_lvl+0x73/0xa3
[ 461.439534] ? dump_stack+0x13/0x1f
[ 461.439909] ? should_fail.cold+0x4a/0x57
[ 461.440346] ? ext4_should_fail.cold+0x11f/0x135
[ 461.440833] ? __ext4_iget+0x407/0x1410
[ 461.441245] ? ext4_lookup+0x1be/0x350
[ 461.441650] ? __lookup_slow+0xb9/0x1f0
[ 461.442070] ? lookup_slow+0x46/0x70
[ 461.442463] ? walk_component+0x13e/0x230
[ 461.442890] ? path_lookupat.isra.0+0x8f/0x200
[ 461.443369] ? filename_lookup+0xd6/0x240
[ 461.443798] ? vfs_statx+0xa6/0x200
[ 461.444186] ? do_statx+0x48/0xc0
[ 461.444546] ? __might_sleep+0x56/0xc0
[ 461.444950] ? should_fail_usercopy+0x19/0x30
[ 461.445424] ? strncpy_from_user+0x33/0x2a0
[ 461.445870] ? getname_flags+0x95/0x330
[ 461.446288] ? switch_fpu_return+0x27/0x1e0
[ 461.446736] ? __x64_sys_statx+0x90/0xd0
[ 461.447160] ? do_syscall_64+0x3b/0x90
[ 461.447563] ? entry_SYSCALL_64_after_hwframe+0x63/0xcd
[ 461.448122] </TASK>
[ 461.448395] EXT4-fs error (device pmem0): ext4_lookup:1840: inode #262146: comm stat: iget: checksum invalid

Thanks,
Yi.


Zhang Yi (12):
ext4: add debugfs interface
ext4: introduce fault injection facility
ext4: add several checksum fault injection
ext4: add bitmaps I/O fault injection
ext4: add inode I/O fault injection
ext4: add extent block I/O fault injection
ext4: add dirblock I/O fault injection
ext4: call ext4_xattr_get_block() when getting xattr block
ext4: add xattr block I/O fault injection
ext4: add symlink block I/O fault injection
ext4: add journal related fault injection
ext4: remove simulate fail facility

fs/ext4/Kconfig | 9 ++
fs/ext4/balloc.c | 14 ++-
fs/ext4/bitmap.c | 4 +
fs/ext4/dir.c | 3 +
fs/ext4/ext4.h | 181 +++++++++++++++++++++++++++++--------
fs/ext4/ext4_jbd2.c | 22 +++--
fs/ext4/ext4_jbd2.h | 5 +
fs/ext4/extents.c | 7 ++
fs/ext4/ialloc.c | 24 +++--
fs/ext4/inode.c | 26 ++++--
fs/ext4/namei.c | 14 ++-
fs/ext4/super.c | 7 +-
fs/ext4/symlink.c | 4 +
fs/ext4/sysfs.c | 183 +++++++++++++++++++++++++++++++++++--
fs/ext4/xattr.c | 216 +++++++++++++++++++-------------------------
15 files changed, 515 insertions(+), 204 deletions(-)

--
2.31.1

2022-11-10 02:05:36

by Zhang Yi

Subject: [PATCH v2 04/12] ext4: add bitmaps I/O fault injection

Add inode and block bitmap reading I/O fault injections, we can specify
the group to inject, the reading function will return -EIO immediately
instead of submitting I/O.

Signed-off-by: Zhang Yi <[email protected]>
---
fs/ext4/balloc.c | 10 ++++++++++
fs/ext4/ext4.h | 6 ++++++
fs/ext4/ialloc.c | 20 +++++++++++++-------
fs/ext4/sysfs.c | 2 ++
4 files changed, 31 insertions(+), 7 deletions(-)

diff --git a/fs/ext4/balloc.c b/fs/ext4/balloc.c
index 8ff4b9192a9f..ff5c90f4386d 100644
--- a/fs/ext4/balloc.c
+++ b/fs/ext4/balloc.c
@@ -501,6 +501,16 @@ ext4_read_block_bitmap_nowait(struct super_block *sb, ext4_group_t block_group,
*/
set_buffer_new(bh);
trace_ext4_read_block_bitmap_load(sb, block_group, ignore_locked);
+ err = ext4_fault_block_bitmap_io(sb, block_group);
+ if (err) {
+ unlock_buffer(bh);
+ ext4_error_err(sb, -err, "Cannot read block bitmap - "
+ "block_group = %u, block_bitmap = %llu",
+ block_group, bitmap_blk);
+ ext4_mark_group_bitmap_corrupted(sb, block_group,
+ EXT4_GROUP_INFO_BBITMAP_CORRUPT);
+ goto out;
+ }
ext4_read_bh_nowait(bh, REQ_META | REQ_PRIO |
(ignore_locked ? REQ_RAHEAD : 0),
ext4_end_bitmap_read);
diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h
index 4c85cf977bea..589d901e8946 100644
--- a/fs/ext4/ext4.h
+++ b/fs/ext4/ext4.h
@@ -1518,6 +1518,9 @@ enum ext4_fault_bits {
EXT4_FAULT_DIRBLOCK_CSUM, /* directory block */
EXT4_FAULT_DIRIDX_CSUM, /* directory index block */
EXT4_FAULT_XATTR_CSUM, /* xattr block */
+ /* inject metadata IO error*/
+ EXT4_FAULT_IBITMAP_EIO, /* inode bitmap block */
+ EXT4_FAULT_BBITMAP_EIO, /* block bitmap block */
EXT4_FAULT_MAX
};

@@ -1617,6 +1620,9 @@ EXT4_FAULT_INODE_FN(DIRBLOCK_CSUM, dirblock_csum, 1)
EXT4_FAULT_INODE_FN(DIRIDX_CSUM, dirindex_csum, 1)
EXT4_FAULT_INODE_FN(XATTR_CSUM, xattr_csum, 1)

+EXT4_FAULT_GRP_FN(IBITMAP_EIO, inode_bitmap_io, -EIO)
+EXT4_FAULT_GRP_FN(BBITMAP_EIO, block_bitmap_io, -EIO)
+
/*
* fourth extended-fs super-block data in memory
*/
diff --git a/fs/ext4/ialloc.c b/fs/ext4/ialloc.c
index e9bc46684106..e299aa80a718 100644
--- a/fs/ext4/ialloc.c
+++ b/fs/ext4/ialloc.c
@@ -194,16 +194,16 @@ ext4_read_inode_bitmap(struct super_block *sb, ext4_group_t block_group)
* submit the buffer_head for reading
*/
trace_ext4_load_inode_bitmap(sb, block_group);
+ err = ext4_fault_inode_bitmap_io(sb, block_group);
+ if (err) {
+ unlock_buffer(bh);
+ goto read_err;
+ }
ext4_read_bh(bh, REQ_META | REQ_PRIO, ext4_end_bitmap_read);
ext4_simulate_fail_bh(sb, bh, EXT4_SIM_IBITMAP_EIO);
if (!buffer_uptodate(bh)) {
- put_bh(bh);
- ext4_error_err(sb, EIO, "Cannot read inode bitmap - "
- "block_group = %u, inode_bitmap = %llu",
- block_group, bitmap_blk);
- ext4_mark_group_bitmap_corrupted(sb, block_group,
- EXT4_GROUP_INFO_IBITMAP_CORRUPT);
- return ERR_PTR(-EIO);
+ err = -EIO;
+ goto read_err;
}

verify:
@@ -211,6 +211,12 @@ ext4_read_inode_bitmap(struct super_block *sb, ext4_group_t block_group)
if (err)
goto out;
return bh;
+read_err:
+ ext4_error_err(sb, -err, "Cannot read inode bitmap - "
+ "block_group = %u, inode_bitmap = %llu",
+ block_group, bitmap_blk);
+ ext4_mark_group_bitmap_corrupted(sb, block_group,
+ EXT4_GROUP_INFO_IBITMAP_CORRUPT);
out:
put_bh(bh);
return ERR_PTR(err);
diff --git a/fs/ext4/sysfs.c b/fs/ext4/sysfs.c
index 7773c5504174..b177263592ff 100644
--- a/fs/ext4/sysfs.c
+++ b/fs/ext4/sysfs.c
@@ -578,6 +578,8 @@ char *ext4_fault_names[EXT4_FAULT_MAX] = {
"dir_block_checksum", /* EXT4_FAULT_DIRBLOCK_CSUM */
"dir_index_block_checksum", /* EXT4_FAULT_DIRIDX_CSUM */
"xattr_block_checksum", /* EXT4_FAULT_XATTR_CSUM */
+ "inode_bitmap_eio", /* EXT4_FAULT_IBITMAP_EIO */
+ "block_bitmap_eio", /* EXT4_FAULT_BBITMAP_EIO */
};

static int ext4_fault_available_show(struct seq_file *m, void *v)
--
2.31.1

2022-11-10 02:05:41

by Zhang Yi

Subject: [PATCH v2 03/12] ext4: add several checksum fault injection

Add 8 checksum fault injections, include group descriptors, inode
bitmap, block bitmap, inode, extent block, directory leaf block,
directory index block and xattr block. They are visable in
"available_faults" debugfs interface, and can be set and enabled in the
"inject_faults" interface.

Signed-off-by: Zhang Yi <[email protected]>
---
fs/ext4/bitmap.c | 4 ++++
fs/ext4/ext4.h | 18 ++++++++++++++++++
fs/ext4/extents.c | 2 ++
fs/ext4/inode.c | 2 ++
fs/ext4/namei.c | 4 ++++
fs/ext4/super.c | 7 ++++---
fs/ext4/sysfs.c | 9 ++++++++-
fs/ext4/xattr.c | 15 +++++++++------
8 files changed, 51 insertions(+), 10 deletions(-)

diff --git a/fs/ext4/bitmap.c b/fs/ext4/bitmap.c
index f63e028c638c..c857cff280bb 100644
--- a/fs/ext4/bitmap.c
+++ b/fs/ext4/bitmap.c
@@ -26,6 +26,8 @@ int ext4_inode_bitmap_csum_verify(struct super_block *sb, ext4_group_t group,

if (!ext4_has_metadata_csum(sb))
return 1;
+ if (ext4_fault_inode_bitmap_csum(sb, group))
+ return 0;

provided = le16_to_cpu(gdp->bg_inode_bitmap_csum_lo);
calculated = ext4_chksum(sbi, sbi->s_csum_seed, (__u8 *)bh->b_data, sz);
@@ -65,6 +67,8 @@ int ext4_block_bitmap_csum_verify(struct super_block *sb, ext4_group_t group,

if (!ext4_has_metadata_csum(sb))
return 1;
+ if (ext4_fault_block_bitmap_csum(sb, group))
+ return 0;

provided = le16_to_cpu(gdp->bg_block_bitmap_csum_lo);
calculated = ext4_chksum(sbi, sbi->s_csum_seed, (__u8 *)bh->b_data, sz);
diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h
index 7a030b0b51c7..4c85cf977bea 100644
--- a/fs/ext4/ext4.h
+++ b/fs/ext4/ext4.h
@@ -1509,6 +1509,15 @@ struct ext4_orphan_info {
#define FAULT_NOTSET (U64_MAX)

enum ext4_fault_bits {
+ /* inject checksum error */
+ EXT4_FAULT_GRPDESC_CSUM, /* group descriptor */
+ EXT4_FAULT_IBITMAP_CSUM, /* inode bitmap block */
+ EXT4_FAULT_BBITMAP_CSUM, /* block bitmap block */
+ EXT4_FAULT_INODE_CSUM, /* inode */
+ EXT4_FAULT_EXTENT_CSUM, /* extent block */
+ EXT4_FAULT_DIRBLOCK_CSUM, /* directory block */
+ EXT4_FAULT_DIRIDX_CSUM, /* directory index block */
+ EXT4_FAULT_XATTR_CSUM, /* xattr block */
EXT4_FAULT_MAX
};

@@ -1599,6 +1608,15 @@ static inline int ext4_fault_##name(struct super_block *sb, unsigned long ino, \

#endif /* CONFIG_EXT4_FAULT_INJECTION */

+EXT4_FAULT_GRP_FN(GRPDESC_CSUM, groupdesc_csum, 1)
+EXT4_FAULT_GRP_FN(IBITMAP_CSUM, inode_bitmap_csum, 1)
+EXT4_FAULT_GRP_FN(BBITMAP_CSUM, block_bitmap_csum, 1)
+EXT4_FAULT_INODE_FN(INODE_CSUM, inode_csum, 1)
+EXT4_FAULT_INODE_FN(EXTENT_CSUM, extent_csum, 1)
+EXT4_FAULT_INODE_FN(DIRBLOCK_CSUM, dirblock_csum, 1)
+EXT4_FAULT_INODE_FN(DIRIDX_CSUM, dirindex_csum, 1)
+EXT4_FAULT_INODE_FN(XATTR_CSUM, xattr_csum, 1)
+
/*
* fourth extended-fs super-block data in memory
*/
diff --git a/fs/ext4/extents.c b/fs/ext4/extents.c
index f1956288307f..0d07e5cf4dab 100644
--- a/fs/ext4/extents.c
+++ b/fs/ext4/extents.c
@@ -65,6 +65,8 @@ static int ext4_extent_block_csum_verify(struct inode *inode,

if (!ext4_has_metadata_csum(inode->i_sb))
return 1;
+ if (ext4_fault_extent_csum(inode->i_sb, inode->i_ino))
+ return 0;

et = find_ext4_extent_tail(eh);
if (et->et_checksum != ext4_extent_block_csum(inode, eh))
diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
index 2b5ef1b64249..8bfbc8d100b4 100644
--- a/fs/ext4/inode.c
+++ b/fs/ext4/inode.c
@@ -90,6 +90,8 @@ static int ext4_inode_csum_verify(struct inode *inode, struct ext4_inode *raw,
cpu_to_le32(EXT4_OS_LINUX) ||
!ext4_has_metadata_csum(inode->i_sb))
return 1;
+ if (ext4_fault_inode_csum(inode->i_sb, inode->i_ino))
+ return 0;

provided = le16_to_cpu(raw->i_checksum_lo);
calculated = ext4_inode_csum(inode, raw, ei);
diff --git a/fs/ext4/namei.c b/fs/ext4/namei.c
index d5daaf41e1fc..4960ef9f05a0 100644
--- a/fs/ext4/namei.c
+++ b/fs/ext4/namei.c
@@ -398,6 +398,8 @@ int ext4_dirblock_csum_verify(struct inode *inode, struct buffer_head *bh)

if (!ext4_has_metadata_csum(inode->i_sb))
return 1;
+ if (ext4_fault_dirblock_csum(inode->i_sb, inode->i_ino))
+ return 0;

t = get_dirent_tail(inode, bh);
if (!t) {
@@ -493,6 +495,8 @@ static int ext4_dx_csum_verify(struct inode *inode,

if (!ext4_has_metadata_csum(inode->i_sb))
return 1;
+ if (ext4_fault_dirindex_csum(inode->i_sb, inode->i_ino))
+ return 0;

c = get_dx_countlimit(inode, dirent, &count_offset);
if (!c) {
diff --git a/fs/ext4/super.c b/fs/ext4/super.c
index 7950904fbf04..4ab2f1ad0dd4 100644
--- a/fs/ext4/super.c
+++ b/fs/ext4/super.c
@@ -3194,11 +3194,12 @@ static __le16 ext4_group_desc_csum(struct super_block *sb, __u32 block_group,
int ext4_group_desc_csum_verify(struct super_block *sb, __u32 block_group,
struct ext4_group_desc *gdp)
{
- if (ext4_has_group_desc_csum(sb) &&
- (gdp->bg_checksum != ext4_group_desc_csum(sb, block_group, gdp)))
+ if (!ext4_has_group_desc_csum(sb))
+ return 1;
+ if (ext4_fault_groupdesc_csum(sb, block_group))
return 0;

- return 1;
+ return gdp->bg_checksum == ext4_group_desc_csum(sb, block_group, gdp);
}

void ext4_group_desc_csum_set(struct super_block *sb, __u32 block_group,
diff --git a/fs/ext4/sysfs.c b/fs/ext4/sysfs.c
index a400b2164b10..7773c5504174 100644
--- a/fs/ext4/sysfs.c
+++ b/fs/ext4/sysfs.c
@@ -570,7 +570,14 @@ void ext4_unregister_sysfs(struct super_block *sb)

#ifdef CONFIG_EXT4_FAULT_INJECTION
char *ext4_fault_names[EXT4_FAULT_MAX] = {
- /* empty */
+ "group_desc_checksum", /* EXT4_FAULT_GRPDESC_CSUM */
+ "inode_bitmap_checksum", /* EXT4_FAULT_IBITMAP_CSUM */
+ "block_bitmap_checksum", /* EXT4_FAULT_BBITMAP_CSUM */
+ "inode_checksum", /* EXT4_FAULT_INODE_CSUM */
+ "extent_block_checksum", /* EXT4_FAULT_EXTENT_CSUM */
+ "dir_block_checksum", /* EXT4_FAULT_DIRBLOCK_CSUM */
+ "dir_index_block_checksum", /* EXT4_FAULT_DIRIDX_CSUM */
+ "xattr_block_checksum", /* EXT4_FAULT_XATTR_CSUM */
};

static int ext4_fault_available_show(struct seq_file *m, void *v)
diff --git a/fs/ext4/xattr.c b/fs/ext4/xattr.c
index 36d6ba7190b6..46a87ae9fdc8 100644
--- a/fs/ext4/xattr.c
+++ b/fs/ext4/xattr.c
@@ -152,14 +152,17 @@ static int ext4_xattr_block_csum_verify(struct inode *inode,
struct buffer_head *bh)
{
struct ext4_xattr_header *hdr = BHDR(bh);
- int ret = 1;
+ int ret;
+
+ if (!ext4_has_metadata_csum(inode->i_sb))
+ return 1;
+ if (ext4_fault_xattr_csum(inode->i_sb, inode->i_ino))
+ return 0;

- if (ext4_has_metadata_csum(inode->i_sb)) {
- lock_buffer(bh);
- ret = (hdr->h_checksum == ext4_xattr_block_csum(inode,
+ lock_buffer(bh);
+ ret = (hdr->h_checksum == ext4_xattr_block_csum(inode,
bh->b_blocknr, hdr));
- unlock_buffer(bh);
- }
+ unlock_buffer(bh);
return ret;
}

--
2.31.1

2022-11-10 02:05:49

by Zhang Yi

Subject: [PATCH v2 05/12] ext4: add inode I/O fault injection

Add I/O fault injection when reading raw inode from disk, we can
specify the inode to inject, __ext4_get_inode_loc() will return -EIO
immediately instead of submitting I/O, note that it doesn't handle
the readhead case.

Signed-off-by: Zhang Yi <[email protected]>
---
fs/ext4/ext4.h | 2 ++
fs/ext4/inode.c | 18 ++++++++++++------
fs/ext4/sysfs.c | 1 +
3 files changed, 15 insertions(+), 6 deletions(-)

diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h
index 589d901e8946..29a819a186f7 100644
--- a/fs/ext4/ext4.h
+++ b/fs/ext4/ext4.h
@@ -1521,6 +1521,7 @@ enum ext4_fault_bits {
/* inject metadata IO error*/
EXT4_FAULT_IBITMAP_EIO, /* inode bitmap block */
EXT4_FAULT_BBITMAP_EIO, /* block bitmap block */
+ EXT4_FAULT_INODE_EIO, /* inode */
EXT4_FAULT_MAX
};

@@ -1622,6 +1623,7 @@ EXT4_FAULT_INODE_FN(XATTR_CSUM, xattr_csum, 1)

EXT4_FAULT_GRP_FN(IBITMAP_EIO, inode_bitmap_io, -EIO)
EXT4_FAULT_GRP_FN(BBITMAP_EIO, block_bitmap_io, -EIO)
+EXT4_FAULT_INODE_FN(INODE_EIO, inode_io, -EIO)

/*
* fourth extended-fs super-block data in memory
diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
index 8bfbc8d100b4..8c611ad6dac1 100644
--- a/fs/ext4/inode.c
+++ b/fs/ext4/inode.c
@@ -4570,19 +4570,25 @@ static int __ext4_get_inode_loc(struct super_block *sb, unsigned long ino,
* Read the block from disk.
*/
trace_ext4_load_inode(sb, ino);
+ if (ext4_fault_inode_io(sb, ino)) {
+ unlock_buffer(bh);
+ blk_finish_plug(&plug);
+ goto err;
+ }
ext4_read_bh_nowait(bh, REQ_META | REQ_PRIO, NULL);
blk_finish_plug(&plug);
wait_on_buffer(bh);
ext4_simulate_fail_bh(sb, bh, EXT4_SIM_INODE_EIO);
- if (!buffer_uptodate(bh)) {
- if (ret_block)
- *ret_block = block;
- brelse(bh);
- return -EIO;
- }
+ if (!buffer_uptodate(bh))
+ goto err;
has_buffer:
iloc->bh = bh;
return 0;
+err:
+ if (ret_block)
+ *ret_block = block;
+ brelse(bh);
+ return -EIO;
}

static int __ext4_get_inode_loc_noinmem(struct inode *inode,
diff --git a/fs/ext4/sysfs.c b/fs/ext4/sysfs.c
index b177263592ff..68b2c02bb266 100644
--- a/fs/ext4/sysfs.c
+++ b/fs/ext4/sysfs.c
@@ -580,6 +580,7 @@ char *ext4_fault_names[EXT4_FAULT_MAX] = {
"xattr_block_checksum", /* EXT4_FAULT_XATTR_CSUM */
"inode_bitmap_eio", /* EXT4_FAULT_IBITMAP_EIO */
"block_bitmap_eio", /* EXT4_FAULT_BBITMAP_EIO */
+ "inode_eio", /* EXT4_FAULT_INODE_EIO */
};

static int ext4_fault_available_show(struct seq_file *m, void *v)
--
2.31.1

2022-11-10 02:09:41

by Zhang Yi

Subject: [PATCH v2 07/12] ext4: add dirblock I/O fault injection

Add directory block reading I/O fault injection, we can specify the
inode and logical block to inject. It will return -EIO immediately
instead of submitting I/O in readdir cases, but in the
__ext4_find_entry() procedure, it's hard to inject error precisely due
to the batch count reading, so it simulate error by clearing the
buffer's uptodate flag after I/O complete.

Signed-off-by: Zhang Yi <[email protected]>
---
fs/ext4/dir.c | 3 +++
fs/ext4/ext4.h | 2 ++
fs/ext4/namei.c | 4 ++++
fs/ext4/sysfs.c | 1 +
4 files changed, 10 insertions(+)

diff --git a/fs/ext4/dir.c b/fs/ext4/dir.c
index 3985f8c33f95..1cf2b89c9dcd 100644
--- a/fs/ext4/dir.c
+++ b/fs/ext4/dir.c
@@ -196,6 +196,9 @@ static int ext4_readdir(struct file *file, struct dir_context *ctx)
&file->f_ra, file,
index, 1);
file->f_ra.prev_pos = (loff_t)index << PAGE_SHIFT;
+ err = ext4_fault_dirblock_io(inode, map.m_lblk);
+ if (err)
+ goto errout;
bh = ext4_bread(NULL, inode, map.m_lblk, 0);
if (IS_ERR(bh)) {
err = PTR_ERR(bh);
diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h
index 9c1dcbed59e6..aaa3a29cd0e7 100644
--- a/fs/ext4/ext4.h
+++ b/fs/ext4/ext4.h
@@ -1523,6 +1523,7 @@ enum ext4_fault_bits {
EXT4_FAULT_BBITMAP_EIO, /* block bitmap block */
EXT4_FAULT_INODE_EIO, /* inode */
EXT4_FAULT_EXTENT_EIO, /* extent block */
+ EXT4_FAULT_DIRBLOCK_EIO, /* directory block */
EXT4_FAULT_MAX
};

@@ -1626,6 +1627,7 @@ EXT4_FAULT_GRP_FN(IBITMAP_EIO, inode_bitmap_io, -EIO)
EXT4_FAULT_GRP_FN(BBITMAP_EIO, block_bitmap_io, -EIO)
EXT4_FAULT_INODE_FN(INODE_EIO, inode_io, -EIO)
EXT4_FAULT_INODE_PBLOCK_FN(EXTENT_EIO, extent_io, -EIO)
+EXT4_FAULT_INODE_LBLOCK_FN(DIRBLOCK_EIO, dirblock_io, -EIO)

/*
* fourth extended-fs super-block data in memory
diff --git a/fs/ext4/namei.c b/fs/ext4/namei.c
index 4960ef9f05a0..fa754f1ba4a6 100644
--- a/fs/ext4/namei.c
+++ b/fs/ext4/namei.c
@@ -140,6 +140,8 @@ static struct buffer_head *__ext4_read_dirblock(struct inode *inode,

if (ext4_simulate_fail(inode->i_sb, EXT4_SIM_DIRBLOCK_EIO))
bh = ERR_PTR(-EIO);
+ else if (ext4_fault_dirblock_io(inode, block))
+ bh = ERR_PTR(-EIO);
else
bh = ext4_bread(NULL, inode, block, 0);
if (IS_ERR(bh)) {
@@ -1663,6 +1665,8 @@ static struct buffer_head *__ext4_find_entry(struct inode *dir,
if ((bh = bh_use[ra_ptr++]) == NULL)
goto next;
wait_on_buffer(bh);
+ if (ext4_fault_dirblock_io(dir, bh->b_blocknr))
+ clear_buffer_uptodate(bh);
if (!buffer_uptodate(bh)) {
EXT4_ERROR_INODE_ERR(dir, EIO,
"reading directory lblock %lu",
diff --git a/fs/ext4/sysfs.c b/fs/ext4/sysfs.c
index 871da7c8c2c6..82178c9eb5b6 100644
--- a/fs/ext4/sysfs.c
+++ b/fs/ext4/sysfs.c
@@ -582,6 +582,7 @@ char *ext4_fault_names[EXT4_FAULT_MAX] = {
"block_bitmap_eio", /* EXT4_FAULT_BBITMAP_EIO */
"inode_eio", /* EXT4_FAULT_INODE_EIO */
"extent_block_eio", /* EXT4_FAULT_EXTENT_EIO */
+ "dir_block_eio", /* EXT4_FAULT_DIRBLOCK_EIO */
};

static int ext4_fault_available_show(struct seq_file *m, void *v)
--
2.31.1

2022-11-10 02:09:41

by Zhang Yi

Subject: [PATCH v2 02/12] ext4: introduce fault injection facility

Introduce fault injection feature for ext4, it depends on the standard
fault-injection (CONFIG_FAULT_INJECTION) facility. User could test and
reinforce ext4 by introduce errors like checksum error, metadata I/O
error, journal error, etc. We could also inject precision fault by set
filters, such as group, inode, logical block of an inode, physical
block of filesystem, and so on.

This patch just add fault injection frame and 6 debugfs interfaces, does
not introduce any concrete faults, later patch will do this
step-by-step. Lists of debugfs interfaces:

- available_faults: show available faults that we can inject.
- inject_faults: set faults, can set multiple at one time.
- inject_inode: set the inode filter, matches all inodes if not set.
- inject_group: set the block group filter, similar to inject_inode.
- inject_logical_block: set the logical block filter for one inode.
- inject_physical_block: set the physical block filter for the fs.

Signed-off-by: Zhang Yi <[email protected]>
---
fs/ext4/Kconfig | 9 +++
fs/ext4/ext4.h | 98 ++++++++++++++++++++++++++++++++
fs/ext4/sysfs.c | 148 ++++++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 255 insertions(+)

diff --git a/fs/ext4/Kconfig b/fs/ext4/Kconfig
index 86699c8cab28..2c01c9b335c3 100644
--- a/fs/ext4/Kconfig
+++ b/fs/ext4/Kconfig
@@ -101,6 +101,15 @@ config EXT4_DEBUG
If you select Y here, then you will be able to turn on debugging
using dynamic debug control for mb_debug() / ext_debug() msgs.

+config EXT4_FAULT_INJECTION
+ bool "Ext4 fault injection support"
+ depends on EXT4_DEBUG && FAULT_INJECTION_DEBUG_FS
+ help
+ Enables fault injecton facility. Allow test ext4 by injecting
+ failures like checksum error, EIO, etc. The injection could be
+ filtered by block group, inode, logical block of file, pyhsical
+ block, and so on.
+
config EXT4_KUNIT_TESTS
tristate "KUnit tests for ext4" if !KUNIT_ALL_TESTS
depends on EXT4_FS && KUNIT
diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h
index 53099ffe307f..7a030b0b51c7 100644
--- a/fs/ext4/ext4.h
+++ b/fs/ext4/ext4.h
@@ -37,6 +37,7 @@
#include <linux/falloc.h>
#include <linux/percpu-rwsem.h>
#include <linux/fiemap.h>
+#include <linux/fault-inject.h>
#ifdef __KERNEL__
#include <linux/compat.h>
#endif
@@ -1504,6 +1505,100 @@ struct ext4_orphan_info {
* file blocks */
};

+#ifdef CONFIG_EXT4_FAULT_INJECTION
+#define FAULT_NOTSET (U64_MAX)
+
+enum ext4_fault_bits {
+ EXT4_FAULT_MAX
+};
+
+struct ext4_fault_attr {
+ struct fault_attr fa_attr;
+ struct dentry *fa_dir;
+ /* filter config */
+ u64 fa_group; /* group number */
+ u64 fa_ino; /* inode number */
+ u64 fa_lblock; /* logical block number */
+ u64 fa_pblock; /* pyhsical block number */
+ /* inject fault operations bitmap */
+ DECLARE_BITMAP(fail_ops, EXT4_FAULT_MAX);
+};
+
+extern void ext4_init_fault_inject(struct super_block *sb);
+extern bool ext4_should_fail(struct super_block *sb, unsigned int bit,
+ u64 group, u64 ino, u64 lblock, u64 pblock);
+
+#define EXT4_FAULT_FN(bit, name, errno) \
+static inline int ext4_fault_##name(struct super_block *sb) \
+{ \
+ bool ret = ext4_should_fail(sb, EXT4_FAULT_##bit, FAULT_NOTSET, \
+ FAULT_NOTSET, FAULT_NOTSET, FAULT_NOTSET); \
+ return (ret && errno) ? (int)errno : (int)ret; \
+}
+#define EXT4_FAULT_GRP_FN(bit, name, errno) \
+static inline int ext4_fault_##name(struct super_block *sb, ext4_group_t group) \
+{ \
+ bool ret = ext4_should_fail(sb, EXT4_FAULT_##bit, group, \
+ FAULT_NOTSET, FAULT_NOTSET, FAULT_NOTSET); \
+ return (ret && errno) ? (int)errno : (int)ret; \
+}
+#define EXT4_FAULT_INODE_FN(bit, name, errno) \
+static inline int ext4_fault_##name(struct super_block *sb, unsigned long ino) \
+{ \
+ bool ret = ext4_should_fail(sb, EXT4_FAULT_##bit, FAULT_NOTSET, \
+ ino ? : FAULT_NOTSET, FAULT_NOTSET, \
+ FAULT_NOTSET); \
+ return (ret && errno) ? (int)errno : (int)ret; \
+}
+#define EXT4_FAULT_INODE_LBLOCK_FN(bit, name, errno) \
+static inline int ext4_fault_##name(struct inode *inode, ext4_lblk_t lblock) \
+{ \
+ bool ret = ext4_should_fail(inode->i_sb, EXT4_FAULT_##bit, FAULT_NOTSET,\
+ inode->i_ino, lblock, FAULT_NOTSET); \
+ return (ret && errno) ? (int)errno : (int)ret; \
+}
+#define EXT4_FAULT_INODE_PBLOCK_FN(bit, name, errno) \
+static inline int ext4_fault_##name(struct super_block *sb, unsigned long ino, \
+ ext4_fsblk_t pblock) \
+{ \
+ bool ret = ext4_should_fail(sb, EXT4_FAULT_##bit, FAULT_NOTSET, \
+ ino ? : FAULT_NOTSET, FAULT_NOTSET, pblock);\
+ return (ret && errno) ? (int)errno : (int)ret; \
+}
+
+#else
+static inline void ext4_init_fault_inject(struct super_block *sb)
+{
+}
+#define EXT4_FAULT_FN(bit, name, errno) \
+static inline int ext4_fault_##name(struct super_block *sb) \
+{ \
+ return 0; \
+}
+#define EXT4_FAULT_GRP_FN(bit, name, errno) \
+static inline int ext4_fault_##name(struct super_block *sb, ext4_group_t group) \
+{ \
+ return 0; \
+}
+#define EXT4_FAULT_INODE_FN(bit, name, errno) \
+static inline int ext4_fault_##name(struct super_block *sb, unsigned long ino) \
+{ \
+ return 0; \
+}
+#define EXT4_FAULT_INODE_LBLOCK_FN(bit, name, errno) \
+static inline int ext4_fault_##name(struct inode *inode, ext4_lblk_t lblock) \
+{ \
+ return 0; \
+}
+#define EXT4_FAULT_INODE_PBLOCK_FN(bit, name, errno) \
+static inline int ext4_fault_##name(struct super_block *sb, unsigned long ino, \
+ ext4_fsblk_t pblock) \
+{ \
+ return 0; \
+}
+
+#endif /* CONFIG_EXT4_FAULT_INJECTION */
+
/*
* fourth extended-fs super-block data in memory
*/
@@ -1710,6 +1805,9 @@ struct ext4_sb_info {
u64 s_dax_part_off;
#ifdef CONFIG_EXT4_DEBUG
unsigned long s_simulate_fail;
+#endif
+#ifdef CONFIG_EXT4_FAULT_INJECTION
+ struct ext4_fault_attr s_fault_attr;
#endif
/* Record the errseq of the backing block device */
errseq_t s_bdev_wb_err;
diff --git a/fs/ext4/sysfs.c b/fs/ext4/sysfs.c
index f3e4049ec50e..a400b2164b10 100644
--- a/fs/ext4/sysfs.c
+++ b/fs/ext4/sysfs.c
@@ -553,6 +553,8 @@ int ext4_register_sysfs(struct super_block *sb)
}
if (ext4_debugfs_root)
sbi->s_debug = debugfs_create_dir(sb->s_id, ext4_debugfs_root);
+ if (sbi->s_debug)
+ ext4_init_fault_inject(sb);
return 0;
}

@@ -566,6 +568,152 @@ void ext4_unregister_sysfs(struct super_block *sb)
kobject_del(&sbi->s_kobj);
}

+#ifdef CONFIG_EXT4_FAULT_INJECTION
+char *ext4_fault_names[EXT4_FAULT_MAX] = {
+ /* empty */
+};
+
+static int ext4_fault_available_show(struct seq_file *m, void *v)
+{
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(ext4_fault_names); i++)
+ seq_printf(m, "%s\n", ext4_fault_names[i]);
+
+ return 0;
+}
+DEFINE_SHOW_ATTRIBUTE(ext4_fault_available);
+
+static int ext4_fault_ops_show(struct seq_file *m, void *v)
+{
+ struct super_block *sb = m->private;
+ struct ext4_fault_attr *attr = &EXT4_SB(sb)->s_fault_attr;
+ int bit = 0;
+
+ for_each_set_bit(bit, attr->fail_ops, EXT4_FAULT_MAX)
+ seq_printf(m, "%s\n", ext4_fault_names[bit]);
+
+ return 0;
+}
+
+static int ext4_fault_ops_open(struct inode *inode, struct file *file)
+{
+ struct super_block *sb = inode->i_private;
+ struct ext4_fault_attr *attr = &EXT4_SB(sb)->s_fault_attr;
+ int ret;
+
+ ret = single_open(file, ext4_fault_ops_show, sb);
+ if (ret)
+ return ret;
+
+ if (file->f_flags & O_TRUNC)
+ bitmap_zero(attr->fail_ops, EXT4_FAULT_MAX);
+ return ret;
+}
+
+static int ext4_fault_ops_release(struct inode *inode, struct file *file)
+{
+ return single_release(inode, file);
+}
+
+static ssize_t ext4_fault_ops_write(struct file *file, const char __user *buffer,
+ size_t count, loff_t *ppos)
+{
+ struct seq_file *m = file->private_data;
+ struct super_block *sb = m->private;
+ struct ext4_fault_attr *attr = &EXT4_SB(sb)->s_fault_attr;
+ char fault_buf[32] = { };
+ char *fault_op;
+ int i;
+
+ if (count >= sizeof(fault_buf)) {
+ ext4_msg(sb, KERN_ERR, "fault operation too long %zu", count);
+ return -EINVAL;
+ }
+ if (copy_from_user(fault_buf, buffer, count))
+ return -EFAULT;
+
+ fault_op = strstrip(fault_buf);
+ for (i = 0; i < ARRAY_SIZE(ext4_fault_names); i++) {
+ if (!strcmp(fault_op, ext4_fault_names[i])) {
+ __set_bit(i, attr->fail_ops);
+ break;
+ }
+ }
+ *ppos += count;
+ return count;
+}
+
+static const struct file_operations ext4_fault_ops_fops = {
+ .open = ext4_fault_ops_open,
+ .read = seq_read,
+ .write = ext4_fault_ops_write,
+ .llseek = seq_lseek,
+ .release = ext4_fault_ops_release,
+};
+
+
+/*
+ * Inject fault injection for one operation, it could be filtered by the
+ * group, inode, logical block and physical block. Return true if we should
+ * inject fault.
+ */
+bool ext4_should_fail(struct super_block *sb, unsigned int bit,
+ u64 group, u64 ino, u64 lblock, u64 pblock)
+{
+ struct ext4_sb_info *sbi = EXT4_SB(sb);
+ struct ext4_fault_attr *attr = &sbi->s_fault_attr;
+
+ if (!test_bit(bit, attr->fail_ops))
+ return false;
+
+#define EXT4_FAIL_FILTER_MATCH(conf, check) \
+ ((conf == FAULT_NOTSET) || (check == FAULT_NOTSET) || (conf == check))
+
+ if (!EXT4_FAIL_FILTER_MATCH(attr->fa_group, group))
+ return false;
+ if (!EXT4_FAIL_FILTER_MATCH(attr->fa_ino, ino))
+ return false;
+ if (!EXT4_FAIL_FILTER_MATCH(attr->fa_lblock, lblock))
+ return false;
+ if (!EXT4_FAIL_FILTER_MATCH(attr->fa_pblock, pblock))
+ return false;
+
+ return should_fail(&attr->fa_attr, 1);
+}
+
+void ext4_init_fault_inject(struct super_block *sb)
+{
+ struct ext4_sb_info *sbi = EXT4_SB(sb);
+ struct ext4_fault_attr *attr = &sbi->s_fault_attr;
+ struct dentry *parent = sbi->s_debug;
+ struct dentry *dir;
+
+ attr->fa_attr = (struct fault_attr) FAULT_ATTR_INITIALIZER;
+ attr->fa_ino = FAULT_NOTSET;
+ attr->fa_group = FAULT_NOTSET;
+ attr->fa_lblock = FAULT_NOTSET;
+ attr->fa_pblock = FAULT_NOTSET;
+ memset(attr->fail_ops, 0, sizeof(attr->fail_ops));
+
+ dir = fault_create_debugfs_attr("fault_inject", parent, &attr->fa_attr);
+ if (IS_ERR(dir)) {
+ ext4_msg(sb, KERN_ERR, "failed to initialize fault_injection %ld",
+ PTR_ERR(dir));
+ return;
+ }
+ attr->fa_dir = dir;
+ debugfs_create_file("available_faults", 0400, dir, sb,
+ &ext4_fault_available_fops);
+ debugfs_create_file("inject_faults", 0600, dir, sb,
+ &ext4_fault_ops_fops);
+ debugfs_create_x64("inject_inode", 0600, dir, &attr->fa_ino);
+ debugfs_create_x64("inject_group", 0600, dir, &attr->fa_group);
+ debugfs_create_x64("inject_logical_block", 0600, dir, &attr->fa_lblock);
+ debugfs_create_x64("inject_physical_block", 0600, dir, &attr->fa_pblock);
+}
+#endif /* CONFIG_EXT4_FAULT_INJECTION */
+
int __init ext4_init_sysfs(void)
{
int ret;
--
2.31.1

2022-11-10 02:26:29

by Zhang Yi

Subject: [PATCH v2 08/12] ext4: call ext4_xattr_get_block() when getting xattr block

We currently open code reading xattr block and checking valid in many
places where getting xattr block, but we already have a helper function
ext4_xattr_get_block(), use this helper can unify all of the getting
xattr block procedure and make them more clean-up.

Signed-off-by: Zhang Yi <[email protected]>
---
fs/ext4/xattr.c | 197 ++++++++++++++++++++----------------------------
1 file changed, 80 insertions(+), 117 deletions(-)

diff --git a/fs/ext4/xattr.c b/fs/ext4/xattr.c
index 46a87ae9fdc8..39c80565c65d 100644
--- a/fs/ext4/xattr.c
+++ b/fs/ext4/xattr.c
@@ -74,6 +74,7 @@
# define ea_bdebug(bh, fmt, ...) no_printk(fmt, ##__VA_ARGS__)
#endif

+static struct buffer_head *ext4_xattr_get_block(struct inode *);
static void ext4_xattr_block_cache_insert(struct mb_cache *,
struct buffer_head *);
static struct buffer_head *
@@ -542,18 +543,11 @@ ext4_xattr_block_get(struct inode *inode, int name_index, const char *name,
ea_idebug(inode, "name=%d.%s, buffer=%p, buffer_size=%ld",
name_index, name, buffer, (long)buffer_size);

- if (!EXT4_I(inode)->i_file_acl)
+ bh = ext4_xattr_get_block(inode);
+ if (!bh)
return -ENODATA;
- ea_idebug(inode, "reading block %llu",
- (unsigned long long)EXT4_I(inode)->i_file_acl);
- bh = ext4_sb_bread(inode->i_sb, EXT4_I(inode)->i_file_acl, REQ_PRIO);
if (IS_ERR(bh))
return PTR_ERR(bh);
- ea_bdebug(bh, "b_count=%d, refcount=%d",
- atomic_read(&(bh->b_count)), le32_to_cpu(BHDR(bh)->h_refcount));
- error = ext4_xattr_check_block(inode, bh);
- if (error)
- goto cleanup;
ext4_xattr_block_cache_insert(ea_block_cache, bh);
entry = BFIRST(bh);
end = bh->b_data + bh->b_size;
@@ -715,22 +709,13 @@ ext4_xattr_block_list(struct dentry *dentry, char *buffer, size_t buffer_size)
ea_idebug(inode, "buffer=%p, buffer_size=%ld",
buffer, (long)buffer_size);

- if (!EXT4_I(inode)->i_file_acl)
- return 0;
- ea_idebug(inode, "reading block %llu",
- (unsigned long long)EXT4_I(inode)->i_file_acl);
- bh = ext4_sb_bread(inode->i_sb, EXT4_I(inode)->i_file_acl, REQ_PRIO);
- if (IS_ERR(bh))
+ bh = ext4_xattr_get_block(inode);
+ if (!bh || IS_ERR(bh))
return PTR_ERR(bh);
- ea_bdebug(bh, "b_count=%d, refcount=%d",
- atomic_read(&(bh->b_count)), le32_to_cpu(BHDR(bh)->h_refcount));
- error = ext4_xattr_check_block(inode, bh);
- if (error)
- goto cleanup;
+
ext4_xattr_block_cache_insert(EA_BLOCK_CACHE(inode), bh);
error = ext4_xattr_list_entries(dentry, BFIRST(bh), buffer,
buffer_size);
-cleanup:
brelse(bh);
return error;
}
@@ -849,18 +834,13 @@ int ext4_get_inode_usage(struct inode *inode, qsize_t *usage)
ea_inode_refs++;
}

- if (EXT4_I(inode)->i_file_acl) {
- bh = ext4_sb_bread(inode->i_sb, EXT4_I(inode)->i_file_acl, REQ_PRIO);
- if (IS_ERR(bh)) {
- ret = PTR_ERR(bh);
- bh = NULL;
- goto out;
- }
-
- ret = ext4_xattr_check_block(inode, bh);
- if (ret)
- goto out;
-
+ bh = ext4_xattr_get_block(inode);
+ if (IS_ERR(bh)) {
+ ret = PTR_ERR(bh);
+ bh = NULL;
+ goto out;
+ }
+ if (bh) {
for (entry = BFIRST(bh); !IS_LAST_ENTRY(entry);
entry = EXT4_XATTR_NEXT(entry))
if (entry->e_value_inum)
@@ -1816,37 +1796,27 @@ static int
ext4_xattr_block_find(struct inode *inode, struct ext4_xattr_info *i,
struct ext4_xattr_block_find *bs)
{
- struct super_block *sb = inode->i_sb;
+ struct buffer_head *bh;
int error;

ea_idebug(inode, "name=%d.%s, value=%p, value_len=%ld",
i->name_index, i->name, i->value, (long)i->value_len);

- if (EXT4_I(inode)->i_file_acl) {
- /* The inode already has an extended attribute block. */
- bs->bh = ext4_sb_bread(sb, EXT4_I(inode)->i_file_acl, REQ_PRIO);
- if (IS_ERR(bs->bh)) {
- error = PTR_ERR(bs->bh);
- bs->bh = NULL;
- return error;
- }
- ea_bdebug(bs->bh, "b_count=%d, refcount=%d",
- atomic_read(&(bs->bh->b_count)),
- le32_to_cpu(BHDR(bs->bh)->h_refcount));
- error = ext4_xattr_check_block(inode, bs->bh);
- if (error)
- return error;
- /* Find the named attribute. */
- bs->s.base = BHDR(bs->bh);
- bs->s.first = BFIRST(bs->bh);
- bs->s.end = bs->bh->b_data + bs->bh->b_size;
- bs->s.here = bs->s.first;
- error = xattr_find_entry(inode, &bs->s.here, bs->s.end,
- i->name_index, i->name, 1);
- if (error && error != -ENODATA)
- return error;
- bs->s.not_found = error;
- }
+ bh = ext4_xattr_get_block(inode);
+ if (!bh || IS_ERR(bh))
+ return PTR_ERR(bh);
+
+ /* Find the named attribute. */
+ bs->bh = bh;
+ bs->s.base = BHDR(bs->bh);
+ bs->s.first = BFIRST(bs->bh);
+ bs->s.end = bs->bh->b_data + bs->bh->b_size;
+ bs->s.here = bs->s.first;
+ error = xattr_find_entry(inode, &bs->s.here, bs->s.end,
+ i->name_index, i->name, 1);
+ if (error && error != -ENODATA)
+ return error;
+ bs->s.not_found = error;
return 0;
}

@@ -2260,9 +2230,15 @@ static struct buffer_head *ext4_xattr_get_block(struct inode *inode)

if (!EXT4_I(inode)->i_file_acl)
return NULL;
+
+ ea_idebug(inode, "reading block %llu",
+ (unsigned long long)EXT4_I(inode)->i_file_acl);
bh = ext4_sb_bread(inode->i_sb, EXT4_I(inode)->i_file_acl, REQ_PRIO);
if (IS_ERR(bh))
return bh;
+
+ ea_bdebug(bh, "b_count=%d, refcount=%d",
+ atomic_read(&(bh->b_count)), le32_to_cpu(BHDR(bh)->h_refcount));
error = ext4_xattr_check_block(inode, bh);
if (error) {
brelse(bh);
@@ -2703,6 +2679,7 @@ int ext4_expand_extra_isize_ea(struct inode *inode, int new_extra_isize,
int error = 0, tried_min_extra_isize = 0;
int s_min_extra_isize = le16_to_cpu(sbi->s_es->s_min_extra_isize);
int isize_diff; /* How much do we need to grow i_extra_isize */
+ struct buffer_head *bh;

retry:
isize_diff = new_extra_isize - EXT4_I(inode)->i_extra_isize;
@@ -2733,19 +2710,12 @@ int ext4_expand_extra_isize_ea(struct inode *inode, int new_extra_isize,
* Enough free space isn't available in the inode, check if
* EA block can hold new_extra_isize bytes.
*/
- if (EXT4_I(inode)->i_file_acl) {
- struct buffer_head *bh;
-
- bh = ext4_sb_bread(inode->i_sb, EXT4_I(inode)->i_file_acl, REQ_PRIO);
- if (IS_ERR(bh)) {
- error = PTR_ERR(bh);
- goto cleanup;
- }
- error = ext4_xattr_check_block(inode, bh);
- if (error) {
- brelse(bh);
- goto cleanup;
- }
+ bh = ext4_xattr_get_block(inode);
+ if (IS_ERR(bh)) {
+ error = PTR_ERR(bh);
+ goto cleanup;
+ }
+ if (bh) {
base = BHDR(bh);
end = bh->b_data + bh->b_size;
min_offs = end - base;
@@ -2892,56 +2862,49 @@ int ext4_xattr_delete_inode(handle_t *handle, struct inode *inode,
false /* skip_quota */);
}

- if (EXT4_I(inode)->i_file_acl) {
- bh = ext4_sb_bread(inode->i_sb, EXT4_I(inode)->i_file_acl, REQ_PRIO);
- if (IS_ERR(bh)) {
- error = PTR_ERR(bh);
- if (error == -EIO) {
- EXT4_ERROR_INODE_ERR(inode, EIO,
- "block %llu read error",
- EXT4_I(inode)->i_file_acl);
- }
- bh = NULL;
- goto cleanup;
+ bh = ext4_xattr_get_block(inode);
+ if (!bh || IS_ERR(bh)) {
+ error = PTR_ERR(bh);
+ bh = NULL;
+ if (error == -EIO) {
+ EXT4_ERROR_INODE_ERR(inode, EIO, "block %llu read error",
+ EXT4_I(inode)->i_file_acl);
}
- error = ext4_xattr_check_block(inode, bh);
- if (error)
- goto cleanup;
-
- if (ext4_has_feature_ea_inode(inode->i_sb)) {
- for (entry = BFIRST(bh); !IS_LAST_ENTRY(entry);
- entry = EXT4_XATTR_NEXT(entry)) {
- if (!entry->e_value_inum)
- continue;
- error = ext4_xattr_inode_iget(inode,
- le32_to_cpu(entry->e_value_inum),
- le32_to_cpu(entry->e_hash),
- &ea_inode);
- if (error)
- continue;
- ext4_xattr_inode_free_quota(inode, ea_inode,
- le32_to_cpu(entry->e_value_size));
- iput(ea_inode);
- }
+ goto cleanup;
+ }

+ if (ext4_has_feature_ea_inode(inode->i_sb)) {
+ for (entry = BFIRST(bh); !IS_LAST_ENTRY(entry);
+ entry = EXT4_XATTR_NEXT(entry)) {
+ if (!entry->e_value_inum)
+ continue;
+ error = ext4_xattr_inode_iget(inode,
+ le32_to_cpu(entry->e_value_inum),
+ le32_to_cpu(entry->e_hash),
+ &ea_inode);
+ if (error)
+ continue;
+ ext4_xattr_inode_free_quota(inode, ea_inode,
+ le32_to_cpu(entry->e_value_size));
+ iput(ea_inode);
}

- ext4_xattr_release_block(handle, inode, bh, ea_inode_array,
- extra_credits);
- /*
- * Update i_file_acl value in the same transaction that releases
- * block.
- */
- EXT4_I(inode)->i_file_acl = 0;
- error = ext4_mark_inode_dirty(handle, inode);
- if (error) {
- EXT4_ERROR_INODE(inode, "mark inode dirty (error %d)",
- error);
- goto cleanup;
- }
- ext4_fc_mark_ineligible(inode->i_sb, EXT4_FC_REASON_XATTR, handle);
}
- error = 0;
+
+ ext4_xattr_release_block(handle, inode, bh, ea_inode_array,
+ extra_credits);
+ /*
+ * Update i_file_acl value in the same transaction that releases
+ * block.
+ */
+ EXT4_I(inode)->i_file_acl = 0;
+ error = ext4_mark_inode_dirty(handle, inode);
+ if (error) {
+ EXT4_ERROR_INODE(inode, "mark inode dirty (error %d)",
+ error);
+ goto cleanup;
+ }
+ ext4_fc_mark_ineligible(inode->i_sb, EXT4_FC_REASON_XATTR, handle);
cleanup:
brelse(iloc.bh);
brelse(bh);
--
2.31.1

2022-11-10 02:27:55

by Zhang Yi

Subject: [PATCH v2 12/12] ext4: remove simulate fail facility

Now that we have fault injection support for ext4, it could replace
current simulate fail facility entirely, so this patch remove all
ext4_simulate_fail() interface and supports.

Signed-off-by: Zhang Yi <[email protected]>
---
fs/ext4/balloc.c | 4 +---
fs/ext4/ext4.h | 38 --------------------------------------
fs/ext4/ialloc.c | 4 +---
fs/ext4/inode.c | 6 ++----
fs/ext4/namei.c | 10 +++-------
fs/ext4/sysfs.c | 6 ------
6 files changed, 7 insertions(+), 61 deletions(-)

diff --git a/fs/ext4/balloc.c b/fs/ext4/balloc.c
index ff5c90f4386d..999e66d9dc45 100644
--- a/fs/ext4/balloc.c
+++ b/fs/ext4/balloc.c
@@ -384,8 +384,7 @@ static int ext4_validate_block_bitmap(struct super_block *sb,
if (buffer_verified(bh))
goto verified;
if (unlikely(!ext4_block_bitmap_csum_verify(sb, block_group,
- desc, bh) ||
- ext4_simulate_fail(sb, EXT4_SIM_BBITMAP_CRC))) {
+ desc, bh))) {
ext4_unlock_group(sb, block_group);
ext4_error(sb, "bg %u: bad block bitmap checksum", block_group);
ext4_mark_group_bitmap_corrupted(sb, block_group,
@@ -537,7 +536,6 @@ int ext4_wait_block_bitmap(struct super_block *sb, ext4_group_t block_group,
if (!desc)
return -EFSCORRUPTED;
wait_on_buffer(bh);
- ext4_simulate_fail_bh(sb, bh, EXT4_SIM_BBITMAP_EIO);
if (!buffer_uptodate(bh)) {
ext4_error_err(sb, EIO, "Cannot read block bitmap - "
"block_group = %u, block_bitmap = %llu",
diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h
index 96b805992ea5..74b5b36c39d3 100644
--- a/fs/ext4/ext4.h
+++ b/fs/ext4/ext4.h
@@ -1849,9 +1849,6 @@ struct ext4_sb_info {
struct percpu_rw_semaphore s_writepages_rwsem;
struct dax_device *s_daxdev;
u64 s_dax_part_off;
-#ifdef CONFIG_EXT4_DEBUG
- unsigned long s_simulate_fail;
-#endif
#ifdef CONFIG_EXT4_FAULT_INJECTION
struct ext4_fault_attr s_fault_attr;
#endif
@@ -1966,41 +1963,6 @@ static inline int ext4_test_mount_flag(struct super_block *sb, int bit)
return test_bit(bit, &EXT4_SB(sb)->s_mount_flags);
}

-
-/*
- * Simulate_fail codes
- */
-#define EXT4_SIM_BBITMAP_EIO 1
-#define EXT4_SIM_BBITMAP_CRC 2
-#define EXT4_SIM_IBITMAP_EIO 3
-#define EXT4_SIM_IBITMAP_CRC 4
-#define EXT4_SIM_INODE_EIO 5
-#define EXT4_SIM_INODE_CRC 6
-#define EXT4_SIM_DIRBLOCK_EIO 7
-#define EXT4_SIM_DIRBLOCK_CRC 8
-
-static inline bool ext4_simulate_fail(struct super_block *sb,
- unsigned long code)
-{
-#ifdef CONFIG_EXT4_DEBUG
- struct ext4_sb_info *sbi = EXT4_SB(sb);
-
- if (unlikely(sbi->s_simulate_fail == code)) {
- sbi->s_simulate_fail = 0;
- return true;
- }
-#endif
- return false;
-}
-
-static inline void ext4_simulate_fail_bh(struct super_block *sb,
- struct buffer_head *bh,
- unsigned long code)
-{
- if (!IS_ERR(bh) && ext4_simulate_fail(sb, code))
- clear_buffer_uptodate(bh);
-}
-
/*
* Error number codes for s_{first,last}_error_errno
*
diff --git a/fs/ext4/ialloc.c b/fs/ext4/ialloc.c
index e299aa80a718..a45f0c0aaa3a 100644
--- a/fs/ext4/ialloc.c
+++ b/fs/ext4/ialloc.c
@@ -99,8 +99,7 @@ static int ext4_validate_inode_bitmap(struct super_block *sb,
goto verified;
blk = ext4_inode_bitmap(sb, desc);
if (!ext4_inode_bitmap_csum_verify(sb, block_group, desc, bh,
- EXT4_INODES_PER_GROUP(sb) / 8) ||
- ext4_simulate_fail(sb, EXT4_SIM_IBITMAP_CRC)) {
+ EXT4_INODES_PER_GROUP(sb) / 8)) {
ext4_unlock_group(sb, block_group);
ext4_error(sb, "Corrupt inode bitmap - block_group = %u, "
"inode_bitmap = %llu", block_group, blk);
@@ -200,7 +199,6 @@ ext4_read_inode_bitmap(struct super_block *sb, ext4_group_t block_group)
goto read_err;
}
ext4_read_bh(bh, REQ_META | REQ_PRIO, ext4_end_bitmap_read);
- ext4_simulate_fail_bh(sb, bh, EXT4_SIM_IBITMAP_EIO);
if (!buffer_uptodate(bh)) {
err = -EIO;
goto read_err;
diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
index 8c611ad6dac1..20546338bc2a 100644
--- a/fs/ext4/inode.c
+++ b/fs/ext4/inode.c
@@ -4578,7 +4578,6 @@ static int __ext4_get_inode_loc(struct super_block *sb, unsigned long ino,
ext4_read_bh_nowait(bh, REQ_META | REQ_PRIO, NULL);
blk_finish_plug(&plug);
wait_on_buffer(bh);
- ext4_simulate_fail_bh(sb, bh, EXT4_SIM_INODE_EIO);
if (!buffer_uptodate(bh))
goto err;
has_buffer:
@@ -4835,9 +4834,8 @@ struct inode *__ext4_iget(struct super_block *sb, unsigned long ino,
sizeof(gen));
}

- if ((!ext4_inode_csum_verify(inode, raw_inode, ei) ||
- ext4_simulate_fail(sb, EXT4_SIM_INODE_CRC)) &&
- (!(EXT4_SB(sb)->s_mount_state & EXT4_FC_REPLAY))) {
+ if (!ext4_inode_csum_verify(inode, raw_inode, ei) &&
+ (!(EXT4_SB(sb)->s_mount_state & EXT4_FC_REPLAY))) {
ext4_error_inode_err(inode, function, line, 0,
EFSBADCRC, "iget: checksum invalid");
ret = -EFSBADCRC;
diff --git a/fs/ext4/namei.c b/fs/ext4/namei.c
index fa754f1ba4a6..e410e4c0357a 100644
--- a/fs/ext4/namei.c
+++ b/fs/ext4/namei.c
@@ -138,9 +138,7 @@ static struct buffer_head *__ext4_read_dirblock(struct inode *inode,
return ERR_PTR(-EFSCORRUPTED);
}

- if (ext4_simulate_fail(inode->i_sb, EXT4_SIM_DIRBLOCK_EIO))
- bh = ERR_PTR(-EIO);
- else if (ext4_fault_dirblock_io(inode, block))
+ if (ext4_fault_dirblock_io(inode, block))
bh = ERR_PTR(-EIO);
else
bh = ext4_bread(NULL, inode, block, 0);
@@ -187,8 +185,7 @@ static struct buffer_head *__ext4_read_dirblock(struct inode *inode,
* caller is sure it should be an index block.
*/
if (is_dx_block && type == INDEX) {
- if (ext4_dx_csum_verify(inode, dirent) &&
- !ext4_simulate_fail(inode->i_sb, EXT4_SIM_DIRBLOCK_CRC))
+ if (ext4_dx_csum_verify(inode, dirent))
set_buffer_verified(bh);
else {
ext4_error_inode_err(inode, func, line, block,
@@ -199,8 +196,7 @@ static struct buffer_head *__ext4_read_dirblock(struct inode *inode,
}
}
if (!is_dx_block) {
- if (ext4_dirblock_csum_verify(inode, bh) &&
- !ext4_simulate_fail(inode->i_sb, EXT4_SIM_DIRBLOCK_CRC))
+ if (ext4_dirblock_csum_verify(inode, bh))
set_buffer_verified(bh);
else {
ext4_error_inode_err(inode, func, line, block,
diff --git a/fs/ext4/sysfs.c b/fs/ext4/sysfs.c
index da725e128c89..11fc508de336 100644
--- a/fs/ext4/sysfs.c
+++ b/fs/ext4/sysfs.c
@@ -227,9 +227,6 @@ EXT4_RW_ATTR_SBI_UI(warning_ratelimit_interval_ms, s_warning_ratelimit_state.int
EXT4_RW_ATTR_SBI_UI(warning_ratelimit_burst, s_warning_ratelimit_state.burst);
EXT4_RW_ATTR_SBI_UI(msg_ratelimit_interval_ms, s_msg_ratelimit_state.interval);
EXT4_RW_ATTR_SBI_UI(msg_ratelimit_burst, s_msg_ratelimit_state.burst);
-#ifdef CONFIG_EXT4_DEBUG
-EXT4_RW_ATTR_SBI_UL(simulate_fail, s_simulate_fail);
-#endif
EXT4_RO_ATTR_SBI_ATOMIC(warning_count, s_warning_count);
EXT4_RO_ATTR_SBI_ATOMIC(msg_count, s_msg_count);
EXT4_RO_ATTR_ES_UI(errors_count, s_error_count);
@@ -294,9 +291,6 @@ static struct attribute *ext4_attrs[] = {
ATTR_LIST(first_error_time),
ATTR_LIST(last_error_time),
ATTR_LIST(journal_task),
-#ifdef CONFIG_EXT4_DEBUG
- ATTR_LIST(simulate_fail),
-#endif
ATTR_LIST(mb_prefetch),
ATTR_LIST(mb_prefetch_limit),
ATTR_LIST(last_trim_minblks),
--
2.31.1

2022-11-10 02:29:57

by Zhang Yi

Subject: [PATCH v2 09/12] ext4: add xattr block I/O fault injection

Add I/O fault injection when reading xattr block, we can specify the
inode to inject. ext4_xattr_get_block() will return -EIO immediately
instead of submitting I/O.

Signed-off-by: Zhang Yi <[email protected]>
---
fs/ext4/ext4.h | 2 ++
fs/ext4/sysfs.c | 1 +
fs/ext4/xattr.c | 4 ++++
3 files changed, 7 insertions(+)

diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h
index aaa3a29cd0e7..94894daef595 100644
--- a/fs/ext4/ext4.h
+++ b/fs/ext4/ext4.h
@@ -1524,6 +1524,7 @@ enum ext4_fault_bits {
EXT4_FAULT_INODE_EIO, /* inode */
EXT4_FAULT_EXTENT_EIO, /* extent block */
EXT4_FAULT_DIRBLOCK_EIO, /* directory block */
+ EXT4_FAULT_XATTR_EIO, /* xattr block */
EXT4_FAULT_MAX
};

@@ -1628,6 +1629,7 @@ EXT4_FAULT_GRP_FN(BBITMAP_EIO, block_bitmap_io, -EIO)
EXT4_FAULT_INODE_FN(INODE_EIO, inode_io, -EIO)
EXT4_FAULT_INODE_PBLOCK_FN(EXTENT_EIO, extent_io, -EIO)
EXT4_FAULT_INODE_LBLOCK_FN(DIRBLOCK_EIO, dirblock_io, -EIO)
+EXT4_FAULT_INODE_FN(XATTR_EIO, xattr_io, -EIO)

/*
* fourth extended-fs super-block data in memory
diff --git a/fs/ext4/sysfs.c b/fs/ext4/sysfs.c
index 82178c9eb5b6..842e4f60fb69 100644
--- a/fs/ext4/sysfs.c
+++ b/fs/ext4/sysfs.c
@@ -583,6 +583,7 @@ char *ext4_fault_names[EXT4_FAULT_MAX] = {
"inode_eio", /* EXT4_FAULT_INODE_EIO */
"extent_block_eio", /* EXT4_FAULT_EXTENT_EIO */
"dir_block_eio", /* EXT4_FAULT_DIRBLOCK_EIO */
+ "xattr_block_eio", /* EXT4_FAULT_XATTR_EIO */
};

static int ext4_fault_available_show(struct seq_file *m, void *v)
diff --git a/fs/ext4/xattr.c b/fs/ext4/xattr.c
index 39c80565c65d..3a066c1ddd5c 100644
--- a/fs/ext4/xattr.c
+++ b/fs/ext4/xattr.c
@@ -2231,6 +2231,10 @@ static struct buffer_head *ext4_xattr_get_block(struct inode *inode)
if (!EXT4_I(inode)->i_file_acl)
return NULL;

+ error = ext4_fault_xattr_io(inode->i_sb, inode->i_ino);
+ if (error)
+ return ERR_PTR(error);
+
ea_idebug(inode, "reading block %llu",
(unsigned long long)EXT4_I(inode)->i_file_acl);
bh = ext4_sb_bread(inode->i_sb, EXT4_I(inode)->i_file_acl, REQ_PRIO);
--
2.31.1

2022-11-10 02:31:10

by Zhang Yi

Subject: [PATCH v2 11/12] ext4: add journal related fault injection

Add journal start, getting create/write access, and dirty metadata fault
injection. The journal start fault injections return -ENOMEM directly,
other 3 injections will abort the journal and return -EROFS.

Signed-off-by: Zhang Yi <[email protected]>
---
fs/ext4/ext4.h | 12 ++++++++++++
fs/ext4/ext4_jbd2.c | 22 ++++++++++++++++------
fs/ext4/ext4_jbd2.h | 5 +++++
fs/ext4/sysfs.c | 5 +++++
4 files changed, 38 insertions(+), 6 deletions(-)

diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h
index 813127cfd3c0..96b805992ea5 100644
--- a/fs/ext4/ext4.h
+++ b/fs/ext4/ext4.h
@@ -1526,6 +1526,12 @@ enum ext4_fault_bits {
EXT4_FAULT_DIRBLOCK_EIO, /* directory block */
EXT4_FAULT_XATTR_EIO, /* xattr block */
EXT4_FAULT_SYMLINK_EIO, /* symlink block */
+ /* journal error */
+ EXT4_FAULT_JOURNAL_START, /* journal start inode */
+ EXT4_FAULT_JOURNAL_START_SB, /* journal start sb */
+ EXT4_FAULT_JOURNAL_CREATE_ACCESS, /* journal get create access */
+ EXT4_FAULT_JOURNAL_WRITE_ACCESS, /* journal get write access */
+ EXT4_FAULT_JOURNAL_DIRTY_METADATA, /* journal dirty meta data */
EXT4_FAULT_MAX
};

@@ -1633,6 +1639,12 @@ EXT4_FAULT_INODE_LBLOCK_FN(DIRBLOCK_EIO, dirblock_io, -EIO)
EXT4_FAULT_INODE_FN(XATTR_EIO, xattr_io, -EIO)
EXT4_FAULT_INODE_FN(SYMLINK_EIO, symlink_io, -EIO)

+EXT4_FAULT_INODE_FN(JOURNAL_START, journal_start, -ENOMEM)
+EXT4_FAULT_FN(JOURNAL_START_SB, journal_start_sb, -ENOMEM)
+EXT4_FAULT_INODE_PBLOCK_FN(JOURNAL_CREATE_ACCESS, journal_create_access, -EROFS)
+EXT4_FAULT_INODE_PBLOCK_FN(JOURNAL_WRITE_ACCESS, journal_write_access, -EROFS)
+EXT4_FAULT_INODE_PBLOCK_FN(JOURNAL_DIRTY_METADATA, journal_dirty_metadata, -EROFS)
+
/*
* fourth extended-fs super-block data in memory
*/
diff --git a/fs/ext4/ext4_jbd2.c b/fs/ext4/ext4_jbd2.c
index 8e1fb18f465e..e0972dea7463 100644
--- a/fs/ext4/ext4_jbd2.c
+++ b/fs/ext4/ext4_jbd2.c
@@ -95,6 +95,9 @@ handle_t *__ext4_journal_start_sb(struct super_block *sb, unsigned int line,

trace_ext4_journal_start(sb, blocks, rsv_blocks, revoke_creds,
_RET_IP_);
+ err = ext4_fault_journal_start_sb(sb);
+ if (err)
+ return ERR_PTR(err);
err = ext4_journal_check_start(sb);
if (err < 0)
return ERR_PTR(err);
@@ -232,7 +235,9 @@ int __ext4_journal_get_write_access(const char *where, unsigned int line,
ext4_check_bdev_write_error(bh->b_bdev->bd_super);

if (ext4_handle_valid(handle)) {
- err = jbd2_journal_get_write_access(handle, bh);
+ err = ext4_fault_journal_write_access(sb, 0, bh->b_blocknr);
+ if (!err)
+ err = jbd2_journal_get_write_access(handle, bh);
if (err) {
ext4_journal_abort_handle(where, line, __func__, bh,
handle, err);
@@ -320,7 +325,9 @@ int __ext4_journal_get_create_access(const char *where, unsigned int line,
if (!ext4_handle_valid(handle))
return 0;

- err = jbd2_journal_get_create_access(handle, bh);
+ err = ext4_fault_journal_create_access(sb, 0, bh->b_blocknr);
+ if (!err)
+ err = jbd2_journal_get_create_access(handle, bh);
if (err) {
ext4_journal_abort_handle(where, line, __func__, bh, handle,
err);
@@ -338,7 +345,7 @@ int __ext4_handle_dirty_metadata(const char *where, unsigned int line,
handle_t *handle, struct inode *inode,
struct buffer_head *bh)
{
- int err = 0;
+ int err = 0, fa_err = 0;

might_sleep();

@@ -346,9 +353,12 @@ int __ext4_handle_dirty_metadata(const char *where, unsigned int line,
set_buffer_prio(bh);
set_buffer_uptodate(bh);
if (ext4_handle_valid(handle)) {
- err = jbd2_journal_dirty_metadata(handle, bh);
- /* Errors can only happen due to aborted journal or a nasty bug */
- if (!is_handle_aborted(handle) && WARN_ON_ONCE(err)) {
+ if (bh->b_bdev->bd_super)
+ fa_err = ext4_fault_journal_dirty_metadata(bh->b_bdev->bd_super,
+ 0, bh->b_blocknr);
+ if (!fa_err)
+ err = jbd2_journal_dirty_metadata(handle, bh);
+ if (!is_handle_aborted(handle) && (WARN_ON_ONCE(err) || fa_err)) {
ext4_journal_abort_handle(where, line, __func__, bh,
handle, err);
if (inode == NULL) {
diff --git a/fs/ext4/ext4_jbd2.h b/fs/ext4/ext4_jbd2.h
index db2ae4a2b38d..b0a996f306bb 100644
--- a/fs/ext4/ext4_jbd2.h
+++ b/fs/ext4/ext4_jbd2.h
@@ -323,6 +323,11 @@ static inline handle_t *__ext4_journal_start(struct inode *inode,
int blocks, int rsv_blocks,
int revoke_creds)
{
+ int err;
+
+ err = ext4_fault_journal_start(inode->i_sb, inode->i_ino);
+ if (err)
+ return ERR_PTR(err);
return __ext4_journal_start_sb(inode->i_sb, line, type, blocks,
rsv_blocks, revoke_creds);
}
diff --git a/fs/ext4/sysfs.c b/fs/ext4/sysfs.c
index dfccd9f04fbb..da725e128c89 100644
--- a/fs/ext4/sysfs.c
+++ b/fs/ext4/sysfs.c
@@ -585,6 +585,11 @@ char *ext4_fault_names[EXT4_FAULT_MAX] = {
"dir_block_eio", /* EXT4_FAULT_DIRBLOCK_EIO */
"xattr_block_eio", /* EXT4_FAULT_XATTR_EIO */
"symlink_block_eio", /* EXT4_FAULT_SYMLINK_EIO */
+ "journal_start", /* EXT4_FAULT_JOURNAL_START */
+ "journal_start_sb", /* EXT4_FAULT_JOURNAL_START_SB */
+ "journal_get_create_access", /* EXT4_FAULT_JOURNAL_CREATE_ACCESS */
+ "journal_get_write_access", /* EXT4_FAULT_JOURNAL_WRITE_ACCESS */
+ "journal_dirty_metadata", /* EXT4_FAULT_JOURNAL_DIRTY_METADATA */
};

static int ext4_fault_available_show(struct seq_file *m, void *v)
--
2.31.1

2022-11-10 02:36:35

by Zhang Yi

Subject: [PATCH v2 06/12] ext4: add extent block I/O fault injection

Add inode extent block reading I/O fault injection, we can specify the
inode and physical metadata block to inject, it will return -EIO
immediately instead of submitting I/O.

Signed-off-by: Zhang Yi <[email protected]>
---
fs/ext4/ext4.h | 2 ++
fs/ext4/extents.c | 5 +++++
fs/ext4/sysfs.c | 1 +
3 files changed, 8 insertions(+)

diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h
index 29a819a186f7..9c1dcbed59e6 100644
--- a/fs/ext4/ext4.h
+++ b/fs/ext4/ext4.h
@@ -1522,6 +1522,7 @@ enum ext4_fault_bits {
EXT4_FAULT_IBITMAP_EIO, /* inode bitmap block */
EXT4_FAULT_BBITMAP_EIO, /* block bitmap block */
EXT4_FAULT_INODE_EIO, /* inode */
+ EXT4_FAULT_EXTENT_EIO, /* extent block */
EXT4_FAULT_MAX
};

@@ -1624,6 +1625,7 @@ EXT4_FAULT_INODE_FN(XATTR_CSUM, xattr_csum, 1)
EXT4_FAULT_GRP_FN(IBITMAP_EIO, inode_bitmap_io, -EIO)
EXT4_FAULT_GRP_FN(BBITMAP_EIO, block_bitmap_io, -EIO)
EXT4_FAULT_INODE_FN(INODE_EIO, inode_io, -EIO)
+EXT4_FAULT_INODE_PBLOCK_FN(EXTENT_EIO, extent_io, -EIO)

/*
* fourth extended-fs super-block data in memory
diff --git a/fs/ext4/extents.c b/fs/ext4/extents.c
index 0d07e5cf4dab..504ed35ffeaf 100644
--- a/fs/ext4/extents.c
+++ b/fs/ext4/extents.c
@@ -566,6 +566,11 @@ __read_extent_tree_block(const char *function, unsigned int line,

if (!bh_uptodate_or_lock(bh)) {
trace_ext4_ext_load_extent(inode, pblk, _RET_IP_);
+ err = ext4_fault_extent_io(inode->i_sb, inode->i_ino, pblk);
+ if (err) {
+ unlock_buffer(bh);
+ goto errout;
+ }
err = ext4_read_bh(bh, 0, NULL);
if (err < 0)
goto errout;
diff --git a/fs/ext4/sysfs.c b/fs/ext4/sysfs.c
index 68b2c02bb266..871da7c8c2c6 100644
--- a/fs/ext4/sysfs.c
+++ b/fs/ext4/sysfs.c
@@ -581,6 +581,7 @@ char *ext4_fault_names[EXT4_FAULT_MAX] = {
"inode_bitmap_eio", /* EXT4_FAULT_IBITMAP_EIO */
"block_bitmap_eio", /* EXT4_FAULT_BBITMAP_EIO */
"inode_eio", /* EXT4_FAULT_INODE_EIO */
+ "extent_block_eio", /* EXT4_FAULT_EXTENT_EIO */
};

static int ext4_fault_available_show(struct seq_file *m, void *v)
--
2.31.1

2022-11-10 02:37:49

by Zhang Yi

Subject: [PATCH v2 10/12] ext4: add symlink block I/O fault injection

Add I/O fault injection when reading symlink block, user could specify
which inode to inject error. It will return -EIO immediately instead of
submitting I/O.

Signed-off-by: Zhang Yi <[email protected]>
---
fs/ext4/ext4.h | 2 ++
fs/ext4/symlink.c | 4 ++++
fs/ext4/sysfs.c | 1 +
3 files changed, 7 insertions(+)

diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h
index 94894daef595..813127cfd3c0 100644
--- a/fs/ext4/ext4.h
+++ b/fs/ext4/ext4.h
@@ -1525,6 +1525,7 @@ enum ext4_fault_bits {
EXT4_FAULT_EXTENT_EIO, /* extent block */
EXT4_FAULT_DIRBLOCK_EIO, /* directory block */
EXT4_FAULT_XATTR_EIO, /* xattr block */
+ EXT4_FAULT_SYMLINK_EIO, /* symlink block */
EXT4_FAULT_MAX
};

@@ -1630,6 +1631,7 @@ EXT4_FAULT_INODE_FN(INODE_EIO, inode_io, -EIO)
EXT4_FAULT_INODE_PBLOCK_FN(EXTENT_EIO, extent_io, -EIO)
EXT4_FAULT_INODE_LBLOCK_FN(DIRBLOCK_EIO, dirblock_io, -EIO)
EXT4_FAULT_INODE_FN(XATTR_EIO, xattr_io, -EIO)
+EXT4_FAULT_INODE_FN(SYMLINK_EIO, symlink_io, -EIO)

/*
* fourth extended-fs super-block data in memory
diff --git a/fs/ext4/symlink.c b/fs/ext4/symlink.c
index 3d3ed3c38f56..5392e707418e 100644
--- a/fs/ext4/symlink.c
+++ b/fs/ext4/symlink.c
@@ -39,6 +39,8 @@ static const char *ext4_encrypted_get_link(struct dentry *dentry,
caddr = EXT4_I(inode)->i_data;
max_size = sizeof(EXT4_I(inode)->i_data);
} else {
+ if (ext4_fault_symlink_io(inode->i_sb, inode->i_ino))
+ return ERR_PTR(-EIO);
bh = ext4_bread(NULL, inode, 0, 0);
if (IS_ERR(bh))
return ERR_CAST(bh);
@@ -97,6 +99,8 @@ static const char *ext4_get_link(struct dentry *dentry, struct inode *inode,
if (!bh || !ext4_buffer_uptodate(bh))
return ERR_PTR(-ECHILD);
} else {
+ if (ext4_fault_symlink_io(inode->i_sb, inode->i_ino))
+ return ERR_PTR(-EIO);
bh = ext4_bread(NULL, inode, 0, 0);
if (IS_ERR(bh))
return ERR_CAST(bh);
diff --git a/fs/ext4/sysfs.c b/fs/ext4/sysfs.c
index 842e4f60fb69..dfccd9f04fbb 100644
--- a/fs/ext4/sysfs.c
+++ b/fs/ext4/sysfs.c
@@ -584,6 +584,7 @@ char *ext4_fault_names[EXT4_FAULT_MAX] = {
"extent_block_eio", /* EXT4_FAULT_EXTENT_EIO */
"dir_block_eio", /* EXT4_FAULT_DIRBLOCK_EIO */
"xattr_block_eio", /* EXT4_FAULT_XATTR_EIO */
+ "symlink_block_eio", /* EXT4_FAULT_SYMLINK_EIO */
};

static int ext4_fault_available_show(struct seq_file *m, void *v)
--
2.31.1

2022-12-14 14:03:38

by Zhang Yi

Subject: Re: [PATCH v2 00/12] ext4: enhance simulate fail facility

Hello, is anybody have advice?

Thanks,
Yi.

On 2022/11/10 10:25, Zhang Yi wrote:
> Changes since v1:
> - Fix format error in ext4_fault_ops_write().
>
> Now we can test ext4's reliability by simulating fail facility introduced
> in commit 46f870d690fe ("ext4: simulate various I/O and checksum errors
> when reading metadata"), it can simulate checksum error or I/O error
> when reading metadata from disk. But it is functional limited, it cannot
> set failure times, probability, filters, etc. Fortunately, we already
> have common fault-injection frame in Linux, so above limitation could be
> easily supplied by using it in ext4. This patch set add ext4
> fault-injection facility to replace the old frame, supply some kinds of
> checksum error and I/O error, and also add group, inode, physical
> block and inode logical block filters. After this patch set, we could
> inject failure more precisly. The facility could be used to do fuzz
> stress test include random errors, and it also could be used to
> reprodece issues more conveniently.
>
> Patch 1: add debugfs for preparing.
> Patch 2: introduce the fault-injection frame for ext4.
> Patch 3-11: add various kinds of faults and also do some cleanup.
> Patch 12: remove the old simulating facility.
>
> It provides a debugfs interface in ext4/<disk>/fault_inject, besides the
> common config interfaces, we give 6 more.
> - available_faults: present available faults we can inject.
> - inject_faults: set faults, can set multiple at a time.
> - inject_inode: set the inode filter, matches all inodes if not set.
> - inject_group: set the block group filter, similar to inject_inode.
> - inject_logical_block: set the logical block filter for one inode.
> - inject_physical_block: set the physical block filter.
>
> Current we add 20 available faults list below, include 8 kinds of
> metadata checksum error, 7 metadata I/O error and 5 journal error.
> After we have this facility, more other faults could be added easily
> in the future.
> - group_desc_checksum
> - inode_bitmap_checksum
> - block_bitmap_checksum
> - inode_checksum
> - extent_block_checksum
> - dir_block_checksum
> - dir_index_block_checksum
> - xattr_block_checksum
> - inode_bitmap_eio
> - block_bitmap_eio
> - inode_eio
> - extent_block_eio
> - dir_block_eio
> - xattr_block_eio
> - symlink_block_eio
> - journal_start
> - journal_start_sb
> - journal_get_create_access
> - journal_get_write_access
> - journal_dirty_metadata
>
> For example: inject inode metadata checksum error on file 'foo'.
>
> $ mkfs.ext4 -F /dev/pmem0
> $ mount /dev/pmem0 /mnt
> $ mkdir /mnt/dir
> $ touch /mnt/dir/foo
> $ ls -i /mnt/dir/foo
> 262146 /mnt/foo
>
> $ echo 100 > /sys/kernel/debug/ext4/pmem0/fault_inject/probability
> $ echo 1 > /sys/kernel/debug/ext4/pmem0/fault_inject/times
> $ echo 262146 > /sys/kernel/debug/ext4/pmem0/fault_inject/inject_inode
> $ echo inode_checksum > /sys/kernel/debug/ext4/pmem0/fault_inject/inject_faults
> $ echo 1 > /sys/kernel/debug/ext4/pmem0/fault_inject/enable
> $ echo 3 > /proc/sys/vm/drop_caches ##drop cache
> $ stat /mnt/dir/foo
> stat: cannot statx '/mnt/dir/foo': Bad message
>
> The kmesg print the injection location.
>
> [ 461.433817] FAULT_INJECTION: forcing a failure.
> [ 461.433817] name fault_inject, interval 1, probability 100, space 0, times 1
> ...
> [ 461.438609] Call Trace:
> [ 461.438875] <TASK>
> [ 461.439116] ? dump_stack_lvl+0x73/0xa3
> [ 461.439534] ? dump_stack+0x13/0x1f
> [ 461.439909] ? should_fail.cold+0x4a/0x57
> [ 461.440346] ? ext4_should_fail.cold+0x11f/0x135
> [ 461.440833] ? __ext4_iget+0x407/0x1410
> [ 461.441245] ? ext4_lookup+0x1be/0x350
> [ 461.441650] ? __lookup_slow+0xb9/0x1f0
> [ 461.442070] ? lookup_slow+0x46/0x70
> [ 461.442463] ? walk_component+0x13e/0x230
> [ 461.442890] ? path_lookupat.isra.0+0x8f/0x200
> [ 461.443369] ? filename_lookup+0xd6/0x240
> [ 461.443798] ? vfs_statx+0xa6/0x200
> [ 461.444186] ? do_statx+0x48/0xc0
> [ 461.444546] ? __might_sleep+0x56/0xc0
> [ 461.444950] ? should_fail_usercopy+0x19/0x30
> [ 461.445424] ? strncpy_from_user+0x33/0x2a0
> [ 461.445870] ? getname_flags+0x95/0x330
> [ 461.446288] ? switch_fpu_return+0x27/0x1e0
> [ 461.446736] ? __x64_sys_statx+0x90/0xd0
> [ 461.447160] ? do_syscall_64+0x3b/0x90
> [ 461.447563] ? entry_SYSCALL_64_after_hwframe+0x63/0xcd
> [ 461.448122] </TASK>
> [ 461.448395] EXT4-fs error (device pmem0): ext4_lookup:1840: inode #262146: comm stat: iget: checksum invalid
>
> Thanks,
> Yi.
>
>
> Zhang Yi (12):
> ext4: add debugfs interface
> ext4: introduce fault injection facility
> ext4: add several checksum fault injection
> ext4: add bitmaps I/O fault injection
> ext4: add inode I/O fault injection
> ext4: add extent block I/O fault injection
> ext4: add dirblock I/O fault injection
> ext4: call ext4_xattr_get_block() when getting xattr block
> ext4: add xattr block I/O fault injection
> ext4: add symlink block I/O fault injection
> ext4: add journal related fault injection
> ext4: remove simulate fail facility
>
> fs/ext4/Kconfig | 9 ++
> fs/ext4/balloc.c | 14 ++-
> fs/ext4/bitmap.c | 4 +
> fs/ext4/dir.c | 3 +
> fs/ext4/ext4.h | 181 +++++++++++++++++++++++++++++--------
> fs/ext4/ext4_jbd2.c | 22 +++--
> fs/ext4/ext4_jbd2.h | 5 +
> fs/ext4/extents.c | 7 ++
> fs/ext4/ialloc.c | 24 +++--
> fs/ext4/inode.c | 26 ++++--
> fs/ext4/namei.c | 14 ++-
> fs/ext4/super.c | 7 +-
> fs/ext4/symlink.c | 4 +
> fs/ext4/sysfs.c | 183 +++++++++++++++++++++++++++++++++++--
> fs/ext4/xattr.c | 216 +++++++++++++++++++-------------------------
> 15 files changed, 515 insertions(+), 204 deletions(-)
>