Received-SPF: pass (google.com: domain of linux-kernel+bounces-179486-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
From: Daniel Gomez <da.gomez@samsung.com>
To: "hughd@google.com" <hughd@google.com>, "akpm@linux-foundation.org"
	<akpm@linux-foundation.org>, "willy@infradead.org" <willy@infradead.org>,
	"jack@suse.cz" <jack@suse.cz>, "mcgrof@kernel.org" <mcgrof@kernel.org>
CC: "linux-mm@kvack.org" <linux-mm@kvack.org>, "linux-xfs@vger.kernel.org"
	<linux-xfs@vger.kernel.org>, "djwong@kernel.org" <djwong@kernel.org>,
	"Pankaj Raghav" <p.raghav@samsung.com>, "dagmcr@gmail.com"
	<dagmcr@gmail.com>, "yosryahmed@google.com" <yosryahmed@google.com>,
	"baolin.wang@linux.alibaba.com" <baolin.wang@linux.alibaba.com>,
	"ritesh.list@gmail.com" <ritesh.list@gmail.com>,
	"lsf-pc@lists.linux-foundation.org" <lsf-pc@lists.linux-foundation.org>,
	"david@redhat.com" <david@redhat.com>, "chandan.babu@oracle.com"
	<chandan.babu@oracle.com>, "linux-kernel@vger.kernel.org"
	<linux-kernel@vger.kernel.org>, "brauner@kernel.org" <brauner@kernel.org>,
	Daniel Gomez <da.gomez@samsung.com>
Subject: [PATCH 02/12] shmem: add per-block uptodate tracking for large
 folios
Thread-Topic: [PATCH 02/12] shmem: add per-block uptodate tracking for large
	folios
Thread-Index: AQHapozCeJW3qmZrtEOp+8upfVo2+A==
Date: Wed, 15 May 2024 05:57:24 +0000
Message-ID: <20240515055719.32577-3-da.gomez@samsung.com>
In-Reply-To: <20240515055719.32577-1-da.gomez@samsung.com>
Accept-Language: en-US, en-GB
Content-Language: en-US
x-ms-exchange-messagesentrepresentingtype: 1
x-ms-exchange-transport-fromentityheader: Hosted
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk
X-Mailing-List: linux-kernel@vger.kernel.org
List-Id: <linux-kernel.vger.kernel.org>
List-Subscribe: <mailto:linux-kernel+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:linux-kernel+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrIKsWRmVeSWpSXmKPExsWy7djPc7rH/FzSDI4161rMWb+GzeL/3mOM
	Fq8Pf2K0uHRUzuJs3282i6/rfzFbXH7CZ/H0Ux+LxezpzUwWl3fNYbO4t+Y/q8WuPzvYLfa9
	3stscWPCU0aLg6c62C1+/wDKbt8V6SDosXPWXXaPBZtKPTav0PLYtKqTzWPTp0nsHidm/Gbx
	2PnQ0mPyjeWMHh+f3mLxeL/vKpvHmQVH2D0+b5IL4InisklJzcksSy3St0vgymhpm8hY0OBS
	0fe9g6mB8aB5FyMnh4SAicT5ef/Zuxi5OIQEVjBKrDz5gA3C+cIocbq/jwnC+cwosXvTfTaY
	ltudhxkhEssZJU5cO8ACV7Xo+B1mCOcMo8TyNUtYIZyVjBILdj1kB+lnE9CU2HdyE9hKEYHb
	jBJPT50Bc5gFTrJK/Nm8E2gYB4ewQIBEzz5DEFNEIFTi6p8gkF4RAT2JFet3MYOEWQRUJa41
	BoKEeQUsJdb2b2UBsTkFrCTuzv8KdiqjgKzEo5W/wNYyC4hL3HoynwniBUGJRbP3MEPYYhL/
	dj2Eek1H4uz1J4wQtoHE1qX7WCBsZYn179qYIOboSdyYOoUNwtaWWLbwNTPEDYISJ2c+AYeE
	hMBBLon/z7azQjS7SEzYtw9qqLDEq+Nb2CFsGYnTk3tYJjBqz0Jy3ywkO2Yh2TELyY4FjCyr
	GMVTS4tz01OLjfJSy/WKE3OLS/PS9ZLzczcxAlPl6X/Hv+xgXP7qo94hRiYOxkOMEhzMSiK8
	ImnOaUK8KYmVValF+fFFpTmpxYcYpTlYlMR5VVPkU4UE0hNLUrNTUwtSi2CyTBycUg1MzG4H
	X9Q05rXOamb4cMgmuv3Rno8y8ce/XbHyOmbNcLTEzn1n5eI4oydFDxofeEzjj3+tmHCjzexc
	HdshvWWNX2ZnsIlfkK10PvV8m5+JzOz+edscPigc9dq6svy7yY3dT3iYbsgm5hjwbdZ9m/dS
	8G/E7VMMZbHb5j1X3VVxqfpWrHbvqfn6Dpl/WwsrflSIz5nRmZrE/NmSKfPsLI+fc1UUDyZL
	/JyyyP6myQ/3GgU+5S0qFU8DtExLd10tfr7v1oEFgUanj2rec2E0W23s433UeebNQqkb+cmf
	UlY/u5P96tTCGVxSUoKR2SHhcraaO7Z+rnzLcs/55YYvdYrHmOOkMg6YM3kc/LlNzeatEktx
	RqKhFnNRcSIAOTw8zQQEAAA=
X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrLKsWRmVeSWpSXmKPExsVy+t/xu7rH/FzSDP5cELSYs34Nm8X/vccY
	LV4f/sRocemonMXZvt9sFl/X/2K2uPyEz+Lppz4Wi9nTm5ksLu+aw2Zxb81/Votdf3awW+x7
	vZfZ4saEp4wWB091sFv8/gGU3b4r0kHQY+esu+weCzaVemxeoeWxaVUnm8emT5PYPU7M+M3i
	sfOhpcfkG8sZPT4+vcXi8X7fVTaPMwuOsHt83iQXwBOlZ1OUX1qSqpCRX1xiqxRtaGGkZ2hp
	oWdkYqlnaGwea2VkqqRvZ5OSmpNZllqkb5egl9HSNpGxoMGlou97B1MD40HzLkZODgkBE4nb
	nYcZQWwhgaWMEpNui0PEZSQ2frnKCmELS/y51sXWxcgFVPORUeLNur8sEM4ZRonZW7YxQzgr
	GSUePWpmBmlhE9CU2HdyEztIQkTgNqPE01NnwBxmgZOsEgdOf2YHqRIW8JOYunsZ2BIRgVCJ
	PXubmSBsPYkV63cBTeLgYBFQlbjWGAgS5hWwlFjbv5UF4lZLiUuvtoPdzSlgJXF3/lc2EJtR
	QFbi0cpfYOOZBcQlbj2ZzwTxg4DEkj3nmSFsUYmXj/9B/aYjcfb6E0YI20Bi69J9LBC2ssT6
	d21MEHP0JG5MncIGYWtLLFv4mhniHkGJkzOfsExglJ6FZN0sJC2zkLTMQtKygJFlFaNIamlx
	bnpusZFecWJucWleul5yfu4mRmC623bs55YdjCtffdQ7xMjEwXiIUYKDWUmEVyTNOU2INyWx
	siq1KD++qDQntfgQoykwiCYyS4km5wMTbl5JvKGZgamhiZmlgamlmbGSOK9nQUeikEB6Yklq
	dmpqQWoRTB8TB6dUA1PwQyvn3HrXnjO/RQR3blKfHnY5+IaEic41pxMJqy9v/rGTa+/iiG9c
	/MuiSjeuCXuRyKHbeLFmx6y3kW5fFz6/qnYuae9npvAVGj6y1rsdNlZ/vT3BxKvmmE7r+XDZ
	C1NtNJV0Fr3a2lU4RegB03o5qzcTBJvnm8YstzP/Ulb/8HyB24tc0wLFfTOstLo/38xX+Oug
	fEb+j7WHHu+ttfJxd3xqL1+yX+xjLfpey/Lgwdrb3B+3uEt+kTwl/+1639X4t0pK0cqvP6Yu
	OsC5vnP6acYl/RbvH0StzF++iE/USPPmbabGQoXky8v3bTARTGiJz2yxX/DKS2aBrXTw2qpo
	bsXgufNu9u7611vvNU2JpTgj0VCLuag4EQBY2qAJAAQAAA==
X-CMS-MailID: 20240515055726eucas1p2a795fc743373571bfc3349f9e1ef3f9e
X-Msg-Generator: CA
X-RootMTR: 20240515055726eucas1p2a795fc743373571bfc3349f9e1ef3f9e
X-EPHeader: CA
CMS-TYPE: 201P
X-CMS-RootMailID: 20240515055726eucas1p2a795fc743373571bfc3349f9e1ef3f9e
References: <20240515055719.32577-1-da.gomez@samsung.com>
	<CGME20240515055726eucas1p2a795fc743373571bfc3349f9e1ef3f9e@eucas1p2.samsung.com>

Based on iomap per-block dirty and uptodate state track, add support
for shmem_folio_state struct to track the uptodate state per-block for
large folios.

Signed-off-by: Daniel Gomez <da.gomez@samsung.com>
---
 mm/shmem.c | 195 +++++++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 189 insertions(+), 6 deletions(-)

diff --git a/mm/shmem.c b/mm/shmem.c
index 94ab99b6b574..4818f9fbd328 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -131,6 +131,124 @@ struct shmem_options {
 #define SHMEM_SEEN_QUOTA 32
 };
=20
+/*
+ * Structure allocated for each folio to track per-block uptodate state.
+ *
+ * Like buffered-io iomap_folio_state struct but only for uptodate.
+ */
+struct shmem_folio_state {
+	spinlock_t state_lock;
+	unsigned long state[];
+};
+
+static inline bool sfs_is_fully_uptodate(struct folio *folio)
+{
+	struct inode *inode =3D folio->mapping->host;
+	struct shmem_folio_state *sfs =3D folio->private;
+
+	return bitmap_full(sfs->state, i_blocks_per_folio(inode, folio));
+}
+
+static inline bool sfs_is_block_uptodate(struct shmem_folio_state *sfs,
+					 unsigned int block)
+{
+	return test_bit(block, sfs->state);
+}
+
+/**
+ * sfs_get_last_block_uptodate - find the index of the last uptodate block
+ * within a specified range
+ * @folio: The folio
+ * @first: The starting block of the range to search
+ * @last: The ending block of the range to search
+ *
+ * Returns the index of the last uptodate block within the specified range=
 If
+ * a non uptodate block is found at the start, it returns UINT_MAX.
+ */
+static unsigned int sfs_get_last_block_uptodate(struct folio *folio,
+						unsigned int first,
+						unsigned int last)
+{
+	struct inode *inode =3D folio->mapping->host;
+	struct shmem_folio_state *sfs =3D folio->private;
+	unsigned int nr_blocks =3D i_blocks_per_folio(inode, folio);
+	unsigned int aux =3D find_next_zero_bit(sfs->state, nr_blocks, first);
+
+	/*
+	 * Exceed the range of possible last block and return UINT_MAX if a non
+	 * uptodate block is found at the beginning of the scan.
+	 */
+	if (aux =3D=3D first)
+		return UINT_MAX;
+
+	return min_t(unsigned int, aux - 1, last);
+}
+
+static void sfs_set_range_uptodate(struct folio *folio,
+				   struct shmem_folio_state *sfs, size_t off,
+				   size_t len)
+{
+	struct inode *inode =3D folio->mapping->host;
+	unsigned int first_blk =3D off >> inode->i_blkbits;
+	unsigned int last_blk =3D (off + len - 1) >> inode->i_blkbits;
+	unsigned int nr_blks =3D last_blk - first_blk + 1;
+	unsigned long flags;
+
+	spin_lock_irqsave(&sfs->state_lock, flags);
+	bitmap_set(sfs->state, first_blk, nr_blks);
+	if (sfs_is_fully_uptodate(folio))
+		folio_mark_uptodate(folio);
+	spin_unlock_irqrestore(&sfs->state_lock, flags);
+}
+
+static struct shmem_folio_state *sfs_alloc(struct inode *inode,
+					   struct folio *folio)
+{
+	struct shmem_folio_state *sfs =3D folio->private;
+	unsigned int nr_blocks =3D i_blocks_per_folio(inode, folio);
+	gfp_t gfp =3D GFP_KERNEL;
+
+	if (sfs || nr_blocks <=3D 1)
+		return sfs;
+
+	/*
+	 * sfs->state tracks uptodate flag when the block size is smaller
+	 * than the folio size.
+	 */
+	sfs =3D kzalloc(struct_size(sfs, state, BITS_TO_LONGS(nr_blocks)), gfp);
+	if (!sfs)
+		return sfs;
+
+	spin_lock_init(&sfs->state_lock);
+	if (folio_test_uptodate(folio))
+		bitmap_set(sfs->state, 0, nr_blocks);
+	folio_attach_private(folio, sfs);
+
+	return sfs;
+}
+
+static void sfs_free(struct folio *folio, bool force)
+{
+	if (!folio_test_private(folio))
+		return;
+
+	if (!force)
+		WARN_ON_ONCE(sfs_is_fully_uptodate(folio) !=3D
+			     folio_test_uptodate(folio));
+
+	kfree(folio_detach_private(folio));
+}
+
+static void shmem_set_range_uptodate(struct folio *folio, size_t off,
+				     size_t len)
+{
+	struct shmem_folio_state *sfs =3D folio->private;
+
+	if (sfs)
+		sfs_set_range_uptodate(folio, sfs, off, len);
+	else
+		folio_mark_uptodate(folio);
+}
 #ifdef CONFIG_TMPFS
 static unsigned long shmem_default_max_blocks(void)
 {
@@ -1487,7 +1605,7 @@ static int shmem_writepage(struct page *page, struct =
writeback_control *wbc)
 		}
 		folio_zero_range(folio, 0, folio_size(folio));
 		flush_dcache_folio(folio);
-		folio_mark_uptodate(folio);
+		shmem_set_range_uptodate(folio, 0, folio_size(folio));
 	}
=20
 	swap =3D folio_alloc_swap(folio);
@@ -1769,13 +1887,16 @@ static int shmem_replace_folio(struct folio **folio=
p, gfp_t gfp,
 	if (!new)
 		return -ENOMEM;
=20
+	if (folio_get_private(old))
+		folio_attach_private(new, folio_detach_private(old));
+
 	folio_get(new);
 	folio_copy(new, old);
 	flush_dcache_folio(new);
=20
 	__folio_set_locked(new);
 	__folio_set_swapbacked(new);
-	folio_mark_uptodate(new);
+	shmem_set_range_uptodate(new, 0, folio_size(new));
 	new->swap =3D entry;
 	folio_set_swapcache(new);
=20
@@ -2063,6 +2184,12 @@ static int shmem_get_folio_gfp(struct inode *inode, =
pgoff_t index,
=20
 alloced:
 	alloced =3D true;
+
+	if (!sfs_alloc(inode, folio) && folio_test_large(folio)) {
+		error =3D -ENOMEM;
+		goto unlock;
+	}
+
 	if (folio_test_pmd_mappable(folio) &&
 	    DIV_ROUND_UP(i_size_read(inode), PAGE_SIZE) <
 					folio_next_index(folio) - 1) {
@@ -2104,7 +2231,7 @@ static int shmem_get_folio_gfp(struct inode *inode, p=
goff_t index,
 		for (i =3D 0; i < n; i++)
 			clear_highpage(folio_page(folio, i));
 		flush_dcache_folio(folio);
-		folio_mark_uptodate(folio);
+		shmem_set_range_uptodate(folio, 0, folio_size(folio));
 	}
=20
 	/* Perhaps the file has been truncated since we checked */
@@ -2773,8 +2900,8 @@ shmem_write_end(struct file *file, struct address_spa=
ce *mapping,
 			folio_zero_segments(folio, 0, from,
 					from + copied, folio_size(folio));
 		}
-		folio_mark_uptodate(folio);
 	}
+	shmem_set_range_uptodate(folio, 0, folio_size(folio));
 	folio_mark_dirty(folio);
 	folio_unlock(folio);
 	folio_put(folio);
@@ -2782,6 +2909,59 @@ shmem_write_end(struct file *file, struct address_sp=
ace *mapping,
 	return copied;
 }
=20
+static void shmem_invalidate_folio(struct folio *folio, size_t offset,
+				   size_t len)
+{
+	/*
+	 * If we're invalidating the entire folio, clear the dirty state
+	 * from it and release it to avoid unnecessary buildup of the LRU.
+	 */
+	if (offset =3D=3D 0 && len =3D=3D folio_size(folio)) {
+		WARN_ON_ONCE(folio_test_writeback(folio));
+		folio_cancel_dirty(folio);
+		sfs_free(folio, true);
+	}
+}
+
+static bool shmem_release_folio(struct folio *folio, gfp_t gfp_flags)
+{
+	if (folio_test_dirty(folio) && !sfs_is_fully_uptodate(folio))
+		return false;
+
+	sfs_free(folio, false);
+	return true;
+}
+
+/*
+ * shmem_is_partially_uptodate checks whether blocks within a folio are
+ * uptodate or not.
+ *
+ * Returns true if all blocks which correspond to the specified part
+ * of the folio are uptodate.
+ */
+static bool shmem_is_partially_uptodate(struct folio *folio, size_t from,
+					size_t count)
+{
+	struct shmem_folio_state *sfs =3D folio->private;
+	struct inode *inode =3D folio->mapping->host;
+	unsigned int first, last;
+
+	if (!sfs)
+		return false;
+
+	/* Caller's range may extend past the end of this folio */
+	count =3D min(folio_size(folio) - from, count);
+
+	/* First and last blocks in range within folio */
+	first =3D from >> inode->i_blkbits;
+	last =3D (from + count - 1) >> inode->i_blkbits;
+
+	if (sfs_get_last_block_uptodate(folio, first, last) !=3D last)
+		return false;
+
+	return true;
+}
+
 static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *t=
o)
 {
 	struct file *file =3D iocb->ki_filp;
@@ -3533,7 +3713,7 @@ static int shmem_symlink(struct mnt_idmap *idmap, str=
uct inode *dir,
 			goto out_remove_offset;
 		inode->i_op =3D &shmem_symlink_inode_operations;
 		memcpy(folio_address(folio), symname, len);
-		folio_mark_uptodate(folio);
+		shmem_set_range_uptodate(folio, 0, folio_size(folio));
 		folio_mark_dirty(folio);
 		folio_unlock(folio);
 		folio_put(folio);
@@ -4523,7 +4703,10 @@ static const struct address_space_operations shmem_a=
ops =3D {
 #ifdef CONFIG_MIGRATION
 	.migrate_folio	=3D migrate_folio,
 #endif
-	.error_remove_folio =3D shmem_error_remove_folio,
+	.error_remove_folio    =3D shmem_error_remove_folio,
+	.invalidate_folio      =3D shmem_invalidate_folio,
+	.release_folio         =3D shmem_release_folio,
+	.is_partially_uptodate =3D shmem_is_partially_uptodate,
 };
=20
 static const struct file_operations shmem_file_operations =3D {
--=20
2.43.0