Hi all,
this series switches the core block layer code and all users of the
existing bvec kmap helpers to use kmap_local_page. Drivers that
currently use open coded kmap_atomic calls will converted in a follow
on series.
To do so a new kunmap variant is added that calls
flush_kernel_dcache_page. I'm not entirely sure where to call
flush_dcache_page vs flush_kernel_dcache_page, so I've tried to follow
the documentation here, but additional feedback would be welcome.
Changes since v1:
- add more/better comments
- add a new kunmap_local_dirty helper to feal with
flush(_kernel)_dcache_page
Diffstat:
arch/mips/include/asm/mach-rc32434/rb.h | 2 -
block/bio-integrity.c | 14 +++-----
block/bio.c | 37 ++++++----------------
block/blk-map.c | 2 -
block/bounce.c | 39 +++++-------------------
block/t10-pi.c | 16 +++------
drivers/block/ps3disk.c | 19 +----------
drivers/block/rbd.c | 15 +--------
drivers/md/dm-writecache.c | 5 +--
include/linux/bio.h | 42 -------------------------
include/linux/bvec.h | 52 ++++++++++++++++++++++++++++++--
include/linux/highmem-internal.h | 7 ++++
include/linux/highmem.h | 10 ++++--
13 files changed, 102 insertions(+), 158 deletions(-)
memcpy_to_page can write to potentially mapped page cache pages, so
use kunmap_local_dirty to make sure flush_kernel_dcache_pages is
called.
Signed-off-by: Christoph Hellwig <[email protected]>
---
include/linux/highmem.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/include/linux/highmem.h b/include/linux/highmem.h
index 65f548db4f2d..d0497c0daf80 100644
--- a/include/linux/highmem.h
+++ b/include/linux/highmem.h
@@ -333,7 +333,7 @@ static inline void memcpy_to_page(struct page *page, size_t offset,
VM_BUG_ON(offset + len > PAGE_SIZE);
memcpy(to + offset, from, len);
- kunmap_local(to);
+ kunmap_local_dirty(page, to);
}
static inline void memzero_page(struct page *page, size_t offset, size_t len)
--
2.30.2
Fix the include guards to match the file naming.
Signed-off-by: Christoph Hellwig <[email protected]>
Reviewed-by: Bart Van Assche <[email protected]>
Reviewed-by: Chaitanya Kulkarni <[email protected]>
Reviewed-by: Ira Weiny <[email protected]>
---
include/linux/bvec.h | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/include/linux/bvec.h b/include/linux/bvec.h
index ff832e698efb..883faf5f1523 100644
--- a/include/linux/bvec.h
+++ b/include/linux/bvec.h
@@ -4,8 +4,8 @@
*
* Copyright (C) 2001 Ming Lei <[email protected]>
*/
-#ifndef __LINUX_BVEC_ITER_H
-#define __LINUX_BVEC_ITER_H
+#ifndef __LINUX_BVEC_H
+#define __LINUX_BVEC_H
#include <linux/bug.h>
#include <linux/errno.h>
@@ -183,4 +183,4 @@ static inline void bvec_advance(const struct bio_vec *bvec,
}
}
-#endif /* __LINUX_BVEC_ITER_H */
+#endif /* __LINUX_BVEC_H */
--
2.30.2
Add a helper that calls flush_kernel_dcache_page before unmapping the
local mapping. flush_kernel_dcache_page is required for all pages
potentially mapped into userspace that were written to using kmap*,
so having a helper that does the right thing can be very convenient.
Signed-off-by: Christoph Hellwig <[email protected]>
---
include/linux/highmem-internal.h | 7 +++++++
include/linux/highmem.h | 4 ++++
2 files changed, 11 insertions(+)
diff --git a/include/linux/highmem-internal.h b/include/linux/highmem-internal.h
index 7902c7d8b55f..bd37706db147 100644
--- a/include/linux/highmem-internal.h
+++ b/include/linux/highmem-internal.h
@@ -224,4 +224,11 @@ do { \
__kunmap_local(__addr); \
} while (0)
+#define kunmap_local_dirty(__page, __addr) \
+do { \
+ if (!PageSlab(__page)) \
+ flush_kernel_dcache_page(__page); \
+ kunmap_local(__addr); \
+} while (0)
+
#endif
diff --git a/include/linux/highmem.h b/include/linux/highmem.h
index 832b49b50c7b..65f548db4f2d 100644
--- a/include/linux/highmem.h
+++ b/include/linux/highmem.h
@@ -93,6 +93,10 @@ static inline void kmap_flush_unused(void);
* On HIGHMEM enabled systems mapping a highmem page has the side effect of
* disabling migration in order to keep the virtual address stable across
* preemption. No caller of kmap_local_page() can rely on this side effect.
+ *
+ * If data is written to the returned kernel mapping, the callers needs to
+ * unmap the mapping using kunmap_local_dirty(), else kunmap_local() should
+ * be used.
*/
static inline void *kmap_local_page(struct page *page);
--
2.30.2
There is no need to include genhd.h from a random arch header, and not
doing so prevents the possibility for nasty include loops.
Signed-off-by: Christoph Hellwig <[email protected]>
Reviewed-by: Chaitanya Kulkarni <[email protected]>
Reviewed-by: Ira Weiny <[email protected]>
---
arch/mips/include/asm/mach-rc32434/rb.h | 2 --
1 file changed, 2 deletions(-)
diff --git a/arch/mips/include/asm/mach-rc32434/rb.h b/arch/mips/include/asm/mach-rc32434/rb.h
index d502673a4f6c..34d179ca020b 100644
--- a/arch/mips/include/asm/mach-rc32434/rb.h
+++ b/arch/mips/include/asm/mach-rc32434/rb.h
@@ -7,8 +7,6 @@
#ifndef __ASM_RC32434_RB_H
#define __ASM_RC32434_RB_H
-#include <linux/genhd.h>
-
#define REGBASE 0x18000000
#define IDT434_REG_BASE ((volatile void *) KSEG1ADDR(REGBASE))
#define UART0BASE 0x58000
--
2.30.2
Add a helper to call kmap_local_page on a bvec. There is no need for
an unmap helper given that kunmap_local accept any address in the mapped
page.
Signed-off-by: Christoph Hellwig <[email protected]>
Reviewed-by: Chaitanya Kulkarni <[email protected]>
Reviewed-by: Ira Weiny <[email protected]>
---
include/linux/bvec.h | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/include/linux/bvec.h b/include/linux/bvec.h
index 883faf5f1523..f8710af18eef 100644
--- a/include/linux/bvec.h
+++ b/include/linux/bvec.h
@@ -7,6 +7,7 @@
#ifndef __LINUX_BVEC_H
#define __LINUX_BVEC_H
+#include <linux/highmem.h>
#include <linux/bug.h>
#include <linux/errno.h>
#include <linux/limits.h>
@@ -183,4 +184,16 @@ static inline void bvec_advance(const struct bio_vec *bvec,
}
}
+/**
+ * bvec_kmap_local - map a bvec into the kernel virtual address space
+ * @bvec: bvec to map
+ *
+ * Must be called on single-page bvecs only. Call kunmap_local on the returned
+ * address to unmap.
+ */
+static inline void *bvec_kmap_local(struct bio_vec *bvec)
+{
+ return kmap_local_page(bvec->bv_page) + bvec->bv_offset;
+}
+
#endif /* __LINUX_BVEC_H */
--
2.30.2
There is no need to disable interrupts in bio_copy_block, and the local
only mappings helps to avoid any sort of problems with stray writes
into the bio data.
Signed-off-by: Christoph Hellwig <[email protected]>
Reviewed-by: Ira Weiny <[email protected]>
---
drivers/md/dm-writecache.c | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
diff --git a/drivers/md/dm-writecache.c b/drivers/md/dm-writecache.c
index aecc246ade26..93ca454eaca9 100644
--- a/drivers/md/dm-writecache.c
+++ b/drivers/md/dm-writecache.c
@@ -1205,14 +1205,13 @@ static void memcpy_flushcache_optimized(void *dest, void *source, size_t size)
static void bio_copy_block(struct dm_writecache *wc, struct bio *bio, void *data)
{
void *buf;
- unsigned long flags;
unsigned size;
int rw = bio_data_dir(bio);
unsigned remaining_size = wc->block_size;
do {
struct bio_vec bv = bio_iter_iovec(bio, bio->bi_iter);
- buf = bvec_kmap_irq(&bv, &flags);
+ buf = bvec_kmap_local(&bv);
size = bv.bv_len;
if (unlikely(size > remaining_size))
size = remaining_size;
@@ -1230,7 +1229,7 @@ static void bio_copy_block(struct dm_writecache *wc, struct bio *bio, void *data
memcpy_flushcache_optimized(data, buf, size);
}
- bvec_kunmap_irq(buf, &flags);
+ kunmap_local(buf);
data = (char *)data + size;
remaining_size -= size;
--
2.30.2
Use memzero_bvec to zero each segment in the bio instead of manually
mapping and zeroing the data.
Signed-off-by: Christoph Hellwig <[email protected]>
Reviewed-by: Chaitanya Kulkarni <[email protected]>
Reviewed-by: Ira Weiny <[email protected]>
---
block/bio.c | 9 ++-------
1 file changed, 2 insertions(+), 7 deletions(-)
diff --git a/block/bio.c b/block/bio.c
index 44205dfb6b60..1d7abdb83a39 100644
--- a/block/bio.c
+++ b/block/bio.c
@@ -495,16 +495,11 @@ EXPORT_SYMBOL(bio_kmalloc);
void zero_fill_bio(struct bio *bio)
{
- unsigned long flags;
struct bio_vec bv;
struct bvec_iter iter;
- bio_for_each_segment(bv, bio, iter) {
- char *data = bvec_kmap_irq(&bv, &flags);
- memset(data, 0, bv.bv_len);
- flush_dcache_page(bv.bv_page);
- bvec_kunmap_irq(data, &flags);
- }
+ bio_for_each_segment(bv, bio, iter)
+ memzero_bvec(&bv);
}
EXPORT_SYMBOL(zero_fill_bio);
--
2.30.2
Use the bvec helpers instead of open coding the copy.
Signed-off-by: Christoph Hellwig <[email protected]>
---
drivers/block/ps3disk.c | 19 +++----------------
1 file changed, 3 insertions(+), 16 deletions(-)
diff --git a/drivers/block/ps3disk.c b/drivers/block/ps3disk.c
index f374ea2c67ce..b7d4c3efd7a8 100644
--- a/drivers/block/ps3disk.c
+++ b/drivers/block/ps3disk.c
@@ -83,26 +83,13 @@ static void ps3disk_scatter_gather(struct ps3_storage_device *dev,
unsigned int offset = 0;
struct req_iterator iter;
struct bio_vec bvec;
- unsigned int i = 0;
- size_t size;
- void *buf;
rq_for_each_segment(bvec, req, iter) {
- unsigned long flags;
- dev_dbg(&dev->sbd.core, "%s:%u: bio %u: %u sectors from %llu\n",
- __func__, __LINE__, i, bio_sectors(iter.bio),
- iter.bio->bi_iter.bi_sector);
-
- size = bvec.bv_len;
- buf = bvec_kmap_irq(&bvec, &flags);
if (gather)
- memcpy(dev->bounce_buf+offset, buf, size);
+ memcpy_from_bvec(dev->bounce_buf + offset, &bvec);
else
- memcpy(buf, dev->bounce_buf+offset, size);
- offset += size;
- flush_kernel_dcache_page(bvec.bv_page);
- bvec_kunmap_irq(buf, &flags);
- i++;
+ memcpy_to_bvec(&bvec, dev->bounce_buf + offset);
+ offset += bvec.bv_len;
}
}
--
2.30.2
These two helpers are entirely unused now.
Signed-off-by: Christoph Hellwig <[email protected]>
---
include/linux/bio.h | 42 ------------------------------------------
1 file changed, 42 deletions(-)
diff --git a/include/linux/bio.h b/include/linux/bio.h
index d2b98efb5cc5..8070f3f77c14 100644
--- a/include/linux/bio.h
+++ b/include/linux/bio.h
@@ -5,7 +5,6 @@
#ifndef __LINUX_BIO_H
#define __LINUX_BIO_H
-#include <linux/highmem.h>
#include <linux/mempool.h>
#include <linux/ioprio.h>
/* struct bio, bio_vec and BIO_* flags are defined in blk_types.h */
@@ -519,47 +518,6 @@ static inline void bio_clone_blkg_association(struct bio *dst,
struct bio *src) { }
#endif /* CONFIG_BLK_CGROUP */
-#ifdef CONFIG_HIGHMEM
-/*
- * remember never ever reenable interrupts between a bvec_kmap_irq and
- * bvec_kunmap_irq!
- */
-static inline char *bvec_kmap_irq(struct bio_vec *bvec, unsigned long *flags)
-{
- unsigned long addr;
-
- /*
- * might not be a highmem page, but the preempt/irq count
- * balancing is a lot nicer this way
- */
- local_irq_save(*flags);
- addr = (unsigned long) kmap_atomic(bvec->bv_page);
-
- BUG_ON(addr & ~PAGE_MASK);
-
- return (char *) addr + bvec->bv_offset;
-}
-
-static inline void bvec_kunmap_irq(char *buffer, unsigned long *flags)
-{
- unsigned long ptr = (unsigned long) buffer & PAGE_MASK;
-
- kunmap_atomic((void *) ptr);
- local_irq_restore(*flags);
-}
-
-#else
-static inline char *bvec_kmap_irq(struct bio_vec *bvec, unsigned long *flags)
-{
- return page_address(bvec->bv_page) + bvec->bv_offset;
-}
-
-static inline void bvec_kunmap_irq(char *buffer, unsigned long *flags)
-{
- *flags = 0;
-}
-#endif
-
/*
* BIO list management for use by remapping drivers (e.g. DM or MD) and loop.
*
--
2.30.2
Add helpers to perform common memory operation on a bvec.
Signed-off-by: Christoph Hellwig <[email protected]>
Reviewed-by: Chaitanya Kulkarni <[email protected]>
Reviewed-by: Ira Weiny <[email protected]>
---
include/linux/bvec.h | 33 +++++++++++++++++++++++++++++++++
1 file changed, 33 insertions(+)
diff --git a/include/linux/bvec.h b/include/linux/bvec.h
index f8710af18eef..f9fa43b940ff 100644
--- a/include/linux/bvec.h
+++ b/include/linux/bvec.h
@@ -196,4 +196,37 @@ static inline void *bvec_kmap_local(struct bio_vec *bvec)
return kmap_local_page(bvec->bv_page) + bvec->bv_offset;
}
+/**
+ * memcpy_from_bvec - copy data from a bvec
+ * @bvec: bvec to copy from
+ *
+ * Must be called on single-page bvecs only.
+ */
+static inline void memcpy_from_bvec(char *to, struct bio_vec *bvec)
+{
+ memcpy_from_page(to, bvec->bv_page, bvec->bv_offset, bvec->bv_len);
+}
+
+/**
+ * memcpy_to_bvec - copy data to a bvec
+ * @bvec: bvec to copy to
+ *
+ * Must be called on single-page bvecs only.
+ */
+static inline void memcpy_to_bvec(struct bio_vec *bvec, const char *from)
+{
+ memcpy_to_page(bvec->bv_page, bvec->bv_offset, from, bvec->bv_len);
+}
+
+/**
+ * memzero_bvec - zero all data in a bvec
+ * @bvec: bvec to zero
+ *
+ * Must be called on single-page bvecs only.
+ */
+static inline void memzero_bvec(struct bio_vec *bvec)
+{
+ memzero_page(bvec->bv_page, bvec->bv_offset, bvec->bv_len);
+}
+
#endif /* __LINUX_BVEC_H */
--
2.30.2
Use memzero_bvec instead of reimplementing it.
Signed-off-by: Christoph Hellwig <[email protected]>
Acked-by: Ilya Dryomov <[email protected]>
Reviewed-by: Ira Weiny <[email protected]>
---
drivers/block/rbd.c | 15 ++-------------
1 file changed, 2 insertions(+), 13 deletions(-)
diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c
index 531d390902dd..501a10bfb1fe 100644
--- a/drivers/block/rbd.c
+++ b/drivers/block/rbd.c
@@ -1219,24 +1219,13 @@ static void rbd_dev_mapping_clear(struct rbd_device *rbd_dev)
rbd_dev->mapping.size = 0;
}
-static void zero_bvec(struct bio_vec *bv)
-{
- void *buf;
- unsigned long flags;
-
- buf = bvec_kmap_irq(bv, &flags);
- memset(buf, 0, bv->bv_len);
- flush_dcache_page(bv->bv_page);
- bvec_kunmap_irq(buf, &flags);
-}
-
static void zero_bios(struct ceph_bio_iter *bio_pos, u32 off, u32 bytes)
{
struct ceph_bio_iter it = *bio_pos;
ceph_bio_iter_advance(&it, off);
ceph_bio_iter_advance_step(&it, bytes, ({
- zero_bvec(&bv);
+ memzero_bvec(&bv);
}));
}
@@ -1246,7 +1235,7 @@ static void zero_bvecs(struct ceph_bvec_iter *bvec_pos, u32 off, u32 bytes)
ceph_bvec_iter_advance(&it, off);
ceph_bvec_iter_advance_step(&it, bytes, ({
- zero_bvec(&bv);
+ memzero_bvec(&bv);
}));
}
--
2.30.2
Using local kmaps slightly reduces the chances to stray writes, and
the bvec interface cleans up the code a little bit.
Signed-off-by: Christoph Hellwig <[email protected]>
---
block/t10-pi.c | 16 ++++++----------
1 file changed, 6 insertions(+), 10 deletions(-)
diff --git a/block/t10-pi.c b/block/t10-pi.c
index d910534b3a41..00c203b2a921 100644
--- a/block/t10-pi.c
+++ b/block/t10-pi.c
@@ -147,11 +147,10 @@ static void t10_pi_type1_prepare(struct request *rq)
break;
bip_for_each_vec(iv, bip, iter) {
- void *p, *pmap;
unsigned int j;
+ void *p;
- pmap = kmap_atomic(iv.bv_page);
- p = pmap + iv.bv_offset;
+ p = bvec_kmap_local(&iv);
for (j = 0; j < iv.bv_len; j += tuple_sz) {
struct t10_pi_tuple *pi = p;
@@ -161,8 +160,7 @@ static void t10_pi_type1_prepare(struct request *rq)
ref_tag++;
p += tuple_sz;
}
-
- kunmap_atomic(pmap);
+ kunmap_local(p);
}
bip->bip_flags |= BIP_MAPPED_INTEGRITY;
@@ -195,11 +193,10 @@ static void t10_pi_type1_complete(struct request *rq, unsigned int nr_bytes)
struct bvec_iter iter;
bip_for_each_vec(iv, bip, iter) {
- void *p, *pmap;
unsigned int j;
+ void *p;
- pmap = kmap_atomic(iv.bv_page);
- p = pmap + iv.bv_offset;
+ p = bvec_kmap_local(&iv);
for (j = 0; j < iv.bv_len && intervals; j += tuple_sz) {
struct t10_pi_tuple *pi = p;
@@ -210,8 +207,7 @@ static void t10_pi_type1_complete(struct request *rq, unsigned int nr_bytes)
intervals--;
p += tuple_sz;
}
-
- kunmap_atomic(pmap);
+ kunmap_local(p);
}
}
}
--
2.30.2
Rewrite the actual bounce buffering loop in __blk_queue_bounce to that
the memcpy_to_bvec helper can be used to perform the data copies.
Signed-off-by: Christoph Hellwig <[email protected]>
---
block/bounce.c | 19 +++++++------------
1 file changed, 7 insertions(+), 12 deletions(-)
diff --git a/block/bounce.c b/block/bounce.c
index 7e9e666c04dc..05fc7148489d 100644
--- a/block/bounce.c
+++ b/block/bounce.c
@@ -239,24 +239,19 @@ void __blk_queue_bounce(struct request_queue *q, struct bio **bio_orig)
* because the 'bio' is single-page bvec.
*/
for (i = 0, to = bio->bi_io_vec; i < bio->bi_vcnt; to++, i++) {
- struct page *page = to->bv_page;
+ struct page *bounce_page;
- if (!PageHighMem(page))
+ if (!PageHighMem(to->bv_page))
continue;
- to->bv_page = mempool_alloc(&page_pool, GFP_NOIO);
- inc_zone_page_state(to->bv_page, NR_BOUNCE);
+ bounce_page = mempool_alloc(&page_pool, GFP_NOIO);
+ inc_zone_page_state(bounce_page, NR_BOUNCE);
if (rw == WRITE) {
- char *vto, *vfrom;
-
- flush_dcache_page(page);
-
- vto = page_address(to->bv_page) + to->bv_offset;
- vfrom = kmap_atomic(page) + to->bv_offset;
- memcpy(vto, vfrom, to->bv_len);
- kunmap_atomic(vfrom);
+ flush_dcache_page(to->bv_page);
+ memcpy_from_bvec(page_address(bounce_page), to);
}
+ to->bv_page = bounce_page;
}
trace_block_bio_bounce(*bio_orig);
--
2.30.2
Using local kmaps slightly reduces the chances to stray writes, and
the bvec interface cleans up the code a little bit.
Signed-off-by: Christoph Hellwig <[email protected]>
---
block/bio-integrity.c | 14 ++++++--------
1 file changed, 6 insertions(+), 8 deletions(-)
diff --git a/block/bio-integrity.c b/block/bio-integrity.c
index 4b4eb8964a6f..8f54d49dc500 100644
--- a/block/bio-integrity.c
+++ b/block/bio-integrity.c
@@ -172,18 +172,16 @@ static blk_status_t bio_integrity_process(struct bio *bio,
iter.prot_buf = prot_buf;
__bio_for_each_segment(bv, bio, bviter, *proc_iter) {
- void *kaddr = kmap_atomic(bv.bv_page);
+ void *kaddr = bvec_kmap_local(&bv);
- iter.data_buf = kaddr + bv.bv_offset;
+ iter.data_buf = kaddr;
iter.data_size = bv.bv_len;
-
ret = proc_fn(&iter);
- if (ret) {
- kunmap_atomic(kaddr);
- return ret;
- }
+ kunmap_local(kaddr);
+
+ if (ret)
+ break;
- kunmap_atomic(kaddr);
}
return ret;
}
--
2.30.2
Use the proper helpers instead of open coding the copy.
Signed-off-by: Christoph Hellwig <[email protected]>
---
block/bio.c | 28 ++++++++--------------------
1 file changed, 8 insertions(+), 20 deletions(-)
diff --git a/block/bio.c b/block/bio.c
index 1d7abdb83a39..c14d2e66c084 100644
--- a/block/bio.c
+++ b/block/bio.c
@@ -1186,27 +1186,15 @@ EXPORT_SYMBOL(bio_advance);
void bio_copy_data_iter(struct bio *dst, struct bvec_iter *dst_iter,
struct bio *src, struct bvec_iter *src_iter)
{
- struct bio_vec src_bv, dst_bv;
- void *src_p, *dst_p;
- unsigned bytes;
-
while (src_iter->bi_size && dst_iter->bi_size) {
- src_bv = bio_iter_iovec(src, *src_iter);
- dst_bv = bio_iter_iovec(dst, *dst_iter);
-
- bytes = min(src_bv.bv_len, dst_bv.bv_len);
-
- src_p = kmap_atomic(src_bv.bv_page);
- dst_p = kmap_atomic(dst_bv.bv_page);
-
- memcpy(dst_p + dst_bv.bv_offset,
- src_p + src_bv.bv_offset,
- bytes);
-
- kunmap_atomic(dst_p);
- kunmap_atomic(src_p);
-
- flush_dcache_page(dst_bv.bv_page);
+ struct bio_vec src_bv = bio_iter_iovec(src, *src_iter);
+ struct bio_vec dst_bv = bio_iter_iovec(dst, *dst_iter);
+ unsigned int bytes = min(src_bv.bv_len, dst_bv.bv_len);
+ void *src_buf;
+
+ src_buf = bvec_kmap_local(&src_bv);
+ memcpy_to_bvec(&dst_bv, src_buf);
+ kunmap_local(src_buf);
bio_advance_iter_single(src, src_iter, bytes);
bio_advance_iter_single(dst, dst_iter, bytes);
--
2.30.2
Use memcpy_to_bvec instead of opencoding the logic.
Signed-off-by: Christoph Hellwig <[email protected]>
---
block/bounce.c | 20 ++------------------
1 file changed, 2 insertions(+), 18 deletions(-)
diff --git a/block/bounce.c b/block/bounce.c
index 94081e013c58..7e9e666c04dc 100644
--- a/block/bounce.c
+++ b/block/bounce.c
@@ -67,18 +67,6 @@ static __init int init_emergency_pool(void)
__initcall(init_emergency_pool);
-/*
- * highmem version, map in to vec
- */
-static void bounce_copy_vec(struct bio_vec *to, unsigned char *vfrom)
-{
- unsigned char *vto;
-
- vto = kmap_atomic(to->bv_page);
- memcpy(vto + to->bv_offset, vfrom, to->bv_len);
- kunmap_atomic(vto);
-}
-
/*
* Simple bounce buffer support for highmem pages. Depending on the
* queue gfp mask set, *to may or may not be a highmem page. kmap it
@@ -86,7 +74,6 @@ static void bounce_copy_vec(struct bio_vec *to, unsigned char *vfrom)
*/
static void copy_to_high_bio_irq(struct bio *to, struct bio *from)
{
- unsigned char *vfrom;
struct bio_vec tovec, fromvec;
struct bvec_iter iter;
/*
@@ -104,11 +91,8 @@ static void copy_to_high_bio_irq(struct bio *to, struct bio *from)
* been modified by the block layer, so use the original
* copy, bounce_copy_vec already uses tovec->bv_len
*/
- vfrom = page_address(fromvec.bv_page) +
- tovec.bv_offset;
-
- bounce_copy_vec(&tovec, vfrom);
- flush_dcache_page(tovec.bv_page);
+ memcpy_to_bvec(&tovec, page_address(fromvec.bv_page) +
+ tovec.bv_offset);
}
bio_advance_iter(from, &from_iter, tovec.bv_len);
}
--
2.30.2
Use memcpy_from_bvec instead of open coding the logic.
Signed-off-by: Christoph Hellwig <[email protected]>
---
block/blk-map.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/block/blk-map.c b/block/blk-map.c
index 3743158ddaeb..d1448aaad980 100644
--- a/block/blk-map.c
+++ b/block/blk-map.c
@@ -400,7 +400,7 @@ static void bio_copy_kern_endio_read(struct bio *bio)
struct bvec_iter_all iter_all;
bio_for_each_segment_all(bvec, bio, iter_all) {
- memcpy(p, page_address(bvec->bv_page), bvec->bv_len);
+ memcpy_from_bvec(p, bvec);
p += bvec->bv_len;
}
--
2.30.2
Christoph,
> this series switches the core block layer code and all users of the
> existing bvec kmap helpers to use kmap_local_page. Drivers that
> currently use open coded kmap_atomic calls will converted in a follow
> on series.
Looks OK to me.
Reviewed-by: Martin K. Petersen <[email protected]>
--
Martin K. Petersen Oracle Linux Engineering
On 6/15/21 6:24 AM, Christoph Hellwig wrote:
> +/**
> + * bvec_kmap_local - map a bvec into the kernel virtual address space
> + * @bvec: bvec to map
> + *
> + * Must be called on single-page bvecs only. Call kunmap_local on the returned
> + * address to unmap.
> + */
> +static inline void *bvec_kmap_local(struct bio_vec *bvec)
> +{
> + return kmap_local_page(bvec->bv_page) + bvec->bv_offset;
> +}
Hi Christoph,
Would it be appropriate to add WARN_ON_ONCE(bvec->bv_offset >=
PAGE_SIZE) in this function?
Thanks,
Bart.
Hi Christoph,
On 6/15/21 6:24 AM, Christoph Hellwig wrote:
> Use the bvec helpers instead of open coding the copy.
>
> Signed-off-by: Christoph Hellwig <[email protected]>
> ---
> drivers/block/ps3disk.c | 19 +++----------------
> 1 file changed, 3 insertions(+), 16 deletions(-)
I tested your patch set applied to v5.13-rc6 on PS3 and it seemed to be
working OK.
I did some rsync's, some dd's, some fsck's, etc. If you have anything
you could suggest that you think would exercise your changes I could
try that also.
Tested-by: Geoff Levand <[email protected]>
On Thu, Jun 17, 2021 at 08:01:57PM -0700, Ira Weiny wrote:
>
> > + flush_kernel_dcache_page(__page); \
>
> Is this required on 32bit systems? Why is kunmap_flush_on_unmap() not
> sufficient on 64bit systems? The normal kunmap_local() path does that.
>
> I'm sorry but I did not see a conclusion to my query on V1. Herbert implied the
> he just copied from the crypto code.[1] I'm concerned that this _dirty() call
> is just going to confuse the users of kmap even more. So why can't we get to
> the bottom of why flush_kernel_dcache_page() needs so much logic around it
> before complicating the general kernel users.
>
> I would like to see it go away if possible.
This thread may be related:
https://lwn.net/Articles/240249/
Cheers,
--
Email: Herbert Xu <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
On Tue, Jun 15, 2021 at 03:24:39PM +0200, Christoph Hellwig wrote:
> Add a helper that calls flush_kernel_dcache_page before unmapping the
> local mapping. flush_kernel_dcache_page is required for all pages
> potentially mapped into userspace that were written to using kmap*,
> so having a helper that does the right thing can be very convenient.
>
> Signed-off-by: Christoph Hellwig <[email protected]>
> ---
> include/linux/highmem-internal.h | 7 +++++++
> include/linux/highmem.h | 4 ++++
> 2 files changed, 11 insertions(+)
>
> diff --git a/include/linux/highmem-internal.h b/include/linux/highmem-internal.h
> index 7902c7d8b55f..bd37706db147 100644
> --- a/include/linux/highmem-internal.h
> +++ b/include/linux/highmem-internal.h
> @@ -224,4 +224,11 @@ do { \
> __kunmap_local(__addr); \
> } while (0)
>
> +#define kunmap_local_dirty(__page, __addr) \
I think having to store the page and addr to return to kunmap_local_dirty() is
going to be a pain in some code paths. Not a show stopper but see below...
> +do { \
> + if (!PageSlab(__page)) \
Was there some clarification why the page can't be a Slab page? Or is this
just an optimization?
> + flush_kernel_dcache_page(__page); \
Is this required on 32bit systems? Why is kunmap_flush_on_unmap() not
sufficient on 64bit systems? The normal kunmap_local() path does that.
I'm sorry but I did not see a conclusion to my query on V1. Herbert implied the
he just copied from the crypto code.[1] I'm concerned that this _dirty() call
is just going to confuse the users of kmap even more. So why can't we get to
the bottom of why flush_kernel_dcache_page() needs so much logic around it
before complicating the general kernel users.
I would like to see it go away if possible.
Ira
[1] https://lore.kernel.org/lkml/[email protected]/
> + kunmap_local(__addr); \
> +} while (0)
> +
> #endif
> diff --git a/include/linux/highmem.h b/include/linux/highmem.h
> index 832b49b50c7b..65f548db4f2d 100644
> --- a/include/linux/highmem.h
> +++ b/include/linux/highmem.h
> @@ -93,6 +93,10 @@ static inline void kmap_flush_unused(void);
> * On HIGHMEM enabled systems mapping a highmem page has the side effect of
> * disabling migration in order to keep the virtual address stable across
> * preemption. No caller of kmap_local_page() can rely on this side effect.
> + *
> + * If data is written to the returned kernel mapping, the callers needs to
> + * unmap the mapping using kunmap_local_dirty(), else kunmap_local() should
> + * be used.
> */
> static inline void *kmap_local_page(struct page *page);
>
> --
> 2.30.2
>
On Fri, Jun 18, 2021 at 11:37:28AM +0800, Herbert Xu wrote:
> On Thu, Jun 17, 2021 at 08:01:57PM -0700, Ira Weiny wrote:
> >
> > > + flush_kernel_dcache_page(__page); \
> >
> > Is this required on 32bit systems? Why is kunmap_flush_on_unmap() not
> > sufficient on 64bit systems? The normal kunmap_local() path does that.
> >
> > I'm sorry but I did not see a conclusion to my query on V1. Herbert implied the
> > he just copied from the crypto code.[1] I'm concerned that this _dirty() call
> > is just going to confuse the users of kmap even more. So why can't we get to
> > the bottom of why flush_kernel_dcache_page() needs so much logic around it
> > before complicating the general kernel users.
> >
> > I would like to see it go away if possible.
>
> This thread may be related:
>
> https://lwn.net/Articles/240249/
Interesting! Thanks!
Digging around a bit more I found:
https://lore.kernel.org/patchwork/patch/439637/
Auditing all the flush_dcache_page() arch code reveals that the mapping field
is either unused, or is checked for NULL. Furthermore, all the implementations
call page_mapping_file() which further limits the page to not be a swap page.
All flush_kernel_dcache_page() implementations appears to operate the same way
in all arch's which define that call.
So I'm confident now that additional !PageSlab(__page) checks are not needed
and this patch is unnecessary. Christoph, can we leave this out of the kmap
API and just fold the flush_kernel_dcache_page() calls back into the bvec code?
Unfortunately, I'm not convinced this can be handled completely by
kunmap_local() nor the mem*_page() calls because there is a difference between
flush_dcache_page() and flush_kernel_dcache_page() in most archs... [parisc
being an exception which falls back to flush_kernel_dcache_page()]...
It seems like the generic unmap path _should_ be able to determine which call
to make based on the page but I'd have to look at that more.
Ira