Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) client-ip=23.128.96.38;
From:   David Wang <00107082@163.com>
To:     liam.howlett@oracle.com
Cc:     akpm@linux-foundation.org, ankitag@nvidia.com,
        bagasdotme@gmail.com, chunn@nvidia.com,
        linux-kernel@vger.kernel.org, linux-mm@kvack.org,
        regressions@lists.linux.dev, surenb@google.com, willy@infradead.org
Subject: Re: [REGRESSION]: mmap performance regression starting with k-6.1
Date:   Sun, 26 Nov 2023 15:18:54 +0800
Message-Id: <20231126071854.19490-1-00107082@163.com>
In-Reply-To: <20231123143452.erzar3sqhg37hjxz@revolver>
References: <20231123143452.erzar3sqhg37hjxz@revolver>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Precedence: bulk


> What this gains us is the ability to remove contention on the mmap lock
> during page faults.  If you were to test contention around that lock,
> you will see a slowdown until you reach v6.4, where per-vma locking
> started to show up.  More benchmarking will show different types of
> fault handling outside of the mmap lock until (I believe) 6.6, where
> most (or all?) types are supported.

I add memory access between mmap and munmap to the simple stress, and timeit.
Following are the numbers measuring total system time for 10,000,000 rounds, it is not very good for 6.7.0-rc2
(The delta column is just "page fault" - "no page fault", roughly the extra time
needed in kernel to deal with page fault, I guess.)

+-----------+------------+---------------+------------------------------------+
|           | page fault | no page fault | delta(kernel time for page fault?) |
+-----------+------------+---------------+------------------------------------+
|   6.0.0   |    64s     |      13s      |                51s                 |
|   6.1.0   |    104s    |      49s      |                55s                 |
| 6.7.0-rc2 |   ~210s    |      67s      |                143s                |
+-----------+------------+---------------+------------------------------------+

Maybe there is something here needed to be tracked.

My test code now is:

	#define MAXN 1024
	struct { void* addr; size_t n; } maps[MAXN];
	void accessit(char *addr, size_t n) {
		for (int i=0; i<n; i+=128) addr[i]=i;
	}
	int main() {
		int i, n, k, r;
		void *p;
		for (i=0; i<MAXN; i++) {
			n = 1024*((rand()%32)+1);
			p = mmap(NULL, n, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0);
			if (p == MAP_FAILED) {
				perror("fail to mmap");
				return -1;
			}
			maps[i].addr = p; 
			maps[i].n = n;

		}
		for (i=0; i<10000000; i++) {
			k = rand()%MAXN;
	#ifdef PAGE_FAULT
			accessit((char*)maps[k].addr, maps[k].n);
	#endif
			r = munmap(maps[k].addr, maps[k].n);
			if (r) {
				perror("fail to munmap");
				return -1;
			}
			n = 1024*((rand()%32)+1);
			p = mmap(NULL, n, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0);
			if (p == MAP_FAILED) {
				perror("fail to mmap");
				return -1;
			}
			maps[k].addr = p; 
			maps[k].n = n;
		}
		for (i=0; i<MAXN; i++) munmap(maps[i].addr, maps[i].n);
		return 0;
	}

Thanks
David Wang