Received-SPF: pass (google.com: domain of linux-kernel+bounces-87283-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Date: Thu, 29 Feb 2024 11:25:13 -0700
From: Greg Edwards <gedwards@ddn.com>
To: Jens Axboe <axboe@kernel.dk>
Cc: "Matthew Wilcox (Oracle)" <willy@infradead.org>,
	Andrew Morton <akpm@linux-foundation.org>,
	"Kirill A . Shutemov" <kirill.shutemov@linux.intel.com>,
	Hugh Dickins <hughd@google.com>, linux-mm@kvack.org,
	linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org,
	linux-kernel@vger.kernel.org, stable@vger.kernel.org,
	qemu-devel@nongnu.org
Subject: Re: [PATCH] block: Remove special-casing of compound pages
Message-ID: <20240229182513.GA17355@bobdog.home.arpa>
References: <20230814144100.596749-1-willy@infradead.org>
 <170198306635.1954272.10907610290128291539.b4-ty@kernel.dk>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <170198306635.1954272.10907610290128291539.b4-ty@kernel.dk>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?tGV8uThtdu2XszHnQzzbq2mRCkKBX/alYZ4UUrleCeWrp5wI1s2/kaJ7zOT5?=
 =?us-ascii?Q?0YUcoGPLLqp9rOisa3vUA5S+MX5+2ke2Fq0kZh6pZMiPkG5i8c+FSS11OsK2?=
 =?us-ascii?Q?LngIbj52XEZm68blg8BL08FvvUW2aQU6AvDKgHmdIDeFfZKBQszBIxRatCHL?=
 =?us-ascii?Q?BIJogVdkHmWWAIyBcK9D4BhRqXQr/C+bs9j09lpCiQlWmJL1r7PDBkWuaDvs?=
 =?us-ascii?Q?oyvcgGLbZhllmZ3z/6/Q8OYKQLkKR5xlEmRbV+VgWqgU0EPwpDTjKIkBtG9z?=
 =?us-ascii?Q?MrvW8+9B/VsQ6Eq24ZMKaZmJtKcKhLH8npGNiPII4blf3r7xXvW+ZKi38WDJ?=
 =?us-ascii?Q?s5NDVtvCLbWuon5bRrAzLZaC2y42uIdO2AEv+6AhcvB2D+kVMkA+lOndTLa7?=
 =?us-ascii?Q?ybH7EN4eBaD1xBZi/zdja2PjW3HxHEAKhA0wu6KpgX1gr1HiyZXY4rXzgPv8?=
 =?us-ascii?Q?wwWTth0e3qCRBHg+Qs/jJqEtC/QkgU0Twt82dobiB1HwOroNW3+T3rrwir2J?=
 =?us-ascii?Q?DEU3LaP1HhoWFYkKKZIr9ZDq9hXtzxRk5wUi5F4xaSoRdOaPAWN1ESMVeIXq?=
 =?us-ascii?Q?hXHYbZrLjTspjR0Go2ehKZwXa0zJUJSr1LRcfF1lsJJs1K+zkBW8zf9pCtT0?=
 =?us-ascii?Q?QdpdKIlZcbjEYta4upqs57/jwHU9bGVV6QhyQQrUDPbwm8X2gEJOrqHICZUC?=
 =?us-ascii?Q?8K4eJ5L/OOPPioTvGtuzRsD6Q1/bKSAsyJ00eeEV5YZT40aw0bhn0yWVbouu?=
 =?us-ascii?Q?LQmE5IWH9BsgWDt7TktDh1W2FfH7c4AbiCQmjkFJJ/Wc3jfQTdUMWx14ui5x?=
 =?us-ascii?Q?Cjhi7wcWl5irmB4vO1nqRGcaNsfpoXbc5S8IjQV7CsTV+LZ01n8kpacr52XO?=
 =?us-ascii?Q?TtYdwRIidvQo8ydaAGrw4y0oW/g8xMQqKa/EztWOXNPhv/18hd+KhPL1zSt0?=
 =?us-ascii?Q?tJdrHM/SuQpv8Tce++noFTE1Gw/+DI1oNX/rAuI4MKr7nC9YNpZmJLaxsPtQ?=
 =?us-ascii?Q?6KBtcssz2XfvgUZsbbAN4xsZgQLXEkQD35/ZlTIRVcki3MfLikSK/XlzDg9L?=
 =?us-ascii?Q?sbWhEYAHOHDfb/Ul/dTlIt6R46Q+tUAzdYBn4Og4neySXqVMqqMX2WRbr7U5?=
 =?us-ascii?Q?4LnOeiwDJxdiFriuMLsoYgweTgveXISBuOUDAPQhyV/zwXJ75Y9l0qjX4ZbO?=
 =?us-ascii?Q?NYYkDc3YxZysr+AmcclsYtgP7fI4Rh58HUxZ70DN/cesbhZuwCwiMf9SewPk?=
 =?us-ascii?Q?e0jlxVu0R8LsVdcAA4g/uC1mgQpn3ppizTILOyzra8GAiha5p+13Wj/2WtHT?=
 =?us-ascii?Q?eaP5p+Vrsryys3lNxjZ2bc0yEUOpIVNEIo7EbHUAeTy6ZjbyJ3ocoXYAhgL5?=
 =?us-ascii?Q?mNVTCtz/10BbwPycqlxJDUyDETbal9LHO9BwOfU1zES+vTRRCEJLCNX7ifPW?=
 =?us-ascii?Q?m/RKXuPMk4pGYPII6SBq6OAxXx73J8VyREuEAbKjyQbOCdCOIlKrTy0Y5K8l?=
 =?us-ascii?Q?OIYbQmhdUQy+9y0S4VZxPh6Ji6YqQS8FQ7N+yoREahUSVe5qRSU5y4N39Lre?=
 =?us-ascii?Q?sKyzKYVveYSKiYvhbBkto+xh5PNbTiCvIvvivl+juj3NKR6375xzJRX3kMSO?=
 =?us-ascii?Q?zg=3D=3D?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0:
	TBjph9xbz8bLxv8DN88bwPsJxVkhz/6U736V3pKAI/c6poe0mXtB1tH6QBMu9AAPV9RgHomz2lA9T3enB4Zrtz00ImmH35xHC1MX07aRwfMWqhEemx5nz+1nXTOkoZYsQFRsGhOkBNv9mrtbPm6yhCN/5zvvRQlH704jYh4kUZttPv3vhNvDbcGArFvUwXt9noSTqp9UaSjkGs4Dkb06L5GYIGrcLDDtO/R9z3gHhl0OiLIhzgcth9QQzDUlb2nKEBsCWhTOuRVae2fa5+3ADAedlmW8nkWH+K1Ih3LaPzSuB5DJe1nWl/TKYtNPr6eu9M3/oE+5kM3uQTC2eKOf5txz4fprVGPh1SHGNe9mr9XM/rXzgEKOcw6z7IzZM/n05AItd1c25HZCHS1YYdYllHZpVNH4jbZoxwK900ADdKhJ+Vw5y9cBBj81JhVZN2m3GuIIe5jbhxQs/8oS7o56h3ua1hnixcUNPjkjXjvMaM2yVH1bPgFXok9QYCCNUSTXJw83eaE+Y5pDROctXgUmwavqffxv+21CpVSYErrs6MByXwMdaiN9ckNI2xXiBUapzLxCDrfan6FrlWZnFr1gvHV/e3JGt23XvmUv9jgT7qYjdi4kI0hX+iPBnBWe4a5scVRBaDb0UoRmIfZ/imdwTg==
X-OriginatorOrg: ddn.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 32333872-1f7a-4e56-6390-08dc3953c754
X-MS-Exchange-CrossTenant-AuthSource: SA3PR19MB8193.namprd19.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Feb 2024 18:25:17.0422
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 753b6e26-6fd3-43e6-8248-3f1735d59bb4
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Ma4BOxL1qcPkrHr69mNgY3HK3mpKn1sbTlmjy5FVkkha7b6KXWpMlsTx4I1diRDxdT8LKrSAUEPmpAgtKTaKNg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA3PR19MB7588
X-BESS-ID: 1709231120-102909-12414-11066-1
X-BESS-VER: 2019.1_20240227.2356
X-BESS-Apparent-Source-IP: 104.47.74.40
X-BESS-Parts: H4sIAAAAAAACA4uuVkqtKFGyUioBkjpK+cVKVkbGRhZAVgZQ0NQ41TAl1dQ42c
	TEIDXN1CItzcQsNTXZPMki0cIk0cxCqTYWAO3T0Y1BAAAA
X-BESS-Outbound-Spam-Score: 0.00
X-BESS-Outbound-Spam-Report: Code version 3.2, rules version 3.2.2.254555 [from 
	cloudscan19-30.us-east-2b.ess.aws.cudaops.com]
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------
	0.00 BSF_BESS_OUTBOUND      META: BESS Outbound 
X-BESS-Outbound-Spam-Status: SCORE=0.00 using account:ESS124931 scores of KILL_LEVEL=7.0 tests=BSF_BESS_OUTBOUND
X-BESS-BRTS-Status:1

On Thu, Dec 07, 2023 at 02:04:26PM -0700, Jens Axboe wrote:
> On Mon, 14 Aug 2023 15:41:00 +0100, Matthew Wilcox (Oracle) wrote:
>> The special casing was originally added in pre-git history; reproducing
>> the commit log here:
>>
>>> commit a318a92567d77
>>> Author: Andrew Morton <akpm@osdl.org>
>>> Date:   Sun Sep 21 01:42:22 2003 -0700
>>>
>>>     [PATCH] Speed up direct-io hugetlbpage handling
>>>
>>>     This patch short-circuits all the direct-io page dirtying logic for
>>>     higher-order pages.  Without this, we pointlessly bounce BIOs up to
>>>     keventd all the time.
>>
>> [...]
>
> Applied, thanks!
>
> [1/1] block: Remove special-casing of compound pages
>       commit: 1b151e2435fc3a9b10c8946c6aebe9f3e1938c55

This commit results in a change of behavior for QEMU VMs backed by hugepages
that open their VM disk image file with O_DIRECT (QEMU cache=none or
cache.direct=on options).  When the VM shuts down and the QEMU process exits,
one or two hugepages may fail to free correctly.  It appears to be a race, as
it doesn't happen every time.

From debugging on 6.8-rc6, when it occurs, the hugepage that fails to free has
a non-zero refcount when it hits the folio_put_testzero(folio) test in
release_pages().  On a failure test iteration with 1 GiB hugepages, the failing
folio had a mapcount of 0, refcount of 35, and folio_maybe_dma_pinned was true.

The problem only occurs when the VM disk image file is opened with O_DIRECT.
When using QEMU cache=writeback or cache.direct=off options, it does not occur.
We first noticed it on the 6.1.y stable kernel when this commit landed there
(6.1.75).

A very simple reproducer without KVM (just boot VM up, then shut it down):

echo 512 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
qemu-system-x86_64 \
	-cpu qemu64 \
	-m 1024 \
	-nographic \
	-mem-path /dev/hugepages/vm00 \
	-mem-prealloc \
	-drive file=test.qcow2,if=none,cache=none,id=drive0 \
	-device virtio-blk-pci,drive=drive0,id=disk0,bootindex=1
rm -f /dev/hugepages/vm00

Some testing notes:

  * occurs with 6.1.75, 6.6.14, 6.8-rc6, and linux-next-20240229
  * occurs with 1 GiB and 2 MiB huge pages, with both hugetlbfs and memfd
  * occurs with QEMU 8.0.y, 8.1.y, 8.2.y, and master
  * occurs with (-enable-kvm -cpu host) or without (-cpu qemu64) KVM

Thanks for your time!

Greg