To:     Eric Wheeler <bcache@lists.ewheeler.net>
Cc:     Coly Li <colyli@suse.de>, linux-block@vger.kernel.org,
        Jonathan Corbet <corbet@lwn.net>,
        Kent Overstreet <kent.overstreet@gmail.com>,
        "open list:DOCUMENTATION" <linux-doc@vger.kernel.org>,
        open list <linux-kernel@vger.kernel.org>,
        "open list:BCACHE (BLOCK LAYER CACHE)" <linux-bcache@vger.kernel.org>,
        "Martin K. Petersen" <martin.petersen@oracle.com>
Subject: Re: [PATCH] bcache: make stripe_size configurable and persistent
 for hardware raid5/6
From:   "Martin K. Petersen" <martin.petersen@oracle.com>
Organization: Oracle Corporation
Message-ID: <yq15yqvw1f0.fsf@ca-mkp.ca.oracle.com>
References: <d3f7fd44-9287-c7fa-ee95-c3b8a4d56c93@suse.de>
        <1561245371-10235-1-git-send-email-bcache@lists.ewheeler.net>
        <200638b0-7cba-38b4-20c4-b325f3cfe862@suse.de>
        <alpine.LRH.2.11.1906241800350.1114@mx.ewheeler.net>
        <8a9131dc-9bf7-a24a-f7b8-35e0c019e905@suse.de>
        <fdb85dc1-eee6-e55e-8e9c-fa1f36b4a37@ewheeler.net>
Date:   Fri, 07 Jan 2022 19:21:28 -0500
In-Reply-To: <fdb85dc1-eee6-e55e-8e9c-fa1f36b4a37@ewheeler.net> (Eric
        Wheeler's message of "Wed, 5 Jan 2022 19:29:05 -0800 (PST)")
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?ik4jk7PQP2G3xXstYWpsC0lmvdAArm4QRRgP1bbYZ1BtMHMkGGnC2z07mYaA?=
 =?us-ascii?Q?4hbGVXB6UZWO3jHkFKh/HELDJ3xnbHdLS9nm4ctTxBAjhDTEj7ZtUrK6gGZV?=
 =?us-ascii?Q?INPluJGFgKkgvi+IxSJRhvLLqgM4o7dM3Ag0+701XGKgAQXoHyuBIewdOBdx?=
 =?us-ascii?Q?0hqQZtPi/W7zMfl5MwaVV4YCyGV/UetGPo/4JTR+zT+oCT/hVoTBxUQsoCMh?=
 =?us-ascii?Q?I5Mf4SFNP+k5dX1pX3pWgN6os2fTIUNec3NKAb71oNhcIql2gJVtAPz0ouao?=
 =?us-ascii?Q?WTD1f6rLIIt0OqreipVEujqBs2CUcnl0x/DuDBjKz9z29fbSF++DLXBh0w1S?=
 =?us-ascii?Q?BR8kYXSnarW/614WwA5Ca8dPSoUiojPVjQCR9AvvX87q/ETqg4t7CT4YMg7s?=
 =?us-ascii?Q?zNXrl9EoajJ4p3YGeLotzX9OmDoDVoBCf1nc7y7a/rNGkd1LsiThC7miX2Zo?=
 =?us-ascii?Q?oUWjX0jcK+iW3pEG7oKRoGnae5924H4gkks7iXIl1hoqbH5eDJl0Q6fW1cuS?=
 =?us-ascii?Q?0RQmziGw+4LpM6ksPWSKvLguzjtKWDzBsjpz9msaQmF1qxvCUqQgAdXMEJ6X?=
 =?us-ascii?Q?LHPwX5Fxt5eIo/D0HaRj8xeeN3utJ3ePYrpwiHKuRg1RtPvnvYJw5WY78RC3?=
 =?us-ascii?Q?Ouo1XOk40Cjjv/acALp3r6L+DyaMCCQikNcCD7ezXbz3L8NTEp/CJn6Wts6r?=
 =?us-ascii?Q?bkkuzenZd9ssW/1d2MjyL8n3+uQyCBGQHLeOzDUHprcPoKpjFBZGjdWuPoMU?=
 =?us-ascii?Q?SM7F9RFWKXYEiscu8jTMA2IK69KMHcnsAFHZWslT51kf7aYUnoXJOvkzWcIj?=
 =?us-ascii?Q?/YWsGJY2/AQzoZI1RqJGnvNm9z5XGUe/mbv4byAtxsx0xTc7/nnBYtZcT/7k?=
 =?us-ascii?Q?n+V2B5YjJVgPimGfLFBYuTuXeyYeExQxB6mM7j9WYV2+p/WCHo/7EOXKD4FA?=
 =?us-ascii?Q?vK8bDvYWQyGA7Pfo/S6WVYhasoJ/4fxmhoGWQ4ljPqV5lBV73ye8orFKqYAT?=
 =?us-ascii?Q?X3B2uqhcXAFOdxiclRHjlpTEUKJuYaFPKV7TRhu7xfvK44w8hogv2WLgEahb?=
 =?us-ascii?Q?uzfxOOb+l31CKUk3isLQxFKLHikieDjBYH2tuIz56pVrEB6TeNUNjzx0laeR?=
 =?us-ascii?Q?NwnTcf1hl2t0cHiM1n+wpwxzJiakRED8YxpllK3rSyAsIXFqPSOhgA6mW/ML?=
 =?us-ascii?Q?Zei+3CO4Ib/zy0naIdPTsvQaBNz673TKQVYtkAXDxBA4VJeLlNcOxiEMGUks?=
 =?us-ascii?Q?FaovtA6zINrDeJdvpVYp4LFg7yoNu7G9ZP2UqMGbTu7K5aRMDTu75dXxZg/q?=
 =?us-ascii?Q?yBMuvAI7ccO/iF+9RSzpEXJK+ePuY00tTYy2rH7SDZeZ1Igw7gNWDAX3Nxh7?=
 =?us-ascii?Q?JqKekuV0muy/gNqYP3IsAifrKPRfZ8/WimSngX8gdLmdntGOv58ZGvL/eVUz?=
 =?us-ascii?Q?a15X8y+DmPoNHdrrpuQrWOLV3yX80hJBsRLG1K9n5jjIiDL+Lbqg6w8K8Fs0?=
 =?us-ascii?Q?ycA2P1Uo3kbAYfJix8OCGNcAyyOKvgn5BHhCdViN5J+4Ors6WSm4/ivf15aQ?=
 =?us-ascii?Q?TuNT4QIYMjd4xgwvkmdFvtpebyRzuhH9oQ+5Uifjcx9qx1To2DAGLBtCXhLt?=
 =?us-ascii?Q?bG4bFK1t2enxMA0ohJx8Llb3ATT/pxzwdLhYsIkDnHv0WwlzN5yULF6BPSFi?=
 =?us-ascii?Q?67ccqg=3D=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 121dc24e-35fc-40ef-d3b5-08d9d23cd227
X-MS-Exchange-CrossTenant-AuthSource: PH0PR10MB4759.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Jan 2022 00:21:31.7206
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: hJrIbXulQ6StTMUefIEOSghXVHMUcHA3M1qvNP8Wn0wFsKt3+1F+kDr7sIm3fxn3mrV4NXGNo7BkY0/h1coTwsAkIRYbPSyVYHSAYQr13x4=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR10MB4680
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10220 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 bulkscore=0 mlxscore=0
 phishscore=0 malwarescore=0 spamscore=0 adultscore=0 suspectscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2110150000
 definitions=main-2201070140
X-Proofpoint-ORIG-GUID: dJYvrOy5FGYEB4YUnP6yYk60QYrINz8x
X-Proofpoint-GUID: dJYvrOy5FGYEB4YUnP6yYk60QYrINz8x
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


Eric,

> Even new new RAID controlers that _do_ provide `io_opt` still do _not_ 
> indicate partial_stripes_expensive (which is an mdraid feature, but Martin 
> please correct me if I'm wrong here).

partial_stripes_expensive is a bcache thing, I am not sure why it needs
a separate flag. It is implied, although I guess one could argue that
RAID0 is a special case since partial writes are not as painful as with
parity RAID.

The SCSI spec states that submitting an I/O that is smaller than io_min
"may incur delays in processing the command". And similarly, submitting
a command larger than io_opt "may incur delays in processing the
command".

IOW, the spec says "don't write less than an aligned multiple of the
stripe chunk size" and "don't write more than an aligned full
stripe". That leaves "aligned multiples of the stripe chunk size but
less than the full stripe width" unaccounted for. And I guess that's
what the bcache flag is trying to capture.

SCSI doesn't go into details about RAID levels and other implementation
details which is why the wording is deliberately vague. But obviously
the expectation is that partial stripe writes are slower than full.

In my book any component in the stack that sees either io_min or io_opt
should try very hard to send I/Os that are aligned multiples of those
values. I am not opposed to letting users manually twiddle the
settings. But I do think that we should aim for the stack doing the
right thing when it sees io_opt reported on a device.

-- 
Martin K. Petersen	Oracle Linux Engineering