Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Subject: Re: [PATCH v1 4/4] swiotlb: panic if nslabs is too small
To:     Robin Murphy <robin.murphy@arm.com>,
        Christoph Hellwig <hch@infradead.org>,
        Yu Zhao <yuzhao@google.com>
Cc:     ak@linux.intel.com, akpm@linux-foundation.org,
        alexander.sverdlin@nokia.com, andi.kleen@intel.com, bp@alien8.de,
        bp@suse.de, cminyard@mvista.com, corbet@lwn.net,
        damien.lemoal@opensource.wdc.com, dave.hansen@linux.intel.com,
        iommu@lists.linux-foundation.org, joe.jin@oracle.com,
        joe@perches.com, keescook@chromium.org, kirill.shutemov@intel.com,
        kys@microsoft.com, linux-doc@vger.kernel.org,
        linux-hyperv@vger.kernel.org, linux-kernel@vger.kernel.org,
        linux-mips@vger.kernel.org, ltykernel@gmail.com,
        michael.h.kelley@microsoft.com, mingo@redhat.com,
        m.szyprowski@samsung.com, parri.andrea@gmail.com,
        paulmck@kernel.org, pmladek@suse.com, rdunlap@infradead.org,
        tglx@linutronix.de, thomas.lendacky@amd.com,
        Tianyu.Lan@microsoft.com, tsbogend@alpha.franken.de,
        vkuznets@redhat.com, wei.liu@kernel.org, x86@kernel.org
References: <20220611082514.37112-5-dongli.zhang@oracle.com>
 <20220820012031.1285979-1-yuzhao@google.com>
 <f8c743d8-fcbe-4ef7-5f86-d63086552ffd@arm.com>
 <YwNn92WP3rP4ylZu@infradead.org>
 <d5016c1e-55d9-4224-278a-50377d4c6454@arm.com>
From:   Dongli Zhang <dongli.zhang@oracle.com>
Message-ID: <82d5b78d-e027-316a-87de-f76f4383d736@oracle.com>
Date:   Mon, 22 Aug 2022 15:27:47 -0700
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.6.1
In-Reply-To: <d5016c1e-55d9-4224-278a-50377d4c6454@arm.com>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?em83bENJNHphSDM5UE5SNjFNQk9zUmQ3VGUydSthZjYxTE1Nb1ErRkk1WUJT?=
 =?utf-8?B?ZDNMd1ZmMlRiemljRlZXTUN0b3JmY3JXSitxSFhlQW5tald1bnRyU2ZBSDFS?=
 =?utf-8?B?S2IxeG5uUW9CNHNEbHZLZnFZRkRzcGRDSzZJMks0QjhJQjZnVlBlN2kzZEdy?=
 =?utf-8?B?RWE5ZGtRTjNtZGlNRDlwSXB6WjZ4RjZaL2ZoQ1ZrU0laVjhsNW5vVUFxSUgw?=
 =?utf-8?B?bEtyVldHUHo4UXJUSzRrd1FGb2tvN1k5My9FL2JHMmlROFFjS3JTb1UrMlZH?=
 =?utf-8?B?WFZqR0dYWW9TQTczZ3FoTUExNFlucVVzbEpocGE1T2Jtb2I3L1pxV2dSU2p0?=
 =?utf-8?B?NW5iZHY1S0MramFvM3RJOHY4ajlJSCtPWU9nNjNQOGFza3BiNUZQTGhzaXJM?=
 =?utf-8?B?VDA0Z1dWSVNqc21pSVcvUjAwL2VrVitsK3RkTGc4YktWVkRhQU1BeFU2SE5v?=
 =?utf-8?B?V2hoZG92bUtsdzJNSXd4aEFRdG1PR0hYazRUNFdhYjc3YWo4cGxLNFBzc1RV?=
 =?utf-8?B?Umkvbldlczc5c2U3Q1orVU84YmhDaHVUQktYMjNxRTJXWFkzZmdMRVVtSmpP?=
 =?utf-8?B?T2VEVmo3UkhtNHZBRlNURThmZmdBcW5FaXhOc2tXMVlKUHljTEZGV05JMHJl?=
 =?utf-8?B?VWx1TFJLVk9wc2JPdmlxWkFyZkF1MG56dlpBLzFQZnZvVitXeTU1MnJPTEtY?=
 =?utf-8?B?YnFBZDRHQzBoaGhvMDFxaUZJcGNMZXlkR2dHTGhpOG5MUmwzdE1KS0lJa1hk?=
 =?utf-8?B?bWQ1ekdqaFJ0VTkyOWZQMU5FTE5nQldhL3hhekJJSDBQMmhPeVJPa1E0KzNR?=
 =?utf-8?B?MHBoS0xpZ3RvbExFK0dtRXZ0TWFVSkt3Y1FyblR2cTI0YlJnRW9aTnFFa1Ns?=
 =?utf-8?B?b0RWU2RTbHY0YzVNM0hZZ0haanJLQVJwYW9OQjRYZXNuUklITGNuSEdFaVV5?=
 =?utf-8?B?c0VrSWI4R0lheG55UDNTU3k1aTViQXFmUUxkL3hjM282a2JkTG1HRE1IaHBv?=
 =?utf-8?B?MzQzenVCTkJsblFjYjc5dHRndTArTnZoU1grTExMUlFhamVTQ3ZGS1VaQm90?=
 =?utf-8?B?NUdNTlB5S0t1S20yc2xKZ200Q2xBZnJJay9mOUpqdzlUOTdtWkhnTG5OUjZR?=
 =?utf-8?B?YzIyb3d0WDlDQ1ZLeTFLZFVud2huT0NnT00xeFpHZ01NUjkzWldrNGxRa0I2?=
 =?utf-8?B?VHRzZjJ1OXl4bDFHSzhpMW1XWVk4T01KU1ZZaS9Tcndkb1pUZ093WjlNcTU5?=
 =?utf-8?B?ZlBxSnBHd0twNTl2UERwL3dsYjJWeTM2Qm9XaFo0QUt1QU4xdUszSmZGdXQy?=
 =?utf-8?B?VTdXbndZZ3l3cEtnTHRFa2g0RTUxa1B6TUI0UXN0b3dmOXZreWhja3UvTEh0?=
 =?utf-8?B?SmVXSE9IcHVCVkpQaU1VMDZWMWZQZU5vQ0Y5U3J0VUQ5Nk55ZDdKdjlrL284?=
 =?utf-8?B?bGMyb3pxZGFWNlN2VXRJY1F3bmlMNjhrRWlxWVhrc2dzb3RQWjJwd0g1RlBY?=
 =?utf-8?B?OURWUEVYZVJGdVlDSGJsZWlRdGl1WmE5K0RiMnNVeHp1U0VrQ21aZXFjcHc3?=
 =?utf-8?B?ZUFhVkNZR1dJR212Z1RrS0FOVXc2WUxPRTlhWnEyb2ZMWW9RMVg5RHJNdlRQ?=
 =?utf-8?B?SjhjZ1ZORU9WYitGVWJkUWs2eFRMNURtdnVRdmNUMkIyMDVhdFBRbjNiK0hJ?=
 =?utf-8?B?T2lFU1dWR0Z5SndVNE5XU3labjNKYlBsZkNhZDJTRXA1Lzc1Y2ZCSC9yVmpa?=
 =?utf-8?B?dFFQZnZGWUdXK3NWTXA2MG5UWHBtaFZmY2p1Wk9SNVlMR2ZhT2NrdUJucjA4?=
 =?utf-8?B?TUxsdVl6OW8zc1dRcW5FR2dEak1aaXJsVmhtZVFmb1hIUndDbU9uSk1TRkYw?=
 =?utf-8?B?c0tkRnhOWmNKcEwzakMrbHJ4WXVuWEwxQmpycTYzamxFU0EzSFZaU3Rrc21M?=
 =?utf-8?B?MHlnbUF6N0ROMmlzRHNXR0Y1V1phdkQ3c1piVEVpd0V6UkhsL3dKbEZWemFo?=
 =?utf-8?B?L3lRWVFuVjZwYjEvVWpUQnVoYXM4aCtpZVVDNXJqOEU5dG1yQS85eUZsOGFN?=
 =?utf-8?B?enlDR2xOU3Z6aklIWHBxalhIbkV6bmNVWmt5ZTBZWW9nNHBMYURBOXNEWEpj?=
 =?utf-8?B?L0tlYUozQ1B0S2FVS2JOMXFtQjVVZ2F2ZTVoOExOQkE3NnZtWjIrbzlmL0Fs?=
 =?utf-8?Q?32cyi4heAdkD5HJ/8haer70=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: a2d83002-0db5-4032-3150-08da848d907a
X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB2663.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Aug 2022 22:27:57.6934
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: IPUL5oK5am5JPsPAm492RW0JdT6cLRQjrlTrS9SYTEuJGtpVdut1SEKhV1fvHMjS/7FwCtzQxOZ3WrHV+YBwdQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN0PR10MB4822
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.517,FMLib:17.11.122.1
 definitions=2022-08-22_14,2022-08-22_02,2022-06-22_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 suspectscore=0
 phishscore=0 malwarescore=0 spamscore=0 mlxlogscore=999 bulkscore=0
 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2207270000 definitions=main-2208220092
X-Proofpoint-ORIG-GUID: B1cui8Hg7tqS0iBlxKsKP-DKlUEbICDo
X-Proofpoint-GUID: B1cui8Hg7tqS0iBlxKsKP-DKlUEbICDo
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Yu, Robin and Christoph,

The mips kernel panic because the SWIOTLB buffer is adjusted to a very small
value (< 1MB, or < 512-slot), so that the swiotlb panic on purpose.

software IO TLB: SWIOTLB bounce buffer size adjusted to 0MB
software IO TLB: area num 1.
Kernel panic - not syncing: swiotlb_init_remap: nslabs = 128 too small


From mips code, the 'swiotlbsize' is set to PAGE_SIZE initially. It is always
PAGE_SIZE unless it is used by CONFIG_PCI or CONFIG_USB_OHCI_HCD_PLATFORM.

Finally, the swiotlb panic on purpose.

189 void __init plat_swiotlb_setup(void)
190 {
... ...
211         swiotlbsize = PAGE_SIZE;
212
213 #ifdef CONFIG_PCI
214         /*
215          * For OCTEON_DMA_BAR_TYPE_SMALL, size the iotlb at 1/4 memory
216          * size to a maximum of 64MB
217          */
218         if (OCTEON_IS_MODEL(OCTEON_CN31XX)
219             || OCTEON_IS_MODEL(OCTEON_CN38XX_PASS2)) {
220                 swiotlbsize = addr_size / 4;
221                 if (swiotlbsize > 64 * (1<<20))
222                         swiotlbsize = 64 * (1<<20);
223         } else if (max_addr > 0xf0000000ul) {
224                 /*
225                  * Otherwise only allocate a big iotlb if there is
226                  * memory past the BAR1 hole.
227                  */
228                 swiotlbsize = 64 * (1<<20);
229         }
230 #endif
231 #ifdef CONFIG_USB_OHCI_HCD_PLATFORM
232         /* OCTEON II ohci is only 32-bit. */
233         if (OCTEON_IS_OCTEON2() && max_addr >= 0x100000000ul)
234                 swiotlbsize = 64 * (1<<20);
235 #endif
236
237         swiotlb_adjust_size(swiotlbsize);
238         swiotlb_init(true, SWIOTLB_VERBOSE);
239 }


Here are some thoughts. Would you mind suggesting which is the right way to go?

1. Will the PAGE_SIZE swiotlb be used by mips when it is only PAGE_SIZE? If it
is not used, why not disable swiotlb completely in the code?

2. The swiotlb panic on purpose when it is less then 1MB. Should we remove that
limitation?

3. ... or explicitly declare the limitation that: "swiotlb should be at least
1MB, otherwise please do not use it"?


The reason I add the panic on purpose is for below case:

The user's kernel is configured with very small swiotlb buffer. As a result, the
device driver may work abnormally. As a result, the issue is reported to a
specific driver's developers, who spend some time to confirm it is swiotlb
issue. Suppose those developers are not familiar with IOMMU/swiotlb, it takes
times until the root cause is identified.

If we panic earlier, the issue will be reported to IOMMU/swiotlb developer. This
is also to sync with the remap failure logic in swiotlb (used by xen).


Thank you very much!

Dongli Zhang

On 8/22/22 5:32 AM, Robin Murphy wrote:
> On 2022-08-22 12:26, Christoph Hellwig wrote:
>> On Mon, Aug 22, 2022 at 10:49:09AM +0100, Robin Murphy wrote:
>>> Hmm, it's possible this might be quietly fixed by 20347fca71a3, but either
>>> way I'm not sure why we would need to panic *before* we've even tried to
>>> allocate anything, when we could simply return with no harm done? If we've
>>> ended up calculating (or being told) a buffer size which is too small to be
>>> usable, that should be no different to disabling SWIOTLB entirely.
>>
>> Hmm.  I think this might be a philosophical question, but I think
>> failing the boot with a clear error report for a configuration that is
>> supposed to work but can't is way better than just panicing later on.
> 
> Depends which context of "supposed to work" you mean there. The most logical
> reason to end up with a tiny SWIOTLB size is because you don't expect to need
> SWIOTLB, therefore if there's now a functional minimum size limit, failing
> gracefully such that the system keeps working as before is correct in that
> context. Even if we assume the expectation goes the other way, then it should be
> on SWIOTLB to adjust the initial allocation size to whatever minimum it now
> needs, which as I say it looks like 20347fca71a3 might do anyway. Creating new
> breakage by panicking instead of making a decision one way or the other was
> never the right answer.
> 
>>> Historically, passing "swiotlb=1" on the command line has been used to save
>>> memory when the user knows SWIOTLB isn't needed. That should definitely not
>>> be allowed to start panicking.
>>
>> I've never seen swiotlb=1 advertized as a way to disable swiotlb.
>> That's always been swiotlb=noforce, which cleanly disables it.
> 
> No, it's probably not been advertised as such, but it's what clearly fell out of
> the available options before "noforce" was added (which was considerably more
> recently than "always"), and the fact is that people *are* still using it even
> today (presumably copy-pasted through Android BSPs since before 4.10).
> 
> Thanks,
> Robin.