Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Subject: Re: [PATCH v2] Revert "swiotlb: panic if nslabs is too small"
To:     Yu Zhao <yuzhao@google.com>
Cc:     Christoph Hellwig <hch@infradead.org>,
        Robin Murphy <robin.murphy@arm.com>,
        Marek Szyprowski <m.szyprowski@samsung.com>,
        iommu@lists.linux.dev, linux-mips@vger.kernel.org,
        linux-kernel <linux-kernel@vger.kernel.org>,
        kernel test robot <lkp@intel.com>,
        Dan Carpenter <dan.carpenter@oracle.com>
References: <20220829232934.3277747-1-yuzhao@google.com>
 <20220831063818.3902572-1-yuzhao@google.com>
 <747f76e1-a5ec-150c-311e-a60396f6f7ab@oracle.com>
 <CAOUHufYHh3B6A8_pAusnW5==r3VhPxy6bOoUHbxe0qvhNGovqA@mail.gmail.com>
From:   Dongli Zhang <dongli.zhang@oracle.com>
Message-ID: <982a4b95-0ab5-f18e-cbaf-1f503a35ada7@oracle.com>
Date:   Wed, 31 Aug 2022 19:18:22 -0700
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.6.1
In-Reply-To: <CAOUHufYHh3B6A8_pAusnW5==r3VhPxy6bOoUHbxe0qvhNGovqA@mail.gmail.com>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?ck56ZERGVDZwSFlQS2RRUVBXN3ZBb0xPVHd6MUZwMEFJbXBjM1NVM203T1Bk?=
 =?utf-8?B?eVByRlY0SlF6RGR4MnBidFZUNzlSVFZjTHBXMFhMWE9OQ1pvcEJIZmdzSS9W?=
 =?utf-8?B?VjI4UFF0TW5FazUzNFY3bkJxNmcvWklsVnZtR0gxbmhMMEpTQU1OcjcvdUNs?=
 =?utf-8?B?d1E5V1NiWmxOMDJ0cldSdS9zVlhqNHRVWTMyVENWcGd3aWlRMkhIM2hrUFhp?=
 =?utf-8?B?VFhLcnRxSUNBekxOL2lRWm5UaXFVeXFoSEI5Qlp4MnNFZitZdHkwZDJWZVZz?=
 =?utf-8?B?aDJNWEJNWk5ac3MwL1k5OVdnVFJNMjFuNlgrNHY0MWNJZ2x4RTJhODJsczJO?=
 =?utf-8?B?aU85K1hNMDRSS1RkaFRNVnRidHoycHJURXhna3UyQXRxSUFIOHJiMGxtVGJq?=
 =?utf-8?B?K3dLOGR1SlFzbWRnOE93UjdrTDRoMWFGZ05raFJBZTM1R29kTzdRRzdsTm9Z?=
 =?utf-8?B?bXk1eWJ5SzIydE9GKys4b0Z4Z1dsckhYbUVoVm1hSnQ2VGs4SURvL2ZzVVBp?=
 =?utf-8?B?SUNzZ0JMUzV3RUdFKzRLcGVMbDlCcTFBTXRwMmE4MENzL0Q0YWhMUDdncGhT?=
 =?utf-8?B?V2d2dVpOVmx5UkFIUy9hQVJwOEVKWDB0QUdkVXhOQUdFZWF4VDIvRkxTaEg1?=
 =?utf-8?B?TFJXVEowaGVrU1V5M2hGMHF4Zll0bHFBL3d0bzZHMEI5V3YvTHkvdG9SQUtP?=
 =?utf-8?B?YUVFcVFIVWl1TjI5SlhDS2R6Tkd4SHdNaGhvdEtOMnNGWWdXc1NVQ1RKbWtP?=
 =?utf-8?B?c1JTMWZPVno3VVJTTnN2SkMrNExtOHVUQTFobUx0RFppUGRqN2VLS2hPcVZo?=
 =?utf-8?B?QitwQzcwY2tSU3FqUU8wNzhVTEsxdUtSTGhaZ29UeksrVFQySFJGWVNLRCth?=
 =?utf-8?B?djlHcnVyV092WHQ2T3hYNU41NURPSnAyZzRtSUNHNE9DNE15dXlGY3VzUExm?=
 =?utf-8?B?R3p2S09PS2VCdVlxVGdDNE9WMkJIQlRDK2o1cmdCbXhHRzFuc3pYaW9nVUk0?=
 =?utf-8?B?MVd1M1FZU3FMWGNNN1A2MEVJcFVXdU9hME5UMzJraEhxamlxWlEwME81QVVN?=
 =?utf-8?B?OU03WERpdHJUU3NZNEVNaVIxQjIrdGF4VzhBUjdBQzI0RVp0L3Vjb1NXdVVM?=
 =?utf-8?B?VThxL0MrWG1VQ1FSSkR1U05qZmRZT2FUS2pJczRYekI5RE92MDZiRFdxSGpZ?=
 =?utf-8?B?amxZSGIwSXRLSkkwUDlLR1ljM2Frb3ArTnd0MWdYeU8rbHpHWUZpM1N6YVkw?=
 =?utf-8?B?VURyZFJWOUx6c1BYMjRVdDI2cEhndXJEdXA3cGlpZkFCc3IvcXFuNEFGQWF0?=
 =?utf-8?B?QTVMTDdobGZuNHMzS3FiSk9Tay8wZW1OVHVTUmkzMEhLK3hMWnQxdVN0SXkx?=
 =?utf-8?B?Q01NZkxKSHM5eUVjWDBhWE1uWFJVZUs2TDFnbXdzSlVEd3ZPazFKRkFhTkJZ?=
 =?utf-8?B?R2hEcHByZFBKQTBCNjZoUzZZeXN4SWp5R3VXNys0SUxLSHIyZkw5dkQ2a2U4?=
 =?utf-8?B?U1lLOC8zSmRGcFFGTi81MmlpZWJMS3JrSmcyMnlZNk5tdzJtKzRvQi84ejJD?=
 =?utf-8?B?OWN4TElWNlgzSjJIUU5uU0NJRlRiREhOSmtTWHZ5M093UnJCbTJCR3R2UTR5?=
 =?utf-8?B?KzFac3pXeEozRDBkbDI0N2hTVDN1WVVFZFdCdTl1STlvYUZ3R2JKcW82UEs5?=
 =?utf-8?B?VlNUb0thbU9NaXFIb2xYU1QxZXlaWE5LSFdUL0RSRTg4MDdDbXhBQ1JmekZt?=
 =?utf-8?B?ajJCMmxIczZyRzh6Mk84bjVtcjJraHFobFJ0N1o3d3RKZWR6UEZFK3NiWlkv?=
 =?utf-8?B?Sndqa0laY1E3UTkveHdhYXpSWUxadEo0bXVmRDV4VWFoUEVtbjZFdnZSYTRC?=
 =?utf-8?B?eSt2UmhtMk9oWXdGaURJT3E5R1FsaUpRM0NtcEZNY0JMd0RvWDVsSm9rRTU2?=
 =?utf-8?B?R29PVG41d1pNQXdxRXFaSWpkVlJHejFxcmI1eGRJS2hReTJmNlY1RFdIbGND?=
 =?utf-8?B?WTIvMXJ6UXBFcE1ISjNvYnJXS3FOR2dhYVZTZCtTYTA3U3NNd1NLaFVMQStV?=
 =?utf-8?B?eFZGWTFFaHA5WXRWeVIwRnZ1WGJnd1UydGRjU0gvUitSUWl3UWlxRU9Vc0ha?=
 =?utf-8?B?cnVUakRSMVVham5iU2tQVExPMEU1Y3k3UXI0ZUVDaWdVZ1NCd24vM1dLZVBa?=
 =?utf-8?Q?Fj6oEN4Tx1TAlsurxaCnW4s=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: fce9969e-c8b2-4658-e8d9-08da8bc04067
X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB2663.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Sep 2022 02:18:25.9067
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 5/aDA54ynmm0uBg0rJq88HUMupc52wpghSOQ4qo7m6AKiqolj3yOwJbHf0R+v90xbiN7yJKi1Lf4MpFrK0sJ3w==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR10MB6112
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.517,FMLib:17.11.122.1
 definitions=2022-08-31_16,2022-08-31_03,2022-06-22_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 suspectscore=0
 mlxlogscore=999 adultscore=0 phishscore=0 spamscore=0 bulkscore=0
 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2207270000 definitions=main-2209010010
X-Proofpoint-GUID: dCy75uRjfrUQeYXo9vjDo_jdNk0lZDyt
X-Proofpoint-ORIG-GUID: dCy75uRjfrUQeYXo9vjDo_jdNk0lZDyt
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Yu,

On 8/31/22 5:24 PM, Yu Zhao wrote:
> On Wed, Aug 31, 2022 at 4:20 PM Dongli Zhang <dongli.zhang@oracle.com> wrote:
>>
>> Hi Yu,
>>
>> As we discussed in the past, the swiotlb panic on purpose
> 
> We should not panic() at all, especially on a platform that has been
> working well since at least 4.14.
> 
> Did you check out this link I previously shared with you?
> https://urldefense.com/v3/__https://lore.kernel.org/r/CAHk-=wit-DmhMfQErY29JSPjFgebx_Ld*pnerc4J2Ag990WwAA@mail.gmail.com/__;Kw!!ACWV5N9M2RV99hQ!PXzSLurBv7VqxI1451TV4zO3_-BYj4grk-HYBsXzSnA6nZcXaBzdsQ-rF2DAqlICSRPMt-efYv_Uu2A2CQ$  

Thanks for sharing! I used to read that in the past. To be honest I am still
confused on when to use BUG/panic and when to not, as I still see many usage in
some patches.

Just about swiotlb, it may panic in many cases during boot, e.g.,:

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1955655

> 
>> because the
>> mips/cavium-octeon/dma-octeon.c requests to allocate only PAGE_SIZE swiotlb
>> buffer.
> 
> What's wrong with PAGE_SIZE swiotlb?

The swiotlb is like a software IOMMU. When the device supports only <=32-bit DMA
while the IOMMU is not available, it may do the below for each DMA operation:

1. The swiotlb inits and pre-allocates many 32-bit memory during boot.
2. For a DMA read/write, if the page is 64-bit but 32-bit DMA is required, the
kernel swaps the 64-bit DMA page with the pre-allocated swiotlb 32-bit memory.
3. After the DMA is complete, the 32-bit memory is swapped back to swiotlb.

The size of pre-allocated swiotlb is fixed during boot. The DMA will fail later
if the pre-allocated swiotlb is not enough.

That's why I think the objective to set swiotlb=PAGE_SIZE is just to minimize
the swiotlb pre-allocation when it is not required. I do not see a reason to
pre-allocate a PAGE_SIZE for DMA.

There was a discussion between Robin and Christoph whether it is a good idea to
disable swiotlb by allocating a small chunk of memory.

https://lore.kernel.org/all/d5016c1e-55d9-4224-278a-50377d4c6454@arm.com/

> 
>> This is smaller than IO_TLB_MIN_SLABS.
> 
> IO_TLB_MIN_SLABS is an arbitrary number, and it's inherited from IA64.
> So does the comment below. What exactly is the rationale behind it?

To be honest I do not have an answer as different people may have different
preferences. Some people may prefer to increase IO_TLB_MIN_SLABS, while some may
prefer to decrease.

That value of IO_TLB_MIN_SLABS was there since I started working on swiotlb.

> 
>> The below comments mentioned that IO_TLB_MIN_SLABS is the "Minimum IO TLB size
>> to bother booting with".
>>
>> 56 /*
>> 57  * Minimum IO TLB size to bother booting with.  Systems with mainly
>> 58  * 64bit capable cards will only lightly use the swiotlb.  If we can't
>> 59  * allocate a contiguous 1MB, we're probably in trouble anyway.
>> 60  */
>> 61 #define IO_TLB_MIN_SLABS ((1<<20) >> IO_TLB_SHIFT)
>>
>>
>> The arm may create swiotlb conditionally. That is, the swiotlb is not
>> initialized if (1) CONFIG_ARM_LPAE is not set (line 273), or (2) max_pfn <=
>> arm_dma_pfn_limit (line 274).
>>
>> arch/arm/mm/init.c
>>
>> 271 void __init mem_init(void)
>> 272 {
>> 273 #ifdef CONFIG_ARM_LPAE
>> 274         swiotlb_init(max_pfn > arm_dma_pfn_limit, SWIOTLB_VERBOSE);
>> 275 #endif
>> 276
>> 277         set_max_mapnr(pfn_to_page(max_pfn) - mem_map);
>>
>>
>> On x86, the swiotlb is not initialized if the memory is small (> MAX_DMA32_PFN,
>> at line 47), or the secure memory is not required.
>>
>> 44 static void __init pci_swiotlb_detect(void)
>> 45 {
>> 46         /* don't initialize swiotlb if iommu=off (no_iommu=1) */
>> 47         if (!no_iommu && max_possible_pfn > MAX_DMA32_PFN)
>> 48                 x86_swiotlb_enable = true;
>> 49
>> 50         /*
>> 51          * Set swiotlb to 1 so that bounce buffers are allocated and used for
>> 52          * devices that can't support DMA to encrypted memory.
>> 53          */
>> 54         if (cc_platform_has(CC_ATTR_HOST_MEM_ENCRYPT))
>> 55                 x86_swiotlb_enable = true;
>> 56
>> 57         /*
>> 58          * Guest with guest memory encryption currently perform all DMA through
>> 59          * bounce buffers as the hypervisor can't access arbitrary VM memory
>> 60          * that is not explicitly shared with it.
>> 61          */
>> 62         if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT)) {
>> 63                 x86_swiotlb_enable = true;
>> 64                 x86_swiotlb_flags |= SWIOTLB_FORCE;
>> 65         }
>> 66 }
> 
> Thanks for the code snippets. But I failed to see a point.
> 
>> Regardless whether the current patch will be reverted, unless there is specific
>> reason (e.g., those PAGE_SIZE will be used), I do not think it is a good idea to
>> allocate <IO_TLB_MIN_SLABS swiotlb buffer.
> 
> For what specific reason? My current understanding is that you want to
> be future/fool-proof. If so, then you got the priority wrong. We need
> to make sure currently working systems continue to work first, then
> future/fool-proof.

My objective was not to prove which option was correct/incorrect. I was just
going to share my understanding of the situation to people on this list, so that
a discussion can continue.

But I do agree with your point on 'priority'.

> 
>> I will wait for the suggestion from
>> the swiotlb maintainer.
> 
> Chris is on vacation. I sure can wait.
> 
> But it sounds like you are unsure about what to do. If so, it's not
> what you claimed "we have already understood everything related to
> swiotlb" previously.

The swiotlb is a component used by many arch (e.g., x86, arm/arm64, mips and
xen) ... while I believe I understand the swiotlb well, I do not understand what
other components are expecting from the swiotlb.

That's why my objective was to trigger the discussion but not to prove my
statement was correct.

The swiotlb is still evolving due to it is more widely used recently (e.g., AMD
SEV). The discussion helps people avoid introducing more issue because there are
many components that may use swiotlb now or in the future.

About what to do, I think it is to either (1) revert the patch or (2) avoid
initializing swiotlb if it is not going to be used. However, I think my thoughts
do not count that much, and that's why I would wait for the suggestion from more
experienced maintainers.

Thank you very much!

Dongli Zhang

> 
>> Since I do not have a mips environment, I am not able to test if the below makes
>> any trouble in your situation at arch/mips/cavium-octeon/dma-octeon.c.
>>
>> @@ -234,6 +234,8 @@ void __init plat_swiotlb_setup(void)
>>                 swiotlbsize = 64 * (1<<20);
>>  #endif
>>
>> -       swiotlb_adjust_size(swiotlbsize);
>> -       swiotlb_init(true, SWIOTLB_VERBOSE);
>> +       if (swiotlbsize != PAGE_SIZE) {
>> +               swiotlb_adjust_size(swiotlbsize);
>> +               swiotlb_init(true, SWIOTLB_VERBOSE);
>> +       }
>>  }
> 
> Please ask the MIPS/Octeon maintainers to check this first.
>