Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <81af5cdf-1a13-db2c-7b7b-cfd86f1271e6@intel.com>
Date:   Tue, 19 Jul 2022 10:16:43 +0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Firefox/91.0 Thunderbird/91.11.0
Subject: Re: [LKP] Re: [io_uring] 584b0180f0:
 phoronix-test-suite.fio.SequentialWrite.IO_uring.Yes.Yes.1MB.DefaultTestDirectory.mb_s
 -10.2% regression
Content-Language: en-US
To:     Jens Axboe <axboe@kernel.dk>,
        kernel test robot <oliver.sang@intel.com>
CC:     LKML <linux-kernel@vger.kernel.org>, <io-uring@vger.kernel.org>,
        <lkp@lists.01.org>, <lkp@intel.com>
References: <20220527092432.GE11731@xsang-OptiPlex-9020>
 <2085bfef-a91c-8adb-402b-242e8c5d5c55@kernel.dk>
 <0d60aa42-a519-12ad-3c69-72ed12398865@intel.com>
 <26d913ea-7aa0-467d-4caf-a93f8ca5b3ff@kernel.dk>
 <9df150bb-f4fd-7857-aea8-b2c7a06a8791@intel.com>
 <7146c853-0ff8-3c92-c872-ce6615baab40@kernel.dk>
From:   Yin Fengwei <fengwei.yin@intel.com>
In-Reply-To: <7146c853-0ff8-3c92-c872-ce6615baab40@kernel.dk>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?bmdKeFg4S1lLMTlKK2ZhYnZZM2xjMmFNRWJlZ2dyT2ZDS0ZOY0tPUTNGVG82?=
 =?utf-8?B?NU1DRjdWZFViOU8zMnQvQzFndWNnZ005OG9ETThHOUdVV1VLQTFxYS9qMVlR?=
 =?utf-8?B?M1E4dUtpRU5OUDh2UmhEc2l1VjFtV3E4TFVOY2ErRDZ4L2NDak9pdzBIRlRG?=
 =?utf-8?B?c0Zyei81RjZ2NE8zMVduZjBQc3RyVnZISExyQ2tUcHlEVVBxZldSSytwbFRQ?=
 =?utf-8?B?aXp2YVVtdFFUT1lzRHgrenA2NUQvcVozeVgvOXU0WGZaS0M5SVQ2OWNueFU5?=
 =?utf-8?B?MWQvWkdhbkhWdHlQWHJTSDh3bmc4MDRmRmVCR0JqM1hOb2V5VGppbVZ2eFpW?=
 =?utf-8?B?dWZiWTdjajh2blFSZVI1Vkx5VU96eGFxYkNzemoyZmdTTGUzNGhXOWdWR1lJ?=
 =?utf-8?B?MFMyUHhUVXU2KzVSaURwTUh0TEEwNDYwMTN3M3c0TFBhR3NKNzNPRUlHNHlv?=
 =?utf-8?B?c1ZqNDYySTVQeUhUMEMzbjNuMnRScE9xL3E3UUxyZ08zU3BUclVLZUlmOUdO?=
 =?utf-8?B?aG9uZWw3MlY1K2JsTk4vNnN6YUZ3S0h4cU9RTWRqdEV0WEtld2UxZTFSMWcv?=
 =?utf-8?B?RW13TnZLMDlrb2g0YlB5OWo1aVQyQXp2MXRQc0FncFFsSC92Rlo0V3lSYXEr?=
 =?utf-8?B?ZEV1WVZHYURLa2psTURocS9wM25zYi96RUcveFVhQTZJTTEyZ0dCSzBQVVdE?=
 =?utf-8?B?Q2lUN2hRck9Lck1RUmZhNGJIU2Z0OGNZcnR4OWY2Nk5STk9JVHZYbTBlZ2JZ?=
 =?utf-8?B?RzVPL2EybVdwbC9sL29Qd2F4dXBRb2k4ZVNLQzB5QzhYbjVkM0hQS0s5S1dS?=
 =?utf-8?B?VWRIUWFhOUJZOHZaK3VWNGJaM0ZUNHJRWkNTWTF2VE5aZEpzNUxLL3JoTHQv?=
 =?utf-8?B?ZDdVbmNZcXhXTEtIalpTdTh2ZnVLakwwOHhpYlR5MEtKUjJBMDA2TTJaUFJq?=
 =?utf-8?B?UFVsRCt2UHI2eVlxRnliWkhBbkZNSjZjaksvODhNNjk3QTd4MWtCU3o1cVJJ?=
 =?utf-8?B?UFhpVnQ1YytuNTd2SjhRbldDU3drWlpGbWNoak9HTGd5NmJIQytwOGZpVWlY?=
 =?utf-8?B?WENQamJhL29aRDlRbXJENnR5bTNYRnhvMExaYXlWeXovQWU1SkdISDUzc2VC?=
 =?utf-8?B?WlFOSDU0QzNVbUt1bG9HdnNpaXNTZmZwVHE0UzNGblFEOHFwTUlDMXZpZ3Nk?=
 =?utf-8?B?aU1qYXkyNDlNdS9GQlBJT0ZLVkh1VktEc0x6cnBLVEpzU2p0eURRQjlYWXlJ?=
 =?utf-8?B?eTA1eVRuZzYyY3JQWjZKNml4bFdINEFqQTdWMFJSZjhyTXdqVGVtd0gzeWJM?=
 =?utf-8?B?UkF1MnZSYWlUUkNYRFdjZFErd0VJWm0rZFBEMWVQVEZ1VTVnT0l6ZXljYkM3?=
 =?utf-8?B?OE01d2d2OGNzelJFVE5pejExTTcvRU9uMndEMG5VQmxsVi9HaEZ3ZWJ1QW01?=
 =?utf-8?B?eG95VmhWVDA3OVJ4NTZqdythSTN1dk53emFZcUNmeUNKdnZ4UjZicGhVbUQy?=
 =?utf-8?B?Ty8vbHNZM25JQTloUkhSRnk5czBqbHJONjVzVlA3cmpYbnVzWER4VGQzcyth?=
 =?utf-8?B?eStXd2ZLWUtqRUlRM2VlY3VYVnljeFhPZXNLODNCRk9OVjY1R0dXWWliL0VJ?=
 =?utf-8?B?SXB1UnJzWmdmZzJNNzZvZ3RNZlpMSi83YzdtWUNKY0FWUUliN290aXAxSGkz?=
 =?utf-8?B?RFhMZE8yMkh1eG5wSXFNd1ZpbWxrT3R3V3BGUFR2VlhIRllOZzdnblR6UHVC?=
 =?utf-8?B?ZTRhQXZQNzRLMy9XbEIxT0ljcDRxTHFka1RaNkg5YlpOek9pSnNxUmUwK05a?=
 =?utf-8?B?M3Y3eDdJWmw0SVhGakU1ZWlrQUJndHZnS3lTNGVlV2UwbG54WUxKbThWUnBJ?=
 =?utf-8?B?VHBYWlh1NUl6Q2NlZkd2Smp1bU9kK1lIRWJ6QkM4bDBrc25kRmFobEYyOUNY?=
 =?utf-8?B?LzdzT3JtdzZtaHRUYkxEUXNsSi9sOHgwcjJQUkR3dnVRU1dRUlltc0hyZ3M5?=
 =?utf-8?B?eEVmUDQwL0QwYVZSTDR2aXl1RzIwUGk2aDhmaFo0SzluVE8zd2hhaHMwU24r?=
 =?utf-8?B?aytMM0dINFBGOE5uLzNNRUM5akttQXhRcE5vYU1pdlk4Zkc1VE5KS3diOXg4?=
 =?utf-8?B?QnVQNUMyVGV1QnpIM3h1bkcxQkpBRmo3THFCUmFYZEdpcUd6dnZyc3Z6OWwy?=
 =?utf-8?B?VFE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: b41b5318-9217-4564-cace-08da692cbdc5
X-MS-Exchange-CrossTenant-AuthSource: CO1PR11MB4820.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Jul 2022 02:16:51.2505
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: yGaxloVGSulsE/rXU/bIyGREbUB5lA/VhiZgQlWodlMfeuNlo9Yu5YZ5O855w3ugjmk4AvqfZFKc89T5TR1MFQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR11MB3388
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-5.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_MED,SPF_HELO_NONE,SPF_NONE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Jens,

On 7/19/2022 12:27 AM, Jens Axboe wrote:
> On 7/17/22 9:30 PM, Yin Fengwei wrote:
>> Hi Jens,
>>
>> On 7/15/2022 11:58 PM, Jens Axboe wrote:
>>> In terms of making this more obvious, does the below also fix it for
>>> you?
>>
>> The regression is still there after applied the change you posted.
> 
> Still don't see the regression here, using ext4. I get about 1020-1045
> IOPS with or without the patch you sent.
> 
> This is running it in a vm, and the storage device is nvme. What is
> hosting your ext4 fs?
Just did more test with vm. The regression can't be reproduced with latest
code (I tried the tag v5.19-rc7) whatever the underneath storage is SATA
or NVME.

But the regression and the debugging patch from me could be reproduced
on both SATA and NVME if use commit 584b0180f0f4d6 as base commit
(584b0180f0f4d6 vs 584b0180f0f4d6 with my debugging patch).


Here is the test result I got:
NVME as host storage:
  5.19.0-rc7:
    write: IOPS=933, BW=937MiB/s (982MB/s)(18.3GiB/20020msec); 0 zone resets
    write: IOPS=993, BW=996MiB/s (1045MB/s)(19.5GiB/20020msec); 0 zone resets
    write: IOPS=1005, BW=1009MiB/s (1058MB/s)(19.7GiB/20020msec); 0 zone resets
    write: IOPS=985, BW=989MiB/s (1037MB/s)(19.3GiB/20020msec); 0 zone resets
    write: IOPS=1020, BW=1024MiB/s (1073MB/s)(20.0GiB/20020msec); 0 zone resets

  5.19.0-rc7 with my debugging patch:
    write: IOPS=988, BW=992MiB/s (1040MB/s)(19.7GiB/20384msec); 0 zone resets
    write: IOPS=995, BW=998MiB/s (1047MB/s)(20.1GiB/20574msec); 0 zone resets
    write: IOPS=996, BW=1000MiB/s (1048MB/s)(19.5GiB/20020msec); 0 zone resets
    write: IOPS=995, BW=998MiB/s (1047MB/s)(19.5GiB/20020msec); 0 zone resets
    write: IOPS=1006, BW=1009MiB/s (1058MB/s)(19.7GiB/20019msec); 0 zone resets

  584b0180f0:
    write: IOPS=1004, BW=1008MiB/s (1057MB/s)(19.7GiB/20020msec); 0 zone resets
    write: IOPS=968, BW=971MiB/s (1018MB/s)(19.4GiB/20468msec); 0 zone resets
    write: IOPS=982, BW=986MiB/s (1033MB/s)(19.3GiB/20020msec); 0 zone resets
    write: IOPS=1000, BW=1004MiB/s (1053MB/s)(20.1GiB/20461msec); 0 zone resets
    write: IOPS=903, BW=906MiB/s (950MB/s)(18.1GiB/20419msec); 0 zone resets

  584b0180f0 with my debugging the patch:
    write: IOPS=1073, BW=1076MiB/s (1129MB/s)(21.1GiB/20036msec); 0 zone resets
    write: IOPS=1131, BW=1135MiB/s (1190MB/s)(22.2GiB/20022msec); 0 zone resets
    write: IOPS=1122, BW=1126MiB/s (1180MB/s)(22.1GiB/20071msec); 0 zone resets
    write: IOPS=1071, BW=1075MiB/s (1127MB/s)(21.1GiB/20071msec); 0 zone resets
    write: IOPS=1049, BW=1053MiB/s (1104MB/s)(21.1GiB/20482msec); 0 zone resets


SATA disk as host storage:
  5.19.0-rc7:
    write: IOPS=624, BW=627MiB/s (658MB/s)(12.3GiB/20023msec); 0 zone resets
    write: IOPS=655, BW=658MiB/s (690MB/s)(12.9GiB/20021msec); 0 zone resets
    write: IOPS=596, BW=600MiB/s (629MB/s)(12.1GiB/20586msec); 0 zone resets
    write: IOPS=647, BW=650MiB/s (682MB/s)(12.7GiB/20020msec); 0 zone resets
    write: IOPS=591, BW=594MiB/s (623MB/s)(12.1GiB/20787msec); 0 zone resets

  5.19.0-rc7 with my debugging patch:
    write: IOPS=633, BW=637MiB/s (668MB/s)(12.6GiB/20201msec); 0 zone resets
    write: IOPS=614, BW=617MiB/s (647MB/s)(13.1GiB/21667msec); 0 zone resets
    write: IOPS=653, BW=657MiB/s (689MB/s)(12.8GiB/20020msec); 0 zone resets
    write: IOPS=618, BW=622MiB/s (652MB/s)(12.2GiB/20033msec); 0 zone resets
    write: IOPS=604, BW=608MiB/s (638MB/s)(12.1GiB/20314msec); 0 zone resets

  584b0180f0:
    write: IOPS=635, BW=638MiB/s (669MB/s)(12.5GiB/20020msec); 0 zone resets
    write: IOPS=649, BW=652MiB/s (684MB/s)(12.8GiB/20066msec); 0 zone resets
    write: IOPS=639, BW=642MiB/s (674MB/s)(13.1GiB/20818msec); 0 zone resets

  584b0180f0 with my debugging patch:
    write: IOPS=850, BW=853MiB/s (895MB/s)(17.1GiB/20474msec); 0 zone resets
    write: IOPS=738, BW=742MiB/s (778MB/s)(15.1GiB/20787msec); 0 zone resets
    write: IOPS=751, BW=755MiB/s (792MB/s)(15.1GiB/20432msec); 0 zone resets


Regards
Yin, Fengwei