> On Sep 11, 2023, at 9:25 PM, Oliver Sang <[email protected]> wrote:
>
> hi, Chuck Lever,
>
> On Fri, Sep 08, 2023 at 02:43:22PM +0000, Chuck Lever III wrote:
>>
>>
>>> On Sep 8, 2023, at 1:26 AM, kernel test robot <[email protected]> wrote:
>>>
>>>
>>>
>>> Hello,
>>>
>>> kernel test robot noticed a -19.0% regression of aim9.disk_src.ops_per_sec on:
>>>
>>>
>>> commit: a2e459555c5f9da3e619b7e47a63f98574dc75f1 ("shmem: stable directory offsets")
>>> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
>>>
>>> testcase: aim9
>>> test machine: 48 threads 2 sockets Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz (Ivy Bridge-EP) with 112G memory
>>> parameters:
>>>
>>> testtime: 300s
>>> test: disk_src
>>> cpufreq_governor: performance
>>>
>>>
>>> In addition to that, the commit also has significant impact on the following tests:
>>>
>>> +------------------+-------------------------------------------------------------------------------------------------+
>>> | testcase: change | aim9: aim9.disk_src.ops_per_sec -14.6% regression |
>>> | test machine | 48 threads 2 sockets Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz (Ivy Bridge-EP) with 112G memory |
>>> | test parameters | cpufreq_governor=performance |
>>> | | test=all |
>>> | | testtime=5s |
>>> +------------------+-------------------------------------------------------------------------------------------------+
>>>
>>>
>>> If you fix the issue in a separate patch/commit (i.e. not just a new version of
>>> the same patch/commit), kindly add following tags
>>> | Reported-by: kernel test robot <[email protected]>
>>> | Closes: https://lore.kernel.org/oe-lkp/[email protected]
>>
>> Hi, several weeks ago we requested that these tests be run
>> again by the robot because they can't be run in environments
>> I have available to me (the tests do not run on Fedora, and
>> I don't have any big iron).
>>
>> We wanted the tests rerun before the patch was committed.
>> There was a deafening silence. So I assumed the work I did
>> then to address the regression was successful, and the
>> patches are now in upstream Linux.
>>
>> This new report is disappointing.
>
> I'm so sorry that I missed the test request for
> https://lore.kernel.org/all/169030957098.157536.9938425508695693348.stgit@manet.1015granger.net/
>
> just FYI, when this auto-bisect done,
> head commit of linus/master: [65d6e954e37872fd9afb5ef3fc0481bb3c2f20f4] was
> tested, it already includes
> 2be4f05af71bb libfs: Remove parent dentry locking in offset_iterate_dir()
>
> in our tests, the regression still exists.
Thanks for clarifying. I wondered about that, of course only just
after clicking "Send".
>> But, I'm still in a position where I can't run this test,
>> and the results don't really indicate where the problem
>> is. So I can't possibly address this issue.
>>
>> Any suggestions, advice, or help would be appreciated.
>
> if you have further fix patch, could you let us know? I will test it.
Well that's the problem. Since I can't run the reproducer, there's
nothing I can do to troubleshoot the problem myself.
Is there any hope in getting this reproducer to run on Fedora?
--
Chuck Lever
hi, Chuck Lever,
On Tue, Sep 12, 2023 at 01:01:29PM +0000, Chuck Lever III wrote:
>
>
> > in our tests, the regression still exists.
>
> Thanks for clarifying. I wondered about that, of course only just
> after clicking "Send".
>
>
> >> But, I'm still in a position where I can't run this test,
> >> and the results don't really indicate where the problem
> >> is. So I can't possibly address this issue.
> >>
> >> Any suggestions, advice, or help would be appreciated.
> >
> > if you have further fix patch, could you let us know? I will test it.
>
> Well that's the problem. Since I can't run the reproducer, there's
> nothing I can do to troubleshoot the problem myself.
>
> Is there any hope in getting this reproducer to run on Fedora?
sorry about this, maybe it's hard for us to support Fedora in a short time.
BTW, we are investigating this regression now, hope that we can update you
later. Thanks
>
>
> --
> Chuck Lever
>
>
Hi Chuck Lever,
On Tue, Sep 12, 2023 at 09:01:29PM +0800, Chuck Lever III wrote:
>
>
> > On Sep 11, 2023, at 9:25 PM, Oliver Sang <[email protected]> wrote:
> >
> > hi, Chuck Lever,
> >
> > On Fri, Sep 08, 2023 at 02:43:22PM +0000, Chuck Lever III wrote:
> >>
> >>
> >>> On Sep 8, 2023, at 1:26 AM, kernel test robot <[email protected]> wrote:
> >>>
> >>>
> >>>
> >>> Hello,
> >>>
> >>> kernel test robot noticed a -19.0% regression of aim9.disk_src.ops_per_sec on:
> >>>
> >>>
> >>> commit: a2e459555c5f9da3e619b7e47a63f98574dc75f1 ("shmem: stable directory offsets")
> >>> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
> >>>
> >>> testcase: aim9
> >>> test machine: 48 threads 2 sockets Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz (Ivy Bridge-EP) with 112G memory
> >>> parameters:
> >>>
> >>> testtime: 300s
> >>> test: disk_src
> >>> cpufreq_governor: performance
> >>>
> >>>
> >>> In addition to that, the commit also has significant impact on the following tests:
> >>>
> >>> +------------------+-------------------------------------------------------------------------------------------------+
> >>> | testcase: change | aim9: aim9.disk_src.ops_per_sec -14.6% regression |
> >>> | test machine | 48 threads 2 sockets Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz (Ivy Bridge-EP) with 112G memory |
> >>> | test parameters | cpufreq_governor=performance |
> >>> | | test=all |
> >>> | | testtime=5s |
> >>> +------------------+-------------------------------------------------------------------------------------------------+
> >>>
> >>>
> >>> If you fix the issue in a separate patch/commit (i.e. not just a new version of
> >>> the same patch/commit), kindly add following tags
> >>> | Reported-by: kernel test robot <[email protected]>
> >>> | Closes: https://lore.kernel.org/oe-lkp/[email protected]
> >> But, I'm still in a position where I can't run this test,
> >> and the results don't really indicate where the problem
> >> is. So I can't possibly address this issue.
> >>
> >> Any suggestions, advice, or help would be appreciated.
> >
> > if you have further fix patch, could you let us know? I will test it.
>
> Well that's the problem. Since I can't run the reproducer, there's
> nothing I can do to troubleshoot the problem myself.
We dug more into the perf and other profiling data from 0Day server
running this case, and it seems that the new simple_offset_add()
called by shmem_mknod() brings extra cost related with slab,
specifically the 'radix_tree_node', which cause the regression.
Here is some slabinfo diff for commit a2e459555c5f and its parent:
23a31d87645c6527 a2e459555c5f9da3e619b7e47a6
---------------- ---------------------------
26363 +40.2% 36956 slabinfo.radix_tree_node.active_objs
941.00 +40.4% 1321 slabinfo.radix_tree_node.active_slabs
26363 +40.3% 37001 slabinfo.radix_tree_node.num_objs
941.00 +40.4% 1321 slabinfo.radix_tree_node.num_slabs
Also the perf profile show some difference
0.01 ±223% +0.1 0.10 ± 28% pp.self.shuffle_freelist
0.00 +0.1 0.11 ± 40% pp.self.xas_create
0.00 +0.1 0.12 ± 27% pp.self.xas_find_marked
0.00 +0.1 0.14 ± 18% pp.self.xas_alloc
0.03 ±103% +0.1 0.17 ± 29% pp.self.xas_descend
0.00 +0.2 0.16 ± 23% pp.self.xas_expand
0.10 ± 22% +0.2 0.27 ± 16% pp.self.rcu_segcblist_enqueue
0.92 ± 35% +0.3 1.22 ± 11% pp.self.kmem_cache_free
0.00 +0.4 0.36 ± 16% pp.self.xas_store
0.32 ± 30% +0.4 0.71 ± 12% pp.self.__call_rcu_common
0.18 ± 27% +0.5 0.65 ± 8% pp.self.kmem_cache_alloc_lru
0.36 ± 79% +0.6 0.96 ± 15% pp.self.__slab_free
0.00 +0.8 0.80 ± 14% pp.self.radix_tree_node_rcu_free
0.00 +1.0 1.01 ± 16% pp.self.radix_tree_node_ctor
Some perf profile from a2e459555c5f is:
- 17.09% 0.09% singleuser [kernel.kallsyms] [k] path_openat
- 16.99% path_openat
- 12.23% open_last_lookups
- 11.33% lookup_open.isra.0
- 9.05% shmem_mknod
- 5.11% simple_offset_add
- 4.95% __xa_alloc_cyclic
- 4.88% __xa_alloc
- 4.76% xas_store
- xas_create
- 2.40% xas_expand.constprop.0
- 2.01% xas_alloc
- kmem_cache_alloc_lru
- 1.28% ___slab_alloc
- 1.22% allocate_slab
- 1.19% shuffle_freelist
- 1.04% setup_object
radix_tree_node_ctor
Please let me know if you need more info.
>
> Is there any hope in getting this reproducer to run on Fedora?
Myself haven't succeeded to reproduce it locally, will keep trying
it tomorrow.
Thanks,
Feng
>
> --
> Chuck Lever
>
>