Date:   Thu, 13 Jan 2022 11:58:16 +0100
From:   Greg KH <gregkh@linuxfoundation.org>
To:     Imran Khan <imran.f.khan@oracle.com>
Cc:     tj@kernel.org, linux-kernel@vger.kernel.org
Subject: Re: [PATCH v3 2/2] kernfs: Reduce contention around global per-fs
 kernfs_rwsem.
Message-ID: <YeAFyOR61+c+FyMn@kroah.com>
References: <20220113104259.1584491-1-imran.f.khan@oracle.com>
 <20220113104259.1584491-3-imran.f.khan@oracle.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20220113104259.1584491-3-imran.f.khan@oracle.com>
Precedence: bulk

On Thu, Jan 13, 2022 at 09:42:59PM +1100, Imran Khan wrote:
> Right now a global per file system based rwsem (kernfs_rwsem)
> synchronizes multiple kernfs operations. On a large system with
> few hundred CPUs and few hundred applications simultaenously trying
> to access sysfs, this results in multiple sys_open(s) contending on
> kernfs_rwsem via kernfs_iop_permission and kernfs_dop_revalidate.
> 
> -   21.42%    21.34%  showgids   [kernel.kallsyms]     [k] up_read
>      21.34% __libc_start_main
>         __GI___libc_open
>         entry_SYSCALL_64_after_hwframe
>         do_syscall_64
>         sys_open
>         do_sys_open
>         do_filp_open
>       - path_openat
>          - 20.05% link_path_walk
>             - 9.76% walk_component
>                  lookup_fast
>                - d_revalidate.part.24
>                   - 9.75% kernfs_dop_revalidate
>                        up_read
>             - 9.46% inode_permission
>                - __inode_permission
>                   - 9.46% kernfs_iop_permission
>                        up_read
>             - 0.83% kernfs_iop_get_link
>                  up_read
>          - 0.80% lookup_fast
>               d_revalidate.part.24
>               kernfs_dop_revalidate
>               up_read
> 
> -   21.31%    21.21%  showgids   [kernel.kallsyms]    [k] down_read
>      21.21% __libc_start_main
>         __GI___libc_open
>         entry_SYSCALL_64_after_hwframe
>         do_syscall_64
>         sys_open
>         do_sys_open
>         do_filp_open
>       - path_openat
>          - 19.78% link_path_walk
>             - 10.62% inode_permission
>                - __inode_permission
>                   - 10.62% kernfs_iop_permission
>                        down_read
>             - 8.45% walk_component
>                  lookup_fast
>                - d_revalidate.part.24
>                   - 8.45% kernfs_dop_revalidate
>                        down_read
>             - 0.71% kernfs_iop_get_link
>                  down_read
>          - 0.72% lookup_fast
>             - d_revalidate.part.24
>                - 0.72% kernfs_dop_revalidate
>                     down_read
>          - 0.71% may_open
>               inode_permission
>               __inode_permission
>               kernfs_iop_permission
>               down_read
> 
> Since permission is specific to a kernfs_node we can use a hashed
> lock to access/modify permission. Also use kernfs reference counting
> to ensure we are accessing/modifying permissions for an existing
> kernfs_node object.
> 
> Using this change brings down the above mentioned down_read/up_read
> numbers to ~8%, thus indicating that contention around kernfs_rwsem
> has reduced to about 1/3rd of earlier value.

Ah, nevermind, you do post the results here, I should have kept reading.

Nice work!

I'll look at these after 5.17-rc1 is out, thanks!

greg k-h