Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Barry Song <21cnbao@gmail.com>
To:     alex.kogan@oracle.com
Cc:     arnd@arndb.de, bp@alien8.de, daniel.m.jordan@oracle.com,
        dave.dice@oracle.com, guohanjun@huawei.com, hpa@zytor.com,
        jglauber@marvell.com, linux-arch@vger.kernel.org,
        linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org,
        linux@armlinux.org.uk, longman@redhat.com, mingo@redhat.com,
        peterz@infradead.org, steven.sistare@oracle.com,
        tglx@linutronix.de, will.deacon@arm.com, x86@kernel.org
Subject: Re: [PATCH v15 0/6] Add NUMA-awareness to qspinlock
Date:   Thu, 30 Sep 2021 17:44:47 +0800
Message-Id: <20210930094447.9719-1-21cnbao@gmail.com>
In-Reply-To: <20210514200743.3026725-1-alex.kogan@oracle.com>
References: <20210514200743.3026725-1-alex.kogan@oracle.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Precedence: bulk

> We have done some performance evaluation with the locktorture module
> as well as with several benchmarks from the will-it-scale repo.
> The following locktorture results are from an Oracle X5-4 server
> (four Intel Xeon E7-8895 v3 @ 2.60GHz sockets with 18 hyperthreaded
> cores each). Each number represents an average (over 25 runs) of the
> total number of ops (x10^7) reported at the end of each run. The 
> standard deviation is also reported in (), and in general is about 3%
> from the average. The 'stock' kernel is v5.12.0,

I assume x5-4 server has the crossbar topology and its numa diameter is
1hop, and all tests were done on this kind of symmetrical topology. Am
I right? 

    ┌─┐                 ┌─┐
    │ ├─────────────────┤ │
    └─┤1               1└┬┘
      │  1           1   │
      │    1       1     │
      │      1   1       │
      │        1         │
      │      1   1       │
      │     1      1     │
      │   1         1    │
     ┌┼┐1             1  ├─┐
     │┼┼─────────────────┤ │
     └─┘                 └─┘


what if the hardware is using the ring topology and other topologies with
2-hops or even 3-hops such as:

     ┌─┐                 ┌─┐
     │ ├─────────────────┤ │
     └─┤                 └┬┘
       │                  │
       │                  │
       │                  │
       │                  │
       │                  │
       │                  │
       │                  │
      ┌┤                  ├─┐
      │┼┬─────────────────┤ │
      └─┘                 └─┘


or:


    ┌───┐       ┌───┐      ┌────┐      ┌─────┐
    │   │       │   │      │    │      │     │
    │   │       │   │      │    │      │     │
    ├───┼───────┼───┼──────┼────┼──────┼─────┤
    │   │       │   │      │    │      │     │
    └───┘       └───┘      └────┘      └─────┘

do we need to consider the distances of numa nodes in the secondary
queue? does it still make sense to treat everyone else equal in
secondary queue?

Thanks
barry