Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <20220825060819.74303-1-liaochang1@huawei.com>
In-Reply-To: <20220825060819.74303-1-liaochang1@huawei.com>
From:   Andy Shevchenko <andy.shevchenko@gmail.com>
Date:   Thu, 25 Aug 2022 11:27:43 +0300
Message-ID: <CAHp75VdyYY08uwJGi0m4-L9=9rLmFzkeJGTi2-NC=prbgKvo3A@mail.gmail.com>
Subject: Re: [PATCH 1/2] genirq: Record dangling hwirq number into struct irq_data
To:     Liao Chang <liaochang1@huawei.com>
Cc:     Thomas Gleixner <tglx@linutronix.de>,
        Marc Zyngier <maz@kernel.org>,
        Samuel Holland <samuel@sholland.org>,
        Bartosz Golaszewski <brgl@bgdev.pl>,
        Michael Kelley <mikelley@microsoft.com>,
        Jianmin Lv <lvjianmin@loongson.cn>,
        Mark Rutland <mark.rutland@arm.com>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

On Thu, Aug 25, 2022 at 9:11 AM Liao Chang <liaochang1@huawei.com> wrote:

Use spell-checker, please. Or ask somebody for proof-reading of your
commit messages and comments in the code.

> Following interrupt allocation process lead to some interrupts are

leads

> mapped in the low-level domain(Arm ITS), but they are never been mapped

never mapped
  ...or...
they have never

> at the higher level.
>
> irq_domain_alloc_irqs_hierarchy(.., nr_irqs, ...)
>   its_irq_domain_alloc(..., nr_irqs, ...)
>     its_alloc_device_irq(..., nr_irqs, ...)
>       bitmap_find_free_region(..., get_count_order(nr_irqs))
>
> Since ITS domain find a region of zero bits, the length of which must

finds

> aligned to power of two. If nr_irqs is 30, the length of zero bits is

the power

> actually 32, but only first 30 bits are really mapped.

the first

> On teardown, low-level domain only free these interrupts that actually

the low-level
   ...or...
domains

> mapped, and leave last interrupts dangling in the ITS domain. Thus the
> ITS device resources are never freed. On device driver reload, dangling
> interrupts prevent ITS domain from allocating enough resource.
>
> irq_domain_free_irqs_hierarchy(..., nr_irqs, ...)
>   its_irq_domain_free(..., irq_base + i, 1)
>     bitmap_release_region(..., irq_base + i, get_count_order(1))
>
> John reported this problem to LKML and Marc provided a solution and fix
> it in the generic code, see the discussion from Link tag. Marc's patch
> fix John's problem, but does not take care of some corner case, look one
> example below.
>
> Step1: 32 interrupts allocated in LPI domain, but return the first 30 to
> higher driver.
>
>    111111111111111111111111111111 11
>   |<------------0~29------------>|30,31|
>
> Step2: interrupt #16~28 are released one by one, then #0~15 and #29~31
> still be there.
>
>    1111111111111111 0000000000000 1  11
>   |<-----0~15----->|<---16~28--->|29|30,31|
>
> Step#: on driver teardown, generic code will invoke ITS domain code
> twice. The first time, #0~15 will be released, the second one, only #29
> will be released(1 align to power of two).
>
>    0000000000000000 0000000000000 0  11
>   |<-----0~15----->|<---16~28--->|29|30,31|
>
> In short summary, the dangling problem stems from the number of released
> hwirq is less than the one of allocated hwirq in ITS domain. In order to

the allocated

> fix this problem, make irq_data record the number of allocated but
> unmapped hwirq. If hwirq followed by some unmapped bits, ITS domain
> record the number of unmapped bits to the last irq_data mapped to higher
> level, when the last hwirq followed by unmapped hwirq is released, some
> dangling bit will be clear eventualy, look back the trivial example

eventually

> above.
>
> Step1: record '2' into the irq_data.dangling of #29 hwirq.
>
>            111111111111111111111111111111 11
>           |<------------0~29------------>|30,31|
> dangling:  000000000000000000000000000002
>
> Step2: no change
>
>           1111111111111111 0000000000000 1  11
>          |<-----0~15----->|<---16~28--->|29|30,31|
> dangling: 0000000000000000 0000000000000 2
>
> Step3: ITS domain will release #30~31 since the irq_data.dangling of #29
> is '2'.
>
>            0000000000000000 0000000000000 0  00
>           |<-----0~15----->|<---16~28--->|29|30,31|
> dangling:  0000000000000000 0000000000000 2

> Fixes: 4615fbc3788dd ("genirq/irqdomain: Don't try to free an interrupt
> that has no mapping")

All tags must be one-liners. I.o.w. do not split a tag to multiple lines.

...

> + * @dangling:          amount of dangling hardware interrupt, Arm ITS allocate
> + *                     hardware interrupt more than expected, aligned to power
> + *                     of two, so that unsued interrupt number become dangling.

unused
becomes

> + *                     Use this field to record dangling bits follwoing @hwirq.

following

-- 
With Best Regards,
Andy Shevchenko