Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
To:     "Zhang, Qiang" <Qiang.Zhang@windriver.com>,
        Hillf Danton <hdanton@sina.com>,
        "axboe@kernel.dk" <axboe@kernel.dk>
Cc:     "syzbot+6cb11ade52aa17095297@syzkaller.appspotmail.com" 
        <syzbot+6cb11ade52aa17095297@syzkaller.appspotmail.com>,
        "io-uring@vger.kernel.org" <io-uring@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
References: <20210524071844.24085-1-qiang.zhang@windriver.com>
 <20210524082536.2032-1-hdanton@sina.com>
 <DM6PR11MB4202B442C4C27740B6EE2D64FF269@DM6PR11MB4202.namprd11.prod.outlook.com>
 <916ad789-c996-258f-d3b7-b41d749618d8@gmail.com>
 <DM6PR11MB4202561CE9ECD5B7F8DD74AFFF259@DM6PR11MB4202.namprd11.prod.outlook.com>
From:   Pavel Begunkov <asml.silence@gmail.com>
Subject: =?UTF-8?B?UmU6IOWbnuWkjTog5Zue5aSNOiBbUEFUQ0hdIGlvLXdxOiBGaXggVUFG?=
 =?UTF-8?Q?_when_wakeup_wqe_in_hash_waitqueue?=
Message-ID: <9af68623-57e4-cef0-bb61-347207fb0c45@gmail.com>
Date:   Mon, 7 Jun 2021 18:38:34 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.10.1
MIME-Version: 1.0
In-Reply-To: <DM6PR11MB4202561CE9ECD5B7F8DD74AFFF259@DM6PR11MB4202.namprd11.prod.outlook.com>
Content-Type: text/plain; charset=UTF-8
Content-Language: en-US
Content-Transfer-Encoding: 8bit
Precedence: bulk

On 5/25/21 3:01 AM, Zhang, Qiang wrote:
[...]
>> Haven't looked at the trace and description, but I do think
>> there is a problem it solves.
>>
>> 1) io_wait_on_hash() -> __add_wait_queue(&hash->wait, &wqe->wait);
>> 2) (note: wqe is a worker) wqe's workers exit dropping refs
>> 3) refs are zero, free io-wq
>> 4) @hash is shared, so other task/wq does wake_up(&wq->hash->wait);
>> 5) it wakes freed wqe
>>
>> step 4) is a bit more trickier than that, tl;dr;
>> wq3:worker1     | locks bit1
>> wq1:worker2     | waits bit1
>> wq2:worker1     | waits bit1
>> wq1:worker3     | waits bit1
>>
>> wq3:worker1     | drop  bit1
>> wq1:worker2     | locks bit1
>> wq1:worker2     | completes all wq1 bit1 work items
>> wq1:worker2     | drop  bit1, exit and free io-wq
>>
>> wq2:worker1     | locks bit1
>> wq1             | free complete
>> wq2:worker1     | drops bit1
>> wq1:worker3     | waked up, even though freed
>>
>> Can be simplified, don't want to waste time on that
> 
> Thanks Pavel
> 
> Your description is better.  I have another question: under what circumstances will three io-wq(wq1, wq2, wq3) be created to share this @hash?

Oops, missed the email. It's created by io_uring, and passed to
io-wq, which is per-task and created on demand by io_uring.

Can be achieved by a snippet just below, where threads
haven't had io_uring instances before.

thread1: ring = create_io_uring();
thread2: submit_sqes(ring);
thread3: submit_sqes(ring);

> 
> This kind of problem also occurs between two io-wq(wq1, wq2). Is the following description OK？

Yep, and I feel like there are cases simpler (and
more likely) than the one I described.

> 
> wq1:worker2     | locks bit1
> wq2:worker1     | waits bit1
> wq1:worker3     | waits bit1
> 
> wq1:worker2     | completes all wq1 bit1 work items
> wq1:worker2     | drop  bit1, exit and free io-wq
> 
> wq2:worker1     | locks bit1
> wq1                       | free complete
> wq2:worker1     | drops bit1
> wq1:worker3     | waked up, even though freed


-- 
Pavel Begunkov