Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <f6f52b55-a991-1179-75cb-bf8fce8f6783@linux.dev>
Date:   Thu, 6 Jul 2023 22:23:49 +0800
MIME-Version: 1.0
Subject: Re: [PATCH v2 1/4] blk-mq: use percpu csd to remote complete instead
 of per-rq csd
Content-Language: en-US
To:     Christoph Hellwig <hch@lst.de>
Cc:     axboe@kernel.dk, ming.lei@redhat.com, tj@kernel.org,
        linux-block@vger.kernel.org, linux-kernel@vger.kernel.org,
        Chengming Zhou <zhouchengming@bytedance.com>
References: <20230629110359.1111832-1-chengming.zhou@linux.dev>
 <20230629110359.1111832-2-chengming.zhou@linux.dev>
 <20230706130735.GA13089@lst.de>
From:   Chengming Zhou <chengming.zhou@linux.dev>
In-Reply-To: <20230706130735.GA13089@lst.de>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Precedence: bulk

On 2023/7/6 21:07, Christoph Hellwig wrote:
> On Thu, Jun 29, 2023 at 07:03:56PM +0800, chengming.zhou@linux.dev wrote:
>> From: Chengming Zhou <zhouchengming@bytedance.com>
>>
>> If request need to be completed remotely, we insert it into percpu llist,
>> and smp_call_function_single_async() if llist is empty previously.
>>
>> We don't need to use per-rq csd, percpu csd is enough. And the size of
>> struct request is decreased by 24 bytes.
>>
>> This way is cleaner, and looks correct, given block softirq is guaranteed to be
>> scheduled to consume the list if one new request is added to this percpu list,
>> either smp_call_function_single_async() returns -EBUSY or 0.
> 
> Please trim your commit logs to 73 characters per line so that they
> are readable in git log output.

Ok, will fix in the next version.

> 
>>  static void blk_mq_request_bypass_insert(struct request *rq,
>> @@ -1156,13 +1157,13 @@ static void blk_mq_complete_send_ipi(struct request *rq)
>>  {
>>  	struct llist_head *list;
>>  	unsigned int cpu;
>> +	call_single_data_t *csd;
>>  
>>  	cpu = rq->mq_ctx->cpu;
>>  	list = &per_cpu(blk_cpu_done, cpu);
>> -	if (llist_add(&rq->ipi_list, list)) {
>> -		INIT_CSD(&rq->csd, __blk_mq_complete_request_remote, rq);
>> -		smp_call_function_single_async(cpu, &rq->csd);
>> -	}
>> +	csd = &per_cpu(blk_cpu_csd, cpu);
>> +	if (llist_add(&rq->ipi_list, list))
>> +		smp_call_function_single_async(cpu, csd);
>>  }
> 
> No need for the list and csd variables here as they are only used
> once.

Yes, should I change like below? Looks like much long code. :-)

if (llist_add(&rq->ipi_list, &per_cpu(blk_cpu_done, cpu)))
	smp_call_function_single_async(cpu, &per_cpu(blk_cpu_csd, cpu));

> 
> But I think this code has a rpboem when it is preemptd between
> the llist_add and smp_call_function_single_async.  We either need a
> get_cpu/put_cpu around them, or instroduce a structure with the list
> and csd, and then you can use one pointer from per_cpu and still ensure
> the list and csd are for the same CPU.
> 

cpu = rq->mq_ctx->cpu; So it's certainly the same CPU, right?

Thanks!