From: Hannes Reinecke <[email protected]>
When the number of hardware queues changes during resetting we should
update the tagset first before using it.
Signed-off-by: Hannes Reinecke <[email protected]>
Signed-off-by: Daniel Wagner <[email protected]>
---
drivers/nvme/host/tcp.c | 14 ++++++--------
1 file changed, 6 insertions(+), 8 deletions(-)
diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index 0a97ba02f61e..32268f24f62a 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -1789,6 +1789,7 @@ static void nvme_tcp_destroy_io_queues(struct nvme_ctrl *ctrl, bool remove)
static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new)
{
int ret;
+ u32 prior_q_cnt = ctrl->queue_count;
ret = nvme_tcp_alloc_io_queues(ctrl);
if (ret)
@@ -1806,14 +1807,7 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new)
ret = PTR_ERR(ctrl->connect_q);
goto out_free_tag_set;
}
- }
-
- ret = nvme_tcp_start_io_queues(ctrl);
- if (ret)
- goto out_cleanup_connect_q;
-
- if (!new) {
- nvme_start_queues(ctrl);
+ } else if (prior_q_cnt != ctrl->queue_count) {
if (!nvme_wait_freeze_timeout(ctrl, NVME_IO_TIMEOUT)) {
/*
* If we timed out waiting for freeze we are likely to
@@ -1828,6 +1822,10 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new)
nvme_unfreeze(ctrl);
}
+ ret = nvme_tcp_start_io_queues(ctrl);
+ if (ret)
+ goto out_cleanup_connect_q;
+
return 0;
out_wait_freeze_timed_out:
--
2.29.2
> From: Hannes Reinecke <[email protected]>
>
> When the number of hardware queues changes during resetting we should
> update the tagset first before using it.
>
> Signed-off-by: Hannes Reinecke <[email protected]>
> Signed-off-by: Daniel Wagner <[email protected]>
> ---
> drivers/nvme/host/tcp.c | 14 ++++++--------
> 1 file changed, 6 insertions(+), 8 deletions(-)
>
> diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
> index 0a97ba02f61e..32268f24f62a 100644
> --- a/drivers/nvme/host/tcp.c
> +++ b/drivers/nvme/host/tcp.c
> @@ -1789,6 +1789,7 @@ static void nvme_tcp_destroy_io_queues(struct nvme_ctrl *ctrl, bool remove)
> static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new)
> {
> int ret;
> + u32 prior_q_cnt = ctrl->queue_count;
>
> ret = nvme_tcp_alloc_io_queues(ctrl);
> if (ret)
> @@ -1806,14 +1807,7 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new)
> ret = PTR_ERR(ctrl->connect_q);
> goto out_free_tag_set;
> }
> - }
> -
> - ret = nvme_tcp_start_io_queues(ctrl);
> - if (ret)
> - goto out_cleanup_connect_q;
> -
> - if (!new) {
> - nvme_start_queues(ctrl);
> + } else if (prior_q_cnt != ctrl->queue_count) {
So if the queue count did not change we don't wait to make sure
the queue g_usage_counter ref made it to zero? What guarantees that it
did?
> if (!nvme_wait_freeze_timeout(ctrl, NVME_IO_TIMEOUT)) {
> /*
> * If we timed out waiting for freeze we are likely to
> @@ -1828,6 +1822,10 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new)
> nvme_unfreeze(ctrl);
> }
>
> + ret = nvme_tcp_start_io_queues(ctrl);
> + if (ret)
> + goto out_cleanup_connect_q;
> +
Did you test this with both heavy I/O, reset loop and ifdown/ifup loop?
If we unquiesce and unfreeze before we start the queues the pending I/Os
may resume before the connect and not allow the connect to make forward
progress.
> return 0;
>
> out_wait_freeze_timed_out:
>
Hi Sagi,
On Fri, Aug 06, 2021 at 12:57:17PM -0700, Sagi Grimberg wrote:
> > - ret = nvme_tcp_start_io_queues(ctrl);
> > - if (ret)
> > - goto out_cleanup_connect_q;
> > -
> > - if (!new) {
> > - nvme_start_queues(ctrl);
> > + } else if (prior_q_cnt != ctrl->queue_count) {
>
> So if the queue count did not change we don't wait to make sure
> the queue g_usage_counter ref made it to zero? What guarantees that it
> did?
Hmm, good point. we should always call nvme_wait_freeze_timeout()
for !new queues. Is this what you are implying?
> > if (!nvme_wait_freeze_timeout(ctrl, NVME_IO_TIMEOUT)) {
> > /*
> > * If we timed out waiting for freeze we are likely to
> > @@ -1828,6 +1822,10 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new)
> > nvme_unfreeze(ctrl);
> > }
> > + ret = nvme_tcp_start_io_queues(ctrl);
> > + if (ret)
> > + goto out_cleanup_connect_q;
> > +
>
> Did you test this with both heavy I/O, reset loop and ifdown/ifup
> loop?
Not sure if this classifies as heavy I/O (on 80 CPU machine)
fio --rw=readwrite --name=test --filename=/dev/nvme16n1 --size=50M \
--direct=1 --bs=4k --numjobs=40 --group_reporting --runtime=4h \
--time_based
and then I installed iptables rules to block the traffic on the
controller side. With this test it is pretty easily to get
the host hanging. Let me know what test you would like to see
from me. I am glad to try to get them running.
> If we unquiesce and unfreeze before we start the queues the pending I/Os
> may resume before the connect and not allow the connect to make forward
> progress.
So the unfreeze should happen after the connect call? What about the
newly created queues? Do they not suffer from the same problem? Isn't
the NVME_TCP_Q_LIVE flag not enough?
Daniel
On 8/9/21 1:52 AM, Daniel Wagner wrote:
> Hi Sagi,
>
> On Fri, Aug 06, 2021 at 12:57:17PM -0700, Sagi Grimberg wrote:
>>> - ret = nvme_tcp_start_io_queues(ctrl);
>>> - if (ret)
>>> - goto out_cleanup_connect_q;
>>> -
>>> - if (!new) {
>>> - nvme_start_queues(ctrl);
>>> + } else if (prior_q_cnt != ctrl->queue_count) {
>>
>> So if the queue count did not change we don't wait to make sure
>> the queue g_usage_counter ref made it to zero? What guarantees that it
>> did?
>
> Hmm, good point. we should always call nvme_wait_freeze_timeout()
> for !new queues. Is this what you are implying?
I think we should always wait for the freeze to complete.
>
>
>>> if (!nvme_wait_freeze_timeout(ctrl, NVME_IO_TIMEOUT)) {
>>> /*
>>> * If we timed out waiting for freeze we are likely to
>>> @@ -1828,6 +1822,10 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new)
>>> nvme_unfreeze(ctrl);
>>> }
>>> + ret = nvme_tcp_start_io_queues(ctrl);
>>> + if (ret)
>>> + goto out_cleanup_connect_q;
>>> +
>>
>> Did you test this with both heavy I/O, reset loop and ifdown/ifup
>> loop?
>
> Not sure if this classifies as heavy I/O (on 80 CPU machine)
>
> fio --rw=readwrite --name=test --filename=/dev/nvme16n1 --size=50M \
> --direct=1 --bs=4k --numjobs=40 --group_reporting --runtime=4h \
> --time_based
>
> and then I installed iptables rules to block the traffic on the
> controller side. With this test it is pretty easily to get
> the host hanging. Let me know what test you would like to see
> from me. I am glad to try to get them running.
Lets add iodepth=128
>> If we unquiesce and unfreeze before we start the queues the pending I/Os
>> may resume before the connect and not allow the connect to make forward
>> progress.
>
> So the unfreeze should happen after the connect call? What about the
> newly created queues? Do they not suffer from the same problem? Isn't
> the NVME_TCP_Q_LIVE flag not enough?
Q_LIVE will protect against the transport itself from queueing, however
when multipath is not used, the transport will return BLK_STS_RESOURCE
which will immediately trigger re-submission, in an endless loop, and
that can prevent forward progress. It is also consistent with what
nvme-pci does.
On Tue, Aug 10, 2021 at 06:00:37PM -0700, Sagi Grimberg wrote:
>
>
> On 8/9/21 1:52 AM, Daniel Wagner wrote:
> > Hi Sagi,
> >
> > On Fri, Aug 06, 2021 at 12:57:17PM -0700, Sagi Grimberg wrote:
> > > > - ret = nvme_tcp_start_io_queues(ctrl);
> > > > - if (ret)
> > > > - goto out_cleanup_connect_q;
> > > > -
> > > > - if (!new) {
> > > > - nvme_start_queues(ctrl);
> > > > + } else if (prior_q_cnt != ctrl->queue_count) {
> > >
> > > So if the queue count did not change we don't wait to make sure
> > > the queue g_usage_counter ref made it to zero? What guarantees that it
> > > did?
> >
> > Hmm, good point. we should always call nvme_wait_freeze_timeout()
> > for !new queues. Is this what you are implying?
>
> I think we should always wait for the freeze to complete.
Don't the queues need to be started in order for the freeze to complete?
Any enqueued requests on the quiesced queues will never complete this
way, so the wait_freeze() will be stuck, right? If so, I think the
nvme_start_queues() was in the correct place already.
>> On 8/9/21 1:52 AM, Daniel Wagner wrote:
>>> Hi Sagi,
>>>
>>> On Fri, Aug 06, 2021 at 12:57:17PM -0700, Sagi Grimberg wrote:
>>>>> - ret = nvme_tcp_start_io_queues(ctrl);
>>>>> - if (ret)
>>>>> - goto out_cleanup_connect_q;
>>>>> -
>>>>> - if (!new) {
>>>>> - nvme_start_queues(ctrl);
>>>>> + } else if (prior_q_cnt != ctrl->queue_count) {
>>>>
>>>> So if the queue count did not change we don't wait to make sure
>>>> the queue g_usage_counter ref made it to zero? What guarantees that it
>>>> did?
>>>
>>> Hmm, good point. we should always call nvme_wait_freeze_timeout()
>>> for !new queues. Is this what you are implying?
>>
>> I think we should always wait for the freeze to complete.
>
> Don't the queues need to be started in order for the freeze to complete?
> Any enqueued requests on the quiesced queues will never complete this
> way, so the wait_freeze() will be stuck, right? If so, I think the
> nvme_start_queues() was in the correct place already.
Exactly what I was trying to point out (poorly though)
On Tue, Aug 10, 2021 at 10:57:58PM -0700, Sagi Grimberg wrote:
> > > I think we should always wait for the freeze to complete.
> >
> > Don't the queues need to be started in order for the freeze to complete?
> > Any enqueued requests on the quiesced queues will never complete this
> > way, so the wait_freeze() will be stuck, right? If so, I think the
> > nvme_start_queues() was in the correct place already.
>
> Exactly what I was trying to point out (poorly though)
Thanks for explaining. I think I got the general idea what the different
states are doing and what the transitions are now. (famous last words).
Anyway, the first three patches are the result of debugging the case of
'prior_ioq_cnt != nr_io_queues'. Starting the queues before updating the
number of queues lookes strange.
I suppose in the case 'prior_ioq_cnt > nr_io_queues',
nvme_tcp_start_io_queues() should be successful and we do the
blk_mq_update_nr_hw_queues(). In the other case we should land in the
error recovery.
Wouldn't it make sense to always exercise the error recovery path if we
detect 'prior_ioq_cnt != nr_io_queues' and reduce the number of things
which can go wrong?
Daniel