Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Fri, 15 Jul 2022 16:28:09 -0400
From:   Rodrigo Vivi <rodrigo.vivi@intel.com>
To:     Mauro Carvalho Chehab <mchehab@kernel.org>
CC:     Andrzej Hajda <andrzej.hajda@intel.com>,
        David Airlie <airlied@linux.ie>,
        <dri-devel@lists.freedesktop.org>, <linux-kernel@vger.kernel.org>,
        "Chris Wilson" <chris@chris-wilson.co.uk>,
        <intel-gfx@lists.freedesktop.org>,
        "Matthew Auld" <matthew.auld@intel.com>
Subject: Re: [Intel-gfx] [PATCH RFC] drm/i915/gt: Retry RING_HEAD reset until
 it sticks
Message-ID: <YtHN2ZXvpuIWYBpo@intel.com>
References: <2378da383d043de17172d928e59da0ec423cae76.1657873550.git.mchehab@kernel.org>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <2378da383d043de17172d928e59da0ec423cae76.1657873550.git.mchehab@kernel.org>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?3mfqARPYdSjzQ3rapdH6tA4ZricKmYSPErs3rJ8FPpGz/zy9CRgZ2IVhOomB?=
 =?us-ascii?Q?MRO8ooYakiFKgIquAMW6LMUaHT4JfVzTC0Z8ragpj1RxrHPTPLWq/k7fMhMx?=
 =?us-ascii?Q?/ZG/L9T+YjB7zdoqK/WTQrVMcx8zUjOcNGWr8tVZ+WJIpb4j6sKWR+tIMijb?=
 =?us-ascii?Q?3WrfFX0oWjPpxQR3j/sSWJMG5F4yhS9GsuftpR8z1pqWphwpJhns1iGclVNe?=
 =?us-ascii?Q?1Sblaq9Km79+NMeaOsWnmLXDGhwi+Fp/sN39mV3tZWZk9Squx9pc0PTjx0IV?=
 =?us-ascii?Q?7f4geirUPAmGYoHmAJ7dxaZeOHvYI3uIadr1a2/u4gATgCn/kDwDKOFVs0Vo?=
 =?us-ascii?Q?TnkjhQHLu4UW1hbhmBewI44OajdaEbdEIuZX4g40Th5/sIFqh4oPPx84sjd5?=
 =?us-ascii?Q?pIYwJ3C6VodLLMEi97691ihxcEJoWOvkUYz93EnHNa1WRFmszI7KkyY+dSIf?=
 =?us-ascii?Q?m/+Z5VZz7o8ONBonQ3Ad/yDW0aoAwp3nmbb7pMzIctrXLMkR5bxiS2xWcLaX?=
 =?us-ascii?Q?qqqfo0yVvYXn1xi1+Ob3KIi3c6NLJOXbriOUtyhKYOQy2sa23bLFTBBmFSwi?=
 =?us-ascii?Q?NYp8J7WSKrrAz3wQ/ZmJCYMphkNxcFeQtiN/Pp+eP25fez9MhqCeBgwvm4OQ?=
 =?us-ascii?Q?gkrJJhmYW0ZEu5grmC3ZRrDt1ssjtOxaitJkoGB8MoRwAPJuTxrPDNb7fSsF?=
 =?us-ascii?Q?3VsDsrmIuO3Tea+wN1rTyLn4SYZ8UNQAriFTaBeR7BdtrsmFmJsrMRef6RyB?=
 =?us-ascii?Q?8Alh+RTstqo+Db6Wu6vk0+GpkLXSD/SGG/mydUjHAXhnn0IVWLo64ryxau6r?=
 =?us-ascii?Q?t30ww3OcUMm9NnKEZlS7K86yc+kwdhnc/eJc4k33JLrbqzJoo/cvwvGoQdU5?=
 =?us-ascii?Q?G005JhlM9EAPELrZfVJtFcnEt87mgzB0Rm4Mv48RH7vsoVvqFCWeFcp4zMvy?=
 =?us-ascii?Q?omnG5s6HBbsVRtp8UzI2rqmCx8EC0BrzJYu6eCsrH3Ags5RlRncVh0KIc+zc?=
 =?us-ascii?Q?PaHG/oVJWObB3EW8NRkag4+gJBWuEgvwMAIOfyqPzD1Ag0nyGsw/PMLYJ/ON?=
 =?us-ascii?Q?mmrmVdNbT/oQ0QD1rE1Eo90ArVGMgv9EbkHP+Y56MGgXVQT+EMCJiV+v4zZy?=
 =?us-ascii?Q?9qckA1h+bb8uiT6x+bXNaMOEzOBsMgXkr46BFto2DDSbyHKmBhgNb68k2Jh3?=
 =?us-ascii?Q?7laX9lmkfJ1+V4rpdSo4Ysiju9/pGC374B2iopgqQMNtrvaaUR4lXka54/8A?=
 =?us-ascii?Q?uolzJChxkCbpPqXbb2afe/4npaP8n0mCD9ENbiRgkdmts+/cemzY/FnhMd13?=
 =?us-ascii?Q?i/9Mu9TZg/dzO6r1EppWbgV+bPfdiMdk2McMR5D7hk6O45Xx/FRYQ9GIp5vf?=
 =?us-ascii?Q?vgckLAOFxtVo4SmrFii2mhtXqdxna4RMxH/4xuxOBov78vpHMkrnaWiJ1nRx?=
 =?us-ascii?Q?FH4C6iLoUgaY/1tHHs4vADJoha4KduhfYXH3AJuDt6nqRLaLRvLjl6xDS/g/?=
 =?us-ascii?Q?8JcnNm7w5g28lDngAMi9DnVu523BXARCnEG0HF5quSwAmIJYyEpwe7gkyJCh?=
 =?us-ascii?Q?tUT1cizVsV7QW9O31kqF3iJ88Z80suEnA39mTetpmRwEw/NWOtlyWpdeGRoD?=
 =?us-ascii?Q?OA=3D=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: fa7013df-9b10-4572-c311-08da66a08a6a
X-MS-Exchange-CrossTenant-AuthSource: MN0PR11MB6059.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Jul 2022 20:28:13.2836
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: ElML3heDenoy386MVqL64V0dzCwEGGgsMyAi0Mh96JM0KDk17v84DVdCJT5WYQNfUU3dR3mcZrqUMcvXLAliWA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN7PR11MB2724
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-5.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Fri, Jul 15, 2022 at 09:26:16AM +0100, Mauro Carvalho Chehab wrote:
> From: Chris Wilson <chris@chris-wilson.co.uk>
> 
> On Haswell, in particular, we see an issue where resets fails because

Can we then make this platform specific?
Only because some older hw doesn't behave like expected we shouldn't
make this a default & global workaround.

> the engine resumes from an incorrect RING_HEAD. Since the RING_HEAD
> doesn't point to the remaining requests to re-run, but may instead point
> into the uninitialised portion of the ring, the GPU may be then fed
> invalid instructions from a privileged context, oft pushing the GPU into
> an unrecoverable hang.
> 
> If at first the write doesn't succeed, try, try again.
> 
> References: https://gitlab.freedesktop.org/drm/intel/-/issues/5432
> Testcase: igt/i915_selftest/hangcheck
> Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
> Cc: Andrzej Hajda <andrzej.hajda@intel.com>
> Cc: Mika Kuoppala <mika.kuoppala@linux.intel.com>
> Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
> ---
>  .../gpu/drm/i915/gt/intel_ring_submission.c   | 44 +++++++++++++------
>  drivers/gpu/drm/i915/i915_utils.h             | 10 +++++
>  2 files changed, 40 insertions(+), 14 deletions(-)
> 
> diff --git a/drivers/gpu/drm/i915/gt/intel_ring_submission.c b/drivers/gpu/drm/i915/gt/intel_ring_submission.c
> index d5d6f1fadcae..cc53feb1f8ed 100644
> --- a/drivers/gpu/drm/i915/gt/intel_ring_submission.c
> +++ b/drivers/gpu/drm/i915/gt/intel_ring_submission.c
> @@ -190,6 +190,7 @@ static bool stop_ring(struct intel_engine_cs *engine)
>  static int xcs_resume(struct intel_engine_cs *engine)
>  {
>  	struct intel_ring *ring = engine->legacy.ring;
> +	ktime_t kt;
>  
>  	ENGINE_TRACE(engine, "ring:{HEAD:%04x, TAIL:%04x}\n",
>  		     ring->head, ring->tail);
> @@ -228,9 +229,20 @@ static int xcs_resume(struct intel_engine_cs *engine)
>  	set_pp_dir(engine);
>  
>  	/* First wake the ring up to an empty/idle ring */
> -	ENGINE_WRITE_FW(engine, RING_HEAD, ring->head);
> +	until_timeout_ns(kt, 2 * NSEC_PER_MSEC) {
> +		ENGINE_WRITE_FW(engine, RING_HEAD, ring->head);
> +		if (ENGINE_READ_FW(engine, RING_HEAD) == ring->head)
> +			break;
> +	}
> +
>  	ENGINE_WRITE_FW(engine, RING_TAIL, ring->head);
> -	ENGINE_POSTING_READ(engine, RING_TAIL);
> +	if (ENGINE_READ_FW(engine, RING_HEAD) != ENGINE_READ_FW(engine, RING_TAIL)) {
> +		ENGINE_TRACE(engine, "failed to reset empty ring: [%x, %x]: %x\n",
> +			     ENGINE_READ_FW(engine, RING_HEAD),
> +			     ENGINE_READ_FW(engine, RING_TAIL),
> +			     ring->head);
> +		goto err;
> +	}

commit message mentions until this point I'm afraid... everything below
(except the new until_timeout_ns) looks like a different patch to me,
or deserves some mention in the commit msg.

>  
>  	ENGINE_WRITE_FW(engine, RING_CTL,
>  			RING_CTL_SIZE(ring->size) | RING_VALID);
> @@ -239,12 +251,16 @@ static int xcs_resume(struct intel_engine_cs *engine)
>  	if (__intel_wait_for_register_fw(engine->uncore,
>  					 RING_CTL(engine->mmio_base),
>  					 RING_VALID, RING_VALID,
> -					 5000, 0, NULL))
> +					 5000, 0, NULL)) {
> +		ENGINE_TRACE(engine, "failed to restart\n");
>  		goto err;
> +	}
>  
> -	if (GRAPHICS_VER(engine->i915) > 2)
> +	if (GRAPHICS_VER(engine->i915) > 2) {
>  		ENGINE_WRITE_FW(engine,
>  				RING_MI_MODE, _MASKED_BIT_DISABLE(STOP_RING));
> +		ENGINE_POSTING_READ(engine, RING_MI_MODE);
> +	}
>  
>  	/* Now awake, let it get started */
>  	if (ring->tail != ring->head) {
> @@ -257,16 +273,16 @@ static int xcs_resume(struct intel_engine_cs *engine)
>  	return 0;
>  
>  err:
> -	drm_err(&engine->i915->drm,
> -		"%s initialization failed; "
> -		"ctl %08x (valid? %d) head %08x [%08x] tail %08x [%08x] start %08x [expected %08x]\n",
> -		engine->name,
> -		ENGINE_READ(engine, RING_CTL),
> -		ENGINE_READ(engine, RING_CTL) & RING_VALID,
> -		ENGINE_READ(engine, RING_HEAD), ring->head,
> -		ENGINE_READ(engine, RING_TAIL), ring->tail,
> -		ENGINE_READ(engine, RING_START),
> -		i915_ggtt_offset(ring->vma));
> +	ENGINE_TRACE(engine,
> +		     "initialization failed; "
> +		     "ctl %08x (valid? %d) head %08x [%08x] tail %08x [%08x] start %08x [expected %08x]\n",
> +		     ENGINE_READ(engine, RING_CTL),
> +		     ENGINE_READ(engine, RING_CTL) & RING_VALID,
> +		     ENGINE_READ(engine, RING_HEAD), ring->head,
> +		     ENGINE_READ(engine, RING_TAIL), ring->tail,
> +		     ENGINE_READ(engine, RING_START),
> +		     i915_ggtt_offset(ring->vma));
> +	GEM_TRACE_DUMP();
>  	return -EIO;
>  }
>  
> diff --git a/drivers/gpu/drm/i915/i915_utils.h b/drivers/gpu/drm/i915/i915_utils.h
> index c10d68cdc3ca..717fb6b9cc15 100644
> --- a/drivers/gpu/drm/i915/i915_utils.h
> +++ b/drivers/gpu/drm/i915/i915_utils.h
> @@ -256,6 +256,16 @@ wait_remaining_ms_from_jiffies(unsigned long timestamp_jiffies, int to_wait_ms)
>  	}
>  }
>  
> +/**
> + * until_timeout_ns - Keep retrying (busy spin) until the duration has passed
> + * @end: temporary var to be used to track the spent time
> + * @timeout_ns: Maximum timeout, in nanosseconds
> + */
> +#define until_timeout_ns(end, timeout_ns) \
> +	for ((end) = ktime_get() + (timeout_ns); \
> +	     ktime_before(ktime_get(), (end)); \
> +	     cpu_relax())
> +

why do we need yet another timeout macro and cannot use any of the existent ways?

>  /**
>   * __wait_for - magic wait macro
>   *
> -- 
> 2.36.1
> 
>