Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) client-ip=23.128.96.38;
Message-ID: <e0bc7eb5-f1b9-4e99-a39f-ca6f31daa9d8@siemens.com>
Date:   Thu, 2 Nov 2023 11:07:45 +0100
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v3 3/5] remoteproc: k3-r5: Add support for IPC-only mode
 for all R5Fs
Content-Language: en-US
To:     Suman Anna <s-anna@ti.com>,
        Bjorn Andersson <bjorn.andersson@linaro.org>,
        Mathieu Poirier <mathieu.poirier@linaro.org>
Cc:     Hari Nagalla <hnagalla@ti.com>, Praneeth Bajjuri <praneeth@ti.com>,
        linux-remoteproc@vger.kernel.org,
        linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org,
        Nishanth Menon <nm@ti.com>,
        "Su, Bao Cheng (RC-CN DF FA R&D)" <baocheng.su@siemens.com>
References: <20220213201246.25952-1-s-anna@ti.com>
 <20220213201246.25952-4-s-anna@ti.com>
From:   Jan Kiszka <jan.kiszka@siemens.com>
In-Reply-To: <20220213201246.25952-4-s-anna@ti.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?ajAwZ1pNNGpCYUFnMlZrY0JKU1ZZb1FnQ3FQV1dST0R2Zkc4NExQQmJSa2dS?=
 =?utf-8?B?ZnRXKzQwQWMzVUNOVUh2OGNRYjk3YzZZSmY3WDR5cEI0TnJldEFzWXVFVmdY?=
 =?utf-8?B?VXJXaXlYcXR5ck5NbTZ1Szh0a3ZGa3liMitjUTF6ZEI5MnhlWW5ITVJCOURs?=
 =?utf-8?B?NVB4bytJN0FqZWRXazFxMEFkczU4VGRvSHhSbFVpOHYyYzhyYjNzb08yckdG?=
 =?utf-8?B?L1lQMjZ5Q0R1c0U5Wmk5b2VCKzAwUitZbytEc2tYelEzVmhuTjYxaG9EeVYy?=
 =?utf-8?B?OXV1LzBxa1ByYnAyWGdLUE1HemsyQUNjMm5OVGw0cUsxZTRwYTlrK01zdlJG?=
 =?utf-8?B?eUIxdnUxNFl6VnRzODdBMkFxVUZMb0tEUEM2TytUa3dIaWlyckNRR0xJaE0z?=
 =?utf-8?B?VFZvRzVJSTlERy9HSlhJSGlDZWcvemRGdm94a3lUTjFSa0dNaklpMy83ZVJh?=
 =?utf-8?B?dlQ1V0o4UUd0RW12VU1kRzErSGVVMC85MzJML2tEd0t3QW1yMnh3L1YxWGNB?=
 =?utf-8?B?SHQwWFhXTFRHTTYzZEovZjFWOUlkT294Q0txS2tzZVZxMmhHLzBKSmd6ZkNE?=
 =?utf-8?B?bDJjdXF6eHdMdDA0bWJpN01LZUJYczh0NllPejBwbVZ4Umh3RFUrQ3R5cnVC?=
 =?utf-8?B?d21HUFNTT2ZxT0NhVUQ0bFJZaUhmaDRRQVoza2p6b1QvQ0hyM1E4SmpqRTVl?=
 =?utf-8?B?TnRWaTFLaE1LNXVoNE0zRWxPYkdKdE54cEp1OTFjZG9zWmticzkyaStKVmFT?=
 =?utf-8?B?ZTNmcE1RNlU3WkJNdTU2eXdHVmxXSzBlKzVBMngyOThTL2ZPa3p6VVhKOVJU?=
 =?utf-8?B?TlNpclJWQlJKeDh4SUE3dkxURUc3WHlNSFJkaHN6dW8zMkxCd0p6WkZsSjVJ?=
 =?utf-8?B?ZnNST2Vkdk9BeHk3bXNzZWgxTTN6bWJObHpWaU1wUXhFVWQ2OFZOeVZLSWgz?=
 =?utf-8?B?K2g5RTRqYVFyc1B1Zyt6a3IyTnpWVE5XbklmRnB4cUJjK0EwSVNwQ0p1dDFG?=
 =?utf-8?B?ejZHSE1qMkx5UG1MNDVWQXFUOHQ3VWd1N0Y1L0IwLzA1ZWpNdElJVFhuTFZX?=
 =?utf-8?B?bm50MWI0QVRKUFNvVWM4YnRaY0xRUG94UlFhUTdrZllzQnNEY1pKcGdpTTgv?=
 =?utf-8?B?NVBFallMOHlXVSsxMVpEUy9IcGR2aU1NWXVTMExPeTJzNzcrTUxyQ3ZrUExF?=
 =?utf-8?B?YVRYWUllYkpIbkJGNVZhM3lVejBYQTRvSkhpUStxd3Y0d2xOKzZjZ1UzMjFS?=
 =?utf-8?B?SDhyL04zMHNhbStmWGdZd2FsMmpXOUxvWmR5Rms2enF5bzkyNTRlQ1FQVjkw?=
 =?utf-8?B?cmE1NUlBam5SSnJqcUxNN0U5dlhWV3crR1pKb242RUV5ZDhLRnpzSHh6aWtR?=
 =?utf-8?B?T3ZDSUoxK2RSV2NvN3dpTUxlYU80T0xXQWxXTC9TRlVJUWl6cVd6T2ozeDdq?=
 =?utf-8?B?NmhJYmI2WDh4Z1dXNWNEd0ZmUjNJOGtiOTZFQTFCYVIzTGFKaTV2Q05RVTIv?=
 =?utf-8?B?MVRyYXZMemN2aXdnYS84cHVpWHhPYlljd3BURTloUUg0UVB5WVNqRmE3NVZW?=
 =?utf-8?B?ejRDbVJhbWw5Z2VHUkZ6bWtQcXpkNStBbTlWZUNnYW5sc0xuNUVLNFJ4aVZh?=
 =?utf-8?B?bVRyb2R1TmF4alVmLzVhUm5zRmJFS3YzaVlTSkhxeTZzQlJRN1hjLzM4eFlk?=
 =?utf-8?B?RXpua2cxOEFPZVlhTHlpbjBoeEs2NUFubjhiRDVjRm1Sb3FrSmozbWp3d1Vn?=
 =?utf-8?B?Ry9GR1pXR21Rdko0TWhncGVYWDJCNElKUkRBNG9uM2ZPelZvUHRHUGNxbFJF?=
 =?utf-8?B?NmQyU3hBM1NuZ2tITFBDV2ExcjMxZ1JnTTVoSjhZclR2UExaaXpzcWNnSUt6?=
 =?utf-8?B?TDlyRHoxOExjTlNIR2ZNWVB0ZGlGRUZoQTVmTGQvZmJ0TGo5aCtlZjdrMUNQ?=
 =?utf-8?B?bWhWR0IrSWNPL25PNzFxTUl5bTNWT2NSajFHM0ZEUitLQlFjZUlDSVFseGl5?=
 =?utf-8?B?K3ZRYXR1c2dGdDB4WGxBV3daQVlyYWRxZ2p1UnI2em8zbjhnb0Y5Qm95Ulhw?=
 =?utf-8?B?SEJqRjlBNUNUcEFPS3cxNnp1ZWhQNWR2TitML2hVUzV0Mk1EOWZFRXZzbDlQ?=
 =?utf-8?Q?pLN25HaZk4/13holwAqBmNJHI?=
X-OriginatorOrg: siemens.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 12da6b09-306c-48c6-5640-08dbdb8b911c
X-MS-Exchange-CrossTenant-AuthSource: AS4PR10MB6181.EURPRD10.PROD.OUTLOOK.COM
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Nov 2023 10:07:48.6730
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 38ae3bcd-9579-4fd4-adda-b42e1495d55a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: MHVsUCyhUa6RsnTJr3cuUrrOOUzEvuq7eriasKLyFxWbSzNmRDOYlfv9ufrottcbcnLprxe4VUMk5/TTgu/vLA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS2PR10MB6749
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,
	URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Thu, 02 Nov 2023 03:08:12 -0700 (PDT)

On 13.02.22 21:12, Suman Anna wrote:
> Add support to the K3 R5F remoteproc driver to configure all the R5F
> cores to be either in IPC-only mode or the traditional remoteproc mode.
> The IPC-only mode expects that the remote processors are already booted
> by the bootloader, and only performs the minimum steps required to
> initialize and deinitialize the virtio IPC transports. The remoteproc
> mode allows the kernel remoteproc driver to do the regular load and
> boot and other device management operations for a R5F core.
> 
> The IPC-only mode for a R5F core is detected and configured at driver
> probe time by querying the System Firmware for the R5F power and reset
> state and/or status and making sure that the R5F core is indeed started
> by the bootloaders, otherwise the device is configured for remoteproc
> mode.
> 
> Support for IPC-only mode is achieved through .attach(), .detach() and
> .get_loaded_rsc_table() callback ops and zeroing out the regular rproc
> ops .prepare(), .unprepare(), .start() and .stop(). The resource table
> follows a design-by-contract approach and is expected to be at the base
> of the DDR firmware region reserved for each remoteproc, it is mostly
> expected to contain only the virtio device and trace resource entries.
> 
> NOTE:
> The driver cannot configure a R5F core for remoteproc mode by any
> means without rebooting the kernel if that R5F core has been started
> by a bootloader. This is the current desired behavior and can be
> enhanced in the future if the feature is needed.
> 

This change surfaced some complex issue in the K3 core: Turning on the
RTI1 watchdog also powers up R5F core 1. And this could happen either in
U-Boot or in the kernel. If the kernel finds the core running, it also
expects a resource table in the reserved RAM. But as the core is
supposed to start via remoteproc, there is none, rather often garbage.
Sometimes, a consistency check catches that, but not always:

[   38.372844] remoteproc remoteproc18: 41000000.r5f is available
[   38.380324] platform 41400000.r5f: R5F core may have been powered on by a different host, programmed state (0) != actual state (1)
[   38.394910] platform 41400000.r5f: configured R5F for IPC-only mode
[   38.401941] platform 41400000.r5f: assigned reserved memory node r5f-dma-memory@a1000000
[   38.476997] remoteproc remoteproc19: 41400000.r5f is available
[   38.484661] remoteproc remoteproc19: attaching to 41400000.r5f
[   38.491092] Unable to handle kernel paging request at virtual address ffff80000dffffff
[   38.503704] Mem abort info:
[   38.509760]   ESR = 0x0000000096000007
[   38.514071]   EC = 0x25: DABT (current EL), IL = 32 bits
[   38.519578]   SET = 0, FnV = 0
[   38.523095]   EA = 0, S1PTW = 0
[   38.526355]   FSC = 0x07: level 3 translation fault
[   38.528974] cal 6f03000.cal: Neither port is configured, no point in staying up
[   38.531775] Data abort info:
[   38.541866]   ISV = 0, ISS = 0x00000007
[   38.545765]   CM = 0, WnR = 0
[   38.548814] swapper pgtable: 4k pages, 48-bit VAs, pgdp=0000000082fdc000
[   38.555831] [ffff80000dffffff] pgd=10000008fffff003, p4d=10000008fffff003, pud=10000008ffffe003, pmd=1000000886609003, pte=0000000000000000
[   38.568623] remoteproc remoteproc18: powering up 41000000.r5f
[   38.569338] Internal error: Oops: 0000000096000007 [#1] PREEMPT SMP
[   38.574440] remoteproc remoteproc18: Booting fw image am65x-mcu-r5f0_0-fw, size 932
[   38.580644] Modules linked in: usbserial ti_cal videobuf2_dma_contig ti_k3_r5_remoteproc(+) videobuf2_memops pci_endpoint_test videobuf2_v4l2 rti_wdt watchdog videobuf2_common at24 st_lsm6dsx_i2c(+) optee_rng st_lsm6dsx kfifo_buf pm16d17 rng_core tee_stmm_efi tpm_ftpm_tee fuse dm_mod ip_tables x_tables ipv6
[   38.589862] remoteproc remoteproc18: remote processor 41000000.r5f is now up
[   38.615533] CPU: 1 PID: 283 Comm: (udev-worker) Not tainted 6.1.54-cip6 #1
[   38.615546] Hardware name: SIMATIC IOT2050 Advanced PG2 (DT)
[   38.615553] pstate: 00000005 (nzcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[   38.641945] pc : rproc_handle_resources.constprop.0+0x8c/0x164
[   38.647788] lr : rproc_boot+0x2fc/0x57c
[   38.651628] sp : ffff800009d53740
[   38.654934] x29: ffff800009d53740 x28: ffff00087f7d77f8 x27: ffff0008048f4c10
[   38.662068] x26: 0000000000000001 x25: ffffffffffffffff x24: ffff80000e000000
[   38.669199] x23: ffff00080227e038 x22: 0000000000000000 x21: ffff8000092bb1b0
[   38.676333] x20: ffff00080227e000 x19: 0000000000000000 x18: 000000000000028e
[   38.683464] x17: 0000000000000000 x16: 000000006d958cac x15: ffffffffffffffff
[   38.690597] x14: ffffffffffffffff x13: ffffffffffffffff x12: ffffffffffffffff
[   38.697728] x11: ffffffffffffffff x10: ffffffffffffffff x9 : ffffffffbfffffff
[   38.704862] x8 : ffffffffffffffff x7 : fffffdffffffffff x6 : ffffffffffdfffff
[   38.711994] x5 : ffff000802be1f00 x4 : ffff80000e000100 x3 : 00000000000000fd
[   38.719127] x2 : 00000000ffffffff x1 : ffff80000e000003 x0 : ffff80000e000000
[   38.726260] Call trace:
[   38.728703]  rproc_handle_resources.constprop.0+0x8c/0x164
[   38.734196]  rproc_boot+0x2fc/0x57c
[   38.737689]  rproc_add+0xcc/0x16c
[   38.741004]  k3_r5_probe+0x44c/0xe14 [ti_k3_r5_remoteproc]
[   38.746501]  platform_probe+0x68/0xc0
[   38.750168]  really_probe+0xbc/0x2dc
[   38.753742]  __driver_probe_device+0x78/0x114
[   38.758099]  driver_probe_device+0xd8/0x15c
[   38.762282]  __driver_attach+0x94/0x19c
[   38.766119]  bus_for_each_dev+0x74/0xd0
[   38.769954]  driver_attach+0x24/0x30
[   38.773529]  bus_add_driver+0x154/0x20c
[   38.777363]  driver_register+0x78/0x130
[   38.781198]  __platform_driver_register+0x28/0x34
[   38.785901]  k3_r5_rproc_driver_init+0x20/0x1000 [ti_k3_r5_remoteproc]
[   38.792437]  do_one_initcall+0x64/0x1dc
[   38.796272]  do_init_module+0x48/0x1d0
[   38.800023]  load_module+0x185c/0x1cc4
[   38.803770]  __do_sys_finit_module+0xa8/0xfc
[   38.808040]  __arm64_sys_finit_module+0x20/0x30
[   38.812571]  invoke_syscall+0x48/0x114
[   38.816320]  el0_svc_common.constprop.0+0xcc/0xec
[   38.821053]  do_el0_svc+0x2c/0xd0
[   38.821077]  el0_svc+0x2c/0x84
[   38.821095]  el0t_64_sync_handler+0xf4/0x120
[   38.831698]  el0t_64_sync+0x18c/0x190

(this crash was with a stable kernel, but same issue with head of tree)

This raises several questions:
 - Is it a hardware property that RTI1 powers up core 1 as well?
 - If so, how can we use both watchdog and remoteproc so that the latter
   loads the firmware for the former? We are currently doing that from 
   U-Boot, but what if that is not desired? Should the watchdog driver 
   take care to not leave core 1 in a different power state behind?
 - Can and should we do more while parsing the resource table to prevent
   such crashes?

Jan

-- 
Siemens AG, Technology
Linux Expert Center