Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751698AbdFHMsv (ORCPT ); Thu, 8 Jun 2017 08:48:51 -0400 Received: from mail-bl2nam02on0079.outbound.protection.outlook.com ([104.47.38.79]:21776 "EHLO NAM02-BL2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751663AbdFHMso (ORCPT ); Thu, 8 Jun 2017 08:48:44 -0400 From: Rafal Ozieblo To: Richard Cochran CC: David Miller , "nicolas.ferre@atmel.com" , "netdev@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "devicetree@vger.kernel.org" , "linux-arm-kernel@lists.infradead.org" , "harini.katakam@xilinx.com" , "andrei.pistirica@microchip.com" Subject: RE: [PATCH v2 4/4] net: macb: Add hardware PTP support Thread-Topic: [PATCH v2 4/4] net: macb: Add hardware PTP support Thread-Index: AQHS26yCMIGfgNxtoESOPovh46jljaIYL5IAgAEBExCAADvagIABhg0A Date: Thu, 8 Jun 2017 12:48:40 +0000 Message-ID: References: <1496413439-12900-1-git-send-email-rafalo@cadence.com> <1496413690-22826-1-git-send-email-rafalo@cadence.com> <20170606183358.GA25220@localhost.localdomain> <20170607132817.GA28329@localhost.localdomain> In-Reply-To: <20170607132817.GA28329@localhost.localdomain> Accept-Language: pl-PL, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-dg-ref: PG1ldGE+PGF0IG5tPSJib2R5LnR4dCIgcD0iYzpcdXNlcnNccmFmYWxvXGFwcGRhdGFccm9hbWluZ1wwOWQ4NDliNi0zMmQzLTRhNDAtODVlZS02Yjg0YmEyOWUzNWJcbXNnc1xtc2ctY2E1MTA4NjEtNGM0OC0xMWU3LWJmNjAtMjhkMjQ0ZjNkMmFkXGFtZS10ZXN0XGNhNTEwODYyLTRjNDgtMTFlNy1iZjYwLTI4ZDI0NGYzZDJhZGJvZHkudHh0IiBzej0iNjIzOCIgdD0iMTMxNDEzOTk3MTg0MjIxOTUzIiBoPSJhaTI4V1lGdVJyaXB2RTV4TjJPT000SUdQVGM9IiBpZD0iIiBibD0iMCIgYm89IjEiLz48L21ldGE+ authentication-results: gmail.com; dkim=none (message not signed) header.d=none;gmail.com; dmarc=none action=none header.from=cadence.com; x-originating-ip: [213.131.238.28] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1;BN3PR07MB2515;7:jqhh/zCE9v4UfOfVd9wwBzxc432+idAhEcJo54gKPRqXf9wH6xofMOKRc2e49FUytbpKjLQs0ZoJwTOXGOeds0R8By+Ifmta29RHNowoGJxoEE8fNgycxWRR19r3FQXDsEpQsn62tAPrn/c2cgWg4RIk2JjAu1y5yGalHUD2lHuLWEYnBmHgWS3qb/czgrrDEpBReGhezlro3Fr5VLECXeGrgghIH6IBKZnNUTojkftlvwYj4LAR6l7f7ko3LEdT+tIsSKTOFn8VL3lYPlb4N0mLTMshc2KL8rLaZMFyPNt6x0rY1nF7lkCjjaHkzg6Bmr5FpGtd69xU0amMP5vygA==;20:pu/XOqyHlgIciwpJCK7b56h8UNXVPQfnh08Sc9Y+6jJ4HirCe+g9eFDBsOGkN+7q6myqw3QixK1z0dKGtvuDQpmqkm7dNG6kFZsK2yB4bGh8T2S2L7acjFuChv4uq2iiN9dcg0maXknB8aCedWXPzmlawP4c68bxkfFdNuiSEOzh+K07d/7cz6MwZQctCYD0QIOtBjbvE2uZ1P71lfsdIoZMDiQa8ioNFtoEVYPgQbachZh5luSO3XqrfHczZMrh x-ms-traffictypediagnostic: BN3PR07MB2515: x-ms-office365-filtering-correlation-id: 6490a470-8fc0-4734-3b76-08d4ae6cb0bc x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001)(2017030254075)(201703131423075)(201703031133081);SRVR:BN3PR07MB2515; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(9452136761055)(258649278758335)(192813158149592)(72806322054110); x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(601004)(2401047)(8121501046)(5005006)(93006095)(93001095)(10201501046)(3002001)(100000703101)(100105400095)(6041248)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123558100)(20161123562025)(20161123564025)(20161123560025)(20161123555025)(6072148)(100000704101)(100105200095)(100000705101)(100105500095);SRVR:BN3PR07MB2515;BCL:0;PCL:0;RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095);SRVR:BN3PR07MB2515; x-forefront-prvs: 0332AACBC3 x-forefront-antispam-report: SFV:NSPM;SFS:(10009020)(6009001)(39840400002)(39450400003)(39850400002)(39400400002)(39410400002)(36092001)(24454002)(478600001)(7736002)(5890100001)(1411001)(102836003)(6116002)(3846002)(74316002)(53546009)(33656002)(99286003)(9686003)(54906002)(55016002)(8666007)(8656002)(2900100001)(229853002)(38730400002)(66066001)(14454004)(305945005)(6246003)(122556002)(53936002)(110136004)(6506006)(5660300001)(77096006)(7696004)(575784001)(86362001)(3280700002)(39060400002)(3660700001)(6916009)(2950100002)(6436002)(93886004)(25786009)(2906002)(189998001)(54356999)(4326008)(50986999)(76176999)(8936002)(81166006)(8676002);DIR:OUT;SFP:1101;SCL:1;SRVR:BN3PR07MB2515;H:BN3PR07MB2516.namprd07.prod.outlook.com;FPR:;SPF:None;MLV:ovrnspm;PTR:InfoNoRecords;LANG:en; spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 X-OriginatorOrg: cadence.com X-MS-Exchange-CrossTenant-originalarrivaltime: 08 Jun 2017 12:48:40.9034 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: d36035c5-6ce6-4662-a3dc-e762e61ae4c9 X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN3PR07MB2515 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by mail.home.local id v58Cmw4S010520 Content-Length: 6030 Lines: 124 > From: Richard Cochran [mailto:richardcochran@gmail.com] > Sent: 7 czerwca 2017 15:28 > To: Rafal Ozieblo > Cc: David Miller ; nicolas.ferre@atmel.com; > netdev@vger.kernel.org; linux-kernel@vger.kernel.org; > devicetree@vger.kernel.org; linux-arm-kernel@lists.infradead.org; > harini.katakam@xilinx.com; andrei.pistirica@microchip.com > Subject: Re: [PATCH v2 4/4] net: macb: Add hardware PTP support > > ********** EXTERNAL MAIL. PLEASE USE CAUTION WHEN CLICKING LINKS > OR OPENING ATTACHMENTS ********** > > On Wed, Jun 07, 2017 at 11:13:36AM +0000, Rafal Ozieblo wrote: > > Please look at following call-stack: > > > > 1. macb_interrupt() // spin_lock(&bp->lock) is taken > > 2. macb_tx_interrupt() > > 3. macb_handle_txtstamp() > > 4. skb_tstamp_tx() > > 5. __skb_tstamp_tx() > > 6. skb_may_tx_timestamp() > > 7. read_lock_bh() // second lock is taken > > Well, you can always drop the lock, or postpone the call to > skb_tstamp_tx() until after the spin lock is released... Postponing is done using worker. Another pros is that skb_tstamp_tx() is called outside interrupt routine. > > > I know that those are different locks and different types. But this could > lead > > to deadlocks. This is the reason of warning I could see. > > Can you please post the lockdep splat? > [ 1264.352242] ================================= [ 1264.356719] [ INFO: inconsistent lock state ] [ 1264.361242] 4.4.0-42139-g418b7a0-dirty #16 Not tainted [ 1264.366502] --------------------------------- [ 1264.371004] inconsistent {HARDIRQ-ON-W} -> {IN-HARDIRQ-W} usage. [ 1264.377204] ptp4l/105 [HC1[1]:SC0[4]:HE0:SE0] takes: [ 1264.382286] (&(&list->lock)->rlock){?.-...}, at: [] skb_queue_tail+0x18/0x34 [ 1264.390622] {HARDIRQ-ON-W} state was registered at: [ 1264.395618] [] lock_acquire+0x124/0x158 [ 1264.400730] [] _raw_spin_lock_bh+0x50/0x5c [ 1264.406095] [] first_packet_length+0x35/0x1b4 [ 1264.411710] [] udp_poll+0x30/0x3c [ 1264.416267] [] sock_poll+0x141/0x148 [ 1264.421092] [] do_sys_poll+0x171/0x300 [ 1264.426111] [] SyS_poll+0x45/0x80 [ 1264.430676] [] system_call+0x40/0x50 [ 1264.435492] [] common_exception_return+0x0/0x83 [ 1264.441274] irq event stamp: 23325 [ 1264.444802] hardirqs last enabled at (23324): [] _raw_spin_unlock_irqrestore+0x4e/0x78 [ 1264.453929] hardirqs last disabled at (23325): [] do_interrupt+0x13/0x90 [ 1264.461732] softirqs last enabled at (23292): [] ip_finish_output2+0x80/0x43c [ 1264.470073] softirqs last disabled at (23294): [] ip_finish_output2+0x12e/0x43c [ 1264.478480] [ 1264.478480] other info that might help us debug this: [ 1264.485211] Possible unsafe locking scenario: [ 1264.485211] [ 1264.491323] CPU0 [ 1264.493883] ---- [ 1264.496439] lock(&(&list->lock)->rlock); [ 1264.500665] [ 1264.503392] lock(&(&list->lock)->rlock); [ 1264.507793] [ 1264.507793] *** DEADLOCK *** [ 1264.507793] [ 1264.514016] 4 locks held by ptp4l/105: [ 1264.517886] #0: (rcu_read_lock_bh){......}, at: [] ip_finish_output2+0x12e/0x43c [ 1264.526675] #1: (rcu_read_lock_bh){......}, at: [] __dev_queue_xmit+0x38/0x5f8 [ 1264.535287] #2: (_xmit_ETHER#2){+.-...}, at: [] sch_direct_xmit+0x51/0x138 [ 1264.543593] #3: (&(&bp->lock)->rlock){-.-...}, at: [] macb_interrupt+0x36/0x564 [ 1264.552316] [ 1264.552316] stack backtrace: [ 1264.556962] CPU: 0 PID: 105 Comm: ptp4l Not tainted 4.4.0-42139-g418b7a0-dirty #16 [ 1264.564654] [ 1264.564654] Stack: d1786130 d75815e3 00001ffc d75815e0 9003811c d7581620 d7558c00 d1c0bd60 [ 1264.564654] 00000004 00000001 00000001 d7581600 900389fc d7581650 d7558c00 d7559160 [ 1264.564654] 00000004 00000000 00000000 d7559160 d04ecc34 d1bf2de0 00000179 d7581620 [ 1264.588339] Call Trace: [ 1264.590894] [] print_usage_bug.part.33+0x1c7/0x1cc [ 1264.596883] [] mark_lock+0x2f4/0x44c [ 1264.601639] [] __lock_acquire+0x460/0x1228 [ 1264.606907] [] lock_acquire+0x124/0x158 [ 1264.611907] [] _raw_spin_lock_irqsave+0x45/0x54 [ 1264.617612] [] skb_queue_tail+0x18/0x34 [ 1264.622594] [] sock_queue_err_skb+0xdc/0x108 [ 1264.628017] [] __skb_complete_tx_timestamp+0x55/0x60 [ 1264.634146] [] __skb_tstamp_tx+0x94/0x98 [ 1264.639215] [] skb_tstamp_tx+0x18/0x1c [ 1264.644110] [] macb_interrupt+0x1e9/0x564 [ 1264.649302] [] handle_irq_event_percpu+0xe9/0x2a4 [ 1264.655179] [] handle_irq_event+0x36/0x50 [ 1264.660343] [] handle_level_irq+0x99/0xb8 [ 1264.665525] [] generic_handle_irq+0x19/0x24 [ 1264.670863] [] do_IRQ+0x35/0x38 [ 1264.675141] [] do_interrupt+0x70/0x90 [ 1264.679959] [] common_exception_return+0x0/0x83 [ 1264.685635] [] macb_start_xmit+0x414/0x418 [ 1264.690905] [] dev_hard_start_xmit+0x2ed/0x450 [ 1264.696521] [] sch_direct_xmit+0x6e/0x138 [ 1264.701720] [] __dev_queue_xmit+0x38d/0x5f8 [ 1264.707058] [] dev_queue_xmit+0x14/0x18 [ 1264.712049] [] neigh_resolve_output+0x180/0x190 [ 1264.717734] [] ip_finish_output2+0x37a/0x43c [ 1264.723186] [] ip_finish_output+0x1b8/0x200 [ 1264.728532] [] ip_mc_output+0x179/0x1cc [ 1264.733532] [] ip_local_out+0x45/0x5c [ 1264.738359] [] ip_send_skb+0x16/0x64 [ 1264.743097] [] udp_send_skb+0x160/0x1c8 [ 1264.748098] [] udp_sendmsg+0x45d/0x5a8 [ 1264.753009] [] inet_sendmsg+0x21/0x3c [ 1264.757801] [] sock_sendmsg+0x1d/0x44 [ 1264.762629] [] SyS_sendto+0x89/0xa4 [ 1264.767288] [] system_call+0x40/0x50 [ 1264.772010] [] common_exception_return+0x0/0x83 > Thanks, > Richard Regards, Rafal