Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1760420AbcLSPdq (ORCPT ); Mon, 19 Dec 2016 10:33:46 -0500 Received: from mx0a-00082601.pphosted.com ([67.231.145.42]:58571 "EHLO mx0a-00082601.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755108AbcLSPdL (ORCPT ); Mon, 19 Dec 2016 10:33:11 -0500 Subject: Re: [PATCHSET v4] blk-mq-scheduling framework To: Paolo Valente References: <1481933536-12844-1-git-send-email-axboe@fb.com> <7A8A5078-E9B8-4EBF-BAB1-9E8EEBF3A043@linaro.org> CC: , Linux-Kernal , Omar Sandoval , Linus Walleij , Ulf Hansson , Mark Brown From: Jens Axboe Message-ID: Date: Mon, 19 Dec 2016 08:33:01 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.5.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset="windows-1252" Content-Transfer-Encoding: 7bit X-Originating-IP: [66.29.164.166] X-ClientProxiedBy: BY2PR13CA0023.namprd13.prod.outlook.com (10.162.223.33) To CY4PR15MB1191.namprd15.prod.outlook.com (10.172.177.13) X-MS-Office365-Filtering-Correlation-Id: 1b545bbb-5878-4b6e-0ec3-08d4282453e0 X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001);SRVR:CY4PR15MB1191; X-Microsoft-Exchange-Diagnostics: 1;CY4PR15MB1191;3:z04Ks4MveHqujbCRnGyCEGn7Rd0K69gEPQujEdw86/82FJr0fNbYrWr6zSIWvFtz2btlAeKu0+DjRFNFZW3P5oVeNfccVa4A27ZmQHt6Y4tfRpNxhdsC1kQP8fnWoL0n8gRU3Gi5XbqIj3cCCQrb3mwR/+SEXgeEv6FKtpfdFTEqezaJ+MfhbbeIXO0dGNJ/CZxihaFCUjcemu3nRv/7LcIKhQ4Ngnv8s1iBM8OkZb6gfRKxQwZTAles3WXhAeJVqmoaJ2KTLGy0gBHy8ErMSQ==;25:oRNLrpfqBD5ch34jj1efZfWjQ8UA+WIoTfr7VC+iuYdfZBCQHxGiwXmOZp+BpywRJesO/yo5Vop4Xnyrhd+NjPerCSO8QKVWrhytOwXREdQc20OharihDKeJiH3eGy543eJi/T+TsmbV9GkfUrmATgqZXunA2ErUIYN5Fuhg27dTCwO1MNUFJqJzhei3A5P9wc9b5YNd7UJw59EthIbOdSyC02dav7/a67wrRl9jGEqgmYzSnNOF/gsDJmi+uELsCw4MqPnpNsi2YtaaW5OkLOn2r7VPyG1FbvfE0Nm4CBsHkKo8V07aYQEjINL+pLanrekpcT40IWRgqrPAxlZsU8gTyF5lmUAxBe0cjM6LvJlnHdCikrPkGPyfEQyr7omiK01jHEqDer7cTVl25EjIJTo4qOEPuYENYQmEEVRspcv71OMcji8pWWIhFKo3X+HC0nTdnXRoRkMvhAnlHNDMbw== X-Microsoft-Exchange-Diagnostics: 1;CY4PR15MB1191;31:CL2B9q/KpVCQP13ULwLflwZA7YdKgAbLgqAIWSv7uguRXAKzU69QAwRou2p3Y9byhvbFPsDlLZqOg1Pv+orG/2jcLpcm9h37/3hpI9kgRA+yZVNmWnpoM8AmsWZaAkCnSjz6FsDQM2EMrhrLnQ3ELtzNjTVgUjsyFxMw5/OMhuQNOFOpbJU4a8kTWWKtnJN5IPSDpigU51b/H+6WUKtyqi4P+mZwSHxy+6v+Rly6C5+aCvSLgXqW1MnSZedLvXdr;20:BayHj0sOpspNSqnMYOvPRXVaWBgg/jNJ54vTnWZtJgoG0TfTxZZDRZcDIqfguTG8g90txFv2ET1IF1xBrBpi8WQB1XASmzC53oDl96RdRGjHAki47ZSZ65VCjEwx9ScIQNXygb5+vOUFGY00O7VqTD8nqFKYHkSNri8hYZbzlXL2WwoV4wGN8vja+v9auoVzp3e4M6WEO6luAZxNWBDx+3FLxqnR0z1T+PKJjbwP0ZnRG3iGRg4rblP2IoxDZ/uu7itnLJwAQte1eKodVLnGexSKX5puMHGk+hisyWi/VRzpdaIEDyZty4AxvcdCXxluz+GHGv5rOxzJ83gELweQx9Iz/DaKTTWtFLYhYy4hszbG2Kp6jjxdVh/BChh+jPQEBGSVlU1p9dBSV9g4M+BlhhjGGOWKPbP0K3gYl61I2h1ccVp69lOkjLQ2uZD3pAcCy/2DInNXLn/OdI9b4CIeVb7PPv73jsOYbF1/sVD7OJrYbxFG7i7SM33h7Cqaj+nw X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(788757137089)(67672495146484); X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040375)(601004)(2401047)(8121501046)(5005006)(3002001)(10201501046)(6041248)(20161123555025)(20161123562025)(20161123560025)(20161123564025)(6072148);SRVR:CY4PR15MB1191;BCL:0;PCL:0;RULEID:;SRVR:CY4PR15MB1191; X-Microsoft-Exchange-Diagnostics: 1;CY4PR15MB1191;4:HGWV9tYGDeweq551nuG/B63lQFZAt5bOy0HoT4N/AzBY3qE10L8vXZxLCVzCvJr+/4gF0plFZvUq4++quORy0fMkD7aberWrI5SaFMCu7sedT7/5gl1tL0UI/v+K9xZime6vmDHb5i8+U28woFJHsqXC6Sl3z0miJFk+Rg5fx4llp8RIcN1IDtoE/hkBh05uhGEU8hnEWK0lektWqfHpYgF5kFZhMW3fkvOCYKhEpaXCxedltKjyD85r7FevnO9hebX2FhnCVGrUHM7U18RDYgs6Y/ekbew2EcM25qUZ4pj7HI5Wyfq74UWj3ebUIVbmtMmGfd8W/JwGJA1mD2fMt8kIr3Ei3mb/FHROj3UljOghhQ3gmA5ePdpJaJ4u7Ka+Sfhd0SkkWoNgn6nAz5zl9zzVaTOMqzuePKQw8x3vkhjjh5B5zv3C2cSLBbGtrlOOd6qo/tfAHtJFBW6RFqlLOUVqaBSf2pGmRMP12oYZ8TFJzmKcnr4l30zVNGCPY7wPLRgcMdNKuvJQbHao9VOj7pP7JzYWw4tMDqyw/9EFmNDGgRhmwN3JXu+aI7DQW1eDBlyqFWR4qgPTviOaJGXXlf1n6EmJRJV2kmLSuuPk2g+VUSNnWSSkShOgTbXRaQwXywU0P7ge6IeAjrk6x+TDZg== X-Forefront-PRVS: 01613DFDC8 X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10019020)(4630300001)(979002)(6049001)(6009001)(7916002)(39450400003)(24454002)(199003)(377454003)(189002)(50986999)(64126003)(65826007)(4326007)(101416001)(2950100002)(2906002)(6666003)(4001350100001)(76176999)(305945005)(7736002)(54356999)(23746002)(230700001)(25786008)(47776003)(86362001)(6116002)(77096006)(31696002)(65806001)(189998001)(66066001)(97736004)(90366009)(229853002)(65956001)(83506001)(6916009)(3846002)(6486002)(110136003)(68736007)(117156001)(8676002)(230783001)(5660300001)(106356001)(31686004)(38730400001)(33646002)(42186005)(50466002)(105586002)(36756003)(92566002)(81156014)(81166006)(6606295002)(969003)(989001)(999001)(1009001)(1019001);DIR:OUT;SFP:1102;SCL:1;SRVR:CY4PR15MB1191;H:[192.168.1.176];FPR:;SPF:None;PTR:InfoNoRecords;A:1;MX:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?Windows-1252?Q?1;CY4PR15MB1191;23:YQDWhdOLSuADXt8whFSmEu07m3u3cKrpaB9WW?= =?Windows-1252?Q?aowep3/YNF3G18XuExBv9aUws6HDtWG8oNf0lAeKgmM4MiHvyW8QESvF?= =?Windows-1252?Q?X8WSMWnH6P2ENsiZaQS0M2bBpeQK12uE02EVwQK97LjU2EMqM7y38Jr+?= =?Windows-1252?Q?tCN1/KxPjTmjmSeofd2zumEqo3Fzl2uUtwSrp+fU+3ly5WhkmqeSivf9?= =?Windows-1252?Q?TxC3rKabn/CQSZGo2rcGa/2QfsF6TaBv34JUWHuFg7DU2YsTNH2q8360?= =?Windows-1252?Q?WFdQ94/2ZmPlBLSzxzggB0o7apB9FTKr7A58izc/hEjyfFOo3ze5wyeu?= =?Windows-1252?Q?fAM0Tr1dOJG7rS9MDmyBZQSKlhcQ3VHGsghgLmw9Xv5pRIt6VjWGpFU5?= =?Windows-1252?Q?YWfWvwOLFNDZIyJP1RKiVOT3Sy/0Zwwkrl11CVB3COGWU5z1h39CPRR5?= =?Windows-1252?Q?U0NRQgKYwrfXbbC8nH05yQLf+PfsdmRKT4OVqAKG5qa4byC9Q7O+Vn4W?= =?Windows-1252?Q?eCAvSRLqfe7BzkwUHW11whQqeng04fvJLH1kzFwAxFrH4TyU/bGGemPO?= =?Windows-1252?Q?H6weT/u2zBif9N4xDlBe95HJGYL6ZSZB6d+Xh2RugeO5AByFXgXHca8a?= =?Windows-1252?Q?Bl0u/rKcnU38pMVZlGXAGT1InVkmM7sq1t/kebhs+DVQup1yLbCOUXzg?= =?Windows-1252?Q?EooO9cJJkq98jsJorKFDuSTmi2NkBQ7aTfXlz3v76Jc0yR3j0AAMJgoE?= =?Windows-1252?Q?GBm1MzTWGlwdY+cgTLGvz7r+w9N5a5DK/uRyDNj7YDzF+seKRQku5RRz?= =?Windows-1252?Q?jqrxTdaSDyObNdgQvRut7ZLef9cizMjESrNJJQ8agG+46oUTjHXgPTov?= =?Windows-1252?Q?mDMin6RU1nwRzAoxTvl1R0w92WVFOt0yNYJKnYerxRRiDWmPo3VWVihN?= =?Windows-1252?Q?uRjvqNhE8l1bCH6O/lKgKRJ2g+0nFvxFeZk/pGxLwlJqtacii/qMQqss?= =?Windows-1252?Q?Dm8dKQMyy8ag7Njwje+fbYmU9lzvLz5M5ilPEdSgLTMFTWcCsAoGKGRD?= =?Windows-1252?Q?fpWTQ+bv+RDGknL7YRGBmgQm6ZTZLlDLxyHVX5kLxrwJkgzmR8QNwM8x?= =?Windows-1252?Q?4c6py0bMjyLCT6ntQAatuZFt7SVM4elgsgaBvqVfCMG8OjBxOLptH0Xq?= =?Windows-1252?Q?+LWJ/YM3u4UrYY/BvpoKcu3Z0Z4EOJOmLmcdQYhfh/onHN69dxQnbzvj?= =?Windows-1252?Q?V2zocpDdBjjdZdoCoY7KN60EkLWbPlsldhpnmH1stt3/xfC5lgdzCfAK?= =?Windows-1252?Q?UER+B0gdVuoGWsDPR9Y7fTGfMnL3edBDhQWgi+4Kc+gjIiVbGDdJ6+X8?= =?Windows-1252?Q?LORh9W+6E94AkUbShvv6d8N8tDUjW3PkCTW7cOEE8Dp0i0ottTAtWvUz?= =?Windows-1252?Q?aMVpYr3qcqgMMkBMB4ztfh/dOxVNsYwVMFfH9XYXo5nCMx2N+lqHaJad?= =?Windows-1252?Q?jRbWjfZCTSDtSpm7MGOiRSSS8sY4xtg4W99jCSwxCrJKE1QWaeZR9ESG?= =?Windows-1252?Q?uIBkb5vRtdKLeTnfQlwRN4l1YinNyd1HOkCPHK11tiQbE8ulU6gAWdQo?= =?Windows-1252?Q?HMwG06mOlxAbUaP9JLpnEw=3D?= X-Microsoft-Exchange-Diagnostics: 1;CY4PR15MB1191;6:HEe6zHIaojQuoeQKDeqiO5ftCoZacmws7diqFpCLbTwFG+eKD9qcf5NBYoI0lZr34cG8FrG8OvWUReQenNDvoVCvNphKmakDPUkFNhMCA1VzHliLaLgva9lYpKDm21mPm5oGG34fvDrs6OIUUz/3NdULar1InyeUDKJ3fyQkUsrPx6qC3Q0Z3EFVqpHe1fn2MeFCtpMnxJS1uo+J3isF9xnti+OYenPkNjIV+8zvHcXtxkDXsHGeTiHj19x5hUvXQ+mH1kAEXTP3bDvPQxD4s0NJ1G4/T7fXp6YiQvaUOyz/k9WnLTJQl2A7H8o4RmszQ/L8o8faBAyLpmRO20WJXac12PoFCfG5iBJ2j0iaU7B4klO+5i9ILD605h1dLFpxfDpi6Ty6APVx5+h9Y0j0Ibrb1I5EX6UqHjzOGTks+uo=;5:QfJRin2FqoLv2EkksWR0hgc1adPqaG5sTx2WmLkRheI/OIcJ3Ms+/ZikgylDefmNpjsKYZ6mHQk4AkUqLaOmH/fjkbZCqQYsLYzViKXNuMISQbn1XaLojGdyukydE2tB3TZgEj8EANWaNHjB+aLjFQ==;24:ul3F7L2ruhMEwcAiqe4d4FD9485LBHAGXLgWOj27DN1OuLUwc2Wtv+R8jg5vuWubP/nhPTTmEEuiYE9X97ulYh4AkqlMarWi3PNZY7eIj/0= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;CY4PR15MB1191;7:1A4/jUQHcm6er+kdPCvSZ7PcoXaT1bvPHWGEdO9sBj6AYdJ4ovVVPNYS3AGPJs35DTjoGa8tbZOT9u6X0u6PC72I+iDoy/lowHyWiQNDpIk0dn/4KaFgJQO+rEZ5gJg3/1B8zHbkyLL06RhzrH8ON8hxS9Gj0vgYwemDgFtl0OrJDzXaTIPPRZv7BtdaYFthgupzgVPIhWR7oUBCY9yGI4RmCaj5tcJfvcRW0f/J9kEkBrA4McWiRDxhBGL6/BxUI2m2C3R3wFVIUrEKUq2GJurA3VY8+00AqvVf1PDc9W3bPMir1Gwfr21CkwaYO+xHOxw3vKJOO4AahBBRfzxgr27M+JVi3dg0MHYkOG7Vk6rjkNmqCWDPxnrc/JUJSyBLfEp5fCDijIqWvM0moOGv5uh7rbhtj/sGottSNsiYFRLs+03jeERtFrpu9Z8ys7+j8hWcWirFco5ufVQlTD6muw==;20:uIMCm3rAJB+xkiutvBhGnfX2khV9QZJFAa5pXMeiL+OjBuIpBCfddjmx1qjnS5QYCgmMNAqoUhtlSS85Q/Kex2clxvrvqvN5UVMf+MC5gPfiozSUQmQ/XunFrf4RWNjzHT9StGY09JRTtV0hyi+l+5ZDrmbZ1zfb0APtRhaEcjg= X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Dec 2016 15:33:04.7931 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR15MB1191 X-OriginatorOrg: fb.com X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2016-12-19_11:,, signatures=0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 2895 Lines: 63 On 12/19/2016 08:20 AM, Jens Axboe wrote: > On 12/19/2016 04:32 AM, Paolo Valente wrote: >> >>> Il giorno 17 dic 2016, alle ore 01:12, Jens Axboe ha scritto: >>> >>> This is version 4 of this patchset, version 3 was posted here: >>> >>> https://marc.info/?l=linux-block&m=148178513407631&w=2 >>> >>> From the discussion last time, I looked into the feasibility of having >>> two sets of tags for the same request pool, to avoid having to copy >>> some of the request fields at dispatch and completion time. To do that, >>> we'd have to replace the driver tag map(s) with our own, and augment >>> that with tag map(s) on the side representing the device queue depth. >>> Queuing IO with the scheduler would allocate from the new map, and >>> dispatching would acquire the "real" tag. We would need to change >>> drivers to do this, or add an extra indirection table to map a real >>> tag to the scheduler tag. We would also need a 1:1 mapping between >>> scheduler and hardware tag pools, or additional info to track it. >>> Unless someone can convince me otherwise, I think the current approach >>> is cleaner. >>> >>> I wasn't going to post v4 so soon, but I discovered a bug that led >>> to drastically decreased merging. Especially on rotating storage, >>> this release should be fast, and on par with the merging that we >>> get through the legacy schedulers. >>> >> >> I'm to modifying bfq. You mentioned other missing pieces to come. Do >> you already have an idea of what they are, so that I am somehow >> prepared to what won't work even if my changes are right? > > I'm mostly talking about elevator ops hooks that aren't there in the new > framework, but exist in the old one. There should be no hidden > surprises, if that's what you are worried about. > > On the ops side, the only ones I can think of are the activate and > deactivate, and those can be done in the dispatch_request hook for > activate, and put/requeue for deactivate. > > Outside of that, some of them have been renamed, and some have been > collapsed (like activate/deactivate), and yet others again work a little > differently (like merging). See the mq-deadline conversion, and just > work through them one at the time. Some more details... Outside of the differences outlined above, a major one is that the old scheduler interfaces invoked almost all of the hooks with the device queue lock held. That's no longer the case on the new framework, you have to setup your own lock(s) for what you need. That's a lot saner. One example is the attempt to merge a bio to an existing request, that would be the ->bio_merge() hook. If you look at mq-deadline, the hook merely grabs its per-queue lock (dd->lock) and calls a blk-mq-sched helper to do the merging. That, in turn, will call ->request_merge(), so that is called with the lock that ->bio_merge() grabs. -- Jens Axboe -- Jens Axboe