Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <7640a2d9-a32d-2fd7-8f64-586edb9b781e@virtuozzo.com>
Date:   Tue, 27 Sep 2022 10:44:20 +0300
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0)
 Gecko/20100101 Thunderbird/102.3.0
Subject: Re: [PATCH v2] mm: Make failslab writable again
To:     Hyeonggon Yoo <42.hyeyoo@gmail.com>
Cc:     Jonathan Corbet <corbet@lwn.net>, Christoph Lameter <cl@linux.com>,
        Pekka Enberg <penberg@kernel.org>,
        David Rientjes <rientjes@google.com>,
        Joonsoo Kim <iamjoonsoo.kim@lge.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        Vlastimil Babka <vbabka@suse.cz>,
        Roman Gushchin <roman.gushchin@linux.dev>, kernel@openvz.org,
        Kees Cook <keescook@chromium.org>,
        Roman Gushchin <guro@fb.com>, Jann Horn <jannh@google.com>,
        Vijayanand Jitta <vjitta@codeaurora.org>,
        linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org,
        linux-mm@kvack.org
References: <20220920121111.1792905-1-alexander.atanasov@virtuozzo.com>
 <Yyr1xONdw8dBgsKr@hyeyoo>
 <30063d97-69f0-bea2-9d59-108140995bfc@virtuozzo.com>
 <YzJIsFZQoCEYntvR@hyeyoo>
From:   Alexander Atanasov <alexander.atanasov@virtuozzo.com>
In-Reply-To: <YzJIsFZQoCEYntvR@hyeyoo>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?QlZ5R2RGcWp5Z2xkWUxyR3NPT1A2bG1aSlY0SUlNc1JqUGxXNWR4U25nL2pX?=
 =?utf-8?B?RHMrS0dyQzV1a3E5TmhjU0Fod2x2M1BMSmtCU3ZvOGVkV2lLK1ZyNmFUWG90?=
 =?utf-8?B?OXpjNmRVcmJTV0hyUTk0M0dqZFBzRzVmUnhqN3Nuc1ErbHVDdlB6QWFNY2FF?=
 =?utf-8?B?OTdyc1doN0N2SE94K1dLNTJNSE9MMkxheG40Tjh1RW4rQnhndHBDRkg4THJL?=
 =?utf-8?B?dllwNkthUmRQTzkxbUZjc1FvZ3p3clpURGRhcHRpTVEvUUZKSGRFM21Xc2tC?=
 =?utf-8?B?N2JqRVQrOGlvSENWS0lsbGx2aWZqNnNDbVlRb3BMaS91ajB6N2RNelY2S0hn?=
 =?utf-8?B?VUtwSGRHZzB4QzZibldraWRJVWdjSHJWQ1k1T3E5US9LaCtpQm93NW1ZREN1?=
 =?utf-8?B?MmtGbmJhRFNiTnhBTDM1bnFWTlJ4TG1WSU1BR2dEU2VtRE93V1ZQZUo5Vlh1?=
 =?utf-8?B?V0hvMGlQZUFUK0dUVUUzcWdQZkdmbHB6RUg3YzUvRzVjeTBMOE15SUs4cVBh?=
 =?utf-8?B?d3JvL0d5ZWc2MVVGK016TkZyb3FCcXk3MEJ2TWl3YWkyV3AweFFSRU4zV1Rw?=
 =?utf-8?B?RHA2ODF6bkkxOXZZTjFEdDhoSGd6VkVqRlBGTk9sbGJNaTFWeEcycmZkQlYx?=
 =?utf-8?B?cmJJR2lhRFQ1ZlhOZ2FyRWM0NjZzYXBJb2pBK0M1RXQ1ZmJvNHViWkNzckd4?=
 =?utf-8?B?TWN4OHpqaFJpQm02S05Oc0RidCsxUWdkSCtBWi9tUWh6TXhyQWprTUNTelE2?=
 =?utf-8?B?N2hFM0ZPNmcxS2tVR1lBM2Z3N3d6QzdHeHJ6U21GVmh4SlZBdEhoVkY5OTM0?=
 =?utf-8?B?QWRKaDUxUzMyN05EdXZlV3Nsb2UrWVdhd2h5TW8xS00rSVVMc2l4eDFtNzRI?=
 =?utf-8?B?VWZnUXY1a2NOTWxRSll6N0dwWTlwUVZKelNqM0ZoTWtXYk50ZDJ5SGg1TXdi?=
 =?utf-8?B?OWZkbVozMjM0L1BPRFdwK1Jpejk0VTJHSnpsd2l3dG5jcW5NWDNaRERQZWNQ?=
 =?utf-8?B?WmN2SmZSM01jSkNtWFEveXBabVplVUVUMXBPV1Z6SjdvY1lmQlQwcGh3N2I1?=
 =?utf-8?B?U0U4c3IvdGxPSGtjaC9Fc29GSzF4bU92OG1vWVBUZGVRM2YxeDJsTFRBdEs3?=
 =?utf-8?B?TW1BbGxqLy9vVmN1YVNqcno3cGlHKzM0emVOclV3QzBGZ21VYUtEdzRsTjVR?=
 =?utf-8?B?eitHamlnS2RueXNFQnZmenZ5dExvWmlEb0hsY214d2hHeVlOTGVRa0J3VDVh?=
 =?utf-8?B?aFAzdFczcktpKzl2WWM5akNqdzRHZXZFZmJka0FuOG9aVGdoL0NlZVlmWFhW?=
 =?utf-8?B?Ny9Nc0k2RTA3QkV2SlBpQ2tSclZ1a21FSTA4UVFqemZTZFRQQ0pleHl4U0ky?=
 =?utf-8?B?SXVJTkk0dDhjN1REeWtsZXFtSUZOcC8yRTVnN1E5N2lCdGdjK3RzQVVWcnU0?=
 =?utf-8?B?SU1vNkhjMTRUZzlGaWN2NkQ1U3FwZ1pacFE2aUQrTm8rNmlRT3ViSll0QUFT?=
 =?utf-8?B?Yzk2SlovZUZMMHJJdEJ4SHhPODBYZ0dmTXNrMkt2d2k1dkZOYjN0VXUwK1U4?=
 =?utf-8?B?bFdLRFRHTExSTEIwTHlvcWFHQ3VpU3BPK1JXdVJFV3RDc0lua2pUQ1RwSGdJ?=
 =?utf-8?B?VjliN3BFWjZQcGk5U09BUHpWWjF3U1lBbXR0WWJvaXp6bXBpTVY1UVRPSmdz?=
 =?utf-8?B?dHZkM0hTdnZqM1hYUlF1RkJTN2NCalc1dnhoYzV3UlRmY3lybkpFWk1EUTFM?=
 =?utf-8?B?Z3Q4VkFEb3ZVWGVhRWdWQ2hMbm9XbFAzV01jbG8xME1aOThXNVVYaFE5MXJS?=
 =?utf-8?B?dEdnM2wvM2RIVkNpN0c2VUI4YVlzVStQOXhsSFREQU02aGlRbjBhcW13UlR6?=
 =?utf-8?B?aWpvK0FsK2JWVEM1TGE2aHFWMTZTb1NJMDVhUGVPbDZRcDZSVXcycFRNbFZt?=
 =?utf-8?B?TXM4ZWlLZUFxUDB4QWYxdzBhdDcyYXNsRStHUUtJRzlTSVFMSE9EQyt5TlNr?=
 =?utf-8?B?Y1lqcXJOSEZCSk14VEhHa3JQNmNaTVFmUm5CWjZBSGpkSUpUNmk1TS93NmlC?=
 =?utf-8?B?ZW9XcE1BUVFlVCtlNnpvdjNXUjJSMEN0MmpjVnphazFoL3ZzZVljOWZWSGpC?=
 =?utf-8?B?UitsekM2SEhnOTFBVXJqNnRhVksveWYwd3VQUFlPcDEySVFNS0xoblVtZGJw?=
 =?utf-8?B?OHc9PQ==?=
X-OriginatorOrg: virtuozzo.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 734f5144-f419-46fb-2f74-08daa05c1906
X-MS-Exchange-CrossTenant-AuthSource: VE1PR08MB4765.eurprd08.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Sep 2022 07:44:24.5023
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: uPS1Rdhyjw62iuMs7PDnEVyqb30nn6YC+VsDTmJ/0QMHJbUFjQRGwvbKhWqT6+3+f+9v6bH/Z6cFa/rm6m1K6ssFagkCKyABQiSziMivko+FIjx/v2rSLB7+9kL5TJMO
X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM9PR08MB6259
X-Spam-Status: No, score=-3.4 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,NICE_REPLY_A,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hello,

On 27.09.22 3:49, Hyeonggon Yoo wrote:
> On Fri, Sep 23, 2022 at 10:34:28AM +0300, Alexander Atanasov wrote:
>> Hello,
>>
>> On 21.09.22 14:30, Hyeonggon Yoo wrote:
>>> On Tue, Sep 20, 2022 at 03:11:11PM +0300, Alexander Atanasov wrote:
>>>> In (060807f841ac mm, slub: make remaining slub_debug related attributes
>>>> read-only) failslab was made read-only.
>>>> I think it became a collateral victim to the two other options for which
>>>> the reasons are perfectly valid.
>>>> Here is why:
>>>>    - sanity_checks and trace are slab internal debug options,
>>>>      failslab is used for fault injection.
>>>>    - for fault injections, which by presumption are random, it
>>>>      does not matter if it is not set atomically. And you need to
>>>>      set atleast one more option to trigger fault injection.
>>>>    - in a testing scenario you may need to change it at runtime
>>>>      example: module loading - you test all allocations limited
>>>>      by the space option. Then you move to test only your module's
>>>>      own slabs.
>>>>    - when set by command line flags it effectively disables all
>>>>      cache merges.
>>>
>>> Maybe we can make failslab= boot parameter to consider cache filtering?
>>>
>>> With that, just pass something like this:
>>> 	failslab=X,X,X,X,cache_filter slub_debug=A,<cache-name>>
>>
>>> Users should pass slub_debug=A,<cache-name> anyway to prevent cache merging.
>>
>> It will be good to have this in case you want to test cache that is used
>> early. But why push something to command line option only when it can be
>> changed at runtime?
> 
> Hmm okay. I'm not against changing it writable. (it looks okay to me.)

Okay. Good to know that.

> Just wanted to understand your use case!
> Can you please elaborate why booting with slub_debug=A,<your cache name>
> and enabling cache_filter after boot does not work?

I didn't say it does not work - it does work but requires reboot. You 
may want to test variations of caches for example. Cache A, Cache B ... 
C and so on one by one. Reboots might be fast these days with VMs but 
you may not be able to test everything in a VM. And ... reboots used to 
be the signature move of one Other OS.

> Or is it trying to changnig these steps,
> 
> FROM
> 	1. booting with slub_debug=A,<cache name>
> 	2. write to cache_filter to enable cache filtering
> 	3. setup probability, interval, times, size
> 
> TO
> 
> 	1. write to failslab attribute of <cache name> (may fail it has alias)
> 	2. write to cache_filter to enable cache filtering
> 	3. setup probability, interval, times, size
> ?
> 
> as you may know, SLAB_FAILSLAB does nothing whens
> cache_filter is disabled, and you should pass slub_debug=A,<cache name> anyway

Okay , i think there awaits another problem:
bool __should_failslab(struct kmem_cache *s, gfp_t gfpflags)
{
...

         if (failslab.cache_filter && !(s->flags & SLAB_FAILSLAB))
                 return false;
...
	return should_fail(&failslab.attr, s->object_size);
}

So if you do not have cache_filter set ... you go to should_fail for all 
slabs.
I've been hit by that and spend a lot of time trying to understand why i 
got crashes at random places. And the reason was that i read an old 
documentation that said cache_filter is writable and i blindly wrote 1 
to it. If the intent is to only work with cache filter set - then i will 
update the patch to do so. This is the only place where SLAB_FAILSLAB is 
explicitly tested, other places check it as part of SLAB_NEVER_MERGE.

But even for all caches it is kind of possible to test with size(space) 
which is in turn useful because you need to figure out how you handle 
failures from external caches - external to your code under test and you 
don't want to keep track for all of them (same goes for too much options 
in command line).


> to prevent doing cache merging with <cache name>.

Or you can pass SLAB_FAILSLAB from your module when creating the cache 
to prevent merge when under test.


-- 
Regards,
Alexander Atanasov