Received: by 2002:ab2:710b:0:b0:1ef:a325:1205 with SMTP id z11csp1434630lql; Tue, 12 Mar 2024 18:27:04 -0700 (PDT) X-Forwarded-Encrypted: i=3; AJvYcCW5nW+eunRAfQpbxncdgdFTyu7Ntksz5e8m7bunf+VLw+qbWrJjVhI+Ec+Ooj0wPQbV05BcrNj+VdDUysCf1m2WnqzzGTNJfQOZXH2DBg== X-Google-Smtp-Source: AGHT+IHAkJjE6o6VcOVrnxoXTXcXccrUPxS3C3+v3eC2ZXXbo0JM9pnoRYBfFCpIUWbG6PHp5nv0 X-Received: by 2002:a50:f61e:0:b0:566:4654:4fac with SMTP id c30-20020a50f61e000000b0056646544facmr7753863edn.5.1710293224252; Tue, 12 Mar 2024 18:27:04 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1710293224; cv=pass; d=google.com; s=arc-20160816; b=lieJ1hUN2XdYaAHjmDWrgyA+ycWS2MaOJGxat8TvSxdaGMXNiwyNWLOFZBGxlmThA/ YiWdcAGMGD2xiLZRhCsYP+xRKAbX3189RLm3gTzWWXZH/lfQdiXA11BAdY0tw5aWLAeC +rwGPWPkcT7cYy89Zc6NFy7hjeH32UuE1TA/V0YfyqjkPLagiCKafmnX7dsu31mI0ex2 XoZsqGjyTOHmraCM1drRRyx+K40seTR4IVQBTBQeam4lGDr6kCXpHm7KIqMEkd2UMlGp PLjYvo811Sl5mKTwopD6R4s8+/8qAdfFIDcbQgXIy+WqeQVhKyb1ZPeHpoeeDDnSF64f R2ZQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=KGM3mRROwFt6krNu3XhZIaE8fo5xphzgJ2jbr0QsLYo=; fh=7XehbsakU3DNfctgZlwS4XiVVrNXJeaX3Zomx9nO2A8=; b=r9yWvzb+QjZ3+NHmYjhVtNKMDHKN6v5Q1AHApW0rKjpKiE99gNK+kXC2pPghfItJP7 tUkm0bpd9686VuzlgbUVF/3XpKvTs+fp0UQ4aOHsJo6ltLLHMvrBrIbmOIUJ6xSZ6YBY D0aJ1hkZYL9bHrCzabH6wqNZlJTqTwh+dcda8axRdsmDfJmLigdd5BtVYvM5mMAqihEu XXinMsxMe7DQUqipaYGFBweTkEWFnLlWLjiOAEvG5qFcjxtiJg8a1knU8GiHWkijqCgQ XipvIN63azj9fyBGqrra8pXG7l1ubrPPeoo5qTUIiQ3xufyxtpc+9F5msV+AQleA3WHX HD9g==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=xggoGQ3M; arc=pass (i=1 spf=pass spfdomain=joelfernandes.org dkim=pass dkdomain=joelfernandes.org); spf=pass (google.com: domain of linux-kernel+bounces-101007-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-101007-linux.lists.archive=gmail.com@vger.kernel.org" Return-Path: Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [147.75.80.249]) by mx.google.com with ESMTPS id i2-20020a05640242c200b005684b902082si3184512edc.691.2024.03.12.18.27.04 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Mar 2024 18:27:04 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel+bounces-101007-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249; Authentication-Results: mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=xggoGQ3M; arc=pass (i=1 spf=pass spfdomain=joelfernandes.org dkim=pass dkdomain=joelfernandes.org); spf=pass (google.com: domain of linux-kernel+bounces-101007-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-101007-linux.lists.archive=gmail.com@vger.kernel.org" Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 4CE7F1F2288E for ; Wed, 13 Mar 2024 01:26:49 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 6849ABA2B; Wed, 13 Mar 2024 01:25:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=joelfernandes.org header.i=@joelfernandes.org header.b="xggoGQ3M" Received: from mail-vk1-f182.google.com (mail-vk1-f182.google.com [209.85.221.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0DAF6171CE for ; Wed, 13 Mar 2024 01:25:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.182 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710293122; cv=none; b=RSQ7cXBu9vdQtzSEeCEW+3zjR3qNhVc9mth/FCvljy50KeVRLy2n2CebcE0lw3rslabzbHgbO7v+JAB2vBLRLgegXnte0U1SXrg9/FJHgIhGjEZ5xfqcxARHcK71kEvSRbTojNUbAF9Cu34eh5rbeaY5n+SPeHB2F/N1bTbePJo= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710293122; c=relaxed/simple; bh=OSrAg2NG3ZYsMO+/bzATjsRdKuLLDdxyzjpAKJt8kcI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=GE0F+409MkM6E/EL3b1QjBfG70k93eyBwiBAgmUT6KC3XpzFq2hy3jvZstGhOfpeBKd3/r+f9zlkcU5cU0P05Nc2WSexAj7Rw5YOhJPjLck9jOApCb8L/OoAQCQDjtMzW7+V8Jz6uHwzZ0isEe0Kqx/B0DqRbC1fvAIsfCvL61I= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=joelfernandes.org; spf=pass smtp.mailfrom=joelfernandes.org; dkim=pass (1024-bit key) header.d=joelfernandes.org header.i=@joelfernandes.org header.b=xggoGQ3M; arc=none smtp.client-ip=209.85.221.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=joelfernandes.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=joelfernandes.org Received: by mail-vk1-f182.google.com with SMTP id 71dfb90a1353d-4d419b03fd9so33093e0c.2 for ; Tue, 12 Mar 2024 18:25:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; t=1710293118; x=1710897918; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=KGM3mRROwFt6krNu3XhZIaE8fo5xphzgJ2jbr0QsLYo=; b=xggoGQ3MTZkpILwwcMCVN6uKH5tJd64VfD1xbGeVMh6QOeB/hLapWMRehjfv/aR9+b 2kWRI1PShBPbyeapgG/FTRUSMo8WEFL2XEBlLypNoA3ZntIYwoa3+KgU9ypGMbDBQUWU KhqVdi+jH3F1JF9reNSQGyUfVIpWV+CoP2f4E= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710293118; x=1710897918; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KGM3mRROwFt6krNu3XhZIaE8fo5xphzgJ2jbr0QsLYo=; b=Kt3D+LSRO6qKco0ZPrdb17djq/iGMT5cxSxw/992MpK5FJBFXqmiD4t0M7mtDiMUyz 8jJMs6H6gEoE7hTTCHcSYwm5ikvjLjJ302Sy5GJYuRNaDGfbWurSnL/+XdD9dZXZrwr5 S6BldwQjVXEhw0hxmrn82ZGskK6zuN6Vv7UfhTl59KdRB2GRlLEeYuO5T7/ZaNWTCB9n QK7n37/6u/JRpXBI9eeznvWkRTykxg5/s+fAtfZZMarzj+wUcXkr6RAyWMrztNfPo9pf MrOmh6UOnwCvZbkbhmB7sjQkuGpuvPQC2rS+lCLfjwF2L0JP2CJlkeLa/MLJQ3BrYqc/ YC+w== X-Gm-Message-State: AOJu0Yx+oPcq6+fBOzjzdEZ4FMbgavnrvhXMR/n7p7+Guu5xp7JHFvm4 KFXAk+pbw0ojgbl50CfD4cIq3fc6HLroGX+dWTzEc+xxW3U3L6BvRGrF/XWaySRTA24QvCVWT5r Z X-Received: by 2002:a05:6122:4e8f:b0:4d3:3a8c:13ad with SMTP id gf15-20020a0561224e8f00b004d33a8c13admr7939547vkb.8.1710293118309; Tue, 12 Mar 2024 18:25:18 -0700 (PDT) Received: from joelbox2.. (c-98-249-43-138.hsd1.va.comcast.net. [98.249.43.138]) by smtp.gmail.com with ESMTPSA id u9-20020a05621411a900b00690314356a4sm4162132qvv.80.2024.03.12.18.25.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Mar 2024 18:25:17 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org, Shuah Khan , Nathan Chancellor , Nick Desaulniers , Bill Wendling , Justin Stitt Cc: Suleiman Souhlal , Youssef Esmat , Steven Rostedt , David Vernet , Thomas Gleixner , "Paul E . McKenney" , joseph.salisbury@canonical.com, Dietmar Eggemann , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira , Valentin Schneider , Luca Abeni , Tommaso Cucinotta , Vineeth Pillai , Shuah Khan , Phil Auld , "Joel Fernandes (Google)" , linux-kselftest@vger.kernel.org, llvm@lists.linux.dev Subject: [PATCH v2 07/15] selftests/sched: Add a test to verify that DL server works with core scheduling Date: Tue, 12 Mar 2024 21:24:43 -0400 Message-Id: <20240313012451.1693807-8-joel@joelfernandes.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240313012451.1693807-1-joel@joelfernandes.org> References: <20240313012451.1693807-1-joel@joelfernandes.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit This test verifies that DL server infrastructure gives CFS tasks a fixed bandwidth even when RT tasks are being "core scheduled" on a core. Verify that they are getting the expected bandwidth (and thus not being starved). Also verified that not having core scheduling fixes makes the test fail as the CFS task gets no bandwidth. Sample output: # Runtime of PID 97 is 4.440000 seconds # Runtime of PID 98 is 4.560000 seconds # Runtime of PID 99 is 4.550000 seconds ok 1 PASS Notes about test that generated the sample output: The test runs for 12 seconds. We check the runtimes at 9 seconds. We expect the CFS task (PID 7) to get ~50% of the 9 seconds. The DL server is configured for 50% bandwidth. The RT tasks (PID 98, 99) each get 50% as well, because they run concurrently on 2 hyperthreads of a core. Signed-off-by: Joel Fernandes (Google) --- tools/testing/selftests/sched/Makefile | 13 +- tools/testing/selftests/sched/common.c | 24 ++ tools/testing/selftests/sched/common.h | 8 + .../selftests/sched/cs_dlserver_test.c | 254 ++++++++++++++++++ 4 files changed, 290 insertions(+), 9 deletions(-) create mode 100644 tools/testing/selftests/sched/common.c create mode 100644 tools/testing/selftests/sched/common.h create mode 100644 tools/testing/selftests/sched/cs_dlserver_test.c diff --git a/tools/testing/selftests/sched/Makefile b/tools/testing/selftests/sched/Makefile index 099ee9213557..f491d741cb45 100644 --- a/tools/testing/selftests/sched/Makefile +++ b/tools/testing/selftests/sched/Makefile @@ -1,14 +1,9 @@ # SPDX-License-Identifier: GPL-2.0+ +TEST_GEN_PROGS := cs_dlserver_test -ifneq ($(shell $(CC) --version 2>&1 | head -n 1 | grep clang),) -CLANG_FLAGS += -no-integrated-as -endif +cs_dlserver_test: cs_dlserver_test.c common.c -CFLAGS += -O2 -Wall -g -I./ $(KHDR_INCLUDES) -Wl,-rpath=./ \ - $(CLANG_FLAGS) -LDLIBS += -lpthread - -TEST_GEN_FILES := cs_prctl_test -TEST_PROGS := cs_prctl_test +CFLAGS += $(KHDR_INCLUDES) +CFLAGS += -Wall include ../lib.mk diff --git a/tools/testing/selftests/sched/common.c b/tools/testing/selftests/sched/common.c new file mode 100644 index 000000000000..5cf0022acc8d --- /dev/null +++ b/tools/testing/selftests/sched/common.c @@ -0,0 +1,24 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include "common.h" + +bool hyperthreading_enabled(void) +{ + FILE *file = fopen("/sys/devices/system/cpu/smt/active", "r"); + char smt_active[2]; + + if (file == NULL) { + ksft_print_msg("Could not determine if hyperthreading is enabled\n"); + return false; + } + + if (fgets(smt_active, sizeof(smt_active), file) == NULL) { + perror("Failed to read smt_active"); + return false; + } + fclose(file); + + if (smt_active[0] != '1') + return false; + return true; +} diff --git a/tools/testing/selftests/sched/common.h b/tools/testing/selftests/sched/common.h new file mode 100644 index 000000000000..7bcedbd0ed99 --- /dev/null +++ b/tools/testing/selftests/sched/common.h @@ -0,0 +1,8 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#include +#include +#include +#include "../kselftest.h" + +bool hyperthreading_enabled(void); diff --git a/tools/testing/selftests/sched/cs_dlserver_test.c b/tools/testing/selftests/sched/cs_dlserver_test.c new file mode 100644 index 000000000000..9f2a74a25686 --- /dev/null +++ b/tools/testing/selftests/sched/cs_dlserver_test.c @@ -0,0 +1,254 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Use the DL server infrastructure to give CFS tasks a fixed bandwidth + * even when RT tasks are being "core scheduled" on a core. Verify that + * they are getting the expected bandwidth (and thus not being starved). + * + * Copyright (c) 2024 Google. + * Author: Joel Fernandes + * + * This library is free software; you can redistribute it and/or modify it + * under the terms of version 2.1 of the GNU Lesser General Public License as + * published by the Free Software Foundation. + * + * This library is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License + * for more details. + * + * You should have received a copy of the GNU Lesser General Public License + * along with this library; if not, see . + */ + +#define _GNU_SOURCE + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "common.h" + +enum pid_type {PIDTYPE_PID = 0, PIDTYPE_TGID, PIDTYPE_PGID}; + +#define RUN_TIME 12 // Running time of the test in seconds +#define CORE_ID 0 // Assuming we're pinning processes to the first core +#define DL_SERVER_DEBUGFS "/sys/kernel/debug/sched/fair_server" + +void write_server_debugfs(char *file, char *type, unsigned long value) +{ + char path[1024], buf[1024]; + int fd, n; + + snprintf(path, sizeof(path), "%s/%s/%s", DL_SERVER_DEBUGFS, file, type); + fd = open(path, O_WRONLY); + if (fd == -1) { + perror("Failed to open file for writing"); + return; + } + n = snprintf(buf, sizeof(buf), "%lu\n", value); + n = write(fd, buf, n); + if (n == -1) + perror("Failed to write file"); + + close(fd); +} + +void write_dl_server_params(void) +{ + DIR *dir; + struct dirent *entry; + + if (access(DL_SERVER_DEBUGFS, F_OK) == -1) { + perror("DL server debugfs not found, cannot set DL parameters."); + exit(EXIT_FAILURE); + } + + dir = opendir(DL_SERVER_DEBUGFS); + if (dir == NULL) { + perror("Failed to open directory"); + exit(EXIT_FAILURE); + } + + while ((entry = readdir(dir)) != NULL) { + if (strcmp(entry->d_name, ".") == 0 || strcmp(entry->d_name, "..") == 0) + continue; + + write_server_debugfs(entry->d_name, "period", 100000000); + write_server_debugfs(entry->d_name, "runtime", 50000000); + } + closedir(dir); +} + +void process_func(void) +{ + unsigned long long count = 0; + time_t end; + + // Busy loop for RUN_TIME seconds + end = time(NULL) + RUN_TIME; + while (time(NULL) < end) { + count++; // Just a dummy operation + } +} + +void set_affinity(int cpu_id) +{ + cpu_set_t cpuset; + + CPU_ZERO(&cpuset); + CPU_SET(cpu_id, &cpuset); + CPU_SET(cpu_id + 1, &cpuset); + + if (sched_setaffinity(0, sizeof(cpu_set_t), &cpuset) != 0) { + perror("sched_setaffinity"); + exit(EXIT_FAILURE); + } +} + +void set_sched(int policy, int priority) +{ + struct sched_param param; + + param.sched_priority = priority; + if (sched_setscheduler(0, policy, ¶m) != 0) { + perror("sched_setscheduler"); + exit(EXIT_FAILURE); + } +} + +float get_process_runtime(int pid) +{ + char path[256]; + FILE *file; + long utime, stime; + int fields; + + snprintf(path, sizeof(path), "/proc/%d/stat", pid); + file = fopen(path, "r"); + if (file == NULL) { + perror("Failed to open stat file"); + return -1; // Indicate failure + } + + // Skip the first 13 fields and read the 14th and 15th + fields = fscanf(file, + "%*d %*s %*c %*d %*d %*d %*d %*d %*u %*u %*u %*u %*u %lu %lu", + &utime, &stime); + fclose(file); + + if (fields != 2) { + fprintf(stderr, "Failed to read stat file\n"); + return -1; // Indicate failure + } + + // Calculate the total time spent in the process + long total_time = utime + stime; + long ticks_per_second = sysconf(_SC_CLK_TCK); + float runtime_seconds = total_time * 1.0 / ticks_per_second; + + return runtime_seconds; +} + +int main(void) +{ + float runtime1, runtime2, runtime3; + int pid1, pid2, pid3; + + if (!hyperthreading_enabled()) + ksft_test_result_skip("This test requires hyperthreading to be enabled\n"); + + write_dl_server_params(); + + ksft_print_header(); + ksft_set_plan(1); + + // Create and set up a CFS task + pid1 = fork(); + if (pid1 == 0) { + set_affinity(CORE_ID); + process_func(); + exit(0); + } else if (pid1 < 0) { + perror("fork for p1"); + ksft_exit_fail(); + } + + // Create a new unique cookie for the CFS task + if (prctl(PR_SCHED_CORE, PR_SCHED_CORE_CREATE, pid1, PIDTYPE_TGID, 0) < 0) { + perror("prctl for pid1"); + ksft_exit_fail(); + } + + // Create a new unique cookie for the current process. Future + // forks will inherit this cookie. + if (prctl(PR_SCHED_CORE, PR_SCHED_CORE_CREATE, 0, PIDTYPE_TGID, 0) < 0) { + perror("prctl for current process"); + ksft_exit_fail(); + } + + // Create an RT task which inherits the parent's cookie + pid2 = fork(); + if (pid2 == 0) { + set_affinity(CORE_ID); + set_sched(SCHED_FIFO, 50); + process_func(); + exit(0); + } else if (pid2 < 0) { + perror("fork for p2"); + ksft_exit_fail(); + } + + // Create another RT task which inherits the parent's cookie + pid3 = fork(); + if (pid3 == 0) { + set_affinity(CORE_ID); + set_sched(SCHED_FIFO, 50); + process_func(); + exit(0); + } else if (pid3 < 0) { + perror("fork for p3"); + ksft_exit_fail(); + } + + sleep(RUN_TIME * 3 / 4); + runtime1 = get_process_runtime(pid1); + if (runtime1 != -1) + ksft_print_msg("Runtime of PID %d is %f seconds\n", pid1, runtime1); + else + ksft_exit_fail_msg("Error getting runtime for PID %d\n", pid1); + + runtime2 = get_process_runtime(pid2); + if (runtime2 != -1) + ksft_print_msg("Runtime of PID %d is %f seconds\n", pid2, runtime2); + else + ksft_exit_fail_msg("Error getting runtime for PID %d\n", pid2); + + runtime3 = get_process_runtime(pid3); + if (runtime3 != -1) + ksft_print_msg("Runtime of PID %d is %f seconds\n", pid3, runtime3); + else + ksft_exit_fail_msg("Error getting runtime for PID %d\n", pid3); + + // Make sure runtime1 is within 30% of runtime2 + if (runtime1 < 0.7 * runtime2 || runtime1 > 1.3 * runtime2) + ksft_exit_fail_msg("Runtime of PID %d is not within 30%% of runtime of PID %d\n", + pid1, pid2); + + // Make sure runtime1 is within 30% of runtime3 + if (runtime1 < 0.7 * runtime3 || runtime1 > 1.3 * runtime3) + ksft_exit_fail_msg("Runtime of PID %d is not within 30%% of runtime of PID %d\n", + pid1, pid3); + + waitpid(pid1, NULL, 0); + waitpid(pid2, NULL, 0); + waitpid(pid3, NULL, 0); + + ksft_test_result_pass("PASS\n"); + return 0; +} -- 2.34.1